.. _continuous-integration-with-Hudson/Jenkins: Continuous Integration ====================== Tuleap provides Continuous Integration to teams via a deep integration with Jenkins. Tuleap team recommends to avoid big Jenkins instances shared by many projects and many teams as the security model of Jenkins doesn't allow to have a strict split of areas. That is to say, to guaranty that 2 concurrent teams that share the same Jenkins sever cannot have access to the code of each other. In case of doubt the strategy of "One Tuleap Project, one Jenkins Master" is safe and efficient. Jenkins Configuration --------------------- This section will cover how to configure a Jenkins server to be used efficiently with Tuleap. We assume a fresh Jenkins instance that was just installed. Some adaptations might be needed if you modify an existing Jenkins server (be very careful with authentication to not lock yourself out of Jenkins). .. attention:: Both Jenkins and Tuleap servers **must** be in https and certificate must be either `valid `_ or, at least, trusted. If you cannot have valid certificates: - `Jenkins must trust Tuleap server certificate `_ - `Jenkins must run behind a reverse proxy that does TLS termination `_ - :ref:`Tuleap must trust Jenkins server certificate ` Plugins installation ```````````````````` Two plugins should be installed: - `Tuleap Git Branch Source `_ - `Tuleap Authentication `_ Both are available publicly on the Jenkins plugin marketplace and the installation is done from within Jenkins in "Manage Plugins" section. You might also need to install other plugins related to your pipeline of email notifications, artifact publishing to Artifactory, etc. This is business specific to each project and not covered in this documentation. Plugins configuration ````````````````````` First, you need to associate your Jenkins server with a Tuleap server. This is done in "Manage Jenkins > System configuration". .. figure:: ../images/screenshots/jenkins/configure_tuleap_server.png :align: center :alt: Configure Tuleap server in Jenkins :name: Configure Tuleap server in Jenkins If the connection to the Tuleap server is successful you will see ``Connexion established with these Urls`` message in the Jenkins interface (as in the previous screenshot). Otherwise you will get a stacktrace (Jenkins...) with, hopefully, an error message that will help to diagnose the problem. Most common issues are related to DNS (is your server name valid and can Jenkins access it) and TLS (does jenkins trust Tuleap server). Authentication configuration ```````````````````````````` This section requires that your Tuleap server has :ref:`OAuth2 Server ` plugin installed. First, on your Tuleap server, in one of your project, you need to create a new :ref:`OAuth2 app `. The app will ask for a callback URL. This callback URL is your Jenkins server base URL (eg. https://jenkins.example.com/jenkins) + ``/securityRealm/finishLogin``. The plugin allows the PKCE usage for the authentication. You can force its usage at the creation of the :ref:`OAuth2 app `. Keep the generated Client Secret securely until the next step. .. figure:: ../images/screenshots/jenkins/new_oauth2_app.png :align: center :alt: Register a new OAuth2 app for Jenkins in Tuleap :name: Register a new OAuth2 app for Jenkins in Tuleap Then Jenkins, go In *Manage Jenkins > Configure Global Security*, and select *Tuleap Authentication* and fill: - Client ID - Client Secret With the values provided by Tuleap. .. figure:: ../images/screenshots/jenkins/configure_oidc.png :align: center :alt: Register Tuleap as OpenID Connect provider for Jenkins :name: Register Tuleap as OpenID Connect provider for Jenkins Ensure that *Authorization* (bellow *Authentication section*) is still set to *Anyone can do anything* and click save. You should then be able to login on Jenkins with you Tuleap credentials and still have access to *Manage Jenkins*. .. note:: If you locked yourself out of Jenkins you can start over by `disabling security `_. Authorization configuration ``````````````````````````` This section depends on the previous one. If you don't use Tuleap Authentication, you cannot do the following configurations. In the `Authorization Matrix `_ you can reference: - Tuleap users with their ``login_name`` - Tuleap user groups in the form ``project-shortname#user-group-name`` Most of the time should refer to user groups to ease the maintenance. Once you setup the groups, you only have to deal with people management at Tuleap side without having to bother with their Jenkins permissions anymore. In the following screenshot you have a good start point in term of permissions management with the default groups defined in Tuleap: - Access to Jenkins must be authenticated. Controlled by Tuleap => only the people who have access to Tuleap will have access to Jenkins - The people who can access can only Read your jobs. That correspond to a "Public" project (or "Public including Restricted" if you have restricted users in Tuleap). - The project members of the Tuleap mozilla project can manage jobs, builds and credentials. - The project administrators of Tuleap mozilla project can administer Jenkins (plugins, grants & co). .. figure:: ../images/screenshots/jenkins/configure_authorizations.png :align: center :alt: Reference Tuleap users and groups in Jenkins Authorization Matrix :name: Reference Tuleap users and groups in Jenkins Authorization Matrix At this point you are almost done with Jenkins administration. You might need to install and configure other plugins depending of the content of your pipelines, configure agent to distribute the load and, of course, update jenkins itself but you are done for the permission and user management. .. _continuous_integration_git_branch_source: Tuleap Git / Jenkins integration -------------------------------- Thanks to `Tuleap Git Branch Source `_ Jenkins plugin, most of the integration between the two tools is completely streamlined. The configuration is done once at project level, then every new git repository created in Tuleap will be automatically discovered by Jenkins, branches will be inspected to find ``Jenkinsfile`` and created corresponding pipelines. Whenever a new commit will be pushed, the corresponding job will be triggered on Jenkins. Step 1: Have an access key to your repositories ``````````````````````````````````````````````` In Tuleap, either with a service or personal account that have read access to the project's repositories go in user preferences, "Keys & Tokens" section and generate a new Access Key with both ``Git`` and ``REST`` scopes. .. figure:: ../images/screenshots/jenkins/tgbs_access_key.png :align: center :alt: Create a new access key for build user :name: Create a new access key for build user Step 2: Create a Tuleap Project ``````````````````````````````` In Jenkins, create a new job with type "Tuleap Project". It should be named after your Tuleap project name to ease organisation. .. figure:: ../images/screenshots/jenkins/tgbs_new_project.png :align: center :alt: New Tuleap Project job creation :name: New Tuleap Project job creation Once the job created you should grant it access to Tuleap with the credential you generated at Step 1. Near the credential drop down, you have a "Add" button. Create a new "Project name" credential of type "Tuleap Access Key" and give it a descriptive id so you can find it later. .. figure:: ../images/screenshots/jenkins/tgbs_new_credential.png :align: center :alt: Create a new Tuleap Acccess Key credential :name: Create a new Tuleap Acccess Key credential Once the credential is saved, select it in the "Credentials" dropdown. In the "Project" dropdown right after, select the Tuleap project you want to automate. You can adjust "Behaviours" to match your need. By default we suggest to remove ``*`` from ``Exclude`` field of "Filter by name (with wildcards)" section otherwise nothing will be built at all. .. figure:: ../images/screenshots/jenkins/tgbs_conf.png :align: center :alt: Configure Tuleap Project jenkins job :name: Configure Tuleap Project jenkins job When the configuration is ready, save it. This will trigger a scan of your project to look for git repositories, their branches and ``Jenkinsfile`` to create Jenkins jobs. .. figure:: ../images/screenshots/jenkins/tgbs_scan.png :align: center :alt: Initial scan of Tuleap project by Jenkins :name: Initial scan of Tuleap project by Jenkins When the scan is completed, you will find all the git repositories where Jenkins found a ``Jenkinsfile`` and the status of the builds that were triggered. .. figure:: ../images/screenshots/jenkins/tgbs_project_view.png :align: center :alt: Jenkins jobs in project :name: Jenkins jobs in project .. attention:: On Jenkins, in your project settings, you might also want to adjust "Scan Project Triggers" to a shorter period otherwise you will have to wait for 1 day between a new repository creation and jenkins to discover it. As this will trigger a full analyze of all branches of git repositories of your Tuleap project, you need to find a balance between reactivity and Tuleap server overloading. If you don't create a new repository every other hours, you might want to let 1 day period and trigger manually the scan whenever you create a new repository. Step 3: Tell Tuleap where the Jenkins server is ``````````````````````````````````````````````` The final step is on Tuleap. You need to inform the git server where is the Jenkins server that must be informed about new commits that are pushed. In the administration of the Git service of your project, there is a ``Jenkins`` tab where you set the Jenkins root url. For instance ``https://jenkins.example.com/jenkins``. .. figure:: ../images/screenshots/jenkins/tgbs_tuleap_trigger.png :align: center :alt: Tuleap configuration of Jenkins Trigger :name: Tuleap configuration of Jenkins Trigger That's it ````````` When those 3 steps are completed, you no longer have to worry about Jenkins / Tuleap integration, everything is automated. Continuous Integration service in Tuleap ---------------------------------------- .. note:: The "Continuous Integration" service in Tuleap refers to an historical implementation that was mainly targeting Subversion and CVS. It also provides some widgets that can be used on Project and Personal dashboards. .. _link-hudson-job-with-your-Tuleap-project: Reference Jenkins job with your Tuleap project `````````````````````````````````````````````` In order to link Jenkins job with your project, select the Continuous Integration tab of your project, and then select the 'Add a job' link. You need then to give the URL of the Hudson job you want to associate with your project (for instance: ``https://[my\_ci\_server]:8080/jenkins/job/[my\_job]``). .. figure:: ../images/screenshots/sc_hudsonaddjob.png :align: center :alt: Link Jenkins job with your project :name: Link Jenkins job with your project You may also want to enable the auto trigger of the build for this job after each commit in your project repository (CVS or Subversion). If you have protected your build with a token, you can specify this token. By checking this option, each commit will trigger a build of the associated job, using the pre-commit hook (you don't have anything more to do). By the way, it is possible to link several Jenkins jobs with one Tuleap project. Jenkins jobs and builds ``````````````````````` When you select the Continuous Integration tab of your project, you can see a table with all the jobs associated with your project. For every job, you can see the current status (colored bullet left to the name of the job), the name, the last successful build, the last failed build, if you have enabled SCM trigger or not. Project admins will also see for each job some icons that let them modify the job or delete it (remove the link with Tuleap). .. figure:: ../images/screenshots/sc_hudsonbrowsejobs.png :align: center :alt: Jenkins jobs associated with your project :name: Jenkins jobs associated with your project The name of the job is automatically detected during job creation. But you can change it if needed. This is pretty convenient if you want to make references to Jenkins items (see `Make a reference to a Job`_). Spaces in the name of jobs are not allowed. They are replaced by (\_), in order to allow references. Jenkins References ------------------ It is possible to make references to Jenkins items in Tuleap. There are some predefined references (job, build), but you can also create your own references if needed (see :ref:`reference-overview` for more details about references) Make a reference to a Job ````````````````````````` The keyword to make a reference to a Job is: **job**. To make a reference to a job, you can use the expressions: - job #JobNameToReference (the job must be in the current project) - job #project:JobNameToReference (will make a reference to the job 'JobNameToReference' of the project 'project') - job #project\_num:JobNameToReference (will make a reference to the job 'JobNameToReference' of the project with number 'project\_num') Make a reference to a build ``````````````````````````` The keyword to make a reference to a build is: **build**. To make a reference to a build, you can use the expressions: - build #XXX (there must be only one job associated with the current project, and the referenced build will be the build number 'XXX' of this job) - build #AJob/XXX (will make a reference to build number 'XXX' of job named 'AJob' of the current project) - build #project:AJob/XXX (will make a reference to the build number 'XXX' of the job 'AJob' of project 'project') - build #projet\_num:AJob/XXX (will make a reference to the build number 'XXX' of the job 'AJob' of the project number 'project\_num')