Jenkins core is a central component of the project which serves millions of users, and it is critical to maintain it in a good shape. We want to ensure quality of the integrated changes and continuity of the project, and hence the Jenkins core pull request review and merge process is more sophisticated than for the majority of plugins.
This document applies to the following components:
-
Jenkins core
-
Jenkins modules
-
Libraries included into the Jenkins core
-
Core components like Winstone, Executable WAR, etc.
Jenkins Remoting library is not in the scope for this document, because it is handled as a sub-project with independent maintainer team. Remoting updates in the core are subject to the process though.
-
Mailing list: Jenkins Developer Mailing List
-
Chat for runtime Jenkins release coordination:
#jenkins-releaseon Freenode IRC-
All async communications should go to the developer mailing list
-
-
Contributor
-
Issue Triage Team Member
-
Core Pull Request Reviewer
-
Core Maintainer
-
Release Team Member
Contributors submit pull requests to the Jenkins core and review changes submitted by others. There is no special preconditions to do so. Anyone is welcome to contribute.
Issues Triage Team Members review the incoming issues submitted in Jenkins Jira: bug reports, requests for enhancement, etc. Special permissions are not required to take this role or to contribute. Anyone is welcome to start contributing (see the triage guidelines below).
Core Pull Request Reviewers is a team for contributors who are willing to regularly review Jenkins pull requests and to eventually become Jenkins core maintainers. They get Triage permissions so that they can manage pull requests, request reviews and prepare changelog drafts in the pull request description. Their main responsibility is to triage and review the incoming pull requests, and to guide newcomer contributors who are not familiar with the project’s processes. GitHub team: @jenkinsci/core-pr-reviewers.
Core Maintainers get Write permissions in the repository, and hence they are able to merge pull requests.
Their responsibility is to perform pull request reviews on a regular basis and to merge ready pull requests towards the weekly releases (master branch).
They are also responsible to monitor the weekly release status and to perform triage of critical issues.
GitHub team: @jenkinsci/core.
Release Team Members are responsible for Jenkins weekly and LTS releases.
Led by the Jenkins Release Officer, they initiate releases, prepare changelogs and backport changes into the Stable release line.
Team members get Write permissions in the Jenkins core repository, and they also get permissions to trigger release Pipelines.
-
Contributors. Anyone can participate. There is no precondition except having a GitHub account, just submit pull requests or comment in existing ones!
-
Core Pull Request Reviewers can be nominated by contributors in the developer mailing list. Self-nomination is fine. Decision is made by a consensus in the mailing list or via voting at the governance meeting.
-
Core Maintainers - same process as for Core PR reviewers. All nominees must also sign an Individual Contributor License Agreement before getting the permission in GitHub repositories.
-
Release Team Members are assigned by the Jenkins Release officer
Jenkins core is a mission-critical part of the ecosystem. We need to ensure that submitted pull requests are not only code complete, but also that they do not introduce undesired defects, breaking changes and technical debt. At the same time, we are interested to make the review process as simple as possible for contributors and maintainers.
Pull requests review in Jenkins is not just about reviewing code and accepting them if the code is OK. Core maintainers are expected to ensure feasibility and compatibility of changes, to maintain good quality of the codebase and documentation, and to ensure there is a consensus between contributors.
Reviewers are expected to look at changes critically from a "product management" point of view. It’s not just about the code, but also whether the change makes sense in a global / holistic way, considering existing popular plugins and the way users experience Jenkins overall. Also, not every change needs to be merged into the core. Many features would be better introduced as plugins which have separate release cycles and allow delivering changes faster. We want to extend the Jenkins core and incorporate widely used functionality and extension points there, but we try to keep the core as minimal as possible in terms of dependencies.
The Jenkins project has a long history of backward compatibility. We accept breaking changes when it is needed (security fixes, feature deprecation and removal, etc.), but ultimately the project’s goal is retain as much compatibility as possible. It includes both feature compatibility and binary/API compatibility which is important for the plugin ecosystem. Although we have some tools (like usage-in-plugins) for checking API usages across open-source plugins, there is no way to confirm external usages in 3rd-party proprietary plugins which are also a part of the ecosystem.
The code doesn’t have to be perfect, but we want to ensure that all new code matches basic quality standards: test coverage for newly added functionality and fixes, documentation for newly introduced APIs, the submitted code is readable and matches the code style in the surrounding codebase, etc.
We’re aware that there are existing inconsistencies in the code, and we do not enforce a single code style across the codebase at the moment.
-
New code should follow the (majority) style guide. In Jenkins core we use the these Code Conventions for the Java TM Programming Language as a default code style
-
Updates to existing code should only fix formatting on the lines affected anyway to keep the diff minimal. It helps reviewers focus their attention on the specifics of the change and reduces the risk of a change from one pull request creating a conflict in another pull request.
-
Jenkins documentation is hosted on https://jenkins.io/doc/. When a new user-facing change is added, we should encourage contributors to update documentation in downstream pull requests.
-
Same applies to Jenkins changelogs (weekly, stable) and upgrade guidelines: We have a semi-automated process which is based on pull request summaries and labels. Core maintainers are expected to validate the entries as a part of the pull request review/merge process. See the checklist below in the Merge process section.
Not all changes are discussed before they are submitted as pull requests. Developer mailing lists, Jira issues and JEPs are used for discussions, but sometimes the changes go straight to the pull requests. And we are fine with that, especially for small patches. Pull requests often become a venue to discuss feasibility, underlying technical decisions and design. We are fine with that as well. If there is no consensus about the feasibility and implementation, code reviewers are expected to suggest proper channels for contributors to discuss their contribution.
-
A discussion in the Jenkins Developer Mailing List is the default way to go
-
If no consensus can be reached on the mailing list, voting at the Jenkins Governance Meeting can be used to get a final decision.
Code reviews do NOT pursue the following goals:
-
Accepting/merging any pull request. Not everything is going to be merged, and reviewers are expected to focus on the Jenkins ecosystem integrity first. We guide contributors and help them to get their changes integrated, but it needs cooperation on both sides. It is fine to close invalid and inactive pull requests if there is no activity by a submitter or other contributors.
-
Enforcing a particular coding style. Jenkins core has a complex codebase created by many contributors, and different files have different designs. Our main goal is to firstly have the code readable by other contributors.
-
Make contributors fix issues that are not related to the primary topic of the pull request
-
Create follow-up issues instead, it is fine to reference them in comments so that the contributor might want to pick them up
-
-
Make contributors to have atomic commit history or to squash their pull request
-
Not every contributor is a Git expert, do not request changes in the commit history unless it is necessary
-
Core maintainers can squash PRs during the merge. If you feel this is important, add the squash-merge-me label
-
We want to keep pull requests focused when possible (one feature / fix per pull request), but we can live without it if there is no need to backport changes to the stable baseline.
-
Jenkins core and most of its components use Jenkins Jira as an issue tracker. This issue tracker is open to all Jenkins users. They report defects and requests for enhancements, and then component maintainers triage issues and provide feedback to users. In the case of the Jenkins core, Issue Triage Team and Core Maintainers are roles who are expected to process the incoming issues. These contributors perform initial triage of incoming issues and periodically scrub the issue tracker.
This section provides some tips and tricks about triaging issues submitted to the Jenkins core.
Jenkins core uses the JENKINS project for issue tracking.
This project is shared between the Jenkins core components and plugins,
and the Jenkins core is scattered across multiple components: core, remoting, cli, winstone-jetty, etc.
In addition to it, there is a default _unsorted component which is recommended by default for users
who do not know what is the root cause of an issue they experience.
Searching for all Jenkins core issues is not trivial, and we provide Jira filters for it.
-
Community rating in Jenkins Regular (Weekly) and LTS releases. Such ratings allow users to reference issues they experienced with new Jenkins core releases, and it helps to discover regressions in the core causing instability or unexpected plugin failures.
-
Jenkins core triage board - Lists untriaged and recent issues in the Jenkins core and bundled components.
-
Core maintainers board - Lists unresolved recent regressions, unresolved recent core bugs, and popular new issues. This dashboard can be used to discover issues that might be related to the recent changes in the Jenkins core.
Initial issue triage has the following objectives:
-
Perform initial triage of an issue. Issue triage team members are not expected to perform a full analysis of the issue (though they are welcome to do so!). The main goal is to ensure that an issue report is legitimate and that it contains enough information to be processed. It is fine to request additional information from submitters and/or to refer them to reporting guidelines:
-
Verify the issue component. It is essential to ensure that the
componentfield references the right component (the Jenkins core, a plugin, etc.) so that an issue can be discovered and processed by a component maintainer. When moving an issue, assign the issue to theautomaticassignee so that the maintainer gets a notification. Not all components have a default assignee, and it is perfectly fine to leave the assignee field empty. -
Verify the issue type.
Bugshould be used for bug reports. All other issue types are considered as requests for enhancements, and there is no practical difference for the Jenkins core. -
Verify the issue metadata: Jenkins version, environment, labels, etc. Such metadata is useful for further triage and issue discoverability. There are some labels used in Jenkins Jira dashboard and filters, e.g.
jcasc-compatibility,java11-compatibility,jep-200, etc. Assigning such labels helps users and maintainers to discover issues and act on them. There is no list of such "common labels" recommended for use. Some labels can be found in similar issues or documentation linked from system log entries in the reports. -
Move security issue to the
SECURITYproject. Sometimes the issue reporters do not follow the vulnerability reporting process and report security issues in public. If you see such issues, move them to theSECURITYproject so that the security team takes care of their triage. Note that the required fields are different between projects, so some manual updates might be required when moving them. -
Label regressions and CC stakeholders if an issue is reported as a regression with a clear root cause, please set a
regressionlabel and, if applicable, CC contributors of a change which led to the regression. -
Resolve invalid issues and support requests. Sometimes Jenkins Jira is used as a support portal. We do not want to encourage that. Jenkins Jira is an issue tracker, and we expect reporters to investigate issues on their side to an extent that they can be reviewed by maintainers. For support requests users are expected to use mailing lists, chats and other resources (e.g. Stackoverflow). It is fine to link users to this page.
-
Resolve duplicates. It is often that the same issue is already reported in the Jenkins database. Newly reported duplicates can be just resolved with a
Duplicateresolution, and linked to the original issue.
Further triage focuses on confirming the issue and defining a potential resolution. It can be performed by Issue Triage Team Members if they want to dive deeper, or they can leave it to component maintainers.
Triage objectives:
-
Confirm reported defects. Try to reproduce the issue or analyze the codebase. If the issue is legitimate, it is great to explicitly confirm it in a comment.
-
Nice2Have: Define next steps. If possible, define a potential resolution for the issue. If you do not plan to work on the issue in foreseeable future, it is great to explicitly highlight that by unassigning the issue and inviting the reporter and other contributors to submit a fix.
-
Nice2Have: Highlight newcomer-friendly issues. Newcomer-friendly issues are instrumental for an onboarding new code contributors to the project. They are linked from contributing guidelines. If you see a simple issue but do not plan to work on it, put a
newbie-friendlylabel on it so that somebody could pick it up.
In addition to the initial triage, it is a good practice to sometimes review previously reported issues so that we could minimize backlog of issues and simplify search by users.
-
Triage reopened issues. Same as for newly reported issues, it is great to process reopened issues if they are not acted on by the issue assignees. Often such issues can be resolved with a request to report a new issue if an issue is reopened due to another issue.
-
Resolve stale untriaged issues. Issue reporters may become unresponsive before their issue can be fully triaged. If there is a reported issue which does not contain data for reproducing the issue, it is fine to resolve them after a 2-week timeout with the
IncompleteorCannot reproduceresolution. -
Resolve/update obsolete issues. Sometimes issues become obsolete due to other changes in the Jenkins core (e.g. feature removal), and such issues can be closed. Same for detaching functionality from the Jenkins core and plugins, issues can be reassigned to the new Jira component so that they are removed from the core backlog.
Requests for enhancement (RFEs) include the New Feature and Improvement types in Jenkins Jira.
The process to triage them might be different than bug reports.
because it is not always possible to say whether a request should be implemented in the Jenkins core,
an existing or a new plugin.
In the case of doubt, it is fine to just skip an issue or CC subject matter experts who could advice.
For RFEs which are not related to the Jenkins core or plugins,
it is possible to set the plugin-proposals component.
Note that this component is not regularly scrubbed,
and it can be considered only as a pool of ideas somebody could implement.
It is a good practice to set expectation in a comment when updating the RFE.
Some submitters and users tend to ping triage contributors to ask about the fix ETA. In some cases, they may also assign the issue and keep pinging. It is fine to not answer these questions on such pings and to refer requestors to this document, triage team members are not responsible for handling the ticket after initial triage.
Other materials which might help:
-
Contribute to Jenkins / Code or Contributing to the Jenkins Core - extended version of "feel free to contribute".
-
Jenkins Support Disclaimer Page - for those requesters who expect quick response and SLA.
-
Plugin Adoption Policy - for pings in not actively maintained components.
-
Jenkins Code of Conduct - when it gets ugly.
Jira workflow for the JENKINS project has two similar states: Resolved and Closed.
Historically the issues are rarely being closed, and all dashboard and Jenkins processes interpret resolved issues as closed.
The main difference is that the Resolved issues can be reopened by users while Closed ones can be reopened by admins only.
For triage purposes, it is recommended to use the Resolved state if there is a chance that the issue will be reopened by the reporter or other contributor
(e.g. resolving due to inactivity, disagreement with the resolution, etc.).
Merge process can be initiated once a pull request matches the requirements:
-
Pull request is compliant with requirements to submitters (see the pull request template)
-
There are at least 2 approvals for the pull request and no outstanding requests for change
-
Conversations in the pull request are over OR it is explicit that a reviewer does not block the change (often indicated by line comments attached to an approving PR review, or by using the term "nit", from "nit-picking")
-
Changelog entries in the PR title and/or Proposed changelog entries are correct and reflect the current, final state of the PR
-
Proper changelog labels are set so that the changelog can be generated automatically. List of labels we use for changelog generation is available here.
-
If the change needs administrators to be aware of it when upgrading, the
upgrade-guide-neededlabel is present and there is aProposed upgrade guidelinessection in the PR title (example). This is usually the case when a data migration occurs, a feature has been removed, a significant behavior change is introduced (including when there is a way to opt out), or in general when we expect at least a large minority of admins to benefit from knowing about the change, e.g. to apply a new option. -
If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as
lts-candidateto be considered (see this Jira query).
Jenkins security updates are coordinated with the LTS calendar and if the weekly release on the weekend before an LTS release introduces regressions, users of the weekly line may have to choose between security fixes and a working Jenkins. The Jenkins security team will usually send a "pre-announcement" to the advisories list on Wednesday or Thursday the week before release, but that’s not always doable. For these reasons, the following changes should not be merged during the week before LTS releases (weeks 3, 7, 11, 15, etc. on the page linked above):
-
Changes that could be considered risky (relatively high risk of introducing regressions), as they could make users of Jenkins weekly releases choose between getting security fixes, and having a functioning Jenkins
-
Very large changes (in terms of lines changed), because the Jenkins security team needs to prepare security fixes for the weekly release line in a very short period of time.
If the change is ready but it is not a good time, consider labeling the pull request with the on-hold label.
Make sure to add a comment explaining why it was put on hold.
Once the checklist is passed, a Core PR Reviewer or a Maintainer can mark the pull request for merge.
-
ready-for-mergelabel is set -
An explicit comment is added to the pull request so that other repository watchers are notified. Example: Thanks to all contributors! We consider this change as ready to be merged towards the next weekly release. It may be merged after 24hours if there is no negative feedback
-
Jenkins Security Team uses a different process for security issue fixes. They are reviewed and integrated by the Security team in private repositories. Security hardening and enhancements go through the standard process.
-
Release Team members are permitted to bypass the review/merge process if and only if a change is needed to unblock the security release. Common review process is used otherwise.
-
24 hours waiting period after adding the
ready-for-mergelabel is not required for:-
changes that do not result in changes to the primary functionality, such as typo fixes in documentation or localization files
-
changes which do not affect the production code: Jenkinsfile tweaks, tools inside the repo, etc.
-
broken master build
-
Sometimes we have pull requests which include dozens of commits including many non-substantial changes (merge commits, addressing review comments, etc.). We do not require contributors to spend time on cleaning it up, because core maintainers can squash PRs during the merge. Reviewers can add a squash-merge-me label during reviews to highlight that it is needed.
At the same time, we do not require any pull request to be merged as a single commit. Multiple commits are useful in many cases.
When do we merge pull requests as is?
-
There is only one commit with a reasonable commit message
-
There are multiple atomic commits. Each commit has a reasonable message and can be compiled on its own
-
Example:
-
Commit 1:
[JENKINS-1234] - Reproduce the issue in tests -
Commit 2:
[JENKINS-1234] - Fix the issue by updating Foo
-
-
-
There are multiple commit authors who expressed the desire to keep commit history as is. By default we do not consider multiple authors as a blocker for squash, because GitHub now supports co-authors
When do we squash commits?
-
We squash commits when core maintainers decide to do so (
squash-merge-melabel), usually when the conditions above are not met. -
There is no strong requirement to squash merge pull requests at the moment, so there might be deviations from the merge policy in practice.
Jenkins Weekly releases are managed by the Jenkins Release Team which has access to the dedicated release environment within the Jenkins Infrastructure. References:
Jenkins also offers the LTS Release Line. It is maintained by the Jenkins Release Team which coordinates backporting and release candidate testing. Any Jenkins contributors are welcome to participate in backporting and release candidate testing.
-
Backporting discussions happen through the developer mailing list.
-
Backports are submitted as pull requests with the into-lts label.
-
Release candidate testing is announced in the developer mailing list. Discovered issues should be submitted to Jenkins Jira and then referenced in the release candidate testing thread.
-
Core maintainers board - Lists unresolved recent regressions, unresolved recent core bugs, and popular new issues.