Overview for Plugin Maintainers

This site is the new docs site currently being tested. For the actual docs in use please go to https://www.jenkins.io/doc.

The SECURITY project in the Jenkins Jira is a non-public issue tracker for security vulnerabilities. Public knowledge of these vulnerabilities could put our users at risk, so access to issues is restricted.

We grant access to specific SECURITY issues in Jira to maintainers of affected plugins. They’re often in a much better position to fix issues in their plugins, and the security team needs to coordinate releases with them even if they don’t end up authoring the fix.

We identify maintainers using uploader permissions metadata, as that is the most reliable source for this kind of information in the Jenkins project. This is also where you would explicitly specify who should be the preferred contact for security issues. Learn more.

Make sure your notification email address configured in Jira is up to date and that you don’t ignore notifications sent from Jira. Obsolete email addresses and ignored emails will delay the initial contact.

Support for Maintainers

In general, the Jenkins security team helps maintainers understand reports, provides relevant documentation and examples, and reviews proposed security fixes.

Jira

The Jenkins security team can grant access to the private SECURITY issue in Jira to additional users upon request. This is the preferred information sharing platform between reporters, maintainers, and the Jenkins security team.

Maintainers have limited permissions in this tracker, because we use issue statuses to keep track of issues. If you need the issue status or field values to be changed, please request such a change in a comment.
GitHub

The Jenkins security team can create private GitHub repositories to support fix development and review. Any number of GitHub users can be added to these repositories (Read or Write permission) upon request. Jenkins security team members will be available to review and test security fixes.

Artifactory

To support staging of security updates (see below), the Jenkins security team will create a private Maven staging repository.

Recommended Practices

Due to the sensitive nature of security vulnerabilities, the following practices should be followed in resolving the issue and releasing the plugin:

Practices for Issue Handling

  • Be responsive. Not responding to reports of security vulnerabilities will not make them go away. Even if the reporter does not set a coordinated disclosure deadline, the Jenkins security team will publish security vulnerabilities in security advisories even if no fix exists. If you’re no longer maintaining the plugin, please let us know so we can proceed appropriately.

  • Respect embargoes. A few security issues are embargoed, which means information about them must not be shared with others, typically because a new class of issue was discovered and similar vulnerabilities exist in other components.

  • Keep any information gained from the SECURITY tracker private until a version of the plugin containing the fix is released, as it may put users at risk.

  • Keep the source code for your fix private until a fixed version of your plugin has been released, e.g. by pushing it to GitHub, even it’s just your fork that nobody watches. If you need assistance in fixing the vulnerability, ask for it in the SECURITY issue. The Jenkins security team also offers to review your security fix; please ask for a private repository in the jenkinsci-cert GitHub organization to be created, and submit your fix there.

  • Coordinate the date and time of the release of the fixed plugin with the Jenkins security team. This will allow for assigning a CVE ID prior to release, and publishing a security advisory informing users of the security vulnerability, and how to address it.

Practices for Fix Development

Security fixes should be developed on a branch that will be submitted as a pull request in a private repository in the jenkinsci-cert GitHub organization to facilitate fix review.

  • Keep the security fix minimal. This is not an opportunity to address some old TODO comments, refactor code, or modernize the plugin. All of that can be done in a separate update; either at least a week or two before the security update would be published, or after the security update is published. In some cases, this even means implementing a "hack" solution, and holding off on the superior fix until after the security update is published. This limits the potential risk of regressions in the security update due to unrelated changes. Please also refrain from reformatting code you’re not otherwise changing to make it easier to review the security fix.

  • Keep the security update minimal. If the target branch of the security fix has unrelated changes, branch off from the previous release (specifically the prepare for next development iteration commit in case of manual releases, and the tagged commit in case of JEP-229 CD) and target that branch instead, or make sure to release the plugin’s unrelated changes a week or two before the security release. This also limits the potential risk of regressions in the security update due to unrelated changes.

  • Keep source code private until the security updates are published. The staging instructions below prevent pushing the release commits, which would result in premature disclosure due to delays in the infrastructure, or the staging process. Once you’ve staged a plugin release to a staging repository, push the branch/commits and release tag to the private repository in the jenkinsci-cert GitHub organization only. From there, the commits and tag will be pushed to the public repository after the security update is made public, typically within an hour.

  • No late changes. Don’t add changes to the security release unless they’re essential for the security fix to work after the fix has been reviewed and approved. Otherwise this will invalidate the testing and review that has happened before.

See also How We Fix Security Issues for general guidelines.

Process Overview

The following is a rough approximation of the typical recommended lifecycle of a plugin security issue in the SECURITY tracker:

  1. Someone reports an issue in a plugin.

  2. The security team evaluates the report.

    1. If we’re confident it is not a security vulnerability, the issue is moved to the JENKINS project.

  3. We determine the maintainer and/or security contact. For the purposes of this document, only plugins maintained by people not also on the security team are considered here. Maintainers and security contacts are decided based on release permissions.

  4. We assign the issue to the plugin maintainers (resulting in access to the issue), and notify them in a comment. If necessary, plugin maintainers are contacted repeatedly, in Jira and/or directly via email, to notify them of this issue.

  5. The plugin maintainer reviews the reported issue.

  6. If they identify it as an actual security vulnerability, they (and, if requested, the security team) work to resolve the issue.

    1. The security team provides a private repository for that work in the jenkinsci-cert GitHub organization.

    2. Work usually happens on a branch, and a corresponding pull request will be used for review.

  7. A date and time of the release is coordinated between the security team and maintainers. The security team handles CVE ID assignment, advance notification of users, and creation of the security advisory.

  8. The security fix is merged. For details, see Merge the Fix below.

  9. A version of the plugin containing the fix is uploaded to a staging repository (see Stage with Maven below). The release tag is created and the tag and updated branch are pushed to the private GitHub repository.

  10. The staged release is published by the security team, and the corresponding security advisory is published and announced. The source code is pushed to the public GitHub repository. The issue in the issue tracker is closed.

Staging Procedure

Staging includes all the steps described below and is typically done a few days before the scheduled advisory. This strikes a balance between allowing the longest possible time for reviews and testing, while minimizing the risk of releases failing on the intended release date and acknowledging time constraints and different time zones of everyone involved.

The Jenkins security team needs to prepare a Maven staging repository before security updates can be staged, so follow the instructions below only once you know the Maven repository to stage to. Make sure to follow instructions provided by the Jenkins security team if they deviate from the instructions below.

Backporting

At this point, consider the need for backports. Some administrators may be unable to install the latest version of your plugin. Usually this is because the plugin depends on a recent weekly release of Jenkins. Users of the LTS line of Jenkins releases will not be offered a plugin update that requires a weekly version of Jenkins higher than the LTS baseline, even when they’re already on the latest version of Jenkins available to them.

We recommend that all security fixes are made available to both current weekly and current LTS releases. If the current LTS release is the first in its line (e.g., 2.375.1), also consider making the security fix available to users of the previous release, as many administrators may not immediately update.

Additionally, the security fixes should generally be made available to users who are already on the latest version of the plugin available to them, rather than an earlier version. This will make it easy to update to a version with the fix through the plugin manager, without potentially losing features by doing what amounts to a manual downgrade.

Backporting Example

Suppose it is early 2023 and you’re preparing a security fix for a plugin you maintain. The most recent LTS releases are Jenkins 2.361.4 and Jenkins 2.375.1, and the current weekly release is 2.387. The plugin’s latest release, version 2.2, has a Jenkins dependency of 2.380 because it makes use of a new feature in that release. The most recent lower Jenkins dependency of the plugin was Jenkins 2.361.3 in plugin version 2.0.

In this case, instances on both 2.361.4 and 2.375.1 will be able to install a security fix provided on top of version 2.0 of the plugin. Instances on the weekly release line will be able to install a security fix provided on top of version 2.2. So the plugin should get updates with the security fix based on top of version 2.2 (e.g., 2.2.1 or 2.3) and 2.0 (e.g., 2.0.1 or 2.0.0.1, whichever version would be between 2.0 and the next release that already exists).

Now consider the case of the plugin version 2.1 previously raising the core dependency from 2.361.2 to Jenkins 2.370 (before 2.2 raised it to 2.380). In this case, three separate releases are now needed to cover users of Jenkins 2.361.4, 2.375.1, and 2.387:

  • Jenkins 2.361.4 needs a backport on top of plugin version 2.0 (e.g., 2.0.1 or 2.0.0.1) with a Jenkins dependency of 2.361.2.

  • Jenkins 2.375.1 needs a backport on top of plugin version 2.1 (e.g., 2.1.1 or 2.1.0.1) with a Jenkins dependency of 2.370. While they could in theory install the backport on top of version 2.0, this would effectively be a downgrade, and remove all features added between versions 2.0 and 2.1 of the plugin (if this is even possible without data loss).

  • Only weekly releases like 2.387 will be able to update to the next regular plugin release (e.g., 2.2.1 or 2.3) with a Jenkins dependency of 2.380.

Merge the Fix

First, prepare the release branch: If there are unrelated, unreleased changes on the default branch, create a new branch based on the previous release. For plugins with manual release process this is the prepare for next development iteration commit. For plugins delivered with JEP-229 CD automated releases, it is the commit that got tagged as a release.

If all unreleased changes are unlikely to introduce regressions, e.g., minor documentation fixes or similar, it’s also reasonable to include those changes in the security fix. In general we recommend caution when doing this however: even safe-looking dependency updates can easily cause regressions and make administrators choose between a plugin release that is safe and one that works. It is always safest to not include unrelated changes in security fixes.

Next, use the Git command line to squash-merge pull request branches in the private jenkinsci-cert repository (git merge --squash <BRANCH>).

Do not merge using the GitHub UI: It would by default add a reference to the private PR (like #1), which is confusing (or at best useless) when seen in the public repository.

As commit message, use just the ID of the security issue (for example [SECURITY-12345]), without further details. At this point, it may not be clear exactly how the fix will be announced and documented, and discrepancies between the commit message and security advisory would be confusing. If you didn’t develop the fix, make sure to credit the original author by adding --author='Actual Author <author@example.org>' to the git commit command. To find out what name and email address to put there, see git log <BRANCH>.

Backporting

Backport Branch Preparation

Similar to when there are unreleased changes on the default branch, a new branch needs to be created when backporting, unless a branch with the correct baseline already exists, e.g., from an earlier security update.

Create a new branch based on the release that the fix should be backported to:

If the plugin recently switched from manual releases to JEP-229 CD automated releases (or vice versa), each branch may require different procedures.

The branch is conventionally named something like 2.18.x (if it’s based on 2.18) or 1234.x (if it’s based on the JEP-229 version 1234.vabdef123).

Plugins with manual releases

For plugins with manual release process, you could set the version number used for releases from the backport branch at this point, for example:

mvn versions:set -DnewVersion=2.18.1-SNAPSHOT -DgenerateBackupPoms=false

This is optional, as the version number can still be specified when staging.

Plugins with JEP-229 CD

For plugins delivered with JEP-229 CD automated releases, you also need to manually specify a custom version number for the backport branch. Otherwise, the automatically generated, linearly increasing version number would not indicate the existence of a backport branch.

Edit all applicable pom.xml files (usually just one) and add a prefix corresponding to the baseline version prefix to the <version>.[1] For example, suppose the plugin should get a backport on top of the version 12345.vabdef123. The pom.xml needs to be changed as follows:

--- a/pom.xml
+++ b/pom.xml
@@ -10,12 +10,12 @@
   <artifactId>very-cool-plugin</artifactId>
-  <version>${changelist}</version>
+  <version>12345.${changelist}</version>
   <packaging>hpi</packaging>

Commit this change, e.g.: git commit -a -m "Prepare for 12345.x" As a result, the backport will typically have a version like 12345.12347.vabdef123.

In case of manually managed version prefixes, keep those. For example:

--- a/pom.xml
+++ b/pom.xml
@@ -10,12 +10,12 @@
  <artifactId>very-cool-plugin</artifactId>
-  <version>${revision}.${changelist}</version>
+  <version>${revision}.12345.${changelist}</version>
  <packaging>hpi</packaging>
Applying the Fix

When backporting the fix to an earlier line, you need to use git cherry-pick -x <commit-id>, specifying the commit that you merged earlier, instead of git merge --squash <branch>. Trying to squash-merge the fix branch will also merge in all unrelated changes between the baseline that the fix was developed on, and the earlier line you’re trying to merge it into.

Beware of potential merge conflicts if the security fix changes lines that were modified between the baseline against which the fix was developed, and the version you’re backporting to. In extreme cases, it can be useful to basically develop the fix multiple times, with individual PRs each targeting a different (backporting) branch, to let reviewers confirm conflict resolution was done correctly.

It’s a good practice to run mvn clean verify now, even when there are no merge conflicts, to ensure that all tests pass.

Stage with Maven

Plugins with manual releases

For Maven-based plugins that are usually released manually using mvn release:prepare release:perform, use the following command to stage the plugin release. REPONAME is a placeholder for the name of the Maven staging repository that is provided by the Jenkins security team.

mvn -DstagingRepository=maven.jenkins-ci.org::default::https://repo.jenkins-ci.org/REPONAME -DpushChanges=false -DlocalCheckout=true org.apache.maven.plugins:maven-release-plugin:3.0.0:prepare org.apache.maven.plugins:maven-release-plugin:3.0.0:stage

The staging repository name (REPONAME) is never a GitHub URL or GitHub repository name.

Always use release:stage instead of release:perform. While the latter also supports specifying a different repository, it’s not a necessary parameter, so typos in the system property can result in accidental uploads to the public repository, disclosing any vulnerabilities early.

Plugins with otherwise automated releases (JEP-229 CD)

For Maven-based plugins that use JEP-229, use the following commands to stage the plugin release. REPONAME is a placeholder for the name of the Maven staging repository that is provided by the Jenkins security team.

Build and deploy the release

mvn -Dset.changelist -DaltDeploymentRepository=maven.jenkins-ci.org::default::https://repo.jenkins-ci.org/REPONAME deploy

The staging repository name (REPONAME) is never a GitHub URL or GitHub repository name.
If your account is allowed to upload plugin releases, be very careful to enter the -DaltDeploymentRepository argument correctly, as any typo in the name would end up deploying to the regular Maven repository, creating a public release.
Create the tag

git tag "$( mvn -q help:evaluate -Dexpression=project.version -DforceStdout -Dset.changelist )"

The specific syntax depends on your shell. This command works for bash and zsh. You can also just run mvn … first, then copy the output and provide it as argument to git tag manually.

Since this release is staged from a private GitHub repository, be careful not to merge changes into the public GitHub repository that would trigger further releases until after the private commits have been made public after advisory publication. Otherwise, administrators may not be offered the staged release containing the security fix, but another release created from the public repository.

Push Commits and Tags

After uploading, push the release commits/branch(es) and tag(s) to the private GitHub repository in the jenkinsci-cert GitHub organization, but NOT to the public (jenkinsci) repository.

# Command format
git push <remote> [<branch or tag> ...]

# Example
git push jenkinsci-cert master v2.6.3.x matrix-auth-2.6.6 matrix-auth-2.6.3.1

The Jenkins security team will typically push these tags and branches to the public repository after the security advisory has been published, or will comment on the private SECURITY issue asking the maintainer to do it otherwise.

Keep Documentation Private

Documentation (outside the plugin source code, e.g. in the Jenkins wiki or in GitHub releases metadata) should only be published once the security advisory has been published.

Resources

1. mvn versions:set does not work, it really needs to be done manually.