Infrastructure
As an independent open source project, the Jenkins project maintains most of its own infrastructure including services which help to keep the project running. The kind of things that fall into "infrastructure" can span from operating virtual machines, containers, configuring network or developing and maintaining project-specific applications to make the development of Jenkins core and plugins more efficient.
Because we strongly believe in Open Source principles, we also apply them to our infrastructure. As such we consider ourself as an open infrastructure project where everybody is invited to learn, share, contribute.
Team
Jenkins infrastructure is maintained by the Jenkins Infrastructure team. This team consists of Jenkins contributors and volunteers who provide a best-effort support depending on their availability and other commitments. We are always looking for contributors who are interested to improve Jenkins services and to help with maintaining them.
Meetings
Jenkins Infra team has a weekly meeting on Tuesday, 14:30 UTC. This event is a regularly scheduled meeting to sync on the various ongoing infrastructure initiatives within the Jenkins project. This meeting is open to everyone who is interested. See the Event Calendar for meeting links.
You can find Infra team meeting agenda and meeting notes on jenkins-infra/documentation.
Infrastructure
Provisioning
The Jenkins project runs the most of its infrastructure on Azure, sponsored by Continuous Delivery Foundation. This infrastructure is provisioned from the repository jenkins-infra/azure using Terraform and using AKS charts.
While we try to stick to Azure, we still have machines sponsored by different organizations such as OSUOSL, AWS, Rackspace or CloudBees.
Configuration Management
Infrastructure
The Jenkins project’s infrastructure is managed and orchestrated via a git repository named jenkins-infra/jenkins-infra using Puppet. We are also using a Kubernetes cluster orchestrated with jenkins-infra/charts using Helm & Helmfiles.
Monitoring
Infrastructure is monitored via Datadog. Datadog configuration is automated with Terraform on jenkins-infra/datadog
We also provide different dashboards to share the health of our systems.
Services
Another major aspect of the Jenkins Infrastructure project, is the maintenance of all services either provided by us or third parties. Here is a non-exhaustive list of services that we provide and maintain.
Self-hosted services in Kubernetes
| Service | Issue tracker | Repository and links |
|---|---|---|
- |
||
Infra CI - https://infra.ci.jenkins.io/ |
- |
|
Release CI - https://release.ci.jenkins.io/ |
||
Private artifacts - https://repo.private.jenkins.io/ |
||
Keycloak identity management |
||
LDAP |
||
SSL certificate renewal |
||
Plugin Site API Backend |
||
Rating metric |
||
Self-hosted services
| Service | Issue tracker | Repository and links |
|---|---|---|
Mirrorbits helm chart, mirror list, mirror stats, Running a mirror |
||
|
old mirrors.jenkins.io Puppet scripts (search for |
|
Code published with GitHub pages |
||
VPN |
|
SaaS
| Service | Provider | Issue tracker | Repository |
|---|---|---|---|
JFrog |
- |
||
GitHub |
- |
||
Datadog |
|||
Pagerduty |
- |
||
GitLab |
- |
||
Fastly |
- |
||
Namecheap |
- |
||
Linux Foundation |
- |
Contributing
Our infrastructure is an open infrastructure project made by and for the Jenkins community. In other words, it’s a contributors driven project. While we can’t share publicly everything like secrets, certificates,… we still try to be as transparent as possible so that everybody can understand and improve our infrastructure without having privileged accesses. If you have any idea that could help the infra or interest the community, feel free to make suggestions.
In order to contribute to infrastructure project, we ask people to follow the next steps
Pick up a task => Communicate => Implement => Deploy => Review
Pick Up A Task
To keep track of the work that needs to be done on the Jenkins infrastructure project, we use the Github help desk repository. If you want to contribute, the first step is to find the issue in this project that you want to work on, assign it to you, then communicate about it (see Communicate).
If you can’t find an appropriate issue, please create a new one with a clear description:
-
Why (what is the problem to solve - high level value)?
-
What (what your proposal to solve the problem)?
-
How (what are the technical changes to do)?
You may also specify components and finally you can communicate about it, using the suggestions from the next section.
|
Good First Issues
If you want to start contributing on the Jenkins infrastructure,
you can find a list of "first good issues" to be considered (they all have the label |
Communicate
Before any implementation, it’s important to verify that first, there is (still) a need for some implementation and then that no work has been done in the past. The best way for that is to either look after similar issues, ask on IRC or on the mailing list. You can also join our weekly meetings to discuss and coordinate the changes.
When the subject is too broad or hard to explain in few lines, we write an IEP document which stand for "Infrastructure Enhancement Proposal", the purpose of this document is to explain why we need something, how we want to solve it, and why we took a final decision. Finally, once you have your ticket id, you can start looking for knowledgeable people.
Anyway keep in mind that it’s always better to have too much information than too little and in the end you’ll probably be the best person who can work on your case.
+----------------------------------+
| |
| Pick up or Create INFRA Ticket |
| |
+----+----+------------------------+
| | If no responses after few days
| | promote it on
| | +------------------------------------------+
| | | |
| +--------------------> IRC: Libera Chat #jenkins-infra <----+
| | | | |
| | +------------------------------------------+ |
| | +------------------------------------------+ |
| | | | |
| +--------------------> Mail: jenkins-infra@googlegroups.com <----+
| | | |
| +------------------------------------------+ |
| If the topic is too big |
| |
| +-------------------------------------------+ |
| | | |
+--------------------> IEP: https://github.com/jenkins-infra/iep |--------+
| |
+-------------------------------------------+
Implement
Once there is an agreement about the approach and before any changes, we ask contributors to respect the following rules.
Those rules are just what we consider "best practices" for a contributors driven project and can be adapted depending on specific git repositories.
- Rule #1: Everything is in a git repository under the jenkins-infra organization.
-
So it’s easier for everybody to find/review/audit changes and suggest improvements.
- Rule #2: All changes are validated by at least one regular infra contributor via Pull Request.
-
So we always have different people who understand infrastructure changes.
Rem: Non regular contributors are more than welcome to share their expertise or just ask question which also help to spot incoherences.
Code Reviews Purpose:
-
Educate author and the team about the changes in code being made
-
Discover logic or security issues not covered by tests
-
Gather improvement suggestion for code readability or efficiency
- Rule #3: All Changes are tested on ci.jenkins.io
-
So we feel more comfortable when merging PRs and we avoid regression issues.
- Rule #4: Everything is automated.
-
So we only have one source of truth and we don’t break others people works. And if it’s not possible, then it needs to be well communicated and documented cf. rule #1.
- Rule #5: All changes follow the Github workflow.
Fork project -> Create Feature Branch -> Open Pull Request -> Ask Review -> Merge
Deployment
The deployment step is the only moment where we need approval from someone with elevated permission. As already mentioned, even if we try to be as open as possible, we don’t want to share privileged accesses with every contributors even if we trust them and that mainly for security reasons.