Single jenkins job for all repositories in a Github organization

Question

We own a Github organization with hundreds repositories that are authored by contributors. We would like to setup a Jenkins server that performs certain standard tasks for e

Answer 1

You have more than one requirement here. Let's go through one by one.

a) Jenkins GitHub Organization: This will scan all your GitHub organization, and create as many jobs as are needed to build your repositories because having just one job on Jenkins is not the standard. Basically, you lost history data (Jenkins has no idea it is building different stuff at every iteration). It says on the help "Scans a GitHub organization (or user account) for all repositories matching some defined markers.".

b) Try to see Jenkins as an automator, not something which will host all build/deploy logic. What I do is to create files like "build.sh", "deploy.sh", and so on. This way, I can build and deploy directly from my shell. So only after that I create scripts for Jenkins, that just call build/deploy scripts, no matter what they actually do. Jenkins doesn't need to know. A side effect is that all your projects "can be built the same way", no matter if they are NodeJS, Python, or whatever. Of course, you might need extra dependencies in some cases, and Docker can really help here.

c) I did something similar in the past, having less jobs than repositories/branches/pull-requests. Jenkins is kind of dump, and a few plugins can help here. But in your case, if you really want to have one job, you only need a regular parametrized job. The trick is that your Github Organization global webhook will not point to Jenkins. It needs to point to somewhere else, some code you maintain. This code can parse the Github payload, analyze it, can eventually calls back GitHub ("is there a pull request for this branch? no? then forget it") to enhance its decision tree, and at the end, trigger that single job on Jenkins with all parameters you were able to capture. Such parameters will tell the single job which repo to clone, env to deploy to, and that is it. You already know scripts names, since they are standard.

d) That said, I would ask... do you need a Jenkins? Can this parser-little-software actually clone your repo, and run a few scripts inside a Docker container? A builder-docker-container which has every dependency inside?

e) About "talking back" to GitHub, I did that using Python. There are GitHub libraries so I was able to get stuff back from Jenkins, and do API posts to feed GitHub with build statuses. Since I was actually using a Jenkins instance, my tool was a man-in-the-middle-broker. In your case, for a single job, a Docker container will play the role nicely.

Hopes this helps with a different perspective.

If you actually want to use a Jenkins instance, most of what I said here can still be used.