Job inheritance in Jenkins jobs

20
votes

How do you handle mapping Jenkins jobs to your build process, and have you been able to build in cascading configurations on inheritance?

For any given build I'll have at least three jobs (standard continuous integration/nightly, security scan, coverage) and then some downstream integration testing jobs. The configuration slicer plugin handles some aspects cross jobs but each jobs is still very much its own individual entity with no relationship to the other jobs in its group.

I recently saw QuickBuild and it has job inheritance where a parent jobs can define a standard group of steps and its children can override and specialize. With Jenkins, I have copies of jobs, which is fine until I need to change something. With QuickBuild the relationship between jobs allows me to spread my changes with little effort.

I've been trying to figure out how to handle this in Jenkins. I could use the parameterized build trigger plugin to allow jobs to call others and override aspects. I'd then harvest the data from the called jobs to its caller. I suspect I'll run into a series of problems where there are aspects which I can't override which will force me to implement Jenkins functionality in my own script thus making Jenkins less useful.

How do you handle complexity in your build jobs in Jenkins? Have you heard of any serious problems with QuickBuild?

6
buildjenkinscontinuous-integrationquickbuild
I've been looking at these issues today although without any interest in QuickBuild. In particular I would like different configuration options for C-I and Nightly builds, such as checking out into clean workspaces vs doing a revert+update, how long build artifacts are stored, and how to archive or deploy artifacts -- while maintaining the same list of build instructions for each. The Matrix/multiconfig plugin doesn't give enough options. I think we could manage it with a sub-job that does just the build itself, called with a parameterized workspace -- but it's an added complexity. – CJBrew
If each of your jobs does one aspect of your CI then there shouldn't need to be any duplication of tasks between jobs. We have a build job that has a downstream job which stages which then runs the integration tests. If there's a desire to run some of these staged tests at regular timed intervals, this can be set up within those jobs as timed triggers too. This way the build job just builds, the testing job just does testing and so on. If you use the 'Archive for Cloned Workspace' task then the downstream job can get access to the build artefacts. Can expand in an answer if you'd prefer? – jbjon
I am wondering if one year later, you found a solution to this problem. – sorin
Do you have a suggestion how would I use the inheritance-plugin or the template plugin if I wish to inherit from Pipeline (if it is possible at all) ? – Guy Avraham
@GuyAvraham please take a look at my answer (last one on this question) for details how how to side-step the inheritance problem via design. I haven't tried that plugin because at present I don't have that need, – Peter Kahn

6 Answers

13
votes

I would like to point out to you the release of a plugin that my team has developed and only recently published under open source. It implements full "Inheritance between jobs".

Here for further links that might help you:

2
votes

I had pretty much the same problem. We have a set of jobs that needs to run for our trunk as well as at least two branches. The branches represent our versions, and a new branch is created every few months. Creating new jobs by hand for this is no solution, so I checked out some possibilities.

One possibility is to use the template plugin. This lets you create a hierarchy of jobs of a kind. It provides inheritance for builders, publishers and SCM settings. Might work for some, for me it was not enough.

Second thing I checked out was the Ant Script for job cloning, and his sibling the Bash Script. These are truly great. The idea is to make the script create a new job for, copy all settings from a template job, make changes as you need them. As this is a script it is very flexible and you can do a lot with that. Only drawback is, that this will not result in a real hierarchy, so changes in the template job will not reflect on jobs already cloned, only on jobs that will be created going forward.

Looking at the drawbacks and virtues of those two solutions, a combination of both might work best. You create a template project with some basic settings that will be true for all jobs, and then use a bash or ant script to create jobs depending on that template.

Hope that helps.

1
votes

I was asked what our eventual solution to the problem was... After many months of fighting with our purchasing system we spent around $4000 US on Quickbuild. In a about 2-3 months we had a templated build system in place and were very happy with it. Before I left the company we had several product groups in the system and were automating the release process as well.

Quickbuild was a great product. It should be in the $40k class but it's priced at much less. While I'm sure Jenkins could do this, it would be a bit of a kludge whereas Quickbuild had this functionality baked in. I've implemented complex behaviors on top of products before (e.g. merge tracking in SVN 1.0) and regretted it. Quickbuild was reasonably priced and provided a solid base for our build and test systems.

At present, I'm at a firm using Bamboo and hope its new feature branch feature will provide much of what Quickbuild can do

1
votes

EZ Templates plugin allows you to use any job as a template for other jobs. It is really awesome. All you need is to set the base job as a template:

enter image description here

* Usually you would also disable the base job (like "abstract class").

Then create a new job, set it to use the base job template, and save: enter image description here

Now edit the new job - it will include everything! (and you can override existing configurations).

Note: There's another plugin Template Project for configuration templates, but it was not updated recently (last commit on 2016).

0
votes

We use quickbuild and it seems to work great for most things. I have even been able to use their APIs to write custom plugins. One area where quickbuild is lacking is sonar integration. The sonar team has a Jenkins plugin and not one for quickbuild.

0
votes

Given that the goal is DRY (don't repeat yourself) I presently favor this approach:

  • Use jenkins shared library with jenkins pipeline unit to support TDD
  • Use docker images using groovy/python or whatever language you like to execute complex actions requiring apis etc
  • Keep the actual job pipeline very spartan (basically just for pulling build params and passing them to functions in shared library which may use docker images to do the work.

This works really well an eliminates the DRY issues around complex build jobs.

Shared Pipeline Docker Code Example - vars/releasePipeline.groovy

/**
 * Run image
 * @param closure to run within image
 * @return result from execution
 */
def runRelengPipelineEphemeralDocker(closure) {

    def result

    artifactory.withArtifactoryEnvAuth {
        docker.withRegistry("https://${getDockerRegistry()}", 'docker-creds-id') {
            docker.image(getReleasePipelineImage()).inside {
                result = closure()
            }
        }
    }
    return result
}

Usage example

library 'my-shared-jenkins-library'

releasePipeline.runRelengPipelineEphemeralDocker {
    println "Running ${pythonScript}"
    def command = "${pythonInterpreter} -u ${pythonScript} --cluster=${options.clusterName}"
    sh command
}