Fast-turnaround Jenkins Pipeline script development

4
votes

Is there a way to write Jenkins Pipeline script interactively, like you can run other Groovy in the Jenkins Script Console? Ideally I effectively want "pipeline groovysh in Jenkins".

AFAICS the script console doesn't set up any of the environment needed to make Pipeline work so you can't just run ad-hoc chunks of pipeline jobs there.

I'm aware of the "Replay" option, but it's painfully slow for a remote Jenkins and doesn't integrate well into VCS.

Background: As I'm doing a lot of work on Jenkins pipelines lately I'm finding that the test cycle is becoming painful. Ironic really. Commit, push to working branch, "build with parameters" (or rebuild) a job pointing to the the working branch for the pipleine script, wait for results, examine console log, repeat.

Tedious.

I've improved it a little by having my test job auto-trigger on pushes to my pipeline repo but that's still pretty clumsy.

Firing up a local Jenkins for testing is not very practical - the pipelines need to interact with a private Docker registry, need a specific plugin config, need some specific worker types, etc. It also doesn't help a great deal with the turnaround.

So how do others do it better? jenkins-cli remote control? Other?

1
jenkinsgroovyjenkins-pipeline
Also jenkins-cli doesn't seem to be able to interact with the ssh-agent to avoid prompting for the private key password each and every time... – Craig Ringer
It's not really a solution, rather a more general advice: I split my pipelines into smaller pipelines, had each pipeline archive its artifacts and used copyArtifacts on the next pipeline. Then I wrote another pipeline to orchestrate all the other ones. By having each pipeline reside in its own stage you can reduce much of the traction involved in pipeline testing, because you can restart the orchestrating pipeline from the most recent stage. – towel

1 Answers

1
votes

There are two ways of working on pipelines that I have used that alleviate the painful develop-test-fail-fix cycle that you are experiencing:

1) Unit Test Based Test Method

Write the pipelines in a unit test framework with everything completely mocked using an IDE like IntelliJ IDEA. Create unit tests developed in Groovy using Spock or JUnit and making use of the JenkinsPipelineUnit test framework. This can be OK if you are comfortable writing unit tests and mocking, but many people are not. It worked for me for most of my pipelines.

2) Using Docker and File System SCM Plugin

Spin up Jenkins in Docker on your development machine configuring it as near as possible to your production system using Jenkins Configuration as Code (JCasC). In the Docker container running Jenkins map / bind mount a directory on the host system containing all the pipeline code into the Docker container. You can then use a Jenkins plugin called "File system SCM Plugin". This FS SCM plugin will allow you to configure the pipeline jobs in the Dockerized Jenkins to get their pipeline code directly off the file system on your host mapped into the Docker container. So what you do is open up the pipeline code in an IDE on your host system, edit as much as you like and execute the job in the Dockerized Jenkins without having to commit. Then when done with debugging / developing, commit your code for the production server. This method is based on work and ideas from Oleg Nenashev at CloudBees and it is worth taking a look at his presentation "How to Develop Your Jenkins Pipeline Locally". To set up Jenkins in Docker with most of it configured from YAML have a look through the JCasC README file and also at the Jenkins Docker project to see how to configure the installed plugins.

You may also get some useful information on this subject from the Jenkins Pipeline Authoring special interest group.