rsync alternative to Jenkins Copy Artifacts plugin?

5
votes

I'm working on a set of builds related to our online images (such as wordpress content). Overall it's a large workflow, so it's broken into several jobs.

A couple jobs need to copy a large number of artifacts from other jobs; I've been using the Copy Artifacts plugin but it's too slow for my case and rsync would be much better suited.

Is it possible to effectively get the source artifact directory for the upstream build so that I can pass it to rsync in place of using the Copy Artifacts plugin? I'd like to have a simple script like:

rsync -a --delete $UPSTREAM_ARTIFACT_DIR $WORKSPACE

1
jenkinsjenkins-plugins
There's an issue for the copy artifact plugin to do this for you. Unfortunately it's been sitting since 2013 with no progress. This seems like a big missing feature in copy-artifact-plugin to me. issues.jenkins-ci.org/browse/JENKINS-20814 - NateW
The OP's issue isn't necessarily a request for incremental copies, just because he's suggesting to use rsync to solve the problem. Turns out Jenkins uses a single limited java thread implementation for copying artifacts, and it gets abysmal performance. In my case I'm getting about <200Mbps throughput copying large (>1GB) artifacts from master to agent, whereas I can confirm at the OS level (debian stretch on both sides) using iperf3 that I'm getting 6-9Gbps throughput. I think that ANYONE could benefit by dispatching artifact mgmt to a system call to rsync/scp/etc.. - timblaktu

1 Answers

1
votes

The upstream artifacts are accessible via what appear to be well defined URLs. For example, the following URL enables one to access the last good builds' artifacts:

http://jenkins/job/job_name/lastSuccessfulBuild/artifact/

Can even specify the axis on a multiconfiguration project, if that's required:

http://jenkins/job/job_name/label=foo,arch=x86/lastSuccessfulBuild/artifact/