Reusing workspace across multiple nodes in parallel stages

1
votes

I have parallel stages setup in my Jenkins pipeline script which execute tests on separate nodes (AWS EC2 instances)

e.g.

stage('E2E-PR') {
    parallel {
        stage('Cypress Tests 1') {
            agent { label 'aws_slave_cypress' }
            steps {
                runE2eTests()
            }
        }
        stage('Cypress Tests 2') {
            agent { label 'aws_slave_cypress' }
            steps {
                runE2eTests()
            }
        }
    }
}

I would like to re-use the checked out repo\generated workspace generated from the parent node used at the start of my pipeline rather than each parallel stage checking out it's own copy.

I came across an approach using sequential stages, and nesting stages within stages to share workspaces across multiple stages, which I tried like below

parallel {
    stage('Cypress Tests') {
        agent { label 'aws_slave_cypress' }
        stages {
            stage('Cypress 1') {
                steps {
                    runE2eTests()
                }
            }
            stage('Cypress 2') {
                steps {
                    runE2eTests()
                }
            }
        }
    }
}

But I can see from my Jenkins build output that only one aws instance gets spun up and used for both of my nested stages which doesnt give me the benefit of parallelism.

I have also come across the stash\unstash commands, but I have read these should be used for small files and not for large directories\entire repositories?

What's the right approach here to allow me to have parallel stages across multiple nodes use the same originally generated workspace? Is this possible?

Thanks

1
jenkinsjenkins-pipeline
Can you provide a bit more information about why you'd like to do this? (Shallowly) cloning a git repo isn't that different from stashing and unstashing the files. – flup
So the slave agents used for the tests, just need to be able to reference some test files, so I could use stash\unstash approach thinking about it some more. – mindparse

1 Answers

0
votes

I suppose you need to send generated files to s3 on post step for example. Download these files on the first step of your pipeline.

Think about storage, which you need to share between jenkins agents.