Groovy Script to list S3 files in Jenkins parameter value

Question

I am trying to build a Jenkins Jobs. Where I am using the Extensible plugin of Jenkins. This plugin has an option to specify the Groovy script.

Can anyone help me out to write groovy script to pull the list of files in a bucket?

I want to use one of the file name in Jenkins as a parameter.

Answer 1

You can use this script that I wrote, I'm using it as "active choice parameter" for Jenkins:

def sout = new StringBuilder(), serr = new StringBuilder()
def proc = 'aws s3 ls s3://bucket_name/'.execute()
proc.consumeProcessOutput(sout, serr)
proc.waitForOrKill(2000)
def values = "$sout".split('/')
def trimmedValues
def parameters=[]
values.each {  println "${it}" }
def cleanValues = "$sout".split('PRE')
def last = cleanValues.last().split('2018-12-17')[0]
cleanValues.each {  "${it}".toString(); 
                    trimmedValues = "${it}".trim();
                    parameters<<trimmedValues
                 }
parameters.remove(parameters.size() - 1);
parameters.add(last)
parameters

The "2018-12-17" split is because awscli returns the date at the end of the command:

This way I can use this parameter to determine what artifact/folder I'm taking from the bucket.

Answer 2

I'm using Active Choice Parameter in my Jenkins job with the following code:

def s3_path = "s3://bucket/path/to/files/"
def aws_s3_ls_output = "aws s3 ls ${s3_path} --region us-east-1".execute() \
                       | ['awk', '{ print $NF }'].execute()
//aws_s3_ls_output.waitFor() // might be needed if execution is too long
def files = aws_s3_ls_output.text.tokenize().reverse()

return files