Jenkins: Pipeline sh bad substitution error

Question

A step in my pipeline uploads a .tar to an artifactory server. I am getting a Bad substitution error when passing in env.BUILD_NUMBER, but the same commands works when the number is hard coded. The script is written in groovy through jenkins and is running in the jenkins workspace.

sh 'curl -v --user user:password --data-binary ${buildDir}package${env.BUILD_NUMBER}.tar -X PUT "http://artifactory.mydomain.com/artifactory/release-packages/package${env.BUILD_NUMBER}.tar"'

returns the errors:

[Pipeline] sh
[Package_Deploy_Pipeline] Running shell script
/var/lib/jenkins/workspace/Package_Deploy_Pipeline@tmp/durable-4c8b7958/script.sh: 2: 
/var/lib/jenkins/workspace/Package_Deploy_Pipeline@tmp/durable-4c8b7958/script.sh: Bad substitution
[Pipeline] } //node
[Pipeline] Allocate node : End
[Pipeline] End of Pipeline
ERROR: script returned exit code 2

If hard code in a build number and swap out ${env.BUILD_NUMBER} I get no errors and the code runs successfully.

sh 'curl -v --user user:password --data-binary ${buildDir}package113.tar -X PUT "http://artifactory.mydomain.com/artifactory/release-packages/package113.tar"'

I use ${env.BUILD_NUMBER} within other sh commands within the same script and have no issues in any other places.

Answer 1

This turned out to be a syntax issue. Wrapping the command in ''s caused ${env.BUILD_NUMBER to be passed instead of its value. I wrapped the whole command in "s and escaped the nested. Works fine now.

sh "curl -v --user user:password --data-binary ${buildDir}package${env.BUILD_NUMBER}.tar -X PUT \"http://artifactory.mydomain.com/artifactory/release-packages/package${env.BUILD_NUMBER}.tar\""

Answer 2

In order to Pass groovy parameters into bash scripts in Jenkins pipelines (causing sometimes bad substitions) You got 2 options:

The triple double quotes way [ " " " ] OR the triple single quotes way [ ' ' ' ]

1. In triple double quotes you can render the normal parameter from groovy using ${someVariable} ,if it's environment variable ${env.someVariable} , if it's parameters injected into your job ${params.someVariable}

example:

     def YOUR_APPLICATION_PATH= "${WORKSPACE}/myApp/"

      sh """#!/bin/bash
      cd ${YOUR_APPLICATION_PATH}
      npm install
      """

2. In triple single quotes things getting little bit tricky, you can pass the parameter to environment parameter and using it by "\${someVaraiable}" or concating the groovy parameter using ''' + someVaraiable + '''

examples:

   def YOUR_APPLICATION_PATH= "${WORKSPACE}/myApp/"

   sh '''#!/bin/bash
          cd ''' + YOUR_APPLICATION_PATH + '''
          npm install
    '''

OR

   pipeline{
     agent { node { label "test" } }
     environment {
       YOUR_APPLICATION_PATH = "${WORKSPACE}/myapp/"
     }

     continue...
     continue...
     continue...

     sh '''#!/bin/bash
          cd "\${YOUR_APPLICATION_PATH}"
          npm install
    '''

    //OR
    sh '''#!/bin/bash
          cd "\${env.YOUR_APPLICATION_PATH}"
          npm install
    '''

Answer 3

Actually, you seem to have misunderstood the env variable. In your sh block, you should access ${BUILD_NUMBER} directly.

Reason/Explanation: env represents the environment inside the script. This environment is used/available directly to anything that is executed, e.g. shell scripts.

Please also pay attention to not write anything to env.*, but use withEnv{} blocks instead.

Answer 4

Usually the most common issue for:

Bad substitution

error is to use sh instead of bash.

Especially when using Jenkins, if you're using Execute shell, make sure your Command starts with shebang, e.g. #!/bin/bash -xe or #!/usr/bin/env bash.

Answer 5

I can definitely tell you, it's all about sh shell and bash shell. I fixed this problem by specifying #!/bin/bash -xe as follows:

node {
    stage("Preparing"){
        sh'''#!/bin/bash -xe
            colls=( col1 col2 col3 )
            for eachCol in ${colls[@]}
            do
              echo $eachCol
            done
        '''
    }      
}

Answer 6

I had this same issue when working on a Jenkins Pipeline for Amazon S3 Application upload.

My script was like this:

pipeline {  
  agent any
  parameters {
    string(name: 'Bucket', defaultValue: 's3-pipeline-test', description: 'The name of the Amazon S3 Bucket')
    string(name: 'Prefix', defaultValue: 'my-website', description: 'Application directory in the Amazon S3 Bucket')
    string(name: 'Build', defaultValue: 'public/', description: 'Build directory for the application')
  }  
  stages {  
    stage('Build') {  
      steps {
        echo 'Running build phase'
        sh 'npm install' // Install packages
        sh 'npm run build' // Build project 
        sh 'ls' // List project files
      }  
    }  
    stage('Deploy') {  
      steps {  
        echo 'Running deploy phase'
        withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', accessKeyVariable: 'AWS_ACCESS_KEY_ID', credentialsId: 'AWSCredentials', secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) { 
          sh 'aws s3 ls' // List AWS S3 buckets
          sh 'aws s3 sync "${params.Build}" s3://"${params.Bucket}/${params.Prefix}" --delete' // Sync project files with AWS S3 Bucket project path
        }  
      }   
    }
  }
  post { 
    success {  
      echo 'Deployment to Amazon S3 suceeded'  
    }  
    failure {  
      echo 'Deployment to Amazon S3 failed'  
    }  
  }  
}

Here's how I fixed it:

Seeing that it's an interpolation call of variables, I had to substitute the single quotation marks (' ') in this line of the script:

sh 'aws s3 sync "${params.Build}" s3://"${params.Bucket}/${params.Prefix}" --delete' // Sync project files with AWS S3 Bucket project path

to double quotation marks (" "):

sh "aws s3 sync ${params.Build} s3://${params.Bucket}/${params.Prefix} --delete" // Sync project files with AWS S3 Bucket project path

So my script looked like this afterwards:

pipeline {  
  agent any
  parameters {
    string(name: 'Bucket', defaultValue: 's3-pipeline-test', description: 'The name of the Amazon S3 Bucket')
    string(name: 'Prefix', defaultValue: 'my-website', description: 'Application directory in the Amazon S3 Bucket')
    string(name: 'Build', defaultValue: 'public/', description: 'Build directory for the application')
  }  
  stages {  
    stage('Build') {  
      steps {
        echo 'Running build phase'
        sh 'npm install' // Install packages
        sh 'npm run build' // Build project 
        sh 'ls' // List project files
      }  
    }  
    stage('Deploy') {  
      steps {  
        echo 'Running deploy phase'
        withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', accessKeyVariable: 'AWS_ACCESS_KEY_ID', credentialsId: 'AWSCredentials', secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) { 
          sh 'aws s3 ls' // List AWS S3 buckets
          sh "aws s3 sync ${params.Build} s3://${params.Bucket}/${params.Prefix} --delete" // Sync project files with AWS S3 Bucket project path
        }  
      }   
    }
  }
  post { 
    success {  
      echo 'Deployment to Amazon S3 suceeded'  
    }  
    failure {  
      echo 'Deployment to Amazon S3 failed'  
    }  
  }  
}

That's all

I hope this helps

Answer 7

I was having the issue with showing the {env.MAJOR_VERSION} in an artifactory of jar file . show I approaches by keeping of environment step in Jenkinsfile.

pipeline {
agent any
environment {
MAJOR_VERSION = 1
}

stages {
stage('build') {
  steps {
      sh 'ant -f build.xml -v'
 }
}
}
post {
 always{
   archiveArtifacts artifacts: 'dist/*.jar', fingerprint: true
 }
}
}

I got the issue solved and then it was not showing me bad substitution in my Jenkins build output. so environment step plays a more role in Jenkinsfile.