StreamingTemplateEngine in Jenkins Pipeline vs. in plain groovy

Question

In a Jenkins Pipeline we want to create a config File with variable content and so we use StreamingTemplateEngine. Now we have to build a Config File with optional lines depending on the variable map. This example was our first try (during development / testing first written in plain groovy):

import groovy.text.StreamingTemplateEngine

def vars=[
    "KEY2": "VAL2",
]

templateText='''
FIXKEY=FIXVAL
<%
if(KEY1) out.print "KEY1="+KEY1+"\\n";
if(KEY2) out.print "KEY2="+KEY2+"\\n";
%>
'''

def engine = new StreamingTemplateEngine()
def template=engine.createTemplate(templateText)
configContent = template.make(vars).toString()
println "CONTENT FROM TEMPLATE IS:"
println configContent;

So as "KEY1" does not exist in the map we wanted the resulting config String to be:

FIXKEY=FIXVAL
KEY2=VAL2

But we got this exception:

    Exception in thread "main" groovy.text.TemplateExecutionException: Template execution error at line 4:
         3: <%
     --> 4: if(KEY1) out.print "KEY1="+KEY1+"\n";
         5: if(KEY2) out.print "KEY2="+KEY2+"\n";

    at main.run(main.groovy:34)
    at main.main(main.groovy)
Caused by: groovy.lang.MissingPropertyException: No such property: KEY1 for class: groovy.tmp.templates.StreamingTemplateScript1

So we learned, that every variable used in the template has to be defined in the map, as this code works:

import groovy.text.StreamingTemplateEngine

def vars=[
    "KEY1": "VAL1",
    "KEY2": "VAL2",
]

templateText='''
FIXKEY=FIXVAL
<%
if(KEY1) out.print "KEY1="+KEY1+"\\n";
if(KEY2) out.print "KEY2="+KEY2+"\\n";
%>
'''

def engine = new StreamingTemplateEngine()
def template=engine.createTemplate(templateText)
configContent = template.make(vars).toString()
println "CONTENT FROM TEMPLATE IS:"
println configContent;

But results in:

FIXKEY=FIXVAL
KEY1=VAL1
KEY2=VAL2

Now we could define "KEY2":false in the map, but with a huge amount of vars this would be quite much more work as just to define the neccessary vars and leaving the not needed vars completely out.

After a bit of searching we found this:

StreamingTemplateEngine exception MissingPropertyException

We tried the second solution mentioned in this thread:

import groovy.text.StreamingTemplateEngine

def vars=[
    "KEY2": "VAL2",
].withDefault { false }

templateText='''
FIXKEY=FIXVAL
<%
if(KEY1) out.print "KEY1="+KEY1+"\\n";
if(KEY2) out.print "KEY2="+KEY2+"\\n";
%>
'''

def engine = new StreamingTemplateEngine()
def template=engine.createTemplate(templateText)
configContent = template.make(vars).toString()
println "CONTENT FROM TEMPLATE IS:"
println configContent;

And that works as expected, the resulting config content is:

FIXKEY=FIXVAL
KEY2=VAL2

Perfect, we thought, so now we want to use this snippet in a Jenkins Pipeline, but Jenkins behaves somehow different using this test stage code:

import groovy.text.StreamingTemplateEngine

[...]

           stage('test') {
                def vars=[
                    "KEY2": "VAL2",
                ].withDefault { false }

                templateText='''
    FIXKEY=FIXVAL
    <%
    if(KEY1) out.print "KEY1="+KEY1+"\\n";
    if(KEY2) out.print "KEY2="+KEY2+"\\n";
    %>
    '''
                def engine = new StreamingTemplateEngine()
                def template=engine.createTemplate(templateText)
                configContent = template.make(vars).toString()
                println "CONTENT FROM TEMPLATE IS:"
                println configContent;
            }

But the result in Jenkins is this:

[Pipeline] {
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] echo
15:07:10  CONTENT FROM TEMPLATE IS:
[Pipeline] echo
15:07:10  false
[Pipeline] }

Notice the single "false" ??!!

When "completing" the map like this:

import groovy.text.StreamingTemplateEngine

[...]

           stage('test') {
                def vars=[
                    "KEY1": "VAL1",
                    "KEY2": "VAL2",
                ].withDefault { false }

                templateText='''
    FIXKEY=FIXVAL
    <%
    if(KEY1) out.print "KEY1="+KEY1+"\\n";
    if(KEY2) out.print "KEY2="+KEY2+"\\n";
    %>
    '''
                def engine = new StreamingTemplateEngine()
                def template=engine.createTemplate(templateText)
                configContent = template.make(vars).toString()
                println "CONTENT FROM TEMPLATE IS:"
                println configContent;
            }

the content String is as expected:

[Pipeline] stage
[Pipeline] { (test)
[Pipeline] echo
15:09:06  CONTENT FROM TEMPLATE IS:
[Pipeline] echo
15:09:06  
15:09:06  FIXKEY=FIXVAL
15:09:06  KEY1=VAL1
15:09:06  KEY2=VAL2
15:09:06  
15:09:06  
[Pipeline] }

So why does Jenkins Pipeline Groovy behave different with the same Code Snippet than "Plain" Groovy?

Or is there even a completely different approach to solve the request "variable lines based on var - existence in the map"?

Thx for any hint!
T0mcat

Szymon Stepniak Szymon Stepniak · Accepted Answer · 2019-03-12T15:34:12

The root cause of the problem you face is the fact that the following expression:

def vars=[
    "KEY1": "VAL1",
    "KEY2": "VAL2",
].withDefault { false }

returns an instance of MapWithDefault<K,V> class. This object generates an issue inside the Jenkins pipeline because the pipeline uses Groovy CPS library for continuous-passing style transformation. This mode comes with some limitations. For instance, it requires that all objects you use in the pipeline have to be Serializable.

Pipeline scripts may mark designated methods with the annotation @NonCPS. These are then compiled normally (except for sandbox security checks), and so behave much like “binary” methods from the Java Platform, Groovy runtime, or Jenkins core or plugin code. @NonCPS methods may safely use non-Serializable objects as local variables, though they should not accept nonserializable parameters or return or store nonserializable values. You may not call regular (CPS-transformed) methods, or Pipeline steps, from a @NonCPS method, so they are best used for performing some calculations before passing a summary back to the main script. Note in particular that @Overrides of methods defined in binary classes, such as Object.toString(), should in general be marked @NonCPS since it will commonly be binary code calling them.

Source: https://github.com/jenkinsci/workflow-cps-plugin#technical-design

In the case of Groovy classes, this requirement is satisfied out-of-the-box, because every Groovy class implicitly implements Serializable interface. In the case of Java classes, this interface has to be implemented explicitly. As you can see this MapWithDefault<K,V> class is a Java class and it does not implement Serializable interface.

Solution 1: extract the logic to the `@NonCPS` method

Consider the following example:

import groovy.text.StreamingTemplateEngine

node {

   stage('test') {
        def vars=[
            "KEY2": "VAL2",
        ]

        String templateText='''
        FIXKEY=FIXVAL
        <%
        if(KEY1) out.print "KEY1="+KEY1+"\\n";
        if(KEY2) out.print "KEY2="+KEY2+"\\n";
        %>
    '''

        configContent = parseAsConfigString(templateText, vars)
        println "CONTENT FROM TEMPLATE IS:"
        println configContent;
    }
}

@NonCPS
def parseAsConfigString(String templateText, Map vars) {
    def engine = new StreamingTemplateEngine()
    def template=engine.createTemplate(templateText)
    return template.make(vars.withDefault { false }).toString()
}

In this case, a method parseAsConfigString handles the generation of a config string. Keep in mind that it accepts a normal hash map (which is serializable) and it transforms it to the MapWithDefault inside the @NonCPS method, so non-serializable object is not used outside the context of the @NonCPS method. The StreamingTemplateEngine object is also used inside the method, because this class does not implement Serializable interface, so it could cause some weird problems as well.

Solution 2: use `ConfigObject` instead

Even though the solution with template engine may work for you, I would suggest using ConfigObject instead. This class was designed for representing configuration objects and it has a few useful methods. You can create an instance of ConfigObject from any map, and then you can call prettyPrint() method to generate a String representation of a configuration. Consider the following example:

node {
   stage('test') {
       def map = [
                KEYVAL1: "VAL2",
                FIXKEY: "FIXVAL"
        ]

        def config = new ConfigObject()
        config.putAll(map)

        println config.prettyPrint()
   }
}

The output:

[Pipeline] stage
[Pipeline] { (test)
[Pipeline] echo
KEYVAL1='VAL2'
FIXKEY='FIXVAL'

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

The main difference between both approaches is that strings in the pretty print of a ConfigObject are wrapped with single quotes, which is something I would actually expect. In this approach all you have to do is to prepare a proper map that stores configuration options and transforming it to ConfigObject allows you to print it in the desired form.

StreamingTemplateEngine in Jenkins Pipeline vs. in plain groovy

1 Answers

Solution 1: extract the logic to the @NonCPS method

Solution 2: use ConfigObject instead

Solution 1: extract the logic to the `@NonCPS` method

Solution 2: use `ConfigObject` instead