sed | replace value with environment variable containing double quotes

Question

my environment variable contained double-quotes as follows:

$echo $CONNECT_SASL_JAAS_CONFIG
org.apache.kafka.common.security.plain.PlainLoginModule required username="USER" password="XXXX/YYY";

output from "export" command:

declare -x CONNECT_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"USER\" password=\"XXXX/YYY\";

my template file content is as follows (props.txt):

sasl.jaas.config = JAAS_CONF

I try to replace the JAAS_CONF with the env variable using following command :

 sed  -e  "s/JAAS_CONF/$CONNECT_SASL_JAAS_CONFIG/g;" props.txt

This gives the following error :

/tmp# sed -e "s/JAAS_CONF/$CONNECT_CONSUMER_SASL_JAAS_CONFIG/g;" props.txt sed: -e expression #1, char 168: unknown option to `s'

Any tips on how to fix this ?

UPDATE:

Note:

• "password" contains "/" character
• There is a new-line char at the end of CONNECT_CONSUMER_SASL_JAAS_CONFIG
• I need to need escape the double-doubles back again.

So finally my solution was:

  export TMP_JAAS=${SASL_JAAS_CONFIG//$'\"'/'\\"'}
  export PARSED_JAAS=${TMP_JAAS//$'\n'/\ }
  sed  -e  "s|JAAS_CONF|${PARSED_JAAS//$'\n'/\ }|g" file.cong > /tmp/app.conf

Answer 1

Are you sure that the content of $CONNECT_SASL_JAAS_CONFIG is what you expect ? Make sure that you have the correct content in $CONNECT_SASL_JAAS_CONFIG. Because the index "168" in the error message shows that you have a resulting command line of at least 168 chars. But it should not.

I can reproduce this kind of erro with :

    $ var=xxx/
    $ sed  -e  "s/JAAS_CONF/$var/g;" props.txt
    sed: -e expression #1, char 17: unknown option to `s'

So, try to make sure that you peek the correct variable by printing the content of the variable to make sure it is what you expect and secure your command line with "{}" :

     $ sed  -e  "s/JAAS_CONF/${CONNECT_SASL_JAAS_CONFIG}/g;" props.txt

Answer 2

Seem you are calling wrong variable name "CONNECT_CONSUMER_SASL_JAAS_CONFIG" is not what you show in echo command. Try with:

sed "s/JAAS_CONF/$CONNECT_SASL_JAAS_CONFIG/g" props.txt

Output:

sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule required username="USER" password="XXXXYYY";

EDIT: in case special character in your string, you can use another delimiter like # or ~ or | instead of /. It look like:

sed "s#JAAS_CONF#$CONNECT_SASL_JAAS_CONFIG#g" props.txt

Answer 3

To see how the variable is being expanded in the sed command:

set -xv
sed  -e  "s/JAAS_CONF/$CONNECT_SASL_JAAS_CONFIG/g;" props.txt
set +xv`

This should produce something like:

+ sed -e 's/JAAS_CONF/org.apache.kafka.common.security.plain.PlainLoginModule required username="USER" password="XXXX/YYY";/g;' props.txt
sed: -e expression #1, char 109: unknown option to `s'

The embedded double quotes and forward slash (in the variable) are causing your problems; the forward slash can be addressed with a different delimiter (as @Rachid has pointed out).

The double quotes can be addressed a few ways, one idea:

$ sed  -e  's|JAAS_CONF|'"${CONNECT_SASL_JAAS_CONFIG}"'|g;' props.txt

# or

$ sed  -e  "s|JAAS_CONF|""${CONNECT_SASL_JAAS_CONFIG}""|g;" props.txt 

# will produce:

sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule required username="USER" password="XXXX/YYY";

Where:

• we break the sed command pattern into 3 parts ... `s|JAAS_CONF|' + "${variable}" + '|g;'
• the use of single or double quotes for the 1st and 3rd parts provides the same results (in this case)

Answer 4

Using envsubst:

Assume you have a string STRING which you want to replace with the variable VARIABLE_STRING. Then I would suggest the following approach:

1. substitute all variables of interest with a proper shell variable name (include the $)
2. export all variables of interest
3. Use envsubst to perform the substitution

Here is an example:

$ export VARIABLE_STRING='foo "bar" qux'
$ echo "This is the string 'STRING'" | sed 's/\bSTRING\b/$VARIABLE_STRING/g' | envsubst '$VARIABLE_STRING'
This is the string 'foo "bar" qux'

You can adapt this to your liking. Please read [U&L] Replacing only specific variables with envsubst for further information.

Using GNU awk:

If you cannot make use of envsubst, you can write something in awk. GNU awk can access the environment using the ENVIRON array which is a map from variable name to variable value. Example:

awk 'BEGIN{print ENVIRON["HOME"]}' 

This will print the content of $HOME

So now you could something like this:

1. Create a map from string to replace into environment
2. Loop over the names:

This would look something like this:

awk 'BEGIN{ map["NAME1"]="ENVIRONMENT_NAME1"
            map["NAME2"]="ENVIRONMENT_NAME2"
            map["NAME3"]="ENVIRONMENT_NAME3" }
     { for(i in map) gsub("/<" i "/>",ENVIRON[map[i]]) }
     { print }' file

Similar to the sed in the first solution, I've added a way to ensure that the string NAME1FOO is not being replaced by ENVIRONMENT_NAME1FOO. This by adding word_boundaries /< and /> in the regex.

**Note: this method might create loops. Imagine you have an environment variable:

ENVIRONMENT_NAME1="NAME2"

Then the string NAME1 might be replaced with the content of ENVIRONMENT_NAME2. I say might because everything depends on the order of transversal in the for loop which is awk dependent.