Why openssl generated signature looks different than pyjwt generated signature

Question

I'm trying to verify a JWT signature created using PyJWT but it fails. It works if the signature is created using openssl command.

I have two scripts.

Python script which uses PyJWT and PyOpenSSL to create a JWT. It spits out the JWT on stdout if successful.

Bash script that does two experiments.

Runs Python script and captures the JWT. Decodes the header, payload parts. Constructs the message to sign. Uses openssl commands to create a signature (using dgst), verify using public key.
From the JWT, decodes the heder, payload, signature parts. Uses openssl commands to verify JWT created signature using public key.

Experiment 1 succeeds and 2nd fails.

I have gone through few answers on similar problem and tried to ensure

I use urlsafe b64 encoding
I use openssl dgst command to create hash out of input message which gets signed.
Use echo -n when moving data

Signatures generated using openssl commands succeeds verification. Signature generated using PyJWT fails openssl verification. JWT created using PyJWT decodes successfully using PyJWT.

I've run out of ideas on debugging this. Any help, direction is much appreciated.

jwttest.py (needs PyJwt, PyOpenSSL)

import sys
import base64
import jwt

'''
# Just in case keys are not available
privkey="-----BEGIN PRIVATE KEY-----\n" \
"MIIEuwIBADANBgkqhkiG9w0BAQEFAASCBKUwggShAgEAAoIBAQC8JuSavoT2vyZi\n" \
"2RsQ0BFBySQVbK2JJPXXi1O33D3JZhUjQhyRh4yaG6ubirdM/r0eZeADdD047T8i\n" \
"wXFxH/dCZun7AF1dBcXxx1/Jr1VsNiaymXPrnRUBSrCSjqNEJIEbRiKna/JY8i6q\n" \
"cXZRTOYead8fXrTIWJRUTx5F4MNAXqdZ5v/oAsxXc1E7il55vOBDgKZSW4rV7SRC\n" \
"W4zqQA/PE9FmkrygV2x4Kbzc3aObgjgMYR99o6vyOkIhqycBPivwvyLgymDah07r\n" \
"1OR/u2z2w1zX5wD9NyLx2wqbpFQypDr0bOW4CsXuHcPEo206g//6EYlkGOW16ZZr\n" \
"Rhv9t8ipAgMBAAECgf92U/9xUmBMzepWQDPFXxV7SgRndPGuTpBN/lGoT9qLzqd8\n" \
"hRdybsz+HmjOaW7d/Vbyxx8bDP9zzcDnGsE9Y90c5ZxBPvl4hyj15W1YaexPIb80\n" \
"k01T4HZVWaOyiAIl2M9ZV8JziG/hgG3Yw4KlnrcaaXrpP6ZyWULvwtJHIBOrZsei\n" \
"CIlF/cmmwS1uHmXLGTUmZyusOeMfhcQFIh1p3r6dw9Dt6r022FvuMWrqxfsdEAP0\n" \
"kCfhuyyczPIX50gMnVN07ApqH/aMh+77fa8z6pEhqqQeQTkM6ttVzz8l9bEdaHyd\n" \
"T3IXI0G2BfuokGKLAWEKCTFfJbEODDSaSacua9UCgYEA4x+POFom84xJhqgvQdAL\n" \
"ZbDjNadRA6avJ2lDz6PhPSAzq6lleyUj7wv6p1O+37MriPKqkwMXWfZuPXaYEApv\n" \
"VQRxNCofeLjYoueWRAxIts6//1L3ueW1KmQBqGAC5oq3+vy372EvIubmzAmr5hcb\n" \
"3jmzCBjsWzTS/z3s2DpVa78CgYEA1BLjb5zrpnrMP/JJ5SaE9JzGRZ2wmB6wph83\n" \
"U3YYm9j/Qtih24I48EON7zLz/utzGZstoHEwAe8d0F1mxFslsj0A0Y6nIE1rH0Za\n" \
"MAsjkxlU0e+iaZZ9WSVvg7D3i7+dYVKpvHW7SKKYcqvJWNs7u+qkGUqiGuDoEJPr\n" \
"JM5lhZcCgYBlFXa4recIHjfbJp9huyZXbBKznnQAG/94mvEDSPzGJ77Xd90iobUM\n" \
"f1hfgHZDOYr2uIoSRB3wfC00TcP/36UNQZzgip7XK+2/EzNdtdnnAr2Q9Wwr4IBx\n" \
"DXFvbsvbr4GSw0dZ0vcXoYy24tcO4NoWXbfAstb/ANOnpffzhILgIQKBgAw43nSz\n" \
"RX19vEG/M/UJ6EW0t1SRxvitZB7e07BysO5ibiurEoD1G1T1f7uWYyuA5ExIfjOt\n" \
"8kdaQYydpWuRmTWRgHeTUhxxecf+pPn52l4C6rmCpwiQzL6Tgr7DNzENpQNT4UZk\n" \
"PpvsCv8o2VzOnb2xwy1V+Mu1xIoYDEg9wOoXAoGBAN8jWNrtxMQCgCdjwc5gOgs5\n" \
"fRdGVa2yly4dXCzVJl6DN3sEmIOuIZfMXeSNuA40AR6qT36F72WcCVE63ZzKvqtX\n" \
"WPjy/jPyQ9pp1dBKKRQCtktt7Iovuqfkc/XtGg1vv5PCefR4f2GKPCvCL3mPKOEB\n" \
"QWCwIEiwAse5z50XfN10\n" \
"-----END PRIVATE KEY-----\n"

pubkey = "-----BEGIN PUBLIC KEY-----\n" \
"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvCbkmr6E9r8mYtkbENAR\n" \
"QckkFWytiST114tTt9w9yWYVI0IckYeMmhurm4q3TP69HmXgA3Q9OO0/IsFxcR/3\n" \
"Qmbp+wBdXQXF8cdfya9VbDYmsplz650VAUqwko6jRCSBG0Yip2vyWPIuqnF2UUzm\n" \
"HmnfH160yFiUVE8eReDDQF6nWeb/6ALMV3NRO4peebzgQ4CmUluK1e0kQluM6kAP\n" \
"zxPRZpK8oFdseCm83N2jm4I4DGEffaOr8jpCIasnAT4r8L8i4Mpg2odO69Tkf7ts\n" \
"9sNc1+cA/Tci8dsKm6RUMqQ69GzluArF7h3DxKNtOoP/+hGJZBjltemWa0Yb/bfI\n" \
"qQIDAQAB\n" \
"-----END PUBLIC KEY-----\n"
'''

data={}
'''
data=
{
  "iss": "http://localhost:8080/uaa/oauth/token",
  "user_name": "DBA",
  "nbf": 1592397960,
  "exp": 1621295999
}
'''

def padstr(str):
        padding = len(str) % 4
        if padding:
                data = str
                data += '=' * (4 - padding) 
                return data
        return str

if len(sys.argv) < 3 :
        print "Usage: " + sys.argv[0] + " <private key file>" + " <public key file>"
        exit(1)

with open(sys.argv[1], "r") as f:
        privkey=f.read()
#print privkey

with open(sys.argv[2], "r") as f:
        pubkey=f.read()
#print pubkey

encoded=jwt.encode(data, privkey, algorithm="RS256")
decoded=jwt.decode(encoded, pubkey, algorithms="RS256")

if( decoded != data ):
        print "decoded JWT does not match data"
        print "Data=" + str(data)
        print "Decoded=" + decoded
        exit(1)
else:
        print encoded
        exit(0)

sign_verification.sh

#!/bin/bash

function create_true_digest {
        data_file=$1
        digest_file=$2
        echo "Creating SHA256 digest for $data_file"
        ls -l $data_file
        cat $data_file
        echo
        openssl dgst -sha256 $data_file  | cut -d" " -f2 | xxd -r -p > $digest_file
        echo "digest stored in $digest_file"
        cat ${digest_file} | base64 -w 0 | sed 's/+/-/g' | sed 's/\//_/g' > ${digest_file}.base64
        echo "Base64 encoded digest stored in ${digest_file}.base64"
        ls -l ${digest_file}
        cat ${digest_file}.base64
        echo
        echo "............................................................"
}

function create_digest {
        create_true_digest $1 $2
}

function sign_data {
        data_file=$1
        pvtkey_file=$2
        signature_file=$3
        echo "Signing $data_file with $pvtkey_file"
        ls -l $data_file
        openssl dgst -sha256 -sign $pvtkey_file -out $signature_file $data_file
        #openssl dgst -sha256 -sign $pvtkey_file -binary -out $signature_file $data_file
        echo "Signature stored in $signature_file"
        cat $signature_file | base64 -w 0 | sed 's/+/-/g' | sed 's/\//_/g' > ${signature_file}.base64
        echo "Base64 encoded signature stored in ${signature_file}.base64"
        ls -l ${signature_file} ${signature_file}.base64
        echo "............................................................"
}

function verify_signature {
        signature_file=$1
        pubkey_file=$2
        data_file=$3
        echo "Verifying signature $signature_file with $pubkey_file against $data_file"
        openssl dgst -sha256 -verify $pubkey_file -signature $signature_file $data_file
        echo "............................................................"
}

function experiment_1 {
        echo "Experiment 1: This experiment accepts a message to verify"
        echo "  It uses openssl commands to generate a signature and verifies it"
        echo

        message_to_verify=$1
        #message_to_verify="{\"alg\":\"RS256\",\"typ\":\"JWT\"}.{}"
        pvtkey_file=$2
        pubkey_file=$3

        data_file="sign_data.txt"
        rm -f $data_file
        echo -n $message_to_verify > $data_file
        echo "Data stored in $data_file"

        digest_file="data_digest.bin"
        rm -f $digest_file
        rm -f ${digest_file}.base64
        create_digest $data_file $digest_file

        signature_file="signature_r.bin"
        rm -f $signature_file
        rm -f ${signature_file}.base64
        sign_data $data_file $pvtkey_file $signature_file

        verify_signature $signature_file $pubkey_file $data_file

        echo "............................................................"
}

function experiment_2 {
        echo "Experiment 2: This experiment accepts a message to verify and signature to verify against"
        echo "  It uses openssl commands to verify the signature against the message"
        message_to_verify=$1
        signature=$2 # keep it encoded
        pubkey_file=$3

        data_file="sign_data.txt"
        rm -f $data_file
        echo -n $message_to_verify > $data_file
        echo "Data stored in $data_file"
        cat $data_file
        echo

        signature_file="signature_r.bin"
        rm -f $signature_file
        rm -f ${signature_file}.base64
        echo -n $signature | sed 's/-/+/g' | sed 's/_/\//g' | base64 -di >$signature_file
        echo "Signature stored in $signature_file"
        cat $signature_file | base64 -w 0 | sed 's/+/-/g' | sed 's/\//_/g' > ${signature_file}.base64
        echo "Base64 encoded signature stored in ${signature_file}.base64"
        ls -l ${signature_file} ${signature_file}.base64

        verify_signature $signature_file $pubkey_file $data_file

        echo "............................................................"
}

if [ $# -lt 2 ]
then
        echo "Usage: $0 <private key file> <public key file>"
        exit
fi

pvtkey_file=$1
pubkey_file=$2

# Get Base64URL encoded JWT
# Need python 2.7, PyJWT, PyOpenSSL
JWT=`/usr/local/software/python/python2/bin/python jwttest.py $pvtkey_file $pubkey_file`

# Split parts
echo "Ignore base64 warnings..."
header=`echo -n $JWT | cut -d"." -f1 | sed 's/-/+/g' | sed 's/_/\//g' | base64 -di`
payload=`echo -n $JWT | cut -d"." -f2 | sed 's/-/+/g' | sed 's/_/\//g' | base64 -di`
signature=`echo -n $JWT | cut -d"." -f3` # decoding will be done by routines
message_to_verify="${header}.${payload}"

experiment_1 $message_to_verify $pvtkey_file $pubkey_file
experiment_2 $message_to_verify $signature $pubkey_file

I have tried following steps so far - Python PyCrypto code to create JWT by hand instead if using PyJWT and decode it using PyJWT.decode. It worked fine. - Instrumented PyJWT code just before it attempts to sign the payload just to verify my understanding of message content that is signed. That's correct and that's what I have used in this example. I believe the code I have attached is the smallest reproducible exmple. There may be lot of debug statements, but all it does is accept message, sign it, verify it. — ShaileshM
I just did a python debugging session in JWT code and found hash calculated for the input is different compared to what openssl dgst command calculates. Another mystery. — ShaileshM

ShaileshM ShaileshM · Accepted Answer · 2020-07-20T15:50:15

Resolved. Stupid mistake. I was assuming message to verify the signature against is 'header.payload'. It is 'b64_urlsafe(header).b64_urlsafe(payload)'. I reread the JWT articles I was referring and they do point it out. Sorry to have wasted your bandwidth.

Why openssl generated signature looks different than pyjwt generated signature

1 Answers