Error starting hyperledger-composer network after Fabric and Composer version upgrade

Question

I've come across an error starting the hyperledger-composer network that isn't answered in the composer-wiki.

✖ Starting business network definition. This may take a minute...
Error: Error trying to start business network. Error: No valid responses from any peers. 
Response from attempted peer comms was an error: Error: transaction returned with failure: can't find PEM header: undefined 
Command failed

Checking pre-requisites,

Fabric 1.2
Composer 0.20.4
Node 8.12.0
Docker 18.01.1
"composer network install" was successful, with file appearing in the docker peer at /var/hyperleder/production/chaincodes

After running the "composer network start" command, a "docker ps" shows new docker instance with name:

dev-peer0.org1.example.com-<<business-network-name>>-0.0.7

But any attempt to ping this results in a failure like this:

Error: Error trying to ping. Error: make sure the chaincode <<business-network-name>> has been successfully instantiated and try again: getccdata composerchannel/<<business-network-name>> responded with error: could not find chaincode with name '<<business-network-name>>'

Checking the log of the dev-peer0, it ends with the following:

2018-11-05T05:03:18.227Z [4264161f] ERROR    :Composer                 :Init()                    can't find PEM header: undefined
2018-11-05T05:03:18.227Z [4264161f] VERBOSE  :Composer                 :@PERF Init()              Total (ms) duration for txnID [4264161fc30a61c70884d4c7efb460fea6a755d07bc4852875c393346795227a]: 929.00
2018-11-05T05:03:18.228Z ERROR [lib/handler.js] [composerchannel-4264161f]Calling chaincode Init() returned error response [can't find PEM header: undefined]. Sending ERROR message back to peer

The corresponding error in the peer0 log is a big larger:

2018-11-05 05:03:18.229 UTC [endorser] SimulateProposal -> ERRO 439d [composerchannel][4264161f] failed to invoke chaincode name:"lscc" , error: transaction returned with failure: can't find PEM header: undefined
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
    /opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:202
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
    /opt/gopath/src/github.com/hyperledger/fabric/core/endorser/support.go:131
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
    /opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:173
github.com/hyperledger/fabric/core/endorser.(*Endorser).SimulateProposal
    /opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:287
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
    /opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:501
github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
    /opt/gopath/src/github.com/hyperledger/fabric/core/handlers/auth/filter/expiration.go:61
github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
    /opt/gopath/src/github.com/hyperledger/fabric/core/handlers/auth/filter/filter.go:31
github.com/hyperledger/fabric/protos/peer._Endorser_ProcessProposal_Handler
    /opt/gopath/src/github.com/hyperledger/fabric/protos/peer/peer.pb.go:112
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).processUnaryRPC
    /opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:923
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).handleStream
    /opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1148
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1
    /opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:637
runtime.goexit
    /opt/go/src/runtime/asm_amd64.s:2361
2018-11-05 05:03:18.229 UTC [endorser] SimulateProposal -> DEBU 439e [composerchannel][4264161f] Exit

Since this last worked I have updated composer from 0.19 to 0.20.4, and taken Fabric from 1.1 to 1.2.

Googling suggests that this kind of error "can't find PEM header: undefined" is associated with a change in key signing. After tearing down Fabric I re-ran ./createPeerAdminCard.sh - is there another card or similar that needs to be re-created to accomodate the latest versions?

Are there any errors in the log of the dev-peer0... docker container? Similarly are there errors in the peer0.org1... container? It looks like you are starting v0.0.7 of your network - did v0.0.6 start OK? - if so what changes have you made between the versions? — R Thatcher
I have not seen that error, but if you suspect the card, then I would delete the existing cards using composer card delete then make sure the environment variable FABRIC_VERSION is set to hlfv12 - then run the createPeerAdminCard script again. — R Thatcher
one other comment, you should be using Fabric 1.2.1 specifically with Composer 0.20.4 (maybe that's what you meant, instead of 1.2 Fabric FYI) — Paul O'Mahony
@PaulO'Mahony thanks for the pointer, but I can't see 1.2.1 listed on the Fabric release page: github.com/hyperledger/fabric#releases. Can you confirm? — Capn Sparrow
@CapnSparrow - its there alright -> github.com/hyperledger/fabric/releases/tag/v1.2.1 — Paul O'Mahony

Capn Sparrow Capn Sparrow · Accepted Answer · 2018-11-06T12:45:09

Thanks to @R Thatcher for putting me onto the right direction. This was all down to mismatching cards, and was resolved by clearing out everything and starting again.

Specifically, in /fabric-dev-servers:

./stopFabric.sh 
./teardownFabric.sh
composer card list
composer card delete -c admin@<business-network-name>
composer card delete -c PeerAdmin@hlfv1
./startFabric.sh
./createPeerAdminCard.sh

Then changing into the composer/business-network-name directory:

composer network install --card PeerAdmin@hlfv1 --archiveFile business-network-name\@0.0.7.bna 
composer network start -c PeerAdmin@hlfv1 -n business-network-name -V 0.0.7 -A admin -S adminpw --file networkadmin.card
composer card import --file networkadmin.card --card admin@business-network-name
composer network ping -c admin@business-network-name

So yes, it was about mismatching cards and not cleaning these up as part of a new deployment.

Although not part of the original problem, it's also worth noting that the -A and -S parameters of the composer network start command HAD to be set to admin and adminpw respectively. See composer issue #3781.

Error starting hyperledger-composer network after Fabric and Composer version upgrade

2 Answers