Chaincode instantiation fails in Kubernetes setup for Hyperledger Fabric network

Question

I'm currently trying to port the docker-compose setup for a Hyperledger Fabric network into Kubernetes and keep running into this error when instantiating the chaincode from the cli container using the end-to-end scenario provided in the fabric examples:

2017-11-07 20:49:55.476 UTC [shim] userChaincodeStreamGetter -> ERRO 001 Error trying to connect to local peer: x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "tlsca.org0.example.com")
Error starting Simple chaincode: Error trying to connect to local peer: x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "tlsca.org0.example.com")

Here is my crypto-config.yml:

OrdererOrgs:
  - Name: Orderer
    Domain: example.com
    Specs:
      - Hostname: orderer

PeerOrgs:
  - Name: Org0
    Domain: org0.example.com
    Specs:
      - Hostname: peer0
      - Hostname: peer1
      - Hostname: ca
    Users:
      Count: 2

And here are the environment variables I've used in my Kubernetes manifest for the peer pod:

  env:
    - name: CORE_PEER_ID
      value: peer0.org0.example.com
    - name: CORE_PEER_ADDRESS
      value: peer0.org0.example.com:7051
    - name: CORE_PEER_ADDRESSAUTODETECT
      value: "true"
    - name: CORE_PEER_TLS_SERVERHOSTOVERRIDE
      value: peer0.org0.example.com
    - name: CORE_PEER_GOSSIP_EXTERNALENDPOINT
      value: peer0.org0.example.com:7051
    - name: CORE_PEER_LOCALMSPID
      value: Org0MSP
    - name: CORE_LEDGER_STATE_STATEDATABASE
      value: CouchDB
    - name: CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS
      value: localhost:5984
    - name: CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME
      value: 
    - name: CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD
      value: 
    - name: CORE_VM_ENDPOINT
      value: unix:///host/var/run/docker.sock
    - name: CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE
      value: bridge
    - name: CORE_LOGGING_LEVEL
      value: DEBUG
    - name: CORE_PEER_TLS_ENABLED
      value: "true"
    - name: CORE_PEER_GOSSIP_USELEADERELECTION
      value: "true"
    - name: CORE_PEER_GOSSIP_ORGLEADER
      value: "false"
    - name: CORE_PEER_PROFILE_ENABLED
      value: "true"
    - name: CORE_PEER_TLS_CERT_FILE
      value: /etc/hyperledger/fabric/tls/server.crt
    - name: CORE_PEER_TLS_KEY_FILE
      value: /etc/hyperledger/fabric/tls/server.key
    - name: CORE_PEER_TLS_ROOTCERT_FILE
      value: /etc/hyperledger/fabric/tls/ca.crt

Up until the chaincode instantiation step, everything worked fine - channel creation, joining peers to the channel, anchor peer update, chaincode installation.

3cheesewheel 3cheesewheel · Accepted Answer · 2017-11-21T19:08:25

So I haven't been able to verify against the source code yet (I'll edit my answer when I do) because Fabric has changed a lot and I don't have re-find the parts of the source code that generate this error, but I'm pretty sure I finally managed to figure it out.

So this error can happen under a very specific set of circumstances:

You've generated the correct set of certs, then spun up the pods and attempted to run the end-to-end scenario (which may or may not have succeeded); and
You then regenerated the set of certs using cryptogen, spun up a new set of pods, and attempted to run the end-to-end scenario again without first removing the existing chaincode images.

What happens is, the chaincode image is created with the tls rootcert (as well as some of the CORE_* environment variables baked in. However, if you regenerate the set of certs between runs of the end-to-end scenario without removing the chaincode images, you will end up attempting to verify a new end-entity certificate against the old rootcert that has been baked into the chaincode image, from which the chaincode container has be (re)created from.

I've just successfully verified from the end user side by going through the following steps:

Remove all chaincode containers from Minikube (or whatever cluster you're running Kubernetes on)
Spinning up a new set of pods and running the end-to-end scenario - it should be successful; if not, this is a different problem from the one you have.
Tear down the pods. Regenerate the certs.
Spin up the pods again. Re-run the end-to-end scenario. This time the error should appear.

tl;dr Don't forget to remove your chaincode container images if you regenerate a new set of certs using cryptogen.

Chaincode instantiation fails in Kubernetes setup for Hyperledger Fabric network

1 Answers