google.protobuf.Any field with grpc is always nil

Question

I am using gRPC with golang. I have a very simple proto definition and a gRPC service. The proto definition has a field in Endorsement of type google/protobuf/any. gRPC service is unable to map this field to input value and it's always getting initialised to nil

proto definition:

syntax = "proto3";

option go_package = "service";
option java_multiple_files = true;
option java_package = "io.grpc.consensus";

import "google/protobuf/any.proto";

package service;

service MyService {
  rpc Verify (Payload) returns (Response) {}
}

message Response {
  string policyId =1;
  string txnId =2;
}

message Endorsement {
  string endorserId=1;
  // This is being initialise to nil by gRPC
  google.protobuf.Any data = 2;
  string signature=3;
  bool isVerified=4;
}


message Payload {
  string policyId =1;
  string txnId =2;
  repeated Endorsement endorsements=3;
}

Using this, a simple gRPC service is implemented:

package service

import (
    "log"

    "golang.org/x/net/context"
)

type ServiceServerImpl struct {
}

func NewServiceServerImpl() *ServiceServerImpl {
    return &ServiceServerImpl{}
}

func (s *ServiceServerImpl) Verify(ctx context.Context, txnPayload *Payload) (*Response, error) {
    log.Printf("Got verification request: %s", txnPayload.TxnId)
    for _, endorsement := range txnPayload.Endorsements {
        j, err := endorsement.Data.UnmarshalNew()
        if err != nil {
            log.Print("Error while unmarshaling the endorsement")
        }
        if j==nil {
       //This gets printed as payload's endorsement data is always null for google/protobuf/any type
            log.Print("Data is null for endorsement")
        }
    }
    return &Response{TxnId: txnPayload.TxnId,  PolicyId: txnPayload.PolicyId}, nil
}

Input Data:

{
  "policyId": "9dd97b1e-b76f-4c49-b067-22143c954e75",
  "txnId": "231-4dc0-8e54-58231df6f0ce",
  "endorsements": [
    {
      "endorserId": "67e1dfbd-1716-4d91-94ec-83dde64e4b80",
      "data": {
        "type": "issueTx",
        "userId": 1,
        "transaction": {
            "amount": 10123.50
        }
    },
      "signature": "MEUCIBkooxG2uFZeSEeaf5Xh5hWLxcKGMxCZzfnPshOh22y2AiEAwVLAaGhccUv8UhgC291qNWtxrGawX2pPsI7UUA/7QLM=",
      "isVerified": false
    }
  ]
}

Client:

type Data struct {
    Type        string      `json:"type"`
    UserId      int16       `json:"userId"`
    Transaction Transaction `json:"transaction"`
}

type Transaction struct {
    Amount float32 `json:"amount"`
}

data := &Data{Type: "Buffer", UserId: 1, Transaction: Transaction{Amount: 10123.50}}
    byteData, err := json.Marshal(data)
    if err != nil {
        log.Printf("Could not convert data input to bytes")
    }

    e := &any.Any{
        TypeUrl: "anything",
        Value:   byteData,
    }

    endosement := &Endorsement{EndorserId: "1", Signature: "MEUCIBkooxG2uFZeSEeaf5Xh5hWLxcKGMxCZzfnPshOh22y2AiEAwVLAaGhccUv8UhgC291qNWtxrGawX2pPsI7UUA/7QLM=", Data: e, IsVerified: false}
    var endosements = make([]*Endorsement, 1)
    endosements[0] = endosement
    t := &Payload{TxnId: "123", PolicyId: "456", Endorsements: endosements}

    response, err := c.Verify(context.Background(), t)
    if err != nil {
        log.Fatalf("Error when calling SayHello: %s", err)
    }

How should google/protobuf/any type be Unmarshal for unknown types?

m, err := e.Data.UnmarshalNew()
    if err != nil {
        log.Print("Error while unmarshaling the endorsement")
    }

throws an error: s:"not found"

Answer 1

As you know google.protobuf.Any is messageType, so when you didn't set any value for it, It will be nil.

You must unmarshal your data with your struct, see code below from example of protobuf

      // marshal any
      foo := &pb.Foo{...}
      any, err := anypb.New(foo)
      if err != nil {
        ...
      }
      // unmarshal any 
      foo := &pb.Foo{}
      if err := any.UnmarshalTo(foo); err != nil {
        ...
      }

Or I think you can use it with pointer interface (&interface{}) like this :

      d := &interface{}{}
      if err := endorsement.Data.UnmarshalTo(d); err != nil {
        ...
      }

Answer 2

From the package¹ documentation Unmarshaling an Any:

UnmarshalNew uses the global type registry to resolve the message type and construct a new instance of that message to unmarshal into. In order for a message type to appear in the global registry, the Go type representing that protobuf message type must be linked into the Go binary. For messages generated by protoc-gen-go, this is achieved through an import of the generated Go package representing a .proto file.

The type registry is protoregistry.GlobalTypes. The type lookup is done using the Any.TypeUrl field, which is set to the concrete type's url by the gRPC client when marshalling the original message.

The confusing detail about Any is that it can be any protobuf message, but that protobuf message must be defined somewhere.

Your .proto file has no message definition that matches the data object in your input. It could be that this Data message is defined somewhere else (not in your own proto files), but irrespective of that you have to import the Go package where the generated message is.

Otherwise if the input doesn't come from an already defined proto message, you can add a message definition to your proto yourself, then use UnmarshalTo:

// proto file
message Data {
    string type = 1;
    int user_id = 2;
    Transaction transaction = 3;
}

message Transaction {
    float amount = 1;
}

And then:

    for _, endorsement := range txnPayload.Endorsements {
        data := generated.Data{}
        err := endorsement.Data.UnmarshalTo(&data)
        if err != nil {
            log.Print("Error while unmarshaling the endorsement")
        }
    }

---

If you only need an arbitrary sequence of bytes, i.e. a truly unknown type, then use the proto type bytes and treat it as a JSON payload.

Model it as a Go struct:

type Data struct {
    Type     string `json:"type"`
    UserID   int    `json:"userId"`
    Transaction struct{
        Amount float64 `json:"amount"`
    } `json:"transaction"`

}

Or as a map[string]interface{} if clients could be sending literally anything.

Then in your handler func:

    for _, endorsement := range txnPayload.Endorsements {
        data := Data{} // or `map[string]interface{}`
        err := json.Unmarshal(endorsement.Data, &data)
        if err != nil {
            log.Print("Error while unmarshaling the endorsement")
        }
    }