Call grpc google endpoints from android cause HTTP status code 400 from nginx esp

Question

I am trying to do a GRPC Call from android with grpc-java (flavor lite) on Google Cloud Platform and i have HTTP status code 400 from nginx esp

On My local network it works, with a grpc server in golang.

But On Google Cloud Platform we are using a TCP Load Balancer in front of gRPC Google Cloud Endpoints and our golang backend is deployed with Google Container Engine.

From our first analyze, it appears only when we are using grpc metadata with a JWT token on grpc-java, if we are not sending metadata it works.

My endpoint config.yaml

type: google.api.Service
config_version: 3

name: auth.endpoints.MYPROJECT.cloud.goog

title: auth gRPC API
apis:
- name: api.AuthApi

usage:
  rules:
  # Allow unregistered calls for all methods.
  - selector: "*"
  allow_unregistered_calls: true

My Backend config

Go version on backend 1.9.1 / 1.9.2
grpc-go version : 1.7.1
protoc version : 3.4.0

My Client config

protoc version : 3.4.0
grpc-java on android : 1.6.1 (i will test with 1.7.0)

Sample of Go code

We are using a JWT token from firebase with custom claims, passed on metadata.

// go client sample
md["authorization"] = []string{"bearer " + token}
ctx = metadata.NewOutgoingContext(ctx, md)

** Sample of java code**

Metadata headers = new Metadata();
headers.put(TOKEN_KEY, "Bearer " + token);
authClient.attachHeaders(headers);

blockingStub = MetadataUtils.attachHeaders(blockingStub, headers);

My issue

With a Go Client on GCP it works.

With grpcc (a NodeJS) Client on GCP it works.

With grpc-java on android it fails with this trace :

10-26 11:13:49.340 22025-22025/com.mypackage.customer.debug E/AuthenticationManager: Fail to get the custom token from server                                                                                  
io.grpc.StatusRuntimeException: INTERNAL: HTTP status code 400                                                                                  
invalid content-type: text/html                                                                                  
headers: Metadata(:status=400,server=nginx,date=Thu, 26 Oct 2017 09:13:48 GMT,content-type=text/html,content-length=166)                                                                                 
DATA-----------------------------

<html>                                                                                  
<head><title>400 Bad Request</title></head>                                                                                    
<body bgcolor="white">                                                                                   
<center><h1>400 Bad Request</h1></center>                                                                                   
<hr><center>nginx</center>                                                                                   
</body>                                                                                 
</html>

at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:210)                                                                                      
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:191)                                                                              
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:124)                                                                                    
at com.mypackage.protobuf.AuthApiGrpc$AuthApiBlockingStub.generateToken(AuthApiGrpc.java:163)

on Google Cloud Endpoints i can see this log on my esp :

10.12.0.6 - - [26/Oct/2017:11:13:49 +0000] “-” 400 166 “-” “grpc-java-okhttp/0.0”

Instead of

10.12.0.6 - - [26/Oct/2017:11:13:49 +0000] "POST /api.AuthApi/generateToken HTTP/2.0" 200 95 "-" "grpc-java-okhttp/0.0"

Any idea ?

The authentication token is passed as metadata in case of Java. How does your code structure compare with the sample in the "Calling an authenticated method from gRPC" subchapter of the "Authenticating Users" documentation page? cloud.google.com/endpoints/docs/grpc/authenticating-users-grpc — George
We followed the documentation for authentication, it appears this is an interceptor issue. Check my response below. — user4653455

Unknown Unknown · Accepted Answer · 2017-10-27T09:24:24

Of We found the issue, it was on our grpc logging interceptor, if we log Metadata with a call of headers.toString() it fails.

This is very strange, because it fails only for google cloud endpoints. And it's very odd that a toString() changes something, i will open an issue on grpc-java github.

Our simple interceptor

package com.myproject.android.common.api.grpc.interceptor;

import android.util.Log;

import io.grpc.CallOptions;
import io.grpc.Channel;
import io.grpc.ClientCall;
import io.grpc.ClientInterceptor;
import io.grpc.ForwardingClientCall;
import io.grpc.ForwardingClientCallListener;
import io.grpc.Metadata;
import io.grpc.MethodDescriptor;

/**
* Interceptor that log (via Android logger) gRPC request and response contents.
* <p>
* The content might be truncated if too long.
* </p>
* Created on 18/10/2017.
*/
public class GrpcLoggingInterceptor implements ClientInterceptor {
    private static final String TAG = GrpcLoggingInterceptor.class.getSimpleName();

    @Override
    public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(final MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) {
        return new ForwardingClientCall.SimpleForwardingClientCall<ReqT, RespT>(next.newCall(method, callOptions)) {

            @Override
            public void sendMessage(ReqT message) {
                Log.v(TAG, "--> [REQUEST] " + method.getFullMethodName() + ": \n" + message.toString());
                super.sendMessage(message);
            }

            @Override
            public void start(Listener<RespT> responseListener, Metadata headers) {
//                Log.v(TAG, "--> [HEADERS] " + headers.toString()); // /!\ DO NOT LOG METADATA: causes error 400 on GCP
                ForwardingClientCallListener.SimpleForwardingClientCallListener<RespT> listener = new
                        ForwardingClientCallListener.SimpleForwardingClientCallListener<RespT>(responseListener) {
                            @Override
                            public void onMessage(RespT message) {
                                Log.v(TAG, "<-- [RESPONSE] " + method.getFullMethodName() + " (" + message.toString().length() + " bytes): \n" + message.toString());
                                super.onMessage(message);
                            }
                        };
                super.start(listener, headers);
            }
        };
    }
}

Call grpc google endpoints from android cause HTTP status code 400 from nginx esp

1 Answers