com.google.api.config.ServiceConfigSupplier - Failed to fetch default config version for service (only on localhost)

Question

I'm using Cloud Endpoints Frameworks (2.0.1) for Java as part of my final year project and have been relatively successful with it so far. I don't have any problems when deploying to my appspot.com domain, however, I am running into some problems when deploying locally.

(Any references to my-project-id in the following code blocks are aliases for my actual google cloud project id)

I have a valid openapi descriptor (openapi.json) of an annotated @API class which I am deploying to cloud endpoints using "gcloud service-management deploy openapi.json". The command returns successfully:

Service Configuration [2017-02-23r0] uploaded for service [api.endpoints.<my-project-id>.cloud.goog]

I then map the returned config_id to the correct endpoints_api_service in my app.yaml

endpoints_api_service:
  name: api.endpoints.<my-project-id>.cloud.goog
  config_id: 2017-02-23r0

This service is listed by the gcloud cli tool using "gcloud service-management list"

    NAME                                                           TITLE
    storage-component.googleapis.com                 Google Cloud Storage
    api.endpoints.<my-project-id>.cloud.goog         api.endpoints.<my-project-id>.cloud.goog
    etc...

and "gcloud service-management configs list --service api.endpoints.my-project-id.cloud.goog"

CONFIG_ID        SERVICE_NAME
2017-02-23r0     api.endpoints.<my-project-id>.cloud.goog
... other version configs

and is accessible on my appspot.com domain (I can call the endpoint and receive the correct response)

I am trying to deploy my project on localhost using the maven appengine plugin for java (mvn appengine:devserver), but upon jetty startup I'm hit with the following Exception:

 WARNING: Failed startup of context com.google.appengine.tools.development.DevAppEngineWebAppContext...

com.google.api.config.ServiceConfigException: Failed to fetch default config version for service 'api.endpoints.<my-project-id>.cloud.goog'. No versions exist!
at com.google.api.config.ServiceConfigSupplier.fetchLatestServiceVersion(ServiceConfigSupplier.java:155)
....

The deployment then gets stuck in an endless cycle of trying to start jetty, and being hit with that error message, and restarting etc. Any attempts to access localhost:8080 result in a "503: Service not found" error

I assumed that the local deployment of my app would be able to access the service config that was deployed using "gcloud service-management deploy", in the same way that the appspot.com deployment can, but is this not the case? Looking at the source for ServiceConfigSupplier.getchLatestServiceVersion() I gather that serviceManagement.services().configs().list(my-service-name).execute().getServiceConfigs() is returning an empty list, but why is this only occurring locally?

Extra Information

my ENDPOINTS_SERVICE_NAME environment variable matches 'api.endpoints.my-project-id.cloud.goog'

I noticed that there was an update (1.0.2) to com.google.api.config a few days ago, and it has a dependency on an older version of com.google.api.services.servicemanagement (dependent on v1-rev14-1.22.0 with the newest version being v1-rev340-1.22.0) I doubt this is the problem, but I thought I would mention it, as it contains classes relevant to the exception (ServiceManagement is used by ServiceConfigSupplier, which is throwing the exception). Perhaps there is an inconsistency in where they are looking for the service configs?

I'm quite stumped tbh, it's a bit over my head. I would dislike having to remove Endpoints, as I'm starting to like it, but we also can't really lose usage of our devserver either. I hope someone can shed a little bit of light on this issue.

Did you solve this problem? I'm using Java and have the same issue. — Marcus Demnert
Hi Sam Holden, were you able to solve the issue? A document related to this was recently updated. cloud.google.com/endpoints/docs/frameworks/java/… There is also a sample located here: github.com/GoogleCloudPlatform/java-docs-samples/tree/master/… Let me know! Thank you. — Frank Natividad
@FrankNatividad I am seeing almost this exact error following the quickstart you provided in your link. — Jade
I'm having the same problem but when I first started playing with this stuff in early March 2017 and was running version 135 (I think) of gcloud cli, I could do the mvn appengine:run successfully. I know it doesn't fix the problem but you might be able to rollback using gcloud components update --version 135.0.0 to test it. — Tom Saleeba
I'm writing a complete solution for this issue atm. This is a hidden permissions issue. For now, you can get past this by using gcloud auth application-default login. This will request necessary permissions and update default application credentials. — Frank Natividad

Tom Saleeba Tom Saleeba · Accepted Answer · 2017-04-11T04:36:10

It's not a fix but I was able to work around the problem by using the advice in https://stackoverflow.com/a/41493548/1410035.

Namely, commenting out the ServiceManagementConfigFilter:

b) Comment out the ServiceManagementConfigFilter from web.xml , ie,

<!--
    <filter>
      <filter-name>endpoints-api-configuration</filter-name>
      <filter-class>com.google.api.control.ServiceManagementConfigFilter</filter-class>
   </filter>
-->

<!--    
<filter-mapping>
    <filter-name>endpoints-api-configuration</filter-name>
    <servlet-name>EndpointsServlet</servlet-name>
</filter-mapping>
-->

Note that you have to comment out the filter and the filter-mapping and they aren't right next to each other in the file.

I found that I didn't need to remove the scaling block as mentioned in point 'a' in the linked answer.

com.google.api.config.ServiceConfigSupplier - Failed to fetch default config version for service (only on localhost)

7 Answers