Google appengine cloud tasks alpha handler http 404 error

Question

When I send a task to a task queue it keeps failing and shows a http 404 (not found) error in the logs.

The project has been whitelisted for cloud tasks alpha on flexible.

I can send HTTP post requests to /endpointpath & /tasks/worker locally without any errors.

The endpoint works fine and adds the task to the task queue.

 13:37:41.300 POST 200 0 B 422 ms curl/7.54.0 /endpointspath?key=keyremoved 0.0.0.0 - "POST endpointspath?key=keyremoved" 200 undefined "-" "curl/7.54.0"

The app is running as the default service.

app.go main func:

func main() {
    r := mux.NewRouter()

    r.HandleFunc("/", handler)

    r.HandleFunc("/_ah/health", healthCheckHandler)

    // Task handlers

    r.Path("/tasks/worker").Methods("POST", "GET", "PUT").HandlerFunc(workerTaskHandler)

    // Endpoints

    r.Path("/endpointpath").Methods("POST").HandlerFunc(searchHandler)

    http.Handle("/", r)

    port := 8080
    if portStr := os.Getenv("PORT"); portStr != "" {
        port, _ = strconv.Atoi(portStr)
    }
    log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", port), nil))
}

abbreviated app.yaml:

runtime: go
env: flex

handlers:
- url: /tasks/.*
  script: _go_app
- url: /.*
  script: _go_app

Logs HTTP 404 response when queue dispatches request to worker:

10.0.0.1 - "POST /tasks/worker" 404 200 "-" "AppEngine-Google; (+http://code.google.com/appengine)"
Expand all | Collapse all {
 httpRequest: {
  latency:  "0s"    
  referer:  "-"    
  remoteIp:  "10.0.0.1"    
  requestMethod:  "POST"    
  requestUrl:  "/tasks/worker"    
  responseSize:  "200"    
  status:  404    
  userAgent:  "AppEngine-Google; (+http://code.google.com/appengine)"    
 }
 insertId:  "......."   
 jsonPayload: {
  appLatencySeconds:  "-"    
  latencySeconds:  "0.000"    
  time:  null    
  trace:  "......."    
 }
 labels: {
  appengine.googleapis.com/instance_name:  "......"    
  appengine.googleapis.com/trace_id:  "......."    
  compute.googleapis.com/resource_id:  "......."    
  compute.googleapis.com/resource_name:  "......"    
  compute.googleapis.com/zone:  "us-central1-b"    
 }
 logName:  "projects/projectname/logs/appengine.googleapis.com%2Fnginx.request"   
 receiveTimestamp:  "2017-12-09T10:56:14.794726383Z"   
 resource: {
  labels: {
   module_id:  "default"     
   project_id:  "projectname"     
   version_id:  "....."     
  }
  type:  "gae_app"    
 }
 timestamp:  "2017-12-09T10:56:10.301Z"   
}

The closest I can get GAE to find the tasks/worker url is by setting login:admin in app.yaml (even tho flex doesn't use this for authentication). This returns a 403 unauthorised error.

handlers:
- url: /tasks/.*
  script: _go_app
  login: admin

Here is the 403 response in the logs

{
 httpRequest: {
  latency:  "0s"    
  referer:  "-"    
  remoteIp:  "10.0.0.1"    
  requestMethod:  "POST"    
  requestUrl:  "/tasks/worker"    
  responseSize:  "162"    
  status:  403    
  userAgent:  "AppEngine-Google; (+http://code.google.com/appengine)"    
 }
 insertId:  "....."   
 jsonPayload: {
  appLatencySeconds:  "-"    
  latencySeconds:  "0.000"    
  time:  null    
  trace:  "....."    
 }
 labels: {
  appengine.googleapis.com/instance_name:  "...."    
  appengine.googleapis.com/trace_id:  "...."    
  compute.googleapis.com/resource_id:  "...."    
  compute.googleapis.com/resource_name:  "....."    
  compute.googleapis.com/zone:  "us-central1-b"    
 }
 logName:  "projects/projectname/logs/appengine.googleapis.com%2Fnginx.request"   
 receiveTimestamp:  "2017-12-09T13:35:59.986118082Z"   
 resource: {
  labels: {
   module_id:  "default"     
   project_id:  "projectname"     
   version_id:  "....."     
  }
  type:  "gae_app"    
 }
 timestamp:  "2017-12-09T13:35:54.764Z"   
}

Not sure if it's related but projectname.appspot.com/_ah/health returns this error:

{
 "code": 5,
 "message": "Method does not exist.",
 "details": [
  {
   "@type": "type.googleapis.com/google.rpc.DebugInfo",
   "stackEntries": [],
   "detail": "service_control"
  }
 ]
}

mbudge mbudge · Accepted Answer · 2017-12-11T12:04:00

It turns out endpoints can't run on the same service as task handlers. Task handler url requests are blocked by the ESP proxy if they run on the same service in the flexible environment, and the service has the endpoints service enabled.

Run task handlers on a separate service and do not set "endpoints_api_service:" in the task handler service app.yaml file.

Doing so will prevent the queue from being able to dispatch to workers in the flexible environment.

This isn't mentioned in the app engine documentation which is kinda bizarre.

The "/_ah/health" issue was caused by this path not being set in the open api file. If this path isn't set the url isn't recognised by the proxy.

Google appengine cloud tasks alpha handler http 404 error

1 Answers