GCloud AppEngine Task execution returns 403

Question

using push queues and flexible environment on Google AppEngine I get 403 (Forbidden) error when a task (to be executed on backend service), created with default service, is executed. The task is successfully pushed to queue, confirmed locally, but the execution of the task(s) fails with log:

INFO     2020-12-24 13:42:39,897 module.py:865] default: "POST /tasks/test-handler HTTP/1.1" 403 31
WARNING  2020-12-24 13:42:39,897 taskqueue_stub.py:2158] Task task2 failed to execute. The task has no remaining retries. Failing permanently after 1 retries and 0 seconds

The same happens both locally and on production. However, if a task is created with a cron job then the execution works just fine. I am using dev_appserver.py with Go 1.11 with the following .yaml definitions:

# backend service
service: backend
runtime: go111
instance_class: F2

inbound_services:
- warmup
- default

handlers:
- url: /tasks/.*
  login: admin
  redirect_http_response_code: 301

# default app service
service: default
runtime: go111
instance_class: F2

inbound_services:
- warmup

handlers:
- url: /api/.*
  script: auto
  secure: always
  redirect_http_response_code: 301

Initial API request comes to an /api endpoint which then succesfully pushes a queue message using:

t := taskqueue.NewPOSTTask(taskURL, url.Values{
    "testParam": {strconv.Itoa(testParam)},
})

if _, err := taskqueue.Add(ctx, t, "test-queue"); err != nil {
    return ErrPublishingTaskToQueue
}

My queue.yaml definition (in reality I have many more):

total_storage_limit: 120M
queue:
- name: test-queue
  rate: 1/s
  bucket_size: 100
  max_concurrent_requests: 10
  retry_parameters:
      task_retry_limit: 1

Any ideas why I'd be getting 403 (Forbidden) statuses on task execution if a task is not created via a cron job? The documentation and existing resources on this matter do not help much :/

Answer 1

Managed to make it work. If anyone struggles with getting 403 responses on task execution for push queues on Google AppEngine make sure that you set the right target service. In my example above I was missing target: backend in queue.yaml:

total_storage_limit: 120M
queue:
- name: test-queue
  rate: 1/s
  bucket_size: 100
  max_concurrent_requests: 10
  target: backend
  retry_parameters:
      task_retry_limit: 1

The problem was that the tasks were created with default service which by default means they hit the default service, but should hit backend service. Unfortunately default service had the required endpoint deployed as well, so I got 403 instead of 404.

More details on the target field: https://cloud.google.com/appengine/docs/standard/python/config/queueref#target