Error handlers for App Engine push queues

Question

I'm using error handler to get notifications about every error in App Engine app. It works well for Cloud Endpoints and servlets but I can't figure out how to handle errors in servlets invoked by push queues.

The configuration in web.xml

<error-page>
    <exception-type>java.lang.Exception</exception-type>
    <location>/admin/error-notification</location>
</error-page>

The scenario I want to achieve

a new task is added to the push queue
the task fails and is retried x number of times (<task-retry-limit>x</task-retry-limit>)
once the retry limit is reached the task is removed from the queue, but the error handler is invoked with the stack trace of the last unsuccessful run

The problem is once the task is removed nothing is invoked. Is there any way to configure it like I described above? I don't want to receive notification about every single failure, just the last failure preceding the removal of the task from the push queue.

Can your push queue worker servlet check the X-AppEngine-TaskRetryCount header and initiate its own error notification that way? — tx802
@tx802 You suggest to wrap all the servlet code in try-catch and check whether it's the last retry (when an exception occurs), if so invoke the error notification servlet on my own? — tomrozb
broadly speaking, yes, but without seeing your code it's hard to say more. I don't know if you can manually invoke an error handler though. — tx802

tomrozb tomrozb · Accepted Answer · 2015-06-06T08:48:18

Answer based on @tx802 hint to use X-AppEngine-TaskRetryCount

There's no way a task will be retried without throwing an exception in its previous run, so this is based on a simple hack to mark some exceptions as suppressed. In a custom error handler you have to check whether or not the exception is type of SuppressedException. If it's suppressed you can safely skip this exception, it means the task will be retried soon. If an exception is not wrapped in a SuppressedException it means that retry limit has been reached and this is the final exception which should handled in some way.

public abstract class PushQueueHttpServlet extends HttpServlet {

    @Override
    protected final void doGet(HttpServletRequest req, HttpServletResponse res) throws
            IOException, ServletException {
        try {
            doGet2(req, res);
        } catch (Throwable throwable) {
            String retryCountString = req.getHeader("X-AppEngine-TaskRetryCount");

            int retryCount;
            try {
                retryCount = Integer.parseInt(retryCountString);
            } catch (NumberFormatException e) {
                // Probably running outside of the task queue
                throw throwable;
            }

            if (retryCount == getTaskRetryLimit()) {
                throw throwable;
            } else {
                throw new SuppressedException(throwable);
            }
        }
    }

    protected abstract void doGet2(HttpServletRequest req, HttpServletResponse res) throws
            IOException, ServletException;

    protected abstract int getTaskRetryLimit();
}

SuppressedException class

public class SuppressedException extends IOException {

    public SuppressedException(Throwable cause) {
        super(cause);
    }
}

Error handlers for App Engine push queues

1 Answers