I've tried this integration using two approaches and I can get neither to work.
When using the "Notification Serial Number" setting, which allows you to get away with not having an SSL cert on the GAE end, the callback from GAE to Checkout takes so long to happen that I get DeadlineExceededExceptions after about 1 min 12 secs.
When using the "Notification as XML (Requires an HTTPS URL)" setting, the SSL cert isn't valid and Checkout times out on the connection:
We encountered an error trying to access your server at https://VERSION.APP_ID.appspot.com/googleCheckout -- the error we got is java.io.IOException: Error 'TIMEOUT' connecting to url 'https://VERSION.APP_ID.appspot.com/googleCheckout'.
When going to that URL in a browser, the browser warns that the certificate is invalid, but then claims it should be valid:
The certificate is only valid for the following names:
*.appspot.com , *.*.appspot.com , appspot.com
(Error code: ssl_error_bad_cert_domain)