Fetch large JSON from Datastore

2
votes

I've created an API on Google Cloud Endpoints that getting all datas from a single entity in Datastore. The NoSQL request (A really simple one : Select * from Entity) is performed with Objectify.

This datastore entity is populated with 200 rows (entities) and each row (entity) has a list of children entities of same kind :

  • MEAL:
    • String title
    • int preparationTime
    • List< Ingredient > listOfIngredients (child entities...)
    • ...

So when I fetch API, a JSON is returned. It's size is about 641Ko and it has 17K lines.

When I look at the API explorer, it tells me that request takes 4 seconds to execute : enter image description here

I would like to decrease that time, because it's a really high one... I've already :

  • Increase GAE instance to F2
  • Enable Memcache

It helps a little but I don't think this is the best efficient way...

Should I use Big Query to generate the JSON file faster ? Or maybe there is another solution ?

1
google-app-enginegoogle-cloud-datastoregoogle-cloud-endpoints
Check (in appstats or in real GAE request traces) the duration of the datastore/memcache RPC calls themselves - not much you can do about those as they happen on the GAE backend, outside of your app. JSON data decoding/encoding ops can be lengthy for such big data, check for any unnecessary such ops in your app. Upgrading the instance type further should help for the necessary ones. – Dan Cornilescu
Also look at projection (and maybe keys_only?) queries to reduce the amount of data handled - if you don't actually need all that data in your request response. See cloud.google.com/appengine/docs/python/ndb/projectionqueries – Dan Cornilescu

1 Answers

5
votes

Do you need all the entity in a single request ?

  • if Not, then you can batch fetch entities using Cursor Queries and display as per your need, say for eg: fetch 20 or 30 entities at a time depending on your need.
  • If Yes,
    • Does your meal entity changes often
      • If No, you can generate a json file and store it in GCS, and whenever your entity changes you can update the json file, so that on the client end fetching will be lot faster and using etag header, new content can be pulled easily
      • If Yes, then i think batch fetching is only effective way to pull those many entities