How to stress test google datastore with javascript?

Question

We need to test our infrastructure for resilience ( up to hundreds of thousands of requests per second) and one of the elements in the architecture is google cloud datastore. One of the tests I need to perform is how many entities can be written in a second.

There is a code example here by Google to insert and list elements, but I find that this is slow to write one entity (one or two seconds per insert), even paralelising them in a bash script, it takes quite a bit of time, both from outside of the infrastructure and inside.

Is there any way to test the datastore directly where thousands of requests can be sent per second? Preferably in javascript.

I am not a javascript programmer myself, so apologies if this questions comes across as a little obvious - I hope not.

Answer 1

The Cloud Datastore is designed to simultaneously serve with ease all GAE apps out there and even apps outside GAE or even outside the Google Cloud. So IMHO testing the performance of the datastore itself is kinda pointless, as long as you respect the posted Limits - most likely you'll only get to test the depth of your pockets in the process. Just to get an idea: Are there any Performance SLA for Google Cloud Datastore?

Testing the performance of your application using the datastore is a completely different matter. It depends on your app's implementation, your entities structure and relationships, etc. That's what you want to focus on. But that's testing your app, not the datastore itself.

You may want to look at:

• Best Practices
• Avoiding datastore contention
• Designing for Scale

Answer 2

You might look into this one:

https://artillery.io/

You can install it through Nodejs and run your requests by choosing the target URL through yml file as following:

    config:   
    target: 'https://artillery.io'   
    phases:
        - duration: 60
          arrivalRate: 20   
    defaults:
        headers:
          x-my-service-auth: '987401838271002188298567' 
    scenarios:
      - flow:
        - get:
            url: "/docs"

You can run the above code through the following:

artillery run hello.yml 

You can get more details also about this javascript library through this link:

https://artillery.io/docs/getting-started/#run-a-quick-test

Answer 3

Glad you are using Google Datastore (GDS).

The best way to test from the backend, by using Apache JMeter, with Manual Module (presently service) server instance.

Without knowing much about your Data Store Model, GDS could save one entity per second in an entity group. However, you could increase the throughput performance of the GDS using Job Aggregation Queue Technique. Presently upto 300 requests per second per save has been achieved.