Improving MongoDB Aggregation Pipeline Performance

Question

I am using the MongoDB aggregation pipeline, to generate reports. Here are some quick key infos first:

Machine: 8-Core-CPU, 16 GB RAM
OS: Ubuntu 16.04.1 LTS

MongoDB Version: 3.2.11
Mongo PHP Adapter: 1.6.14
PHP Version: 5.6.30

Amount of documents to aggregate: ~ 10+ million

My aggregation pipeline code - which I write and execute in PHP - looks like this:

// create indexes (ASC and DESC for each aggregation key)
$mongoCollection->createIndex('foo' => 1);
$mongoCollection->createIndex('foo' => -1);
$mongoCollection->createIndex('bar' => 1);
$mongoCollection->createIndex('bar' => -1);

// prepare aggregation (1. group, 2. sort)
$aggregationPipeline = [
    [
        '$group' => [
            '_id' => [
                '$foo' => 'foo',
                '$bar' => 'bar
            ],
            'count' => [
                '$sum' => 1
            ]
        ]
    ],
    [
        '$sort' => [
            'count' => -1
        ]
    ]
];


// run aggregation
$mongoCollection->aggregate($aggregationPipeline);

The problem is: the aggregation is not fast enough! Depending on how many fields I aggregate (in my example there are only 2) the process takes about 90 seconds, often longer.

My goal is: Improving the performance of the aggregations!

My questions:

PHP
- As stated I use PHP to control and run my aggregations.
- => Is it bad practice to run aggregations from PHP, utilizing the PHP-Mongo-Classes and -Methods?
- => Would upgrading to PHP 7.x improve Mongo aggregation performance?
Indexes
- I have the subjective feeling, that adding DESC and ASC indexes (see example code) is NOT improving performance. With or without adding indexes the runtime seems to be almost identical.
- => Is it possible that adding indexes does NOT improve the performance significantly?
CPU-Cores
- While the aggregation runs, I can observe, that only ONE CORE of the CPU is being used.
- => How can I manage/achieve that the aggregation pipeline uses ALL CPU-CORES simultaneously?
Sharding
- I read about MongoDB sharding and that might be another possible way to improve Mongo overall performance.
- => Is it possible and does it make sense to setup/configure sharding on a SINGLE machine?

Thank you very much in advance for any comments, suggestions, critisism and questions!

refer this link : docs.mongodb.com/v3.2/core/aggregation-pipeline-optimization — Sourabh Bhagat
Thank you for the link, Sourabh! My aggregation pipeline already follows all the rules stated there. — Chris

Ori Dar Ori Dar · Accepted Answer · 2017-02-01T22:15:06

PHP

Have no clue, but I imagine it plays a very minor role here. The cardinality of foo and bar - how many documents / bytes the aggregation returns may also have an impact.

Indexes

First, there is no sense in having both ascending and descending indexes for a single field. Second, indexes can be used for a $match aggregation pipeline, but are useless when it comes o $group operations. So your notion is right, indexes can't help you here. You are doing a full scan.

CPU-Cores

You can't run aggregation operation in parallel. You can technically achieve it by controlling the aggregation from outside, by breaking it maybe to sub task. But since you are doing a full scan - again - it's no good in your case.

Sharding

There's no point in having multiple shards on the same machine, competing over the same hardware. You add shards by utilizing more hardware.

You are very limited in resources here 16 GB RAM and 100 M documents. This is probably not enough, especially if your documents are not tiny and you have to go to disk in order to process more documents. I would check the IO utilization during aggregation, and how the WiredTiger cache is behaving (assuming you are using WiredTiger).

In summary, it's probably your limited resources. Client / driver has probably little impact on slowness. Start by running and explain() to your aggregation, while observing how your RAM, disk are behaving.

Improving MongoDB Aggregation Pipeline Performance

1 Answers