MonetDB parallel query high IO write activity

Question

While testing Monetdb on Linux I can see a lot of IO write activity. The data set is 2GB (30 Million rows) and system has 120GB free RAM

The query:

select sum(hits) from (select "hour","minute","second", server, count(*) as hits from iislogs group by server, "hour","minute","second") as xxx;

I run a single query it takes ~400 ms to scan 30 Mil rows, but if i run same query 10 times in parallel each takes 30 s! The 48 CPUs are at 15% and I can see lots of disk writes, no swap activity. I've looking for tweaking some kind of setting or limitation without any luck.

Any idea about why ?

UPDATE:

I've disabled logging and write active has lowered but still having some and no improvement at all.

Hannes Mühleisen Hannes Mühleisen · Accepted Answer · 2015-08-25T05:43:25

You could try to allow more dirty pages in the page cache. We had some success with the following virtual memory settings:

sudo sysctl -w vm.swappiness=0
sudo sysctl -w vm.dirty_expire_centisecs=6000000
sudo sysctl -w vm.dirty_background_ratio=90
sudo sysctl -w vm.dirty_writeback_centisecs=6000000
sudo sysctl -w vm.dirty_ratio=90

MonetDB parallel query high IO write activity

1 Answers