I am building up a Magento eCommerce website, and importing customer profiles from old one. I am using CSV importer, and this process is way slower than I can imagine. It takes almost 4 seconds for just one customer. So far the process has been running for 6+ hours and only 30k customers imported. The CSV file is chunked into several ~10M smaller ones.
For now, I am using Amazon Web Services EC2 instance (micro) for development server. It has 1 vCPU (2.5GHz) and 1GiB memory. But I don't think this can be an issue. I increased php memory limit to 1G.
I've read an article saying that this speed issues of importing products are very common because of the Magento's EAV database system and the heavy PHP API modules [Speeding up Magento Imports]. It says that Magento sends 450 MySQL queries in order to import one single product. I have also seen a workaround using [Magmi] which tries to bypass Magento's API and insert data directly into MySQL tables. However AFAIK it doesn't seem to import customers, but only products and categories. I don't know if they (products and customers) are using the same mechanism.
I disabled caching management and set the option of index management to 'manual update'. Though customer profiles don't really use these processes.
Do you have any suggestion to increase this CSV importing speed?
[Follow-up]
I have found one of the problem sources, Amazon EC2 T2 instances. They use CPU Credits to control maximum CPU usage. For micro instances, the base CPU performance is limited to 10% of its capacity. I used all of the CPU credits, and the server didn't allow me to use the full CPU.
I just checked the importing performance now, and it is importing approximately 2 records per sec. I think this is still too slow, so I will wait more for any other solutions. Perhaps optimizations?
