Process Crashes when importing 5 million rows to Neo4J

Question

Summary I'm importing a 3 node files and 2 relationship files using csv batch-import. I'm running Neo4J 2.0.3 and I have the 2.0 version of the batch importer compiled assembly. The file crashes on the 3rd file and doesn't appear to get to the relationships.

Environment - VM running on an SSD. I think it might be too small?? - Windows 7 64
- 4.5 gb ram - JDK version = jdk1.7.0_45

Here are my Batch.Properties.

cache_type=none use_memory_mapped_buffers=true neostore.nodestore.db.mapped_memory=1G neostore.relationshipstore.db.mapped_memory=1g neostore.propertystore.db.mapped_memory=3G neostore.propertystore.db.strings.mapped_memory=500M neostore.propertystore.db.arrays.mapped_memory=0M neostore.propertystore.db.index.keys.mapped_memory=150M neostore.propertystore.db.index.mapped_memory=150M

batch_import.node_index.clients=exact batch_import.node_index.systems=exact batch_import.node_index.technicalobjectkey=exact batch_import.rel_index.clientrelation=exact

Example csv formats

I'm importing 3 node files and 2 relationship csv files

Here is a sample of the first file it has 2.38 million rows.

ClientId:string:clients name    l:label ClientKey
499999999   JUNE  BUG   CLIENT  0000622P32J13106wfe
499999998   MORTIMER  FELICIEN  CLIENT  0001FKV2FBY35273wfe
499999997   ELIAS  REDMAN   CLIENT  0002SFUVAUI1443wfe
499999996   JITENDRA  ISMAIL    CLIENT  0002SFUVAUI17583wfew

the second file has 6 rows

The third file has 2.8 million rows and looks like this

technicalobjectkey:string:technicalobjectkey
009DLSMO6N0SPREM
IFPQFPE6
P1T63GUGC10SPREM
SMSD8FDX
0T4BIAHX

The process crashes on the third file.

Here is the entire error message.

Using: Importer C:\Users\Steven.Suing\Documents\Neo4j\ClientGraph data/ClientNodes.csv,data/SystemNodes.csv,data/ObjectNodes.csv ,data/MainRelations.csv,data/ObjectToSystemRel.csv

Using Existing Configuration File
.......................
Importing 2381482 Nodes took 114 seconds

Importing 6 Nodes took 0 seconds
...................Exception in thread "main" java.lang.RuntimeException: Writer thread failed
        at org.mapdb.AsyncWriteEngine.checkState(AsyncWriteEngine.java:245)
        at org.mapdb.AsyncWriteEngine.close(AsyncWriteEngine.java:391)
        at org.mapdb.EngineWrapper.close(EngineWrapper.java:72)
        at org.mapdb.EngineWrapper.close(EngineWrapper.java:72)
        at org.mapdb.CacheHashTable.close(CacheHashTable.java:169)
        at org.mapdb.DB.close(DB.java:401)
        at org.neo4j.batchimport.index.MapDbCachingIndexProvider.shutdown(MapDbCachingIndexProvider.java:51)
        at org.neo4j.batchimport.Importer.finish(Importer.java:87)
        at org.neo4j.batchimport.Importer.doImport(Importer.java:239)
        at org.neo4j.batchimport.Importer.main(Importer.java:83)
Caused by: java.lang.RuntimeException: File could not be mapped to memory, common problem on 32bit JVM. Use `DBMaker.newRandomAccessFileDB()` as workaround
        at org.mapdb.Volume$MappedFileVol.makeNewBuffer(Volume.java:496)
        at org.mapdb.Volume$ByteBufferVol.ensureAvailable(Volume.java:245)
        at org.mapdb.StoreDirect.freePhysTake(StoreDirect.java:724)
        at org.mapdb.StoreDirect.physAllocate(StoreDirect.java:408)
        at org.mapdb.StoreDirect.update(StoreDirect.java:270)
        at org.mapdb.EngineWrapper.update(EngineWrapper.java:55)
        at org.mapdb.AsyncWriteEngine.access$101(AsyncWriteEngine.java:68)
        at org.mapdb.AsyncWriteEngine.runWriter(AsyncWriteEngine.java:201)
        at org.mapdb.AsyncWriteEngine$2.run(AsyncWriteEngine.java:143)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)
        at org.mapdb.Volume$MappedFileVol.makeNewBuffer(Volume.java:491)
        ... 9 more
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:885)
        ... 10 more
Exception in thread "MapDB shutdown" java.lang.RuntimeException: Writer thread failed
        at org.mapdb.AsyncWriteEngine.checkState(AsyncWriteEngine.java:245)
        at org.mapdb.AsyncWriteEngine.close(AsyncWriteEngine.java:391)
        at org.mapdb.EngineWrapper.close(EngineWrapper.java:72)
        at org.mapdb.EngineWrapper.close(EngineWrapper.java:72)
        at org.mapdb.CacheHashTable.close(CacheHashTable.java:169)
        at org.mapdb.DBMaker$1.run(DBMaker.java:654)
Caused by: java.lang.RuntimeException: File could not be mapped to memory, common problem on 32bit JVM. Use `DBMaker.newRandomAccessFileDB()` as workaround
        at org.mapdb.Volume$MappedFileVol.makeNewBuffer(Volume.java:496)
        at org.mapdb.Volume$ByteBufferVol.ensureAvailable(Volume.java:245)
        at org.mapdb.StoreDirect.freePhysTake(StoreDirect.java:724)
        at org.mapdb.StoreDirect.physAllocate(StoreDirect.java:408)
        at org.mapdb.StoreDirect.update(StoreDirect.java:270)
        at org.mapdb.EngineWrapper.update(EngineWrapper.java:55)
        at org.mapdb.AsyncWriteEngine.access$101(AsyncWriteEngine.java:68)
        at org.mapdb.AsyncWriteEngine.runWriter(AsyncWriteEngine.java:201)
        at org.mapdb.AsyncWriteEngine$2.run(AsyncWriteEngine.java:143)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)
        at org.mapdb.Volume$MappedFileVol.makeNewBuffer(Volume.java:491)
        ... 9 more
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:885)
        ... 10 more

Jan Kotek Jan Kotek · Accepted Answer · 2014-05-15T10:46:31

What version of MapDB is it using? This is common error on 32bit OS, but you wrote it is 64bit.

Process Crashes when importing 5 million rows to Neo4J

2 Answers