How to reduce the process 's memory usage when I convert a ets table to list in Erlang?

0
votes

I got a serious problem when I try to convert a ets table to a list by ets:tab2list.

I have 1500 items in a ets table and most of them have a key with a value about 20K text data. The table size of the ets table is about 30 MB.

But when I try to convert this table to a list, it will cost about 400 MB's process memory and seems the process's memory usage is related to the table & item's size. When I try to convert a 200 MB table with 10000 items. It spend almost my VM's memory within one line and make my program terminated by the OS.

Is there other good way to handle my case? It seems ets is not good for my use case? or there are other good way to convert a ets table to a list without so much memory consumption?

Thanks you~~

Eric

ps: I use a table to queue the command list of a redis command and will do a batch insert to redis later to avoid busy network operation. Each item looks like 

{index, {["set", "key"], "text"}

and I need a batch command in the form of

[["set", "key1", "text1"], ["set", "key2", "text2"]]

In my case, the text is document about 20000 words

1
erlangets
Why do you need the entire ets table as a list at once? – RichardC
I use a table to queue the command list of a redis command and will do a batch insert to redis later to avoid busy network operation. Each item looks like {index, {["set", "key"], "text"}, and I need a batch command in the form of [ ["set", "key1", "text1"], ["set", "key2", "text2"]]. In my case, the text is document about 20000 words. – 彭煥閎
I see Erlang has a queue in stdlib, can I mix queue and ets to achieve my request? Actually I just need a global memory space to put the commands. No matter what it is. – 彭煥閎
Just take a test by queue. Seems got worse result. – 彭煥閎
I tried to simulate the behavior in a simple file, but the memory usage looks fine. Just cost 10~20 MB memory in a loop. It's really strange why this issue happened inside my current project, but I found if I remove the binary_to_list function in my function, the out of memory issue would not happened as expect.But the binary_to_list works fine in my simple project!! – 彭煥閎

1 Answers

1
votes

You can try to implement a batch aproach with: 

ets:select(Tab, MatchSpec, Limit) -> 
    {[Match], Continuation} | '$end_of_table'

or 

ets:match(Tab, Pattern, Limit) ->
     {[Match], Continuation} | '$end_of_table'

so your batch size its the Limit and you can delete de records after a successfull insert on redis.

Regards.