I am writing a Python script (using Python 3.2) that at some point needs to through a file of around 800.000 lines for each key in a dictionary. The number of keys is around 150.000. The lines in the file are dictionaries of the following format:
{'url': 'http://address.com/document/42/1998', 'referrer': 'http://address.com/search?&q=query1', 'session': '1', 'rank': 2, 'time': 1338447254}
{'url': 'http://address.com/document/55/17', 'referrer': 'http://address.com/search&q=query2', 'session': '1', 'rank': 2, 'time': 13384462462}
For each line in this file I need do some computations and store the result. To be able to read the dictionary and work on it I use eval. This will result in 120.000.000.000 calls to eval, which takes a long time. I am therefore looking for a way to optimize this.
You're welcome to come with all possible suggestions for optimizations. Every bit may have an impact, but I'm mainly interested in eval and the way I read the file. Atm. I am thinking that some other method that eval may perform faster, but I can't make JSON read the format and using split has not turned out well yet. Also the way I read from the file might be optimized. I have tried the method in the following code, along with "with" (was much slower, but consumed less memory). I also tried reading the file in memory using map:
f_chunk = map(eval, codecs.open(chunk_file, "r", encoding="utf-8").readlines())
But this does not really work either.
Anyway, the following part of the script is the heavy one. It is run in multiple processes:
def mine(id, tmp_sessions, chunk_file, work_q, result_q, init_qsize):
#f_chunk = map(eval, codecs.open(chunk_file, "r", encoding="utf-8").readlines())
f_chunk = codecs.open(chunk_file, "r", encoding="utf-8").readlines()
while True:
try:
k = work_q.get()
if k == 'STOP':
work_q.task_done()
break # reached end of queue
except Queue.Empty:
break
#with codecs.open(chunk_file, "r", encoding="utf-8") as f_chunk:
for line in f_chunk:
#try:
jlog_nest = dict()
jlog_nest = eval(line)
#jlog_nest = json.loads(line)
#jlog_nest = line
#jlog_nest = defaultdict(line)
if jlog_nest["session"] == k: # If session is the same
query_nest = prepare_test_cases_lib.extract_query(jlog_nest["referrer"])
for q in tmp_sessions[k]:
if q[0] == query_nest:
url = jlog_nest["url"]
rank = jlog_nest["rank"]
doc_id = prepare_test_cases_lib.extract_document_id(url)
# Increase number of hits on that document, and save its rank
if doc_id in q[1]:
q[1][doc_id][0] += 1
q[1][doc_id][1].append(rank)
else:
q[1][doc_id] = [1, [rank]]
#except:
# print ("error",k)
result_q.put((k, tmp_sessions[k]))
work_q.task_done()
If it helps understanding what happening tmp_session may look like this before the above code is run:
tmp_sessions: {'39': [['q7', {}], ['q2', {}]], '40': [['q2', {}]]}
And after:
tmp_sessions: {'39': [['q7', {}], ['q2', {'133378': [1, [2]]}]], '40': [['q2', {'133378': [1, [2]]}]]}
On a subset of the real data, with 562 keys and 2232 lines in the file I ran pstats, sorted descending by time (this is just the top):
1284892 function calls in 76.810 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
8 0.000 0.000 77.985 9.748 {built-in method exec}
8 1.607 0.201 77.978 9.747 prepare_hard_test_cases.py:29(mine)
1254384 75.051 0.000 76.220 0.000 {built-in method eval}
562 0.008 0.000 0.050 0.000 queues.py:99(put)
8 0.000 0.000 0.029 0.004 codecs.py:685(readlines)
From this it seems that it is indeed eval taking up the time.
Edit: As suggested I tried with literal_eval. I actually found this trying to find a solution, but thought it would be the same as eval. I just ran it. It does produce the same result, but the run time is really bad:
50205868 function calls (37662028 primitive calls) in 121.494 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
8 0.001 0.000 121.494 15.187 {built-in method exec}
8 0.008 0.001 121.493 15.187 <string>:1(<module>)
8 4.935 0.617 121.485 15.186 prepare_hard_test_cases.py:29(mine)
1254384 5.088 0.000 116.425 0.000 ast.py:39(literal_eval)
1254384 1.098 0.000 71.432 0.000 ast.py:31(parse)
1254384 70.333 0.000 70.333 0.000 {built-in method compile}
13798224/1254384 22.996 0.000 39.336 0.000 ast.py:51(_convert)
7526304 8.539 0.000 23.042 0.000 ast.py:63(<genexpr>)
25087680 8.371 0.000 8.371 0.000 {built-in method isinstance}
8 0.001 0.000 0.047 0.006 codecs.py:685(readlines)
Edit 2: I have now tried two new approaches. The first one is by extracting key and values manually from each line, constructing a dictionary to work on. This works a little faster on my test set:
51460252 function calls in 45.207 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
8 0.001 0.000 45.207 5.651 {built-in method exec}
8 0.003 0.000 45.207 5.651 <string>:1(<module>)
8 1.701 0.213 45.203 5.650 prepare_hard_test_cases.py:68(mine)
1254384 5.725 0.000 43.391 0.000 prepare_hard_test_cases.py:36(extractDict)
6271920 23.433 0.000 37.665 0.000 prepare_hard_test_cases.py:20(extractKeyValue)
18819074 11.308 0.000 11.308 0.000 {method 'find' of 'str' objects}
25092651 2.927 0.000 2.927 0.000 {built-in method len}
This is good news, but even better is my second approach using pickle. Now I get:
30091 function calls in 5.285 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
8 0.000 0.000 5.285 0.661 {built-in method exec}
8 0.003 0.000 5.285 0.661 <string>:1(<module>)
8 0.173 0.022 5.281 0.660 prepare_hard_test_cases.py:68(mine)
570 0.001 0.000 5.057 0.009 queues.py:113(get)
2281 3.925 0.002 3.925 0.002 {method 'acquire' of '_multiprocessing.SemLock' objects}
570 1.133 0.002 1.133 0.002 {method 'recv' of '_multiprocessing.PipeConnection' objects}
8 0.029 0.004 0.029 0.004 {built-in method load}
When I get the time I will attempt to apply this approach to the full set.
Any suggestions?
Best regards, Casper
eval(). - Gareth Lattyjsonmodule fail? - Gareth Latty'is not a valid string delimiter -import json; json.loads("{'key': 'value'}")- Jon Clements