Got Exception Error “Exception in thread Thread-13 (most likely raised during interpreter shutdown)”

Question

I wrote a simple script, which using threads to retrieve data from service.

    __author__ = 'Igor'
import requests
import time
from multiprocessing.dummy import Pool as ThreadPool

ip_list = []
good_ip_list = []
bad_ip_list = []
progress = 0

with open('/tmp/ip.txt') as f:
    ip_list = f.read().split()

def process_request(ip):
    global progress
    progress += 1
    if progress % 10000 == 0:
        print 'Processed ip:', progress, '...'
    r = requests.get('http://*****/?ip='+ip, timeout=None)
    if r.status_code == 200:
        good_ip_list.append(ip)
    elif r.status_code == 400:
        bad_ip_list.append(ip)
    else:
        print 'Unknown http code received, aborting'
        exit(1)

pool = ThreadPool(16)
try:
    pool.map(process_request, ip_list)
except:
    for name, ip_list in (('/tmp/out_good.txt', good_ip_list),     ('/tmp/out_bad.txt', bad_ip_list)):
        with open(name, 'w') as f:
            for ip in ip_list:
                print>>f, ip

But after some requests processed (40k-50k) i receive:

Exception in thread Thread-7 (most likely raised during interpreter shutdown): Traceback (most recent call last): Process finished with exit code 0

Tried to change service settings:

        <timeout>999</timeout>
        <connectionlimit>600</connectionlimit>
        <httpthreads>32</httpthreads>
        <workerthreads>128</workerthreads>

but still same error. Can anybody help me - what's wrong?

` print>>f, ip` is this a typo? Is there anything else in the traceback? — Patrick Collins
progress += 1 in language with mutable data and using multiple threads w/o any protection... I stopped looking at that point ;) — iced
@PatrickCollins as i understood - the problem in requests.exceptions.ConnectionError raise, tried to catch it and continue pool.map but same effect — Igor
your code is wrong. same with good/bad_ip_list.append (shared lists accessed from multiple threads w/o any protection). you need to learn how to make multithreaded apps in python first. I can bet that is the reason for error you are getting, but I'm not going to investigate it deeper while it's bad from very start. — iced
simple explanation - when 2 threads are trying to append to the same list in the same time bad things will happen. I have no idea why they advocate languages with mutable data as good for beginners... — iced

Igor Igor · Accepted Answer · 2015-03-16T10:17:37

Thanks to everybody, who helped me in solving this problem. Rewrote the whole code and now it works perfectly:

__author__ = 'kulakov'
import requests
import time
from multiprocessing.dummy import Pool as ThreadPool

ip_list = []
good_ip_list = []
bad_ip_list = []

with open('/tmp/ip.txt') as f:
    ip_list = f.read().split()

s = requests.Session()
def process_request(ip):
    r = s.get('http://*****/?ip='+ip, timeout=None)
    if r.status_code == 200:
        # good_ip_list.append(ip)
        return (ip, True)
    elif r.status_code == 400:
        # bad_ip_list.append(ip)
        return (ip, False)
    else:
        print 'Unknown http code received, aborting'
        exit(1)

pool = ThreadPool(16)
for ip, isOk in pool.imap(process_request, ip_list):
    if isOk:
        good_ip_list.append(ip)
    else:
        bad_ip_list.append(ip)
pool.close()
pool.join()

for name, ip_list in (('/tmp/out_good.txt', good_ip_list),    ('/tmp/out_bad.txt', bad_ip_list)):
    with open(name, 'w') as f:
        for ip in ip_list:
            print>>f, ip

Some new usefull information:

1) It was really bad idea to write data in different threads in a function process_request, now it returns statement(true\false) and ip.

2) keep alive is fully supported by requests, by default, but if you want to use it, you must create instance of an object Session, and apply get method to it only:

s = requests.Session()
r = s.get('http://*****/?ip='+ip, timeout=None)

Got Exception Error “Exception in thread Thread-13 (most likely raised during interpreter shutdown)”

2 Answers