When I stop the script manually in PyCharm, process finished with exit code 137. But I didn't stop the script. Still got the exit code 137. What's the problem?
Python version is 3.6, process finished when running xgboost.train() method.
Exit code 137 means that your process was killed by (signal 9) SIGKILL . In the case you manually stopped it - there's your answer.
If you didn't manually stop the script and still got this error code, then the script was killed by your OS. In most of the cases, it is caused by excessive memory usage.
{In my experience}
this is because of Memory issue. When I try to train ml model using sklearn fit with full data set , it abruptly breaks and gives whereas with small data it works fine.
Process finished with exit code 137 (interrupted by signal 9: SIGKILL) Interestingly this is not caught in Exception block either
It's not always a memory issue. In my case subprocess.Popen was utilized and it was throwing the error as 137 which looks like signalKILL and the cause is definitely not the memory utilization, because during the runtime it was hardly using 1% of memory use. This seems to be a permission issue after more investigation. I simply moved the scripts from /home/ubuntu to the root directory.
In my case, my RAM ran out, whether it is real or virtual.
Split your data into small pieces or expand your virtual memory.
I choose the latter.
Following scipts works on my ubuntu 20.04 TLS.
# disable the use of swap
sudo swapoff -a
# create the SWAP file. Make sure you have enough space on the hard disk.
# here is my size, the total size is bs*count B
sudo dd if=/dev/zero of=/swapfile bs=1024 count=136314880 status=progress
# output:
# 139458259968 bytes (139 GB, 130 GiB) copied, 472 s, 295 MB/s
# 136314880+0 records in
# 136314880+0 records out
# 139586437120 bytes (140 GB, 130 GiB) copied, 472.372 s, 296 MB/s
# Mark the file as SWAP space:
sudo mkswap /swapfile
# output:
# Setting up swapspace version 1, size = 130 GiB (139586433024 bytes)
# no label, UUID=25a565d9-d19c-4913-87a5-f02750ab625d
# enable the SWAP.
sudo swapon /swapfile
# check if SWAP is created
sudo swapon --show
# output:
# NAME TYPE SIZE USED PRIO
# /swapfile file 130G 0B -2
# Once everything is set, you must set the SWAP file as permanent, else you will lose the SWAP after reboot. Run this command:
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
After you run your process, memory will grow.
Here is mine:
Good Luck!
SIGKILL(signal 9). Which could happen for a lot of reasons, but usually by excessive memory use. - umutto