Unable to install package through pip on AWS EMR Notebook

Question

I'm using a Jupyter Notebook connected to an AWS EMR Cluster. The subnets and security groups attached to all nodes of the cluster are able to connect to the internet. However, when I attempt to pip install <package> from the Notebook attached to the EMR cluster, I get the following error:

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6b4c6bd710>: Failed to establish a new connection: [Errno 101] Network is unreachable')': .....

I have also tried to make a GET request to the internet r = requests.get('https://www.google.com'), which was also unsuccessful. I'm unable to figure out what is preventing my connectivity to the internet from my Jupyter Notebook on AWS EMR.

Answer 1

Since you are able to connect to internet with cluster, so it might be the problem with security groups of notebook itself not the cluster.

Are you using default security groups for EMR notebook like described here or are you assigning your own security group? if yes,make sure you have right outbound rules, try to add outbound rules and allow traffic

Answer 2

In my case it turned out to be a problem with the sg ElasticMapReduceEditors-Livy & ElasticMapReduceEditors-Editor I had created, I just added a rule in both sg that would allow all outbound traffic to test if that solved it, luckily it solved it, then I removed both rules and mysteriously my notebooks kept working as usual, I think it was bad behavior from aws.