Installing Numpy (v1.9.0), Pandas and SKlearn on GAE Standard Env (Python 27)

1
votes

I've successfully ran my app using Webapp2 on local server. Now I wish to deploy this on Google App Engine. I'm currently debugging some errors since libraries are not compatible.

To configure the app, I followed the Google guide to add 3rd party plugins e.g. added lib folder and: 

# appengine_config.py
from google.appengine.ext import vendor

# Add any libraries install in the "lib" folder.
vendor.add('lib')

In the lib folder, I installed my requirements using pip install -t lib -r requirements.txt which is 

google-cloud-bigquery
oauth2client==4.1.2
google-api-python-client==1.6.4
pandas-gbq
pandas
scipy
scikit-learn==0.18.2
numpy==1.9.0

When I deploy my app, I see:

Traceback (most recent call last): File "/base/alloc/tmpfs/dynamic_runtimes/python27/54c5883f70296ec8_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle handler = _config_handle.add_wsgi_middleware(self._LoadHandler()) File "/base/alloc/tmpfs/dynamic_runtimes/python27/54c5883f70296ec8_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler handler, path, err = LoadObject(self._handler) File "/base/alloc/tmpfs/dynamic_runtimes/python27/54c5883f70296ec8_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject obj = import(path[0]) File "/base/data/home/apps/s~corded-epigram-579/gr:20171215t102242.406232341485344902/main.py", line 17, in import pandas as pd File "/base/data/home/apps/s~corded-epigram-579/gr:20171215t102242.406232341485344902/lib/pandas/init.py", line 19, in "Missing required dependencies {0}".format(missing_dependencies)) ImportError: Missing required dependencies ['numpy']

which basically tells me that it can't find numpy, but it is there in the lib folder.

I know that google supports numpy v1.6.1 which I can add using app.yaml file, but I need at least 1.9.0 for the pandas package.

Also added a screenshot of my main.py imports which I need to run the code.

enter image description here

2
pythonnumpygoogle-app-engine
Why are you using the old software if you are just starting? – percusse
@percusse what do you mean by old? Standard instead of Flex? I was following tutorials building simple apps using webapp2. I'm considering trying out Django and using that on flex. Will flex be as simple as removing the lib folder, and changing app.yaml to flex? – GRS
Newer Numpy/pandas? GAE supports Py3 if I'm not mistaken since 2016 if I'm not mistaken. – percusse
Thanks, should I then close the question? – GRS

2 Answers

3
votes

App Engine standard environment only supports pure Python libraries, except for the built-in libraries provided by Google. Given that Numpy 1.6.1 doesn't work for you, you will need to consider using the flexible environment or anything platform like Compute Engine.

Depending on your application and its requirements, you might be able to architect your application into multiple App Engine "services". For example, say you needed to do some offline or asynchronous analysis using those libraries, you could run that code in a service that targets the flexible environment and have the rest of your application running in a service in the standard environment. The same project can split be split between environments or even languages by virtue of diving the application into services.

-2
votes

add init.py file in your lib directory and then

appengine_config.py: 

from google.appengine.ext import vendor
import os
vendor.add(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'lib'))#  vendor.add('lib')

then modify your imports to

from lib.something import something