[gdal-dev] Use GDAL on AWS Lambda

classic Classic list List threaded Threaded
5 messages Options
kch
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Use GDAL on AWS Lambda

kch
Hello,

I am looking for any help from anyone who has experience with using GDAL on
AWS Lambda. I am trying to deploy a python function which uses GDAL onto AWS
Lambda. At the moment I am using Serverless to package and deploy my
function. I am at a roadblock because I can't figure out how to get all the
dependencies into my deployment package. My Serverless function packages the
dependencies listed in a requirements.txt using the
serverless-python-requirements plugin and dockerizePip: true option. At
deployment, I get the error:

Serverless: Installing requirements of requirements.txt in .serverless…
Serverless: Docker Image: lambci/lambda:build-python3.6
Error --------------------------------------------------
The directory ‘/.cache/pip/http’ or its parent directory is not owned by the
current user and the cache has been disabled. Please check the permissions
and owner of that directory. If executing pip with sudo, you may want sudo’s
-H flag.
The directory ‘/.cache/pip’ or its parent directory is not owned by the
current user and caching wheels has been disabled. check the permissions and
owner of that directory. If executing pip with sudo, you may want sudo’s -H
flag.
Command “python setup.py egg_info” failed with error code 1 in
/tmp/pip-install-yo9exd91/pygdal/
For debugging logs, run again after setting the “SLS_DEBUG=*” environment
variable.

I have already updated setuptools to the latest version and even included
the latest version of setuptools in requirements.txt. I think the problem is
that at deployment, Serverless does not pick up all the non-python
dependencies of GDAL. I don't know how to work around this.

I am running on Ubuntu 16.04 and Python3.6. Thank you in advance.




--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Use GDAL on AWS Lambda

Jeremy Palmer-3
I have no direct experience, but there are plenty of resources on the web for how to install GDAL on Lambda:


Both are using python 2.7, but the steps should be translatable to 3.6 as your biggest issue is likely to be the size of linked binaries which should be similar under both versions.

Cheers
Jeremy

On Fri, Jul 13, 2018 at 7:31 AM kch <[hidden email]> wrote:
Hello,

I am looking for any help from anyone who has experience with using GDAL on
AWS Lambda. I am trying to deploy a python function which uses GDAL onto AWS
Lambda. At the moment I am using Serverless to package and deploy my
function. I am at a roadblock because I can't figure out how to get all the
dependencies into my deployment package. My Serverless function packages the
dependencies listed in a requirements.txt using the
serverless-python-requirements plugin and dockerizePip: true option. At
deployment, I get the error:

Serverless: Installing requirements of requirements.txt in .serverless…
Serverless: Docker Image: lambci/lambda:build-python3.6
Error --------------------------------------------------
The directory ‘/.cache/pip/http’ or its parent directory is not owned by the
current user and the cache has been disabled. Please check the permissions
and owner of that directory. If executing pip with sudo, you may want sudo’s
-H flag.
The directory ‘/.cache/pip’ or its parent directory is not owned by the
current user and caching wheels has been disabled. check the permissions and
owner of that directory. If executing pip with sudo, you may want sudo’s -H
flag.
Command “python setup.py egg_info” failed with error code 1 in
/tmp/pip-install-yo9exd91/pygdal/
For debugging logs, run again after setting the “SLS_DEBUG=*” environment
variable.

I have already updated setuptools to the latest version and even included
the latest version of setuptools in requirements.txt. I think the problem is
that at deployment, Serverless does not pick up all the non-python
dependencies of GDAL. I don't know how to work around this.

I am running on Ubuntu 16.04 and Python3.6. Thank you in advance.




--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Use GDAL on AWS Lambda

Harris, Tim
In reply to this post by kch
I haven't used Serverless, but I'm running GDAL in AWS Lambda using binary wheels:

https://github.com/youngpm/gdalmanylinux

This repo is based on Sean Gillies's wheel build for Fiona, Rasterio, and Shapely (https://github.com/sgillies/frs-wheel-builds). I believe it has instructions, but the general idea is that it uses an old linux Docker image to build GDAL wheels for a few different Python versions. Once those are built, I use the pip --target option to install my Lambda code, any required Python modules, and the GDAL wheel into a local directory. Then I just zip up that directory, upload to S3, and run my Lambda function. The zip ends up being about 30 MB.

For example, to install the GDAL wheel into the local directory, I would do something like this:
pip install --target ./lambda_zip_dir ./wheels/GDAL-2.3.0-cp36-cp36m-manylinux1_x86_64.whl

Same idea for any other dependencies and for my own Lambda code.

-----Original Message-----
From: gdal-dev [mailto:[hidden email]] On Behalf Of kch
Sent: Thursday, July 12, 2018 1:31 PM
To: [hidden email]
Subject: [gdal-dev] Use GDAL on AWS Lambda

Hello,

I am looking for any help from anyone who has experience with using GDAL on
AWS Lambda. I am trying to deploy a python function which uses GDAL onto AWS
Lambda. At the moment I am using Serverless to package and deploy my
function. I am at a roadblock because I can't figure out how to get all the
dependencies into my deployment package. My Serverless function packages the
dependencies listed in a requirements.txt using the
serverless-python-requirements plugin and dockerizePip: true option. At
deployment, I get the error:

Serverless: Installing requirements of requirements.txt in .serverless…
Serverless: Docker Image: lambci/lambda:build-python3.6
Error --------------------------------------------------
The directory ‘/.cache/pip/http’ or its parent directory is not owned by the
current user and the cache has been disabled. Please check the permissions
and owner of that directory. If executing pip with sudo, you may want sudo’s
-H flag.
The directory ‘/.cache/pip’ or its parent directory is not owned by the
current user and caching wheels has been disabled. check the permissions and
owner of that directory. If executing pip with sudo, you may want sudo’s -H
flag.
Command “python setup.py egg_info” failed with error code 1 in
/tmp/pip-install-yo9exd91/pygdal/
For debugging logs, run again after setting the “SLS_DEBUG=*” environment
variable.

I have already updated setuptools to the latest version and even included
the latest version of setuptools in requirements.txt. I think the problem is
that at deployment, Serverless does not pick up all the non-python
dependencies of GDAL. I don't know how to work around this.

I am running on Ubuntu 16.04 and Python3.6. Thank you in advance.




--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
kch
Reply | Threaded
Open this post in threaded view
|

Re: Use GDAL on AWS Lambda

kch
Tim's solution worked for me. Thank you!



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Use GDAL on AWS Lambda

Matt Hanson-2
A bit late to the party, but there's also geolambda which works in a similar manner as the above solutions. It's got the latest version of GDAL and dependencies, Python 2.7 and 3.6 and packaging scripts for packaging up your code for deployment to lambda:

On Fri, Jul 13, 2018 at 2:17 PM kch <[hidden email]> wrote:
Tim's solution worked for me. Thank you!



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev