How do I get "from osgeo import gdal" working?

[23ccozad]

I've seen discussion #79 related to this, which was very helpful. This is a really great repo, so thanks for putting this together. Similar to @mirouml, I'm trying to import the GDAL python package into my Lambda function, but I can't seem to get it working. Here are the steps I've taken so far:

Dockerfile (including a couple of changes different from the example provided in readme):

# Use the lambgeo/lambda-gdal image to get GDAL binaries and libraries
FROM ghcr.io/lambgeo/lambda-gdal:3.8 as gdal

# Use the base AWS Lambda Python 3.11 image
FROM public.ecr.aws/lambda/python:3.11

# Had to add this line, it was not included in the example in the repo's readme
RUN yum install -y gcc gcc-c++ make zip

# Bring C libs from lambgeo/lambda-gdal image
COPY --from=gdal /opt/lib/ /opt/lib/
COPY --from=gdal /opt/include/ /opt/include/
COPY --from=gdal /opt/share/ /opt/share/
COPY --from=gdal /opt/bin/ /opt/bin/

ENV \
  GDAL_DATA=/opt/share/gdal \
  PROJ_LIB=/opt/share/proj \
  GDAL_CONFIG=/opt/bin/gdal-config \
  GEOS_CONFIG=/opt/bin/geos-config \
  PATH=/opt/bin:$PATH

ENV PACKAGE_PREFIX=/var/task

# Note: I'm leaving out handler.py, I want to edit and test my own handler in the AWS console
# COPY handler.py ${PACKAGE_PREFIX}/handler.py

# Install GDAL Python bindings
RUN python -m pip install GDAL==$(gdal-config --version) -t $PACKAGE_PREFIX

# Create package.zip
RUN cd $PACKAGE_PREFIX && zip -r9q /tmp/package.zip *

I then followed the instructions exactly in readme to create the zip file from the Docker container.

The resulting zip file has the following structure:

package.zip
  |___ bin/
  |___ GDAL-3.8.3.dist-info/
  |___ osgeo/
  |___ osgeo_utils/

I upload this zip file as a new version of my Lambda layer, and update the version of the layer used in my function:

The function's environment variables are set like so:

Then, I run my Lambda function with the code:

import os
import sys

def lambda_handler(event, context):
    # Print the Python path
    print("Python path: ", sys.path)

    # List the contents of /opt
    print(f"Contents of /opt: {os.listdir('/opt')}")

    # List the contents of /opt/share
    print(f"Contents of /opt/share: {os.listdir('/opt/share')}")

    # Try to import osgeo
    try:
        import osgeo
        print("osgeo module imported successfully!")
        from osgeo.gdal import gdal
        print("gdal module imported successfully!")
    except ImportError as e:
        print("Error importing module:", e)

    return {
        'statusCode': 200,
        'body': 'Check the logs for details.'
    }

The output from the function is:

Python path:  ['/var/task', '/opt/python/lib/python3.11/site-packages', '/opt/python', '/var/lang/lib/python3.11/site-packages', '/var/runtime', '/var/lang/lib/python311.zip', '/var/lang/lib/python3.11', '/var/lang/lib/python3.11/lib-dynload', '/var/lang/lib/python3.11/site-packages', '/opt/python/lib/python3.11/site-packages', '/opt/python']
Contents of /opt: ['GDAL-3.8.3.dist-info', 'bin', 'lib', 'osgeo', 'osgeo_utils', 'share']
Contents of /opt/share: ['aclocal', 'bash-completion', 'gdal', 'gtk-doc', 'nghttp2', 'postgresql', 'proj']
Error importing module: No module named 'osgeo'

I'm very new to Lambda and Docker, so any suggestions are much appreciated. Thanks!

[mirouml]

I did make it work using Docker. It was actually very helpful to learn as I had to continue to also use Docker to add ArcGIS as I needed to store some result to ArcGIS Online.

My full workflow to create and get lambda layer running so I can use "from osgeo import gdal":

How to build lambda layer with GDAL and python 3.11

1. Install Docker
2. Create Docker file
3. Run docker
4. Remove container
5. use zip to create lambda layer

Create Docker file

Example for python 3.11 and gdal3.8 with numpy (possibly also rasterio, fiona...):

# Use Python 3.11 base image from AWS Lambda
FROM public.ecr.aws/lambda/python:3.11 AS base

# Bring C libs from lambgeo/lambda-gdal image
FROM ghcr.io/lambgeo/lambda-gdal:3.8-python3.11 AS gdal

# Create a new build stage based on the base image
FROM base AS build

COPY --from=gdal /opt/lib/ /opt/lib/
COPY --from=gdal /opt/include/ /opt/include/
COPY --from=gdal /opt/share/ /opt/share/
COPY --from=gdal /opt/bin/ /opt/bin/

ENV \
  GDAL_DATA=/opt/share/gdal \
  PROJ_LIB=/opt/share/proj \
  GDAL_CONFIG=/opt/bin/gdal-config \
  GEOS_CONFIG=/opt/bin/geos-config \
  PATH=/opt/bin:$PATH

ENV PACKAGE_PREFIX=/opt/python

# Install build tools and GDAL Python bindings
RUN yum -y groupinstall "Development Tools" && \
    yum -y install python3-devel && \
    python -m pip install numpy && \
    python -m pip install GDAL==$(gdal-config --version) -t $PACKAGE_PREFIX && \
    # pip install rasterio fiona shapely geopandas - all together are too big for lambda
    # python -m pip install -t $PACKAGE_PREFIX && \
    yum -y groupremove "Development Tools" && \
    yum -y remove python3-devel && \
    yum -y clean all

# Create layer.zip
RUN cd /opt && zip -r9 /tmp/layer.zip python

# Final stage to create the final image
FROM base

COPY --from=build /tmp/layer.zip /tmp/layer.zip

For GDAL lambda layer

Build

docker build -t my-lambda-layer .

Create container

docker create --name my-lambda-layer-container my-lambda-layer

Copy

docker cp my-lambda-layer-container:/tmp/layer.zip .

Remove container

docker rm my-lambda-layer-container

Create lambda layer

In AWS console go to Lambda - Additional Resources - Layers and Create layer

[marcosci]

Is this still working for you? Neither the example in this repo, nor the two solutions here seem to work for me.
Somehow gdal isn't recognised:

57.86       /usr/bin/ld: skipping incompatible /opt/lib/libgdal.so when searching for -lgdal
57.86       /usr/bin/ld: cannot find -lgdal
57.86       collect2: error: ld returned 1 exit status
57.86       error: command '/usr/bin/gcc' failed with exit code 

[marcosci]

Complaining somehow always helps ... So, somehow gdal-config version returns 3.10.x and we need 3.8.3 (coming from docker-lambda.

So this should work:

# Use the lambgeo/lambda-gdal image to get GDAL binaries and libraries
FROM --platform=linux/amd64 ghcr.io/lambgeo/lambda-gdal:3.8-python3.11 AS gdal

# Use the base AWS Lambda Python 3.11 image
FROM --platform=linux/amd64 public.ecr.aws/lambda/python:3.11

# Had to add this line, it was not included in the example in the repo's readme
RUN yum install -y gcc gcc-c++ make zip libgdal-dev

# Bring C libs from lambgeo/lambda-gdal image
COPY --from=gdal /opt/lib/ /opt/lib/
COPY --from=gdal /opt/include/ /opt/include/
COPY --from=gdal /opt/share/ /opt/share/
COPY --from=gdal /opt/bin/ /opt/bin/

ENV \
    GDAL_DATA=/opt/share/gdal \
    PROJ_LIB=/opt/share/proj \
    GDAL_CONFIG=/opt/bin/gdal-config \
    GEOS_CONFIG=/opt/bin/geos-config \
    PATH=/opt/bin:$PATH \
    LD_LIBRARY_PATH=/opt/lib/:$LD_LIBRARY_PATH

ENV PACKAGE_PREFIX=/var/task

# Install GDAL Python bindings
RUN python -m pip install GDAL==3.8.3 -t $PACKAGE_PREFIX

# Create package.zip
RUN cd $PACKAGE_PREFIX && zip -r9q /tmp/package.zip *

[23ccozad]

Thanks for sharing @mirouml, your approach is very similar. I just ended up figuring out what I was missing. The Lambda layer expects dependencies to be located in \python within the zip file, according to this AWS documentation.

So I changed the structure of my zip file from:

package.zip
  |___ bin/
  |___ GDAL-3.8.3.dist-info/
  |___ osgeo/
  |___ osgeo_utils/

to

package.zip
  |___ python/
      |___ bin/
      |___ GDAL-3.8.3.dist-info/
      |___ osgeo/
      |___ osgeo_utils/

Now, I can import osgeo and gdal without any problems. Hopefully, this discussion can help others get gdal working in their Lambda functions as well. Thanks!