Unable to import module 'lambda_function': cannot import name 'etree' from 'lxml' (/opt/python/lib/python3.11/site-packages/lxml/__init__.py)

Question

I’m trying to run a Python script in AWS Lambda that uses the lxml library, but I’m encountering an import error:

Response
{
  "errorMessage": "Unable to import module 'lambda_function': cannot import name 'etree' from 'lxml' (/opt/python/lib/python3.11/site-packages/lxml/__init__.py)",
  "errorType": "Runtime.ImportModuleError",
  "requestId": "9a37109e-1cc3-4800-8a4e-c70c5c7bfe20",
  "stackTrace": []
}

in my AWS Lambda function?

In my research, I found out that lxml is an OS-dependent library. This seems to be causing issues because I can’t use it as a layer in AWS Lambda, which is installed on a different OS (Windows/Mac/Linux).

Is there a way to install lxml within AWS and add it as a layer for AWS Lambda? How can I resolve this issue and successfully import lxml in my AWS Lambda function?

I use Python 3.11 in here.

Answer 1

When you create a lambda, you specify the architecture it will be running on.

At the moment of this writing, you have a choice of x86_64 and arm64.

Regardless of the architecture, the Lambda runtime uses Amazon Linux (here's the table with Amazon Linux versions for different runtimes). It doesn't use Windows or Mac.

As long as your layer matches the architecture of the Lambda and only uses dependencies provided by the host system, you can totally put all your dependencies into a layer.

When Lambda runtime mounts a layer, it mounts its contents under /opt and adds the directories /opt/python and /opt/python/lib/python3.x/site-packages to sys.path. To create the layer, you need to put all the modules not provided by Lambda runtime under python or python/lib/python3.x/site-packages

The easiest way to create a zip file for the layer is putting all your requirements into requirements.txt and running these commands:

pip install --platform manylinux2014_x86_64 --target=python --implementation cp --python-version 3.11 --only-binary=:all: --upgrade -r requirements.txt
zip -r layer.zip python/

(change manylinux2014_x86_64 to manylinux2014_aarch64 if you're targeting ARM)