AWS Lambda not importing LXML

问题

I am trying to use the LXML module within AWS Lambda and having no luck. I downloaded LXML using the following command:

pip install lxml -t folder

To download it to my lambda function deployment package. I zipped the contents of my lambda function up as I have done with all other lambda functions, and uploaded it to AWS Lambda.

However no matter what I try I get this error when I run the function:

Unable to import module 'handler': /var/task/lxml/etree.so: undefined symbol: PyFPE_jbuf

When I run it locally, I don't have an issues, it is simply when I run in on Lambda where this issue arises.

回答1:

I faced the same issue.

The link posted by Raphaël Braud was helpful and so was this one: https://nervous.io/python/aws/lambda/2016/02/17/scipy-pandas-lambda/

Using the two links I was able to successfully import lxml and other required packages. Here are the steps I followed:

Launch an ec2 machine with Amazon Linux ami

Run the following script to accumulate dependencies:

set -e -o pipefail
sudo yum -y upgrade
sudo yum -y install gcc python-devel libxml2-devel libxslt-devel

virtualenv ~/env && cd ~/env && source bin/activate
pip install lxml
for dir in lib64/python2.7/site-packages \
     lib/python2.7/site-packages
do
if [ -d $dir ] ; then
   pushd $dir; zip -r ~/deps.zip .; popd
fi
done  
mkdir -p local/lib
cp /usr/lib64/ #list of required .so files
local/lib/
zip -r ~/deps.zip local/lib

Create handler and worker files as specified in the link. Sample file contents:

handler.py

import os
import subprocess


libdir = os.path.join(os.getcwd(), 'local', 'lib')

def handler(event, context):
    command = 'LD_LIBRARY_PATH={} python worker.py '.format(libdir)
    output = subprocess.check_output(command, shell=True)

    print output

    return

worker.py:

import lxml

def sample_function( input_string = None):
    return "lxml import successful!"

if __name__ == "__main__":
    result = sample_function()
    print result

Add handler and worker to zip file.

Here is how the structure of the zip file looks after the above steps:

deps 
├── handler.py
├── worker.py 
├── local
│   └── lib
│       ├── libanl.so
│       ├── libBrokenLocale.so
|       ....
├── lxml
│   ├── builder.py
│   ├── builder.pyc
|       ....
├── <other python packages>

Make sure you specify the correct handler name while creating the lambda function. In the above example, it would be- "handler.handler"

Hope this helps!

回答2:

Extending on these answers, I found the following to work well.

The punchline here is having python compile lxml with static libs, and installing in the current directory rather than site-packages.

It also means you can write your python code as usual, without need for a distinct worker.py or fiddling with LD_LIBRARY_PATH

sudo yum groupinstall 'Development Tools'
sudo yum -y install python36-devel python36-pip
sudo ln -s /usr/bin/pip-3.6 /usr/bin/pip3
mkdir lambda && cd lambda
STATIC_DEPS=true pip3 install -t . lxml
zip -r ~/deps.zip *

to take it to the next level, use serverless and docker to handle everything. here is a blog post demonstrating this: https://serverless.com/blog/serverless-python-packaging/

回答3:

AWS Lambda use a special version of Linux (as far as I can see).

Using "pip install a_package -t folder" is the good thing to do usually as it will help to package your dependencies within the archive that will be sent to Lambda, but the libraries, and especially the binary libraries have to be compatible with the version of OS and Python on lambda.

You could use the xml module included in Python : https://docs.python.org/2/library/xml.etree.elementtree.html

If you really need lxml, this link gives some tricks on how to compile shared libraries for Lambda : http://www.perrygeo.com/running-python-with-compiled-code-on-aws-lambda.html

回答4:

Expanding a bit on Mask's answer. In the case of installing lxml in particular, the libxslt and libxml2 libraries are already installed on the AMI that executes the AWS lambda. Therefore it is no need to start a subprocess with a different LD_LIBRARY_PATH as in that answer, it is however necessary to run pip install lxml on an AMI image (it might be possible to cross-compile as well but I don't know how).

Launch an ec2 machine with Amazon Linux ami
Run the following script to accumulate dependencies:
set -e -o pipefail
sudo yum -y upgrade
sudo yum -y install gcc python-devel libxml2-devel libxslt-devel

virtualenv ~/env && cd ~/env && source bin/activate
pip install lxml
for dir in lib64/python2.7/site-packages \
    lib/python2.7/site-packages
do
    if [ -d $dir ] ; then
        pushd $dir; zip -r ~/deps.zip .; popd
    fi
done

Note that the last steps from Marks answer is left out. You can use lxml straight from the python file that contains the handler method.

来源：https://stackoverflow.com/questions/36397171/aws-lambda-not-importing-lxml

标签

python

amazon-web-services

lxml

aws-lambda