How to package SoX binary with MP3 support for a NodeJS AWS Lambda Function with limitations of AWS's Linux AMI?

Question

I've been working on this issue for the past 3 months and am completely stuck.

I am trying to package up my NodeJS AWS Lambda function that will use SoX and it's dependencies to convert audio files to MP3. I can get my code to recognize the custom location of the SoX binary by following instructions referenced here, and here. I ended up adding this code to the start of my Lambda function call to update the process.env PATH variable to include the path to the custom binary.

process.env['PATH'] = process.env['PATH'] + ':' 
+ path.join(process.env['LAMBDA_TASK_ROOT'], 'binaries');

This results in my process.env PATH updating to look like this:

/usr/local/lib64/node-v4.3.x/bin:/usr/local/bin:/usr/bin/:/bin:/var/task/binaries

Which looks correct, binaries is the directory that contains the sox binary I compiled.

Since I'm using NodeJS, I had to modify the sox-audio NPM module so it used the updated process.env variable for the child_process exec and spawn calls. This allows the code to find the binary, but I'm still getting an error during execution.

Sox process exited with code 127 and signal null

I understand that although it can find the SoX binary I included, it is not able to find a command used with SoX, but without more information I can't tell what it is. I was thinking it was because I am not sure if I have all the files included for the binary to work.

In an attempt to make a clean compiled build of SoX with MP3 support, I created a fresh EC2 linux instance and then followed the instructions provided here.

I went line by line to ensure I could get it working, and after installing a few dependencies to enable compiling (such as developer tools), and by exporting the build PATH with export PATH=$PATH:/usr/local/bin I was able to get a full build installed with MP3 support. I tested it and it works exactly like how I need it.

Since AWS Lambda Functions run on the same stripped down version of Linux (Amazon Linux AMI) as AWS EC2 instances, theoretically, if I could export the SoX build and include it in my Lambda Package, then I should be able to get it to work.

That is where I am having trouble. What makes up the built itself? There is a SoX linux executable in /usr/local/binwhich is a single file, but there are also a ton more files there that seem all related to making SoX and its dependencies work. Here is a list of files within /usr/local/bin on the working build I have.

I attempted to export all these files via FTP and then import them into another clean EC2 instance, but even after running export PATH=$PATH:/usr/local/bin SoX would fail to run because of a dependency issue. It is clear that just exporting these files is not enough.

• How do I export and what do I export to take my working build of SoX with MP3 support off the EC2 instance and include it to work within my AWS Lambda Function?
• Is the long list of installed dependencies to get MP3 to work going to make this impossible?
• Are there files in more than just the /usr/local/bin directory that I need to include?

I really don't know where else to go with this. Please help :(

Answer 1

Took me 3 months to finally post, and I figure it out the same day.

After doing some more research I came to the conclusion that I needed to create a static build of SoX with the necessary dependencies to convert MP3. I ended up putting together a script to create a static build based on this blog post, and this blog post.

The reason I combined a bit from both was because I was using Amazons Linux AMI, which is very barebones. So all the debian and apt-get stuff mentioned in the second blog would not likely work. However, I needed to ensure I had a complier installed so I wouldn't run into any errors. Below is what I ended up running. I did the whole thing as root, since privileges are fairly restricted on EC2 Amazon Linux AMI instances. The only thing that needs to be ran as the regular user is the PATH export at the very end.

sudo yum update
sudo yum install gcc44 gcc-c++ libgcc44 cmake –y

# now grab sox and its dependencies
mkdir -p deps
mkdir -p deps/unpacked
mkdir -p deps/built
mkdir -p deps/built/libmad
mkdir -p deps/built/sox
mkdir -p deps/built/lame
wget -O deps/sox-14.4.1.tar.bz2 "http://downloads.sourceforge.net/project/sox/sox/14.4.1/sox-14.4.1.tar.bz2?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fsox%2Ffiles%2Fsox%2F14.4.1%2F&ts=1416316415&use_mirror=heanet"
wget -O deps/libmad-0.15.1b.tar.gz "http://downloads.sourceforge.net/project/mad/libmad/0.15.1b/libmad-0.15.1b.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fmad%2Ffiles%2Flibmad%2F0.15.1b%2F&ts=1416316482&use_mirror=heanet"
wget -O deps/lame-3.99.5.tar.gz "http://downloads.sourceforge.net/project/lame/lame/3.99/lame-3.99.5.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Flame%2Ffiles%2Flame%2F3.99%2F&ts=1416316457&use_mirror=kent"

# unpack the dependencies
pushd deps/unpacked
tar xvfpj ../sox-14.4.1.tar.bz2
tar xvfpz ../libmad-0.15.1b.tar.gz
tar xvfpz ../lame-3.99.5.tar.gz
popd

# build libmad, statically
pushd deps/unpacked/libmad-0.15.1b
./configure --disable-shared --enable-static --prefix=$(realpath ../../built/libmad)
# Patch makefile to remove -fforce-mem
sed s/-fforce-mem//g < Makefile > Makefile.patched
cp Makefile.patched Makefile
make
make install
popd

# build lame, statically
pushd deps/unpacked/lame-3.99.5
./configure --disable-shared --enable-static --prefix=$(realpath ../../built/lame)
make
make install
popd

# build sox, statically
pushd deps/unpacked/sox-14.4.1
./configure --disable-shared --enable-static --prefix=$(realpath ../../built/sox) \
    LDFLAGS="-L$(realpath ../../built/libmad/lib) -L$(realpath ../../built/lame/lib)" \
    CPPFLAGS="-I$(realpath ../../built/libmad/include) -I$(realpath ../../built/lame/include)" \
    --with-mad --with-lame --without-oggvorbis --without-oss --without-sndfile --without-flac  --without-gomp
make -s
make install
popd

cp deps/built/sox/bin/sox .
rm -rf deps/built
rm -rf deps/unpacked

After this I ran the following as my normal user

export PATH=$PATH:/home/ec2-user

I could then run sox! Just typing in sox in the terminal should bring up sox the command list in your terminal window.

Even better, I was able to then download the single sox linux executable file, and then upload it to a brand new EC2 instance. The only things I had to run on the new instance to get it to work was the following:

sudo yum update
sudo yum install gcc44 gcc-c++ libgcc44 cmake –y
export PATH=$PATH:/home/ec2-user

I was worried that I would have to try and build it again without making the yum update or yum installs, since I ultimately want to put this in a Lambda function, but that was not necessary! When I added it to my test Lambda function and uploaded, it worked without an issue according to the cloud logs. That must mean the Amazon Linux AMI used within Lambda containers already has the yum updates and proper compilers installed.