I Published a Layer for Running pandas on AWS Lambda
This article was automatically translated from Japanese by AI. Read the original Japanese version →
Table of Contents
I've published pandas-lambda-layer.
It's a layer for running Python's data analysis library pandas on AWS Lambda.
Why Did I Make It?
Lately at work I've had opportunities to run machine learning on AWS, and I've been using SageMaker.
It's a handy service that makes training, inference, and the preprocessing they require easy to run, though both the time and monetary costs can add up quite a bit. Because it essentially spins up a dedicated EC2 instance automatically and runs a container inside it, the instance startup overhead is annoying—it takes around 4 to 5 minutes. (Note for people familiar with SageMaker: of course, this doesn't apply when using a persistent inference endpoint.)
If the dataset for training or inference is large enough, that startup time isn't much of an issue, but for use cases dealing with smaller data, it becomes a nagging source of friction.
So I figured if the workload is light, I might as well just handle it in Lambda, and put together a layer with just pandas working for now. For preprocessing and the like, having just pandas is surprisingly often all you need. For inference or training, you can use whatever library you prefer.
How to Use It
Clone the repository and run make build and make zip to generate layer.zip. From there, you can open the Lambda Layers screen in the AWS Web Console and upload it directly, or upload it to S3 first and attach it to a Lambda Layer—whichever you prefer.
Some people publish layers they've already uploaded(?) directly on AWS, but using a layer published by someone else without knowing what's inside feels risky, so I chose not to publish the built layer itself. (Plus, I'm not even sure how you inspect the contents of someone else's published layer.)
Also, there are repositories that go as far as handling the upload to AWS using tools like the Serverless Framework, but since I've never used the Serverless Framework and everyone has their own preferred way to upload layer.zip, I left that part out. Personally, I upload it to S3 via the AWS CLI and manage it with CloudFormation.
How the Layer Is Built
Container images replicating the AWS Lambda execution environment are publicly available.
That repository also provides container images for building Layers and executables to run inside Lambda. So all you really have to do is use that to install pandas and zip up the target directory. If you take a look at my repository, you'll see that's literally all it does.
That works fine for pandas on its own, though some other libraries won't run with just that due to shared library (.so file) dependencies. In those cases, you also need to create a lib directory and copy the .so files into it before zipping.
Aren't There Other pandas Layers Out There?
Yeah... I only noticed this repository after I had already finished building my own layer. Still, as I mentioned earlier, using a layer someone else built (without checking its contents) feels a bit questionable. Mine is simple enough that you can easily read and modify it if needed, and you upload layer.zip yourself, so I figured this approach has its own easy appeal.

