Is there a server that provides an Amazon S3 style API locally? [closed]

Answer 1

Have you looked at Cloudian? We use it internally at our company to develop our S3 app. I'm using the Community Edition which is free for up to 10TB of storage. It's got pretty good S3 coverage or at least covers most of the stuff my app uses (I use versioning and multipart uploads so I think my app is advanced). The version-ids and multipart ids etc that it generates are different than those you get from AWS but boto has no complaints so far. It also works with s3fs and other s3 bucket browsers that I have tried.

In my opinion it's a good tool for development against the AWS S3 API and should meet your requirements. You can point your app at your local Cloudian server and then when you are ready for production you can point it back at Amazon. Your mileage may vary... Good luck.

Answer 2

We ran into the problem of testing our S3 based code locally and actually implemented a small Java server, which emulates the S3 object API. As it might be useful to others, we setup a github repo along with a small website: http://s3ninja.net - all OpenSource under the MIT license.

Being quite simple and minimalistic, this tool is perfect for testing and developement purposes. However, to use in in production, one might want to add some security (altough the AWS hashes are already verified in the API - just the GUI is completely unprotected). Also, it doesn't do any replication or scaling. So this wouldn't be a good choice for large setups.

Answer 3

As it was already mentioned: you could try to use Swift as Amazon S3 alternative. Take a look at SwiftFS filesystem, it let you mount OpenStack container stored in Swift as a local filesystem.

Answer 4

This is possible via OpenStack Object Storage (code-named Swift), which is open source software for creating redundant, scalable object storage using clusters of standardized servers, specifically its recently added (optional) S3 API layer, which emulates the S3 REST API on top of Object Storage.

See Configuring Object Storage with the S3 API for the official documentation - a more insightful and illustrated small tutorial regarding the entire setup is available in S3 APIs on OpenStack Swift (which builds on the more complex Installing an OpenStack Swift cluster on EC2 though).

---

Update

An noteworthy alternative is Ceph, which is a unified, distributed storage system designed for excellent performance, reliability and scalability - interestingly it provides all three common storage models, i.e. Object Storage, Block Storage and a File System and the RADOS Gateway provides Amazon S3 and OpenStack Swift compatible interfaces to the RADOS object store [emphasis mine], see RADOS S3 API for details on currently supported S3 API features.

Answer 5

If you want to have an S3-like API but hosting the data yourself the mention of minio is spot on.

If you want to write services that interact with S3 but you want to test them locally for speed you can use Localstack which I don't think I've seen mentioned here. It emulates not only S3 but many other AWS services as well. I wouldn't recommend using it for actual customer files as it is an in-memory datastore for testinng only.

https://github.com/localstack/localstack

https://bluesock.org/~willkg/blog/dev/using_localstack_for_s3.html

Answer 6

Minio will be useful for this. Written in Go and simple to deploy.

Binary downloads here: https://min.io/download

To run as as an S3 server:

minio server path/to/disk/storage

Access key and secret will be output to the console.