If we want use S3 to host Python packages, how can we tell pip where to find the newest version?

问题

we are trying to come up with a solution to have AWS S3 to host and distribute our Python packages.

Basically what we want to do is using "python3 setup.py bdist_wheel" to create a wheel. Upload it to S3. Then any server or any machine can do "pip install $http://path/on/s3". (including a virtualenv in AWS lambda) (We've looked into Pypicloud and thought it's an overkill.)

Creating package and installing from S3 work fine. There is only one issue here: we will release new code and give them different versions. If we host our code on Pypi, you can upgrade some packages to their newest version by calling "pip install package --upgrade".

But if you host your packages on S3, how do you let pip know there's a newer version exists? How do you roll back to an older version by simply giving pip the version number? Is there a way to let pip know where to look for different version of wheels on S3?

回答1:

AWS S3 can be used as a pypi server with minimal configuration and no other additional dependencies.

Suppose you want to use the bucket mypackages for hosting your private packages - awesomepy and rattlesnake.

After you have created the bucket in S3, go to its properties, click on Static website hosting card and enable the option Use this bucket to host a website. Note down the Endpoint mentioned above the radio button.

Below is how you have to arrange the folders and files:

mypackages
    |
    |---awesomepy
    |       |---awesomepy-1.10.0-py27-none-any.whl 
    |       |---awesomepy-1.10.3-py27-none-any.whl 
    |       |---index.html
    |
    |---rattlepy
            |---rattlesnake-0.1.1-py27-none-any.whl 
            |---index.html

The index.html files in the package folders will have content like below

<html>
<head>
    <title>Links</title>
</head>
<body>
    <h1>Links</h1>
    <a href='awesomepy-1.10.0-py27-none-any.whl'>awesomepy-1.10.0-py27-none-any.whl </a>
    <br />
    <a href='awesomepy-1.10.3-py27-none-any.whl'>awesomepy-1.10.3-py27-none-any.whl </a>
</body>
</html>

Wheel packages have to follow proper naming convetion. Refer to PEP 427 for details. For the dummy package - awesomepy-1.10.3-py27-none-any.whl, we are assuming,

Language implementation is Python 2.7 - py27
ABI tag is none - none
Platform is any - any

Finally you can install the packages using the below command. Ensure your packages have different names than that of pypi packages. Prefixing the package name with your company initials is one simple solution.

pip install awesomepy rattlesnake==0.1.1 --extra-index-url=<s3 Endpoint> --trusted-host=<s3 Endpoint without http>

Upgrading packages using pip install awesomepy --upgrade will also work with this approach.

Also, modify your bucket permissions to restrict the packages to intended clients. One way to do this is:
Select Permissions tab in the bucket --> Edit Bucket Policy

回答2:

Shameless plug: I had the exact same problem, and wasn't happy with any of the existing solutions, so I wrote a small Lmabda/Terraform combo that does the trick: https://github.com/berislavlopac/plambdapi

Hope it helps!

来源：https://stackoverflow.com/questions/36022867/if-we-want-use-s3-to-host-python-packages-how-can-we-tell-pip-where-to-find-the

标签

python

amazon-s3

pip