Persistent multiprocess shared cache in Python with stdlib or minimal dependencies

Question

I just tried a Python shelve module as the persistent cache for data fetched from the external service. The complete example is here.

I was wondering what would the best

Answer 1

I'd say you'd want to use some existing caching library, dogpile.cache comes to mind, it has many features already, and you can easily plug in the backends you might need.

dogpile.cache documentation tells the following:

This “get-or-create” pattern is the entire key to the “Dogpile” system, which coordinates a single value creation operation among many concurrent get operations for a particular key, eliminating the issue of an expired value being redundantly re-generated by many workers simultaneously.