How Do I Fetch All Old Items on an RSS Feed?

Question

I\'ve been experimenting with writing my own RSS reader. I can handle the \"parse XML\" bit. The thing I\'m getting stuck on is \"How do I fetch older posts?\"

Mos

Answer 1

All previous answers more or less relied on existing services to still have a copy of that feed or the feed engine to be able to provide older items dynamically.

There's though another, admittedly pro-active and rather theoretical way to do so: Let your feedreader use a caching proxy which semantically understands RSS and/or Atom feeds and caches them on a per-item base up to as many items as you configure.

If the feedreader doesn't poll feeds regularily, the proxy could fetch known feeds time-based on its own to not miss an item in highly volatile feeds like the one from User Friendly which has only one item and changes every day (or at least used to do so). Hence if the feedreadere.g. crashed or lost network connection while you are away for a few days, you might loose items in your feedreader's cache. Having the proxy to fetch those feeds regularily (e.g. from a data center instead from at home or on a server instead of a laptop) allows you to easily run the feedreader only then and when without loosing items which were posted after your feedreader fetched feeds the last time but rotated out again before you fetch them the next time.

I call that concept a Semantic Feed Proxy and I've implemented a proof of concept implementation called sfp. It's though not much more than a proof of concept and I haven't developed it further. (So I'd be happy about hints to projects with similar ideas or purposes. :-)

Answer 2

Further to what David Dean said the RSS/Atom feeds will only contain what the publisher of the feed has up at that moment and someone would need to be actively collecting this informaton in order to have any historical information. Basically Google Reader was doing this for free and when you interacted with it you could retrieve this stored informaton from the google database servers.

Now that they have retired the service, to my knowledge you have two choices. You either have to start collection of this information from your feeds of interest and store the data using XML or some such, or you could pay for this data from one of the companies who sell this type of archived feed information.

I hope this information helps somebody.

Seán

Answer 3

As the other replies here mentioned, a feed may not provide archival data but historical items may be available from another source.

Archive.org’s Wayback Machine has an API to access historical content, including RSS feeds (if their bots have downloaded it). I’ve created the web tool Backfeed that uses this API to regenerate a feed containing concatenated historical items. If you'd like to discuss the implementation in detail please get in touch.

Answer 4

The RSS/Atom standards don't have ways to query older RSS articles.

I'm also working on a RSS reader and decided to build my own RSS archival service (https://app.pub.center). It's free to use the REST API. We charge money for push notifications.

The service daily polls it's catalog of RSS feeds, and caches the articles. Then, you can get these articles back in a chronological order. For example:

Page 1 of The Atlantic https://pub.center/feed/02702624d8a4c825dde21af94e9169773454e0c3/articles?limit=10&page=1

Page 2 of The Atlantic https://pub.center/feed/02702624d8a4c825dde21af94e9169773454e0c3/articles?limit=10&page=2

Answer 5

Another potential solution that might not have been available when the question was originally asked and shouldn't require any specific service.

1. Find the URL of the RSS feed you want and use waybackpack to get the archived urls for that feed.
2. Use FeedReader or a similar library to pull down the archived RSS feed.
3. Take the URLs from each feed and scrape them as you wish. If you're going way back in time it's possible there might be some dead links.

Answer 6

In my experience with RSS, the feed is compiled by the last X items where X is a variable. Certain Feeds may have the full list, but for bandwidth sake most places are likely limiting to just the last few items.

The likely answer for google reader having the old info, is that it is storing it on its side for users later.