How to generate sitemap on a highly dynamic website?

Question

Should a highly dynamic website that is constantly generating new pages use a sitemap? If so, how does a site like stackoverflow.com go about regenerating a sitemap? It seem

Answer 1

I'd like to share my solution here just in case it helps someone as well. It took me reading this question and many others to decide what to do.

My site structure.

Static pages

• Home (Highly dynamic. Cached for 30 mins)
• Artists, Albums, Songs, Playlists and Albums (Paginated List)
• Legal (Static page with Terms etc)

...etc

Dynamic Pages

• Artists, Albums, Songs, Playlists and Albums detail pages

---

My approach.

sitemap.xml: This url generates a with the first item being /sitemap-main.xml. The number of Artists, Albums, Songs etc are counted and divided by 1,000 (number of urls I want in each sitemap. the limit is 50,000). I round this number up.

So for e.g, 1900 songs = 1.9 = 2. I generate. add the urls /sitemap-songs-0.xml and /sitemap-songs-1.xml to the index. I repeat this for all other items. Basically, I am paginating.

The output is returned uncached. I want this to always be fresh.

---

sitemap-main.xml: This lists all the static pages. You can actually use a static file for this as you will only need to update it once in a while.

---

sitemap-songs-0.xml, sitemap-albums-0.xml, etc: I use a single route for this in SlimPhp 2.

$app->get('/sitemap-:type-:page.xml', function ($type, $page) use ($app) {...

I use a simple switch statement to generate the relevant files. If for this page, I got 1,000 items, the limit specified above, I cache the file for 2 Weeks. Else, I only cache it for a few hours.

I guess this can help anyone else implement their own system.