I need a script that can spider a website and return the list of all crawled pages in plain-text or similar format; which I will submit to search engines as sitemap. Can I use W
You can use this perl script to do the trick : http://code.google.com/p/perlsitemapgenerator/