Using Nutch to crawl a specified URL list

Question

I have one million URL list to fetch. I use this list as nutch seeds and use the basic crawl command of Nutch to fetch them. However, I find that Nutch auto

Answer 1

• Delete the crawl and urls directory (if created before)
• Create and Update the seed file ( where URLs are listed 1URL per row)
• Restart the crawling process

Command

nutch crawl urllist -dir crawl -depth 3 -topN 1000000

• urllist - Directory where seed file (url list) is present
• crawl - Directory name

Even if the problem persists, try to delete your nutch folder and restart the whole process.