Designing a web crawler

Question

I have come across an interview question \"If you were designing a web crawler, how would you avoid getting into infinite loops? \" and I am trying to answer it.

How

Answer 1

You'd have to have some sort of hash table to store the results in, you'd just have to check it before each page load.