Using wget to recursively fetch a directory with arbitrary files in it

Question

I have a web directory where I store some config files. I\'d like to use wget to pull those files down and maintain their current structure. For instance, the remote directo

Answer 1

Recursive wget ignoring robots (for websites)

wget -e robots=off -r -np --page-requisites --convert-links 'http://example.com/folder/'

-e robots=off causes it to ignore robots.txt for that domain

-r makes it recursive

-np = no parents, so it doesn't follow links up to the parent folder