How to password protect Scrapyd UI?

Question

I have my website available to public and there is Scrapyd running at port 6800 like http://website.com:6800/

I do not want anyone to see list of my crawlers. I know anyone can easily guess type up port 6800 and can see whats going on.

I have few questions, answer any of them will help me.

1. Is there way to password protect Scrapyd UI?
2. Can I password protect a specific Port on Linux? I know it can be done with IPTables to ONLY ALLOW PARTICULAR IPs but thats not a good solution
3. Should I make changes to Scrapyd's source-code?
4. Can I password protect a specific port only via .htaccess?

Answer 1

You should bind address of the machine that is going to make calls.

If its the localhost which is going to make calls to the endpoints just bind it to 127.0.0.1 and voila, the address doesn't work for external ips.

Answer 2

As of scrapyd version 1.2.0 the default bind address is 127.0.0.1

To add a password protection use this gist which uses nginx as a reverse proxy to add basic authentication to scrapyd.

You may also check scrapyd-authenticated repository.