How to detect fake users ( crawlers ) and cURL

Question

Some other website use cURL and fake http referer to copy my website content. Do we have any way to detect cURL or not real web browser ?

Answer 1

As some have mentioned cURL cannot execute JavaScritp (to my knowledge) so you could possibly try setting someting up like raina77ow suggest but that would not wokrk for other grabbers/donwloaders.

I suggest you try building a bot trap that way you deal with the grabbers/downloaders that can execute JavaScript.

I don't know of any 1 solution to fully prevent this, so my best recommendation would be to try multiple solutions:

1) only allow known user agents such as all mainstream browsers in your .htaccess file

2) Set up your robots.txt to prevent bots

3) Set up a bot trap for bots that do not respect the robots.txt file