How to detect fake users ( crawlers ) and cURL

Question

Some other website use cURL and fake http referer to copy my website content. Do we have any way to detect cURL or not real web browser ?

Answer 1

The way of avoid fake referers is tracking the user

You can track the user by one or more of this methods:

1. Save a cookie in the browser client with some special code (ex: last url visited, a timestamp) and verify it in each response of your server.

2. Same as before but using sessions instead of explicit cookies

For cookies you should add cryptographic security like.

[Cookie]
url => http://someurl/
hash => dsafdshfdslajfd

hash is calulated in PHP by this way

$url = $_COOKIE['url'];
$hash = $_COOKIE['hash'];
$secret = 'This is a fixed secret in the code of your application';

$isValidCookie = (hash('algo', $secret . $url) === $hash);

$isValidReferer = $isValidCookie & ($_SERVER['HTTP_REFERER'] === $url)