I have staging and production apps on Heroku.
For crawler, I set robots.txt file.
After that I got message from Google.
Dear Webmaster
What about serving /robots.txt dynamically using a controller action instead of having a static file? Depending on the environment you allow or disallow search engines to index your application.
/robots.txt