Well you can always deploy hardware solutions as well to create Layer 4-7 firewall rules. You can create specific rules to look for the well known agents of bots crawling the web. However to stop newly created bots you need to know what agent they are using for the bot.