Apple release a new feature of the App Store. The ability for owner of iPhone with non compatible iOS version to download a previous (the last compatible version) of your Ap
Edit: this is no longer applicable with the new iTunes Connect UI, check out Sars' answer for an updated version
After looking for this page for a few weeks, I finally figured out how to access it:
1) First, go to the Manage Your Apps section on iTunes Connect.
2) Click the app that you want to edit.
3) On the right-hand side of the resulting page, click "Rights and Pricing":
4) Scroll down and click the terribly-named "Manage iCloud download settings for this app":
5) A list of all versions of your application will appear. You can check a specific version to prevent it from being downloaded due to "a significant usability issue":
Update for new UI of iTunes Connect (2014).
1) Go to "My Apps" section on iTunes Connect.
2) Click the app that you want to edit.
3) On the top menu, click "Pricing"
4) Click "Manage iCloud download settings for this app" at the bottom of window
It looks like the option has moved again.
I believe I have the answer, after looking in the iTunes Connect Developer guide. It says:
You can also indicate a legal issue with your app, including previous versions, from the footer of the “Rights and Pricing” page. Indicate a legal issue with your app by selecting each version of your app that has a legal issue. You cannot select a version of your app that is ready for sale. If the version of your app that is ready for sale has a legal issue, you must submit an update for the app and indicate the legal issue during submission.
Apparently, indicating a legal issue will prevent users from downloading previous versions.