In the Windows Azure (Preview) Management Portal you can change the configuration options for web sites (see http://www.windowsazure.com/en-us/manage/services/web-sites/how-
The solution for my problem was selecting "Custom" instead of "SQL Azure" from the "SQL Azure / SQL Server / MySQL / Custom" selector for the Entity Framework connection string, even though the database does run on SQL Azure.
[Edit] From a popular comment by @matthew-steeples below:
I would add to this for anyone else having the same issue is that sometimes the config file will have
"instead of", and the Azure Websites needs those to be changed to"