Azure Cloud Service : Scaled in web site (web role) instance recieves requests before iis is ready

Question

Description

I got a web role hosted in a cloud service that is a web site. There are predictible load spikes that we handle with a custom monitoring (a worker role) that will periodically scale in or scale out depending on the predicted load profile. That part is working.

Issue

After a new instance of the web site is mounted by the monitoring, we just had a case of a user whose request reach that new instance before the new instance was fully load. It resulted in an unaccessible web site blank page on the web browser. Given our typical target client it is very bad. This happen when a tester connected (for the first time) to the web site approximately during the scale in process.

Do my question is, is there a way to ensure that a web role has fully loaded IIS7, started the apppools and finished all the web site warmup process before any request is directed to it? Or in a more general way, can we manualy decide wich instance is available to the load balancer ? by that I mean that I assume there is a built-in load balancer over a cloud service automaticaly distributing requests.

Thanks in advance.

Answer 1

See http://blogs.msdn.com/b/kwill/archive/2012/09/19/role-instance-restarts-due-to-os-upgrades.aspx, specifically the Common Issues #5 and associated code.

If your website takes several minutes to warmup (either standard IIS/ASP.NET warmup of precompilation and module loading, or warming up a cache or other app specific tasks) then your clients may experience an outage or random timeouts. After a role instance restarts and your OnStart code completes then your role instance will be put back in the load balancer rotation and will begin receiving incoming requests. If your website is still warming up then all of those incoming requests will queue up and time out. If you only have 2 instances of your web role then IN_0, which is still warming up, will be taking 100% of the incoming requests while IN_1 is being restarted for the Guest OS update. This can lead to a complete outage of your service until your website is finished warming up on both instances. It is recommended to keep your instance in OnStart, which will keep it in the Busy state where it won't receive incoming requests from the load balancer, until your warmup is complete. You can use the following code to accomplish this:

public class WebRole : RoleEntryPoint
{
    public override bool OnStart()
    {
        // For information on handling configuration changes
        // see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
        IPHostEntry ipEntry = Dns.GetHostEntry(Dns.GetHostName());
        string ip = null;
        foreach (IPAddress ipaddress in ipEntry.AddressList)
        {
            if (ipaddress.AddressFamily.ToString() == "InterNetwork")
            {
                ip = ipaddress.ToString();
            }
        }

        string urlToPing = "http://" + ip;
        HttpWebRequest req = HttpWebRequest.Create(urlToPing) as HttpWebRequest;
        WebResponse resp = req.GetResponse();
        return base.OnStart();
    }
}

Answer 2

Late to the party, but Cloud Services sits behind a load balancer. The load balancer has a probe feature that assess if an application instance can receive requests. You can configure the probe to make an http request to the application instance to check if the application is up or down, along with other settings.

-- Information about the load balancer

http://azure.microsoft.com/blog/2014/04/08/microsoft-azure-load-balancing-services/

-- Detail on the configuration in the Service Def

https://msdn.microsoft.com/en-us/library/azure/jj151530.aspx