Azure Compute Service worker becomes “busy” following scale-up

懵懂的女人 提交于 2019-12-06 06:03:41

During a scale operation Azure will send a RoleEnvironmentTopologyChange via the Changing event to all existing instances. This lets those instances discover the new role instance in order to allow communication between the instances. Note that this only happens if you have an internal endpoint defined (if you turn on RDP then you implicitly get an internal endpoint).

By default these topology changes won't affect running instances. However, if you subscribe to the Changing event and you set e.Cancel=True then the role instance will recycle and run your startup tasks again.

For more information on the topology change see http://azure.microsoft.com/blog/2011/01/04/responding-to-role-topology-changes/.

So there are two issues here:

  1. Why is your role not able to recover from a recycle? This is a significant issue and one you must fix in order to have a reliable service. You can start with the troubleshooting workflows at http://blogs.msdn.com/b/kwill/archive/2013/08/09/windows-azure-paas-compute-diagnostics-data.aspx, and in particular Scenario 3 at http://blogs.msdn.com/b/kwill/archive/2013/09/06/troubleshooting-scenario-3-role-stuck-in-busy.aspx.
  2. Why are you recycling your role instances in response to a topology change? Check your Changing event handler and make sure you aren't setting e.Cancel=true.

This is too long for a comment, just adding to what kwill has already told:

My ASP.NET Web Role didn't have e.Cancel = true anywhere but still got restarted (or rather: recycled, the environment being completely re-initialized even before OnStart() was called for 10 minutes, just like after a fresh deployment) after a scale-out. So I went ahead and added an event handler which is just supposed to set what's already a default:

public class WebRole : RoleEntryPoint
{
    public override bool OnStart()
    {
        RoleEnvironment.Changing += (sender, e) =>
        {
            if (e.Changes.Any(change => change is RoleEnvironmentTopologyChange))
            {
                e.Cancel = false;
            }
        };
    }
}

And this helped! The role still becomes busy, but just for a few seconds instead of 15-20 minutes. It seems that only the website in the role restarts (or maybe the whole IIS), but the role doesn't restart, neither is the whole environment reinitialized.

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!