Azure Service Fabric Actors - unhandled exceptions?

Question

Right now our ASF cluster is running:

• Web API project - stateless and public facing
• Actor project - mostly volatile, keeping data in memory, used by certain APIs

We are trying out Application Insights, and I can setup unhandled error tracking like their docs here have for our Web API project.

Issue is, I want this for our Actor project as well.

Is there a global place for catching unhandled errors within an Actor? I know it's new, and maybe that is why I can't find documentation on this.

Right now I'm doing this inside every actor method, but doesn't seem like a great solution:

public async Task DoStuff()
{
    try
    {
        //Do all my stuff
    }
    catch (Exception exc)
    {
        //Send to Windows Event Source
        ActorEventSource.Current.ActorMessage(this, "Unhandled error in {0}: {1}", nameof(DoStuff), exc);

        //Send to Application Insights
        new TelemetryClient().TrackException(exc);

        throw exc;
    }
}

Answer 1

You have a few options:

• Actors do have a built-in ETW provider (Microsoft-ServiceFabric-Actors) that has an ActorMethodThrewException event. You can either:

  • Use an external process to collect ETW events and forward them to Application Insights (e.g. using SLAB or Azure Diagnostics)
  • Use the EventListener class to listen to the events in-process and forward it to App Insights (slightly less reliable, but simpler)

• Use a custom ActorServiceRemotingDispatcher, which is the class responsible for dispatching operations to the actors

  class CustomActorServiceRemotingDispatcher : ActorServiceRemotingDispatcher
  {
      public CustomActorServiceRemotingDispatcher(ActorService actorService) : base(actorService)
      {
      }
  
      public override async Task<byte[]> RequestResponseAsync(IServiceRemotingRequestContext requestContext, ServiceRemotingMessageHeaders messageHeaders,
          byte[] requestBodyBytes)
      {
              try
              {
                  LogServiceMethodStart(...);
  
                  result = await base.RequestResponseAsync(requestContext, messageHeaders, requestBodyBytes).ConfigureAwait(false);
  
                  LogServiceMethodStop(...);
  
                  return result;
              }
              catch (Exception exception)
              {
                  LogServiceMethodException(...);
  
                  throw;
              }
      }
  }
  

  To use this class, you'll need to create a custom ActorService class and override the CreateServiceReplicaListeners method. Note this will override any ActorRemotingProviderAttributes you may be using.

  Side notes:

  • You can also use this method to read your own headers (you'll also need a client-side custom IServiceRemotingClientFactory to add them)
  • The same technique can be applied to Reliable Services (using the ServiceRemotingDispatcher class)

Answer 2

No, there is no global place to get exceptions thrown from an actor in the actor framework today. The framework itself does catch exception that are thrown from methods that are managed by the actor runtime - these are your actor interface methods (the ones that are callable from ActorProxy), timer callbacks, reminder callbacks, and base actor overrides like OnActivateAsync - but they're not exposed through the actor API.