Catching events prior to subscription with FromEventPattern

Question

I'm writing a listener for messages using the Rx framework.

The problem I'm facing is that the library I'm using uses a consumer that publishes events whenever a message has arrived.

I've managed to consume the incoming messages via Observable.FromEventPattern but I have a problem with the messages that are already in the server.

At the moment I have the following chain of commands

1. Create a consumer
2. Create an observable sequence with FromEventPattern and apply needed transformations
3. Tell the consumer to start
4. Subscribe to the sequence

The easiest solution would be to swap steps 3. and 4. but since they happen in different components of the system, it's very hard for me to do so.

Ideally I would like to execute step 3 when step 4 happens (like a OnSubscribe method).

Thanks for your help :)

PS: to add more details, the events are coming from a RabbitMQ queue and I am using the EventingBasicConsumer class found in the RabbitMQ.Client package.

Here you can find the library I am working on. Specifically, this is the class/method giving me problems.

Edit

Here is a stripped version of the problematic code

void Main()
{
    var engine = new Engine();

    var messages = engine.Start();

    messages.Subscribe(m => m.Dump());

    Console.ReadLine();

    engine.Stop();
}

public class Engine
{
    IConnection _connection;
    IModel _channel;

    public IObservable<Message> Start()
    {
        var connectionFactory = new ConnectionFactory();

        _connection = connectionFactory.CreateConnection();
        _channel = _connection.CreateModel();

        EventingBasicConsumer consumer = new EventingBasicConsumer(_channel);

        var observable = Observable.FromEventPattern<BasicDeliverEventArgs>(
                                        a => consumer.Received += a, 
                                        a => consumer.Received -= a)
                                    .Select(e => e.EventArgs);

        _channel.BasicConsume("a_queue", false, consumer);

        return observable.Select(Transform);
    }

    private Message Transform(BasicDeliverEventArgs args) => new Message();

    public void Stop()
    {
        _channel.Dispose();
        _connection.Dispose();
    }
}

public class Message { }

The symptom I experience is that since I invoke BasicConsume before subscribing to the sequence, any message that is in the RabbitMQ queue is fetched but not passed down the pipeline.

Since I don't have "autoack" on, the messages are returned to the queue as soon as the program stops.

Answer 1

As some have noted in the comments, and as you note in the question, the issue is due to the way you're using the RabbitMQ client.

To get around some of these issues, what I actually did was create an ObservableConsumer class. This is an alternative to the EventingBasicConsumer which is in use currently. One reason I did this was to deal with the issue described in the question, but the other thing this does is allow you to re-use this consumer object beyond a single connection/channel instance. This has the benefit of allowing your downstream reactive code to remain wired in spite of transient connection/channel characteristics.

using System;
using System.Collections.Generic;
using System.Reactive.Concurrency;
using System.Reactive.Linq;
using System.Reactive.Subjects;
using RabbitMQ.Client;

namespace com.rabbitmq.consumers
{
    public sealed class ObservableConsumer : IBasicConsumer
    {
        private readonly List<string> _consumerTags = new List<string>();
        private readonly object _consumerTagsLock = new object();
        private readonly Subject<Message> _subject = new Subject<Message>();

        public ushort PrefetchCount { get; set; }
        public IEnumerable<string> ConsumerTags { get { return new List<string>(_consumerTags); } }

        /// <summary>
        /// Registers this consumer on the given queue. 
        /// </summary>
        /// <returns>The consumer tag assigned.</returns>
        public string ConsumeFrom(IModel channel, string queueName)
        {
            Model = channel;
            return Model.BasicConsume(queueName, false, this);
        }

        /// <summary>
        /// Contains an observable of the incoming messages where messages are processed on a thread pool thread.
        /// </summary>
        public IObservable<Message> IncomingMessages
        {
            get { return _subject.ObserveOn(Scheduler.ThreadPool); }
        }

        ///<summary>Retrieve the IModel instance this consumer is
        ///registered with.</summary>
        public IModel Model { get; private set; }

        ///<summary>Returns true while the consumer is registered and
        ///expecting deliveries from the broker.</summary>
        public bool IsRunning
        {
            get { return _consumerTags.Count > 0; }
        }

        /// <summary>
        /// Run after a consumer is cancelled.
        /// </summary>
        /// <param name="consumerTag"></param>
        private void OnConsumerCanceled(string consumerTag)
        {

        }

        /// <summary>
        /// Run after a consumer is added.
        /// </summary>
        /// <param name="consumerTag"></param>
        private void OnConsumerAdded(string consumerTag)
        {

        }

        public void HandleBasicConsumeOk(string consumerTag)
        {
            lock (_consumerTagsLock) {
                if (!_consumerTags.Contains(consumerTag))
                    _consumerTags.Add(consumerTag);
            }
        }

        public void HandleBasicCancelOk(string consumerTag)
        {
            lock (_consumerTagsLock) {
                if (_consumerTags.Contains(consumerTag)) {
                    _consumerTags.Remove(consumerTag);
                    OnConsumerCanceled(consumerTag);
                }
            }
        }

        public void HandleBasicCancel(string consumerTag)
        {
            lock (_consumerTagsLock) {
                if (_consumerTags.Contains(consumerTag)) {
                    _consumerTags.Remove(consumerTag);
                    OnConsumerCanceled(consumerTag);
                }
            }
        }

        public void HandleModelShutdown(IModel model, ShutdownEventArgs reason)
        {
            //Don't need to do anything.
        }

        public void HandleBasicDeliver(string consumerTag,
                                       ulong deliveryTag,
                                       bool redelivered,
                                       string exchange,
                                       string routingKey,
                                       IBasicProperties properties,
                                       byte[] body)
        {
            //Hack - prevents the broker from sending too many messages.
            //if (PrefetchCount > 0 && _unackedMessages.Count > PrefetchCount) {
            //    Model.BasicReject(deliveryTag, true);
            //    return;
            //}

            var message = new Message(properties.HeaderFromBasicProperties()) { Content = body };
            var deliveryData = new MessageDeliveryData()
            {
                ConsumerTag = consumerTag,
                DeliveryTag = deliveryTag,
                Redelivered = redelivered,
            };

            message.Tag = deliveryData;

            if (AckMode != AcknowledgeMode.AckWhenReceived) {
                message.Acknowledged += messageAcknowledged;
                message.Failed += messageFailed;
            }

            _subject.OnNext(message);
        }

        void messageFailed(Message message, Exception ex, bool requeue)
        {
            try {
                message.Acknowledged -= messageAcknowledged;
                message.Failed -= messageFailed;

                if (message.Tag is MessageDeliveryData) {
                    Model.BasicNack((message.Tag as MessageDeliveryData).DeliveryTag, false, requeue);
                }
            }
            catch {}
        }

        void messageAcknowledged(Message message)
        {
            try {
                message.Acknowledged -= messageAcknowledged;
                message.Failed -= messageFailed;

                if (message.Tag is MessageDeliveryData) {
                    var ackMultiple = AckMode == AcknowledgeMode.AckAfterAny;
                    Model.BasicAck((message.Tag as MessageDeliveryData).DeliveryTag, ackMultiple);
                }
            }
            catch {}
        }
    }
}

Answer 2

I think there is no need to actually subscribe to rabbit queue (via BasicConsume) until you have subscribers to your observable. Right now you are starting rabbit subscription right away and push items to observable even if no one has subscribed to it.

Suppose we have this sample class:

class Events {
    public event Action<string> MessageArrived;

    Timer _timer;
    public void Start()
    {
        Console.WriteLine("Timer starting");
        int i = 0;
        _timer = new Timer(_ => {
            this.MessageArrived?.Invoke(i.ToString());
            i++;
        }, null, TimeSpan.Zero, TimeSpan.FromSeconds(1));
    }

    public void Stop() {
        _timer?.Dispose();
        Console.WriteLine("Timer stopped");
    }
}

What you are doing now is basically:

var ev = new Events();
var ob = Observable.FromEvent<string>(x => ev.MessageArrived += x, x => ev.MessageArrived -= x);               
ev.Start();    
return ob;

What you need instead is observable which does exactly that, but only when someone subscribes:

return Observable.Create<string>(observer =>
{
    var ev = new Events();
    var ob = Observable.FromEvent<string>(x => ev.MessageArrived += x, x => ev.MessageArrived -= x);
    // first subsribe
    var sub = ob.Subscribe(observer);
    // then start
    ev.Start();
    // when subscription is disposed - unsubscribe from rabbit
    return new CompositeDisposable(sub, Disposable.Create(() => ev.Stop()));
}); 

Good, but now every subscription to observable will result in separate subscription to rabbit queues, which is not what we need. We can solve that with Publish().RefCount():

return Observable.Create<string>(observer => {
    var ev = new Events();
    var ob = Observable.FromEvent<string>(x => ev.MessageArrived += x, x => ev.MessageArrived -= x);
    var sub = ob.Subscribe(observer);                    
    ev.Start();                
    return new CompositeDisposable(sub, Disposable.Create(() => ev.Stop()));
}).Publish().RefCount(); 

Now what will happen is when first subscriber subscribes to observable (ref count goes from 0 to 1) - code from Observable.Create body is invoked and subscribes to rabbit queue. This subscription is then shared by all subsequent subscribers. When last unsubscribes (ref count goes to zero) - subscription is disposed, ev.Stop is called, and we unsubscribe from rabbit queue.

If so happens that you call Start() (which creates observable in your code) and never subscribe to it - nothing happens and no subscriptions to rabbit is made at all.