Update WPF progress bar while filling DataSet, all using Rx

Question

I'm a bit new in Rx, so please excuse me if this seems silly or obvious...

I have an application which at a certain time, is meant to scan a selected folder and retrieve all files recursively, after which it needs to store them in a database. I would like to display a progress bar during that process while keeping the UI responsive of course. A cancel button would also be nice at a later stage.

I've implemented this using Rx, like so:

// Get a list of all the files
var enumeratedFiles = Directory.EnumerateFiles(targetDirectory, "*.*", SearchOption.AllDirectories);

// prepare the progress bar
            double value = 0;
            progBar.Minimum = 0;
            progBar.Maximum = enumeratedFiles.Count();
            progBar.Value = value;
            progBar.Height = 15;
            progBar.Width = 100;
            statusBar.Items.Add(progBar);

var files = enumeratedFiles.ToObservable()
                .SubscribeOn(TaskPoolScheduler.Default)
                .ObserveOnDispatcher()
                .Subscribe(x =>
                    {
                        myDataSet.myTable.AddTableRow(System.IO.Path.GetFileNameWithoutExtension(x));
                        value++;
                    },
                    () =>
                    {
                        myDataSetTableAdapter.Update(myDataSet.myTable);
                        myDataSetTableAdapter.Fill(myDataSet.myTable);
                        statusBar.Items.Remove(progBar);
                    });

However, with the above example the UI is locked and the progress bar doesn't update during the process. I assume that's because the AddTableRow method is blocking the thread, although I believed that the SubscribeOn(TaskPoolScheduler) was supposed to run the task on a new thread?

I've tried a few different approaches as well, with varying results. For example adding a .Do line:

var files = enumeratedFiles.ToObservable()
                .Do(x => myDataSet.myTable.AddTableRow(System.IO.Path.GetFileNameWithoutExtension(x)))
                .SubscribeOn(TaskPoolScheduler.Default)
                .ObserveOnDispatcher()
                .Subscribe(x =>
                    {
                        value++;
                    },
                    () =>
                    {
                        myDataSetTableAdapter.Update(myDataSet.myTable);
                        myDataSetTableAdapter.Fill(myDataSet.myTable);
                        statusBar.Items.Remove(progBar);
                        btnCancel.Visibility = Visibility.Collapsed;
                    });

this actually shows the progress bar updates, and the UI is not completely locked but it is choppy and the performance crawls down...

I've tried using a BackgroundWorker to do the same job, but the performance is way worse than the Rx approach above (for example, for 21000 files the Rx approach takes a few seconds while the BackgroundWorker a few minutes to finish).

I've also seen similar problems being tackled with Delegates for the progress bar's ValueProperty method, but I would really like to solve this using Rx if possible.

Am I missing something obvious here? Any suggestion would be highly appreciated...

Answer 1

The semantics of what you've written are pretty strange:

1. Do the entire file operation on the UI thread
2. Take those items, then spawn a new thread
3. On that thread, Dispatcher.BeginInvoke
4. In the invoke, add the table row

This doesn't seem like what you really want...

Answer 2

I've found the solution, and a bit more details on what's going on:

The delay while inserting the rows in the DataSet I mentioned is only there in Debug mode, whereas when run without Debug the application doesn't show that delay and the progress bar and number of items are processed many times faster. Silly me for not testing that earlier...

While scanning for the files recursively there is a slight delay (a few seconds for 21000 files) but since it only happens the first time you do that, I failed to notice it in my subsequent tests and I was only focusing on the part that seemed slow to me: the filling of the DataSet. I guess that Directory.EnumerateFiles caches everything in memory so any other attempt to read the same files completes instantly?

Also, it seems that the myDataSetTableAdapter.Fill(myDataSet.myTable) line is not needed, since the .Update method already saves the contents in the database itself.

The final code snippet that worked for me is the following:

progBar.Height = 15;
progBar.Width = 100;
progBar.IsIndeterminate = true;
statusBar.Items.Add(progBar);

var files = Directory.EnumerateFiles(targetDirectory, "*.*", SearchOption.AllDirectories)
            .Where(s => extensions.Contains(System.IO.Path.GetExtension(s))) // "extensions" has been specified further above in the code
            .ToObservable(TaskPoolScheduler.Default)
            .Do(x => myDataSet.myTable.AddTableRow(System.IO.Path.GetFileNameWithoutExtension(x), x, "default")) // my table has 3 columns
            .TakeLast(1)
            .Do(_ => myDataSetmyTableTableAdapter.Update(myDataSet.myTable))
            .ObserveOnDispatcher()
            .Subscribe(xy =>
                {
                    //progBar.Value++; //commented out since I've switched to a marquee progress bar
                },
                () =>
                {
                    statusBar.Items.Remove(progBar);
                    btnCancel.Visibility = Visibility.Collapsed;
                });

This seems to work fine for me, thanks for all the help guys!

edit: I've further expanded the above, to include a Cancel button functionality. If the user clicks on the Cancel button, the process is stopped immediately. I've tried to keep it as elegant as possible, so I've added an Observable from the Click event of the Cancel button, then used .TakeUntil in my existing files Observable above. The code now looks like this:

// Show the Cancel button to allow the user to abort the process
btnCancel.Visibility = Visibility.Visible;

// Set the Cancel click event as an observable so we can monitor it
var cancelClicked = Observable.FromEventPattern<EventArgs>(btnCancel, "Click");

// Use Rx to pick the scanned files from the IEnumerable collection, fill them in the DataSet and finally save the DataSet in the DB
var files = Directory.EnumerateFiles(targetDirectory, "*.*", SearchOption.AllDirectories)
            .Where(s => extensions.Contains(System.IO.Path.GetExtension(s)))
            .ToObservable(TaskPoolScheduler.Default)
            .TakeUntil(cancelClicked)
            .Do(x => ....

Answer 3

Reading your code it seems like you're doing a lot of work in the background and then finally updating the UI at the end. For this kind of work it might be better to define your enumeratedFiles variable as such:

var enumeratedFiles =
    Observable
        .Start(() =>
            Directory
                .EnumerateFiles(
                    targetDirectory, "*.*", SearchOption.AllDirectories),
                        Scheduler.TaskPool)
        .ObserveOnDispatcher();

You will get a background operation that is relatively quick followed by a single UI update. This is a better implementation of your current approach.

If you can figure out how to update the UI for each file returned then try this observable instead:

var enumeratedFiles =
    Directory
        .EnumerateFiles(targetDirectory, "*.*", SearchOption.AllDirectories)
        .ToObservable(Scheduler.TaskPool)
        .ObserveOnDispatcher();

With this option you'll definately need to figure out how to update the UI for each file found.

Let me know if either works for you.