Firebase Android: slow “join” using many listeners, seems to contradict documentation

Question

Implementing an Android+Firebase app, which has a many-to-many relationship: User <-> Widget (Widgets can be shared to multiple users).

Considerations:

1. List all the Widgets that a User has.
2. A User can only see the Widgets which are shared to him/her.
3. Be able to see all Users to whom a given Widget is shared.
4. A single Widget can be owned/administered by multiple Users with equal rights (modify Widget and change to whom it is shared). Similar to how Google Drive does sharing to specific users.

One of the approaches to implement fetching (join-style), would be to go with this advice: https://www.firebase.com/docs/android/guide/structuring-data.html ("Joining Flattened Data"):

// fetch a list of Mary's groups
ref.child("users/mchen/groups").addChildEventListener(new ChildEventListener() {
  @Override
  public void onChildAdded(DataSnapshot snapshot, String previousChildKey) {
    // for each group, fetch the name and print it
    String groupKey = snapshot.getKey();
    ref.child("groups/" + groupKey + "/name").addListenerForSingleValueEvent(new ValueEventListener() {
      @Override
      public void onDataChange(DataSnapshot snapshot) {
        System.out.println("Mary is a member of this group: " + snapshot.getValue());
      }
      @Override
      public void onCancelled(FirebaseError firebaseError) {
        // ignore
      }
    });
  }
});

This prompts the question whether having potentially many listeners will have a negative impact on performance or perhaps would hit some hard limit.

But we get reassured in the doc:

Is it really okay to look up each record individually? Yes. The Firebase protocol uses web sockets, and the client libraries do a great deal of internal optimization of incoming and outgoing requests. Until we get into tens of thousands of records, this approach is perfectly reasonable. In fact, the time required to download the data (i.e. the byte count) eclipses any other concerns regarding connection overhead.

But, to be sure, I've made a little test app which compares 2-approaches:

1. Attaching many ValueEventListener-s to all widgets one-by-one (as per Firebase's "structuring-data" guide mentioned above)
2. Attaching a single ChildEventListener to a node which hosts all the widgets (requires adequate structuring of User's Widgets under one node)

Tested on 4 different devices and android versions (4.x - 5.x). Firebase lib: 'com.firebase:firebase-client-android:2.3.1'.
In the first approach the performance was rather disappointing. I consistently saw ~15-100 events/s. The lowest performance, ~15 events/s was coming up quite often, so looks like it should be taken seriously. In such case, if the User had 100 Widgets, it would take ~6seconds to get info about all Widgets (e.g. scrolling a list). This is too slow. With 1000 Widgets, it often took as much as 40 seconds to fetch their info by separate listeners. Way too slow.

In the second approach I observed 200-3000 events/s. Thus 15-30 times faster than the first approach!
So looks like the reassurance in Firebase doc [...] Until we get into tens of thousands of records, this approach is perfectly reasonable [...] is not really accurate given how slowly it worked.

Given all that, I have 4 inter-related questions.

1. Can anyone confirm/disprove my findings given their own experience/benchmarks? Info regarding other platforms also welcome, as there is a plan to develop this app on multiple platforms.
2. What could be the reason for such dramatic difference of performance? Internal data structures perhaps?
3. Is there any way to improve the performance while still keeping the first (multi-listener) approach?
4. Should the multi-listener approach be ditched completely in favor of a denormalized multi-copy approach which was presented in Firebase+Udacity tutorial (https://github.com/udacity/ShoppingListPlusPlus in the "userLists" node - where they keep per-user copies of shopping list info) ? I ask about implications of this approach, in another question - Firebase: structuring data via per-user copies? Risk of data corruption? .

Any other hints/considerations welcome. TIA.

Answer 1

"This prompts the question whether having potentially many listeners will have a negative impact on performance or perhaps would hit some hard limit."

The answer is the same for Firebase Realtime database as it is for the new Cloud Firestore database.

I doesn't matter how many connections you have or how many listener you have, it matters how much data you are processing on the client side.

So if you have 100 listeners, listening to 100 bits of small data, that's gone a be pretty cheap but if each of those listeners is listening to a stream of constantly changing data, that's gone a get very expensive for the client to deal it very quickly.

And because mobile devices are very different is very difficult to know how many is too many. So if you targeting american users who tend to have very high-end phones, that's will be a different limit if we targeting countries with phones that are much lower powered.

So you can have as many listeners as you want, if you are removing them accordingly to the life-cycle of the activity, as exaplined here.

There is also another approach, to use a addListenerForSingleValueEvent. In thise case, there is no need to remove a listener.