Flutter & Firebase Get more than 10 Firebase Documents into a Stream<List<Map>>

Question

With Flutter and Firestore, I am trying to get more than 10 documents into a Stream<List>. I can do this with a .where clause on a collection mapping the QuerySnapshot. However, the 10 limit is a killer.

I'm using the provider package in my app. So, in building a stream in Flutter with a StreamProvider, I can return a

Stream<List<Map from the entire collection. too expensive. 200 plus docs on these collections and too many users. Need to get more efficient.
Stream<List<Map using a .where from a Collection that returns a Stream List 10 max on the list...doesn't cut the mustard.
Stream<Map from a Document, that returns 1 stream of 1 document.

I need something in between 1 and 2.

I have a Collection with up to 500 Documents, and the user will choose any possible combination of those 500 to view. The user assembles class rosters to view their lists of users.

So I'm looking for a way to get individual streams of, say 30 documents, and then compile them into a List: But I need this List<Stream<Map to be a Stream itself so each individual doc is live, and I can also filter and sort this list of Streams. I'm using the Provider Package, and if possible would like to stay consistent with that. Here's where I am currently stuck:

So, my current effort:

  Future<Stream<List<AttendeeData>>> getStreams() async {
    List<Stream<AttendeeData>> getStreamsOutput = [];
    for (var i = 0; i < teacherRosterList.length; i++) {
      Stream thisStream = await returnTeacherRosterListStream(facility, teacherRosterList[i]);
      getStreamsOutput.add(thisStream);
    }
    return StreamZip(getStreamsOutput).asBroadcastStream();
  }

Feels like I'm cheating below: I get an await error if I put the snapshot directly in Stream thisStream above as Stream is not a future if I await, and if I don't await, it moves too fast and gets a null error.

  Future<Stream<AttendeeData>> returnTeacherRosterListStream(String thisFacility, String thisID) async {
    return facilityList.doc(thisFacility).collection('attendance').doc(thisID).snapshots().map(_teacherRosterListFromSnapshot);
    }
  }

Example of how I'm mapping in _teacherRosterListFromSnapshot (not having any problem here):

  AttendeeData _teacherRosterListFromSnapshot(DocumentSnapshot doc) {
    // return snapshot.docs.map((doc) {

    return AttendeeData(
      id: doc.data()['id'] ?? '',
      authorCreatedUID: doc.data()['authorCreatedUID'] ?? '',
   );
  }

My StreamProvider Logic and the error:

return MultiProvider(
            providers: [
              StreamProvider<List<AttendeeData>>.value(
                  value: DatabaseService(
                teacherRosterList: programList,
                facility: user.claimsFacility,
              ).getStreams()),
            ]

Error: The argument type 'Future<Stream<List>>' can't be assigned to the parameter type 'Stream<List>'.

AttendeeData is my Map Class name.

So, the summary of questions:

Can I even do this? I'm basically Streaming a List of Streams of Maps....is this a thing?
If I can, how do I do it? a. I can't get this into the StreamProvider because getStreams is a Future...how can I overcome this?

I can get the data in using another method from StreamProvider, but it's not behaving like a Stream and the state isn't updating. i'm hoping to just get this into Provider, as I'm comfortable there, and I can manage state very easily that way. However, beggars can't be choosers.

jbryanh jbryanh · Accepted Answer · 2021-02-15T20:40:58

Solved this myself, and since there is a dearth of good start to finish answers, I submit my example for the poor souls who come after me trying to learn these things on their own. I'm a beginner, so this was a slog:

Objective: You have any number of docs in a collection and you want to submit a list of any number of docs by their doc number and return a single stream of a list of those mapped documents. You want more than 10 (firestore limit on .where query), less than all the docs...so somewhere between a QuerySnapshot and a DocumentSnapshot.

Solution: We're going to get a list of QuerySnapshots, we're going to combine them and map them and spit them out as a single stream. So we're getting 10each in chunks (the max) and then some odd number left over. I plug mine into a Provider so I can get it whenever and wherever I want.

So from my provider I call this as the Stream value:

  Stream<List<AttendeeData>> filteredRosterList() {
    var chunks = [];
    for (var i = 0; i < teacherRosterList.length; i += 10) {
      chunks.add(teacherRosterList.sublist(i, i + 10 > teacherRosterList.length ? teacherRosterList.length : i + 10));
    } //break a list of whatever size into chunks of 10.

    List<Stream<QuerySnapshot>> combineList = [];
    for (var i = 0; i < chunks.length; i++) {
   combineList.add(*[point to your collection]*.where('id', whereIn: chunks[i]).snapshots());
    } //get a list of the streams, which will have 10 each.

    CombineLatestStream<QuerySnapshot, List<QuerySnapshot>> mergedQuerySnapshot = CombineLatestStream.list(combineList);
    //now we combine all the streams....but it'll be a list of QuerySnapshots.

    //and you'll want to look closely at the map, as it iterates, consolidates and returns as a single stream of List<AttendeeData>
    return mergedQuerySnapshot.map(rosterListFromTeacherListDocumentSnapshot);
  }

Here's a look at how I mapped it for your reference (took out all the fields for brevity):

  List<AttendeeData> rosterListFromTeacherListDocumentSnapshot(List<QuerySnapshot> snapshot) {
    List<AttendeeData> listToReturn = [];
    snapshot.forEach((element) {
      listToReturn.addAll(element.docs.map((doc) {
        return AttendeeData(
          id: doc.data()['id'] ?? '',
          authorCreatedUID: doc.data()['authorCreatedUID'] ?? '',
        );
      }).toList());
    });
    return listToReturn;
  }

Flutter & Firebase Get more than 10 Firebase Documents into a Stream<List<Map>>

1 Answers