Firestore Paginating data + Snapshot listener

Question

I am working with Firestore right now and have a little bit of a problem with pagination.
Basically, I have a collection (assume 10 items) where each item has some data and a timestamp.

Now, I am fetching the first 3 items like this:

Firestore.firestore()
    .collection("collectionPath")
    .order(by: "timestamp", descending: true)
    .limit(to: 3)
    .addSnapshotListener(snapshotListener())

Inside my snapshot listener, I save the last document from the snapshot, in order to use that as a starting point for my next page.

So, at some time I will request the next page of items like this:

Firestore.firestore()
    .collection("collectionPath")
    .order(by: "timestamp", descending: true)
    .start(afterDocument: lastDocument)
    .limit(to: 3)
    .addSnapshotListener(snapshotListener2()) // Note that this is a new snapshot listener, I don't know how I could reuse the first one

Now I have the items from index 0 to index 5 (in total 6) in my frontend. Neat!

If the document at index 4 now updates its timestamp to the newest timestamp of the whole collection, things start to go down.
Remember that the timestamp determines its position on account of the order clause!

What I expected to happen was, that after the changes are applied, I still show 6 items (and still ordered by their timestamps)

What happened was, that after the changes are applied, I have only 5 items remaining, since the item that got pushed out of the first snapshot is not added to the second snapshot automatically.

Am I missing something about Pagination with Firestore?

EDIT: As requested, I post some more code here:
This is my function to return a snapshot listener. Well, and the two methods I use to request the first page and then the second page I posted already above

private func snapshotListener() -> FIRQuerySnapshotBlock {
    let index = self.index
    return { querySnapshot, error in
        guard let snap = querySnapshot, error == nil else {
            log.error(error)
            return
        }

        // Save the last doc, so we can later use pagination to retrieve further chats
        if snap.count == self.limit {
            self.lastDoc = snap.documents.last
        } else {
            self.lastDoc = nil
        }

        let offset = index * self.limit

        snap.documentChanges.forEach() { diff in
            switch diff.type {
            case .added:
                log.debug("added chat at index: \(diff.newIndex), offset: \(offset)")
                self.tVHandler.dataManager.insert(item: Chat(dictionary: diff.document.data() as NSDictionary), at: IndexPath(row: Int(diff.newIndex) + offset, section: 0), in: nil)

            case .removed:
                log.debug("deleted chat at index: \(diff.oldIndex), offset: \(offset)")
                self.tVHandler.dataManager.remove(itemAt: IndexPath(row: Int(diff.oldIndex) + offset, section: 0), in: nil)

            case .modified:
                if diff.oldIndex == diff.newIndex {
                    log.debug("updated chat at index: \(diff.oldIndex), offset: \(offset)")
                    self.tVHandler.dataManager.update(item: Chat(dictionary: diff.document.data() as NSDictionary), at: IndexPath(row: Int(diff.oldIndex) + offset, section: 0), in: nil)
                } else {
                    log.debug("moved chat at index: \(diff.oldIndex), offset: \(offset) to index: \(diff.newIndex), offset: \(offset)")
                    self.tVHandler.dataManager.move(item: Chat(dictionary: diff.document.data() as NSDictionary), from: IndexPath(row: Int(diff.oldIndex) + offset, section: 0), to: IndexPath(row: Int(diff.newIndex) + offset, section: 0), in: nil)
                }
            }
        }
        self.tableView?.reloadData()
    }
}

So again, I am asking if I can have one snapshot listener that listens for changes in more than one page I requested from Firestore

Answer 1

Well, I contacted the guys over at Firebase Google Group for help, and they were able to tell me that my use case is not yet supported.
Thanks to Kato Richardson for attending to my problem!

For anyone interested in the details, see this thread

Answer 2

I came across the same use case today and I have successfully implemented a working solution in Objective C client. Below is the algorithm if anyone wants to apply in their program and I will really appreciate if google-cloud-firestore team can put my solution on their page.

Use Case: A feature to allow paginating a long list of recent chats along with the option to attach real time listeners to update the list to have chat with most recent message on top.

Solution: This can be made possible by using pagination logic like we do for other long lists and attaching real time listener with limit set to 1:

Step 1: On page load fetch the chats using pagination query as below:

- (void)viewDidLoad {
    [super viewDidLoad];
    // Do any additional setup after loading the view.
     [self fetchChats];
}

-(void)fetchChats {
    __weak typeof(self) weakSelf = self;
     FIRQuery *paginateChatsQuery = [[[self.db collectionWithPath:MAGConstCollectionNameChats]queryOrderedByField:MAGConstFieldNameTimestamp descending:YES]queryLimitedTo:MAGConstPageLimit];
    if(self.arrChats.count > 0){
        FIRDocumentSnapshot *lastChatDocument = self.arrChats.lastObject;
        paginateChatsQuery = [paginateChatsQuery queryStartingAfterDocument:lastChatDocument];
    }
    [paginateChatsQuery getDocumentsWithCompletion:^(FIRQuerySnapshot * _Nullable snapshot, NSError * _Nullable error) {
        if (snapshot == nil) {
            NSLog(@"Error fetching documents: %@", error);
            return;
        }
        ///2. Observe chat updates if not attached
        if(weakSelf.chatObserverState == ChatObserverStateNotAttached) {
            weakSelf.chatObserverState = ChatObserverStateAttaching;
            [weakSelf observeChats];
        }

        if(snapshot.documents.count < MAGConstPageLimit) {
            weakSelf.noMoreData = YES;
        }
        else {
            weakSelf.noMoreData = NO;
        }

        [weakSelf.arrChats addObjectsFromArray:snapshot.documents];
        [weakSelf.tblVuChatsList reloadData];
    }];
}

Step 2: On success callback of "fetchAlerts" method attach the observer for real time updates only once with limit set to 1.

-(void)observeChats {
    __weak typeof(self) weakSelf = self;
    self.chatsListener = [[[[self.db collectionWithPath:MAGConstCollectionNameChats]queryOrderedByField:MAGConstFieldNameTimestamp descending:YES]queryLimitedTo:1]addSnapshotListener:^(FIRQuerySnapshot * _Nullable snapshot, NSError * _Nullable error) {
        if (snapshot == nil) {
            NSLog(@"Error fetching documents: %@", error);
            return;
        }
        if(weakSelf.chatObserverState == ChatObserverStateAttaching) {
            weakSelf.chatObserverState = ChatObserverStateAttached;
        }

        for (FIRDocumentChange *diff in snapshot.documentChanges) {
            if (diff.type == FIRDocumentChangeTypeAdded) {
                ///New chat added
                NSLog(@"Added chat: %@", diff.document.data);
                FIRDocumentSnapshot *chatDoc = diff.document;
                [weakSelf handleChatUpdates:chatDoc];

            }
            else if (diff.type == FIRDocumentChangeTypeModified) {
                NSLog(@"Modified chat: %@", diff.document.data);
                FIRDocumentSnapshot *chatDoc = diff.document;
                [weakSelf handleChatUpdates:chatDoc];
            }
            else if (diff.type == FIRDocumentChangeTypeRemoved) {
                NSLog(@"Removed chat: %@", diff.document.data);
            }
        }
    }];

}

Step 3. On listener callback check for document changes and handle only FIRDocumentChangeTypeAdded and FIRDocumentChangeTypeModified events and ignore the FIRDocumentChangeTypeRemoved event. We are doing this by calling "handleChatUpdates" method for both FIRDocumentChangeTypeAdded and FIRDocumentChangeTypeModified event in which we are first trying to find the matching chat document from local list and if it exist we are removing it from the list and then we are adding the new document received from listener callback and adding it to the beginning of the list.

-(void)handleChatUpdates:(FIRDocumentSnapshot *)chatDoc {
    NSInteger chatIndex = [self getIndexOfMatchingChatDoc:chatDoc];
    if(chatIndex != NSNotFound) {
        ///Remove this object
        [self.arrChats removeObjectAtIndex:chatIndex];
    }
    ///Insert this chat object at the beginning of the array
     [self.arrChats insertObject:chatDoc atIndex:0];

    ///Refresh the tableview
    [self.tblVuChatsList reloadData];
}

-(NSInteger)getIndexOfMatchingChatDoc:(FIRDocumentSnapshot *)chatDoc {
    NSInteger chatIndex = 0;
    for (FIRDocumentSnapshot *chatDocument in self.arrChats) {
        if([chatDocument.documentID isEqualToString:chatDoc.documentID]) {
            return chatIndex;
        }
        chatIndex++;
    }
    return NSNotFound;
}

Step 4. Reload the tableview to see the changes.

Answer 3

my solution is to create 1 maintainer query - listener to observe on those removed item from first query, and we will update it every time there's new message coming.

Answer 4

To make pagination with snapshot listener first we have to create reference point document from the collection.After that we are listening to collection based on that reference point document.

Let's you have a collection called messages and timestamp called createdAt with each document in that collection.

//get messages
getMessages(){

//first we will fetch the very last/latest document.

//to hold listeners
listnerArray=[];

const very_last_document= await this.afs.collectons('messages')
    .ref
    .limit(1)
    .orderBy('createdAt','desc')
    .get({ source: 'server' });

 
 //if very_last.document.empty property become true,which means there is no messages 
  //present till now ,we can go with a query without having a limit

 //else we have to apply the limit

 if (!very_last_document.empty) {

    
    const start = very_last_document.docs[very_last_document.docs.length - 1].data().createdAt;
    //listner for new messages
   //all new message will be registered on this listener
    const listner_1 = this.afs.collectons('messages')
    .ref
    .orderBy('createdAt','desc')
    .endAt(start)     <== this will make sure the query will fetch up to 'start' point(including 'start' point document)
    .onSnapshot(messages => {

        for (const message of messages .docChanges()) {
          if (message .type === "added")
            //do the job...
          if (message.type === "modified")
            //do the job...
          if (message.type === "removed")
           //do the job ....
        }
      },
        err => {
          //on error
        })

    //old message will be registered on this listener
    const listner_2 = this.afs.collectons('messages')
    .ref
    .orderBy('createdAt','desc')
    .limit(20)
    .startAfter(start)   <== this will make sure the query will fetch after the 'start' point
    .onSnapshot(messages => {

        for (const message of messages .docChanges()) {
          if (message .type === "added")
            //do the job...
          if (message.type === "modified")
            //do the job...
          if (message.type === "removed")
           //do the job ....
        }
       this.listenerArray.push(listner_1, listner_2);
      },
        err => {
          //on error
        })
  } else {
    //no document found!
   //very_last_document.empty = true
    const listner_1 = this.afs.collectons('messages')
    .ref
    .orderBy('createdAt','desc')
    .onSnapshot(messages => {

        for (const message of messages .docChanges()) {
          if (message .type === "added")
            //do the job...
          if (message.type === "modified")
            //do the job...
          if (message.type === "removed")
           //do the job ....
        }
      },
        err => {
          //on error
        })
    this.listenerArray.push(listner_1);
  }

}


//to load more messages
LoadMoreMessage(){

//Assuming messages array holding the the message we have fetched


 //getting the last element from the array messages.
 //that will be the starting point of our next batch
 const endAt = this.messages[this.messages.length-1].createdAt

  const listner_2 = this.getService
  .collections('messages')
  .ref
  .limit(20)
  .orderBy('createdAt', "asc")    <== should be in 'asc' order
  .endBefore(endAt)    <== Getting the 20 documnents (the limit we have applied) from the point 'endAt';
.onSnapshot(messages => {

if (messages.empty && this.messages.length)
  this.messages[this.messages.length - 1].hasMore = false;

for (const message of messages.docChanges()) {
  if (message.type === "added") 
  //do the job...

  if (message.type === "modified")
    //do the job

  if (message.type === "removed")
    //do the job
}

},
 err => {
    //on error
 })

 this.listenerArray.push(listner_2)



}