JavaScript is full of incredible features. Generators and generator functions are perfect examples. I’ve known about them for a while now, but until recently, I wasn’t sure how to make use of them in my programs.

After watching Anjana Vakil’s presentation I got a lot of ideas on how to make use of them!

In this blog post, I will show how to implement pagination using JavaScript generators and Firestore.

Generator Functions Overview

A generator function is a special function in JavaScript that can pause execution in the middle of the function body, yield a result, and wait until it is called again with the next() method. It will then pick up execution from after the latest yield statement.

This behavior is exactly what you need if you want to query your database and get results in batches.

Problem

While using Firestore, there is often a need to retrieve all documents from a specific collection. However, this is not always possible with a simple query, especially if there are too many documents in the collection.

// error! collection is to big!
admin.firestore().collection('some/path').get()

Although the Firestore SDK comes with handy methods such as .startWith() and .limit(), the use of JavaScript generators provides an elegant solution for pagination.

Solution Explanation

The first thing we need is a generator function. This function should accept two parameters: a Firebase query and a batch size – number of documents per ‘page’.

export async function* allSnapshotsGenerator(
  query,
  batchSize
) {}

Let’s begin constructing the function body by defining the base query that pulls the first batch of snapshots. In this example I will order documents by ID but you can choose any other field.

let baseQuery = query
   .limit(batchSize)
   .orderBy(FieldPath.documentId());

Each query must start where the previous one left off. For this to happen, we need to store the last snapshot from the previous batch.

let lastSnapshot;

This variable will be undefined during the first query because we don’t know what the first document in the collection is.

It will also be undefined after the last batch arrives. At this point, we will set this variable to null to indicate that the while loop has to exit.

I will use a do {} while () loop to express that the first iteration runs, even if lastSnapshot is undefined. It’s important to note that undefined !== null.

  do {
      // query next batch
  } while (lastSnapshop == null);

During each iteration, we will construct a query and set lastSnapshot for the next batch. The last batch will be empty, and lastSnapshot will be set to null, signaling the end of the loop.

const currentQuery = lastSnapshot
  ? baseQuery.startAfter(lastSnap)
  : baseQuery;
const snapshots = (await currentQuery.get()).docs;
lastSnap = snapshots[snapshots.length - 1] || null;

Now, the most important part – yielding the results! Let’s ensure that we have results in the first place

if (snapshots.length > 0) {
  yield snapshots;
}

Let’s examine the complete solution

export async function* allSnapshotsGenerator(
  query,
  batchSize,
) {
  let baseQuery = query
    .limit(batchSize)
    .orderBy(FieldPath.documentId());
  let lastSnapshot;

  do {
    const currentQuery = lastSnapshot
      ? baseQuery.startAfter(lastSnapshot)
      : baseQuery;
    const snapshots = (await currentQuery.get()).docs;
    lastSnapshot = snapshots[snapshots.length - 1] || null;

    if (snapshots.length > 0) {
      yield snapshots;
    }
  } while (lastSnapshot);

  return;
}

Calling the generator

There are two approaches to using our paginator-generator.

  1. If you need to retrieve all documents at once, you can use a for await loop.
const myQuery = admin.firestore().collection('some/path').get()

for await (const batch of allSnapshotsGenerator(
  myQuery,
  1000
)) {
  const docs = batch.map((s) => s.data());
  // do something with ...
}

2. If you want to control when the next batch is called, you need to create a generator and then call the next() method when needed.

const myQuery = admin.firestore().collection('some/path')
const myPaginator = allSnapshotsGenerator(
    myQuery,
    10
  )) 

const {value, done} = await myPaginator.next()

Summary

In summary, we’ve developed a robust solution for pagination using JavaScript generators and Firestore. The process involves constructing a generator function, initializing a base query for fetching snapshots, and utilizing a loop mechanism with careful handling of the lastSnapshot variable. This ensures the smooth transition between batches and efficient retrieval of documents. We’ve explored two approaches: using a for await loop for fetching all documents at once and creating a generator for more control over when the next batch is called. This comprehensive solution provides flexibility and efficiency in managing large collections of documents in Firestore.

Resources