Ceramics with InstantDB

One of the hardest parts of doing ceramics is figuring out how various glazes interact with one another when a piece gets fired in the kiln.

In the beginning, managing this in my Notes app and a spreadsheet worked well enough. But as I've added more glazes, and various combinations of layering glazes, it's gotten more and more difficult to organize the data.

So, I did what any self-respecting engineer would do in my situation — I spent an entire weekend building an app that probably saves me 2 minutes of busywork a week. Hooray? (You can check out the code here.)

Still, it ended up being a good learning experience. So I wanted to break down the interesting part: how I ended up choosing a graph-based database.

Picking the tech stack

Here's what I needed the app to do:

  • View and manage all my finished pieces
  • View and manage which glazes and glaze combinations were used for each piece
  • View all my glazes and glaze combinations, and see which pieces used these glazes and/or combinations
  • Be able to "tag" pieces and glazes to make it easier to search by clay body (white vs red vs speckled), glaze types (runny vs stable, matte vs glossy), and so on
  • Be able to "favorite" pieces and glazes

The goal was not to spend a lot of time coding; I just wanted to have something that works and is useful, as quickly as possible.

My first instinct in cases like this is to reach for Firebase. It's effectively free for small apps like this one, it's flexible, and I've used it often enough to feel confident that I can get something up and running in minutes.

But I've also used it enough to know its limitations... especially when it comes to modeling relational data. (This is where Supabase generally serves as a nice alternative.)

I also happen to know the founders of Instant, which bills itself as a graph-based Firebase. I thought this may be a good fit to try them out but I don't want to use something new just for the sake of spicing things up — especially if I can just solve my problem with a tool I'm already familiar with.

So, here's how I thought about it:

Using Firebase

To be clear, Firebase offers a full collection of tools, including authentication, storage, cloud functions, and much more. I'm just going to focus on its database, called "Firestore".

From their docs:

Firestore is a NoSQL, document-oriented database. Unlike a SQL database, there are no tables or rows. Instead, you store data in documents, which are organized into collections.

Basically, I can save stuff as JSON.

So looking back at my app requirements, I'm going to need collections for the following objects:

  • glazes
  • combinations (mixtures of layered glazes)
  • pieces (ceramic pieces, like mugs or bowls)
  • (...plus a few more that I'll ignore for now)

Now, how are these objects related to one another?

  • combinations represent a mix of 2 (and maybe even 3) glazes, where each glaze may be applied with 1-3 layers
  • pieces may have one or many glazes and/or combinations (e.g. the inside of a cup may use one glaze, while the outside may use a glaze combination)
  • glazes and combinations may be used in many pieces (e.g. when viewing a glaze or a combination, I'd like to see all the pieces it is used in)

That's already starting to look like a lot of "many-to-many" relationships between my data models. Pieces can use multiple glazes, and glazes have many pieces associated with them. Glaze combinations use multiple glazes, and glazes can be used in many combinations.

Why does this matter? Because the way relational data is typically organized in a NoSQL database like Firestore is through nesting.

For example, consider how an object in the pieces collection might look. Here is the data structure for a ceramic mug that was created with a "Linda" dipping glaze on the inside, and a "Opal/Van" glaze combination on the outside:


{
  id: 'piece_2',
  notes: '...',
  image: '...',
  parts: [
    {
      type: 'glaze',
      location: 'inner',
      glaze: {id: 'glaze_linda_1', name: 'Linda', variant: 'dipping'},
    },
    {
      type: 'combo',
      location: 'outer',
      combo: {
        id: 'opal_van',
        applications: [
          {
            layers: 1,
            glaze: {id: 'glaze_opal_2', name: 'Opal', variant: 'dipping'},
          },
          {
            layers: 3,
            glaze: {id: 'glaze_van_3', name: 'Van', variant: 'brushing'},
          },
        ],
      },
    },
  ],
};

This is the data I'd like to retrieve when viewing the details for an individual piece. If my app only needed to support two views — e.g. the list of all my pieces, plus the details view for each piece — this might work.

But I also want to be able to view my list of glazes, and for each glaze, see which finished pieces have used it. How would I do that?

There are no great options... I likely have to shift towards storing all of my data in top-level collections, referencing related objects by id. Then, perform multiple queries anytime I want to fetch an object and all its nested resources:

// assuming all objects are stored in their own top-level collections...
async function fetchPieceDetails(pieceId) {
  const piece = await db.collection('pieces').doc(pieceId);
  const parts = await db.collection('parts').where('pieceId', '==', pieceId);

  for (const part of parts) {
    if (part.type === 'glaze') {
      const glaze = await db.collection('glazes').doc(part.glazeId);
      // ...
    } else {
      const combo = await db.collection('combos').doc(part.comboId);
      const applications = await db
        .collection('applications')
        .where('comboId', '==', part.comboId);
      for (const application of applications) {
        const glaze = await db.collection('glazes').doc(application.glazeId);
        // ...
      }
      // ...
    }
  }
  // construct object from parts above
}

At this point, I'm basically just recreating a relational database (without any of the benefits of a relational database), and may as well make my life easier and just switch to SQL.

Using Supabase (i.e. PostgreSQL)

Alright, so things fell apart pretty quickly when considering Firebase. It would probably make more sense to use a relational database like Postgres.

Now, I could just set up my own instance of Postgres and manage it myself, but who has time for that? I just want to move fast and get this thing over with. Thankfully, Supabase gives me that option.

I've used Supabase a couple times, usually when it just doesn't make sense to use Firebase. Although some aspects of Supabase feel a bit clunky to me, they do deliver on a few important things:

  1. I get a managed Postgres database for free
  2. They make schema updates and database migrations super easy
  3. I get a nice dashboard that allows me to view, create, edit, and delete all my data

Cool. So now that I can model my data in SQL, what might that look like?

create table `glazes` (
  `id` integer primary key,
  --> etc
);

--> represents a mixture of glazes commonly used on ceramic pieces
create table `combinations` (
  `id` integer primary key,
  --> etc
);

--> represents an application of a glaze (glaze + # of layers) within a combination
create table `applications` (
  `id` integer primary key,
  `layers` integer not null,
  --> which combination is this a part of
  foreign key (`combo_id`) references `combinations`(`id`),
  --> which glaze is used in this application
  foreign key (`glaze_id`) references `glazes`(`id`),
);

create table `pieces` (
  `id` integer primary key,
  --> etc
);

--> represents the glazes and/or combos used to glaze a piece
create table `parts` (
  `id` integer primary key,
  `type` text not null check (`type` in ('combo', 'glaze')),
  foreign key (`piece_id`) references `pieces`(`id`),
  foreign key (`combo_id`) references `combinations`(`id`),
  foreign key (`glaze_id`) references `glazes`(`id`),
);

With this schema, I'm using a bunch of foreign keys to manage relationships between objects. Now, I can more easily perform more complex queries like "show me all the pieces that use glaze glaze_id in any way", by joining across pieces -> parts -> glaze_id or piece -> parts -> combo_id -> applications -> glaze_id.

For example, to do the above in SQL might look something like:

SELECT DISTINCT p.id AS piece_id, p.notes, p.image,
  JSON_AGG(
    JSON_BUILD_OBJECT(
      'type', pp.type,
      'combo_id', pp.combo_id,
      'glaze_id', pp.glaze_id,
      'combo', JSON_BUILD_OBJECT(
        'id', c.id,
        'applications', (
          SELECT JSON_AGG(
            JSON_BUILD_OBJECT(
              'layers', a.layers,
              'glaze', JSON_BUILD_OBJECT(
                'id', a.glaze_id,
                'name', g.name
              )
            )
          )
          FROM applications a
          JOIN glazes g ON a.glaze_id = g.id
          WHERE a.combo_id = c.id
        )
      )
    )
  ) AS parts
FROM pieces p
JOIN parts pp ON p.id = pp.piece_id
JOIN combinations c ON pp.combo_id = c.id
JOIN applications a ON c.id = a.combo_id
JOIN glazes g ON a.glaze_id = g.id
WHERE a.glaze_id = $glazeId
GROUP BY p.id;

This isn't too bad, but SQL queries can get a bit hairy as they grow in complexity. And I'd prefer to avoid writing raw SQL if I can avoid it.

Luckily, Supabase also ships with a convenient GraphQL extension. This means we can write something like this instead:

query GetPiecesByGlazeId($glazeId: Int!) {
  pieces(
    filter: {
      parts: {
        some: {
          type: { equalTo: "combo" },
          combo: {
            applications: {
              some: { glazeId: { equalTo: $glazeId } }
            }
          }
        }
      }
    }
  ) {
    nodes {
      id
      parts {
        nodes {
          type
          combo {
            applications {
              nodes {
                glaze {
                  id
                  name
                }
              }
            }
          }
        }
      }
    }
  }
}

This is already starting to get a little easier. But personally, I'm not the biggest fan of GraphQL. It feels a bit clunky to me, and requires a bit of extra setup which I don't want to spend the time on.

Could it be better?

Even though Postgres solves most of my data modeling issues, it still requires a fair bit of work to get up and running.

Supabase also lacks a few of the nice things that Firebase offers out of the box, such as optimistic updates, offline mode, and (to some extent) realtime updates. Postgres may beat Firebase on the backend, but Firebase wins on the frontend.

Why does this matter? For three reasons.

First, Firebase gives me optimistic updates. Optimistic updates makes my app feel instant. All updates are immediately applied on the client and reconciled with the server. This makes my app feel smooth and saves me a bunch of the boilerplate. To achieve the same effect with Supabase I need to manage pending client-side state changes and handle reversions in the case of failed network requests.

Second, Firebase has realtime subscriptions to queries which means that I never have to worry about "refetching" my data, since the query response data can be relied upon as the source of truth.

And third, Firebase comes with offline support. I'm building a simple mobile app for myself — it sucks to have it stop working just because I'm not connected to the internet. And while offline mode may just be a “nice-to-have” on web, it's table stakes for mobile.

Using Instant

In contrast to Firebase's NoSQL and Supabase's PostgresQL databases, Instant is built on top of a graph database. This means that in addition to the speed and flexibility that Firebase offers, I also get the support for relational data that Supabase provides.

In practice, Instant's querying language (InstaQL) looks a lot like GraphQL. That is, the structure of the query itself represents the structure of the data it will return.

Taking the example above, of querying all ceramic pieces that uses glaze glazeId, it would look something like this:

const {data, isLoading, error} = db.useQuery({
  pieces: {
    parts: {
      glazes: {},
      combos: {
        applications: {glazes: {}},
      },
    },
    $: {
      where: {
        or: [
          {'parts.glazes.id': glazeId},
          {'parts.combos.applications.glazes.id': glazeId},
        ],
      },
    },
  },
});

console.log(data)
=> {
  pieces: [
    {
      parts: [
        {
          glazes: [...], 
          combos: [
              {
                applications: [
                  {glazes: [...]}
                ]
              }
            ]
          ]
        }
      ]
    }
  ]
}

That was... a lot easier.

And even though it may look a bit like GraphQL, it's actually much simpler: there's no configuration or build step needed to get up and running — it's just plain old objects and arrays.

But how does it work? How do I create data in my app, such that it can be queried like this? This is where things took a little getting used to.

Instant handles database writes with "transactions". Transactions support operations like update, delete, link, and unlink. As you might have guessed, the link operation is how the "graph" is created.

Let's see how this would look if I wanted to create a new ceramic piece in my app, such that the query above would return it. For the sake of simplicity, I'm going to assume that the piece I'm adding uses the same glaze combination for both the inside and outside:

import {init, id, tx} from '@instantdb/react-native';

const db = init({appId: "..."});

type Piece = {id: string; notes: string; ...}
type Glaze = {id: string; name: string; ...}
// GlazeApplications represent how many layers of a specific glaze 
// are applied to a GlazeCombo
type GlazeApplication = {id: string; layers: number; glazeId: string; ...}
// GlazeCombos generally consist of 2-3 GlazeApplications
type GlazeCombo = {id: string; ...}

async function createNewGlazeCombo(
  data: Partial<GlazeCombo>, 
  applications: GlazeApplication[]
) {
  const combo = {id: id(), ...data};
  const txns = applications.map(application => {
    const applicationId = id();
    const {glazeId, ...params} = application;
    // this transaction will create the glaze application object, and then link
    // it to both the combo object AND the glaze object in both directions.
    // this allows us to query a glaze and all its combos, as well as a combo
    // and all its glazes, with no additional work.
    return tx.applications[applicationId]
      .update(params)
      .link({combos: combo.id})
      .link({glazes: glazeId})
  });
  // send the transactions to the db (with optimistic update!)
  return db.transact([
    tx.combos[combo.id].update(combo), 
    ...txns
  ]);
}

async function createNewPiece(data: Partial<Piece>, combo: GlazeCombo) {
  const piece = {id: id(), ...data};
  const part = {id: id(), type: "combo", location: "entire_piece"} 
  // send the transactions to the db (with optimistic update!)
  // note that calling update/link on transactions themselves do nothing
  // until they are passed into `db.transact`
  return db.transact([
    tx.pieces[piece.id].update(piece).link({parts: part.id}), 
    tx.parts[part.id].update(part).link({combos: combo.id}),
  ]);
}

What's going on here?

There are a few things that stand out to me as a bit unconventional:

  • Generating unique ids for my data on the client, with the id() helper function
  • Using the tx proxy object to create "transaction chunks"
  • Using link to connect relational data

First, let's look at link. This allows me to create associations between objects, and it works in both directions. Meaning, if I wanted to link a piece to a glaze, I could do either:

db.transact(tx.pieces[piece.id].link({glazes: glaze.id}))

or,

db.transact(tx.glazes[glaze.id].link({pieces: piece.id}))

…and both would do the same thing.

Once I've linked objects, I can query them:

// query all glazes, with the nested pieces that use each glaze
db.useQuery({glazes: {pieces: {}})

and, I can use them in my query filters:

// query all pieces that use the glaze named "Opal"
db.useQuery({
  pieces: {
    $: {where: {'glazes.name': 'Opal'}
  }
})

Pretty cool!

Notice the id() helper. The reason I have to generate my own id in the client is because Instant lets me batch multiple transaction chunks into a single transaction — including transactions that link objects that haven't been created yet — and run these optimistically.

For example, in my createNewPiece function above, I have these transaction chunks:

const txns = [
  tx.pieces[piece.id].update(piece).link({parts: part.id}), 
  tx.parts[part.id].update(part).link({combos: combo.id}),
]

This code handles both creating and linking my new piece to its corresponding parts (in this case, only 1 part; but there could be 2-3). And once I call db.transact, all of this is executed as a batch to update my database.

What's especially cool about this is that the update happens immediately!

Because Instant supports offline mode and optimistic updates, the transaction is first processed in a local datastore before it's sent along to the server. (If the transaction fails on the server, it gets automatically reverted in the local store.)

To understand why this is so nice, here's how I might handle the simple operation of marking a piece as a "favorite" in both Instant and Supabase.

First, using Supabase:

function SupabaseExample({pieceId}) {
  const [piece, setPiece] = React.useState(null);

  React.useEffect(() => {
    const init = async () => {
      const {data} = await supabase.from('pieces').select().eq('id', pieceId);
      setPiece(data[0]);
    };
    init();
  }, []);

  const handleToggleFavorite = async () => {
    const favorite = !piece.favorite;
    // optimistic update
    setPiece({...piece, favorite});
    const {data, error} = await supabase
      .from('pieces')
      .update({favorite})
      .eq('id', pieceId);
    if (error) {
      // revert optimistic update on error
      setPiece({...piece, favorite: !favorite});
    }
  };

  return <PieceDetails piece={piece} onToggleFavorite={handleToggleFavorite} />;
}

Now, using Instant:

function InstantExample({pieceId}) {
  const {data} = db.useQuery({pieces: {$: {where: {id: pieceId}}}});
  const piece = data.pieces[0];

  const handleToggleFavorite = async () => {
    await db.transact(tx.pieces[pieceId].update({favorite: !piece.favorite}));
  };

  return <PieceDetails piece={piece} onToggleFavorite={handleToggleFavorite} />;
}

Because db.useQuery automatically detects changes, and db.transact handles optimistic updates for me, I get the same functionality with half the amount of code!

Conclusion

Though Instant is still in its early days, it happened to be perfect for building my app. It was fast, flexible, and allowed me to build exactly what I needed in a matter of days.

I'll start with what I liked:

  • Querying. The querying syntax feels very intuitive to me, and the ability to have a realtime subscription to a query is really nice.
  • Optimistic updates. I have to admit, optimistic updates were not something I've tended to worry or care much about in previous projects. But getting it "by default" makes for a wonderful developer experience (and much less React state management).
  • A local db. I think the "local database" Instant aims to provide hasn't quite reached its full potential, but I'm beginning to love this idea. I think it will dramatically simplify how product developers manage the "state" of their frontend apps.

However, here's what I'd love to see from Instant:

  • Feature parity with Firebase/Supabase. Instant is still missing some "table stakes" features, like advanced query filters and file storage. (For my app, I ended up using GCP for storage, and to implement "search by text" I just handled filtering in the frontend.)
  • Schema validations. Though Instant's flexibility is nice when I want to move fast, it would be nice to have the option to require certain fields, and perform some validations on the data before it's saved.
  • Improving has_many/belongs_to relationships. Right now, everything Instant returns in its query responses is a list. By default, everything behaves as if it's in a "many-to-many" relationship (although I can configure Instant to enforce "one-to-many" or "one-to-one" relations on the backend). Regardless, this can create some annoying boilerplate code, where I have to do things like post.authors[0] to get the author of a post, even though I know there should only be one.

That being said, I'm pretty excited to see how the product evolves.

To view the full code for this app, I've open sourced it here: https://github.com/reichert621/glazepal