Postgres Write Ahead Log

[redlock]

Hello,
Was just wondering whether using WAL can make realtime faster since you don't need to wait for disk writes.
Thanks.

[kiwicopple]

Hi @redlock we already use the WAL for realtime :). I might be misunderstanding your suggestion though - feel free to add some more details if you think we can improve - we're always looking for better performance.

[kiwicopple]

This gives a rough idea of the internals of Postgres, but roughly: the incoming changes are written to the WAL (hence the name), then persisted to the database and sent to any subscribers.

We're still in the process of benchmarking Realtime, but I suspect it will be able to keep up with most workloads, and that we will be able to significantly improve any bottlenecks. If it doesn't look like it's going to be able to scale to enterprise, then we will go back to the drawing board.

Perhaps you're looking for a way just to broadcast messages, without too much concern on Data Consistency? This is useful for things like sending mouse positions in real time (like Figma does). In those cases you don't really care about saving the data to the database. It's an idea that I've been toying with, but it's a different use-case than the Realtime Server. Let me know what you're building - I'd be interested how Supabase could make it easier.

[redlock]

If you are reading from WAL and sending the data to subscribers then you aren't waiting for the database to persist. Isn't that right?

For our use case. It would be nice to have an option where persistence is optional or data consistency is not a hard requirement. For example in a real time game latest player position is more important than sending the whole sequence of positions and persisting them.
Thanks

[kiwicopple]

reading from WAL and sending the data to subscribers then you aren't waiting for the database to persist

All of this is handled by Postgres - honestly I couldn't tell you the exact sequence without digging into the internals of Postgres.

in a real time game latest player position

makes a lot of sense. Let me know if this is right: you'd want to be broadcasting changes to all participants (clients) frequently, and then occasionally persisting to the database (server), where the server also sends back a verified position (to avoid clients from cheating - "never trust the client").

These scenarios are typically handled well by game engines, though I think it sometimes involves some heavy engineering to make sure it works seamlessly. That being said, I can see some potential opportunities for Supabase here. In my mind, we could do something like this with WebRTC or an Elixir cluster, but I think it would be a large project. Did you already try Supabase with your game? Are you experiencing any bottlenecks?

[redlock]

We haven't used Supabase yet we are evaluating it as a future candidate to replace certain parts of our stack. In particular, fast realtime data sync is more important to us than persistence.

The other important part is client SDKs for different platforms and how easy/difficult is it to build a client to connect to supabase. I have seen this issue bite many SAAS projects (Nakama, Deepstream, etc..) they get overwhelmed with maintaining a multitude of SDK/clients for their platforms and end up unable to keep up the clients with changes to their server.
(Example Nakama Swift SDK hasn't been updated for 2+ years and isn't compatible with their V2 server)

Being able to quickly write a client for some platform if you guys can't support it is a very big advantage and will make it easier to be adopted across different use cases and client bases.

[kiwicopple]

The other important part is client SDKs

The good news is that we try to support existing OSS tools wherever possible. This means that each tool usually has it's own community of client libraries. For example PostgREST has a number of client libraries. We're internally focused on JS libraries, and our community has built them for python, dart, swift, and rust. Even if Supabase went away though, there are other non-Supabase libraries you could use with most of the tools.

fast realtime data sync is more important to us than persistence.

Quite honestly, we may not be the best option for you - we might get there one day, but I think you're going to need global clusters to reduce latency and probably some sort of confict resolution. These are things we'll work on eventually, but they aren't trivial. I'm happy to jump on a call at any time to discuss your architecture, seems interesting.

[kiwicopple]

I was thinking about this more, @thorwebdev interested to get your view on the DX of this.

For some cases like @redlock's we need some shared ephemeral state - sort of like roomservice.dev (although they are also handling conflict resolution). There are more and more use-cases of this because collaboration is becoming so important. We could do this quite easily with the Realtime server because it has Presence and Websockets built in, as long as we avoid Conflict Resolution use-cases for now.

Here is how it might look

/**
 * Example of how to do ephemeral data
 */

 // A client could be any browser tab or mobile etc.
 // It's up to the user to specify some identifying data (supabase-js will create a UUID for every client)
type Client = {
    name: string,
    email?: string
}

// Here I set up some identifying data for this browser tab
let currentClient: Client = {
    name: "Paul",
}

// Here we connect to a room "my-room-name" with my info. 
// A room name can be any string - probably a URL in most cases
const { room, error } = await supabase
    .presence("any-room-name", {
        password: "p@ssword",
        client: currentClient
    })


// Here we broadcast my mouse position to everyone in the room
// The developer could call this as much as want, 
// although we would probably throttle it behind the scenes to send to the server every 50ms by default
room.broadcast({
    mouse: {
        x: 323,
        y: 631
    }
})

// Here we receive data from the users
room.receive((data: Set) => {
  // data is a Set, like this:
  /**
   {
       '04095645-6aa5-4e65-a7e3-2ddbfce480c6': {
           client: {
                name: 'Paul'
           },
           data: {
               mouse: {
                    x: 323,
                    y: 631
                }
           }
       },
       '0369b45d-8b15-4d42-b618-ae439dd8c2da': {
           client: {
                name: 'Thor'
           },
           data: {
               mouse: {
                    x: 323,
                    y: 631
                }
           }
       }
   }
   */
});

A couple of notes:

• No persistence: There is no persistence to the database, although we could add that later
• No conflict resolution: Because every client is unique, there doesn't need to be any conflict resolution. It is not great for things like text editors where we need to handle persistent collaboration (which need CRDTs or OT)
• No offline: It isnt good for offline use-cases (edit a document offline, come online, sync the changes). Hence the naming convention "presence"
• We could tie it in with supabase.auth, but also nice if we keep them decoupled
• This would be helpful for some of the things we need in the Dashboard, so good dogfooding opportunity