Initial support for Supabase Sync

[joelmnz]

In respose to #414

Added User based settings for Sync details

Added a manual "Sync" button to the "Export" screen

Low error handling, requires Supabase project database setup with the expected schema (see README.md in the src/modules/supabasesync/README.md folder for instructions.

Known Issues

• When importing history from json, the updated field is not altered, reset your "Last sync time" to ensure you export everything in Model Settings.
• Deleted chats are NOT deleted from the Sync database. if you reset your "Last Sync Time" all the deleted chats will get imported again.

Nice to Haves

• Manual sync is annoying already, need a quick way to know if we need to sync and if possible do in background
• Model settings (temperature) are not synced, not sure how to find which would be the latest, skipping for now.
• Folders are not synced, I'm not sure if we can due to the data structure, need to have FolderId on the Conversation entity then can sync

[vercel]

@joelmnz is attempting to deploy a commit to the Enrico Pro Team on Vercel.

A member of the Team first needs to authorize it.

[enricoros]

Hi @joelmnz this is an impressive amount of work. I can't review it right away as my current main has a huge pending change that touches everywhere (multi modal support that touches even the Conversation and Messages data structures, and unfortunately updates them), but I'll get to it.

Before that, a couple of questions:

• is any risk for existing users introduced by the change? E.g. data corruption, etc.
• does anyone that's already building/deploying it have to do extra steps (installing things, having API keys, etc), or in other words is this fully transparent to existing devs and processes
• do you plan to maintain this in the future?

[joelmnz]

Hi @enricoros ,

No hurry, I appreciate the opportunity to contribute to this project. It's the best Chat UI I've seen, and I'm keen to help.

Regarding your questions:

1. Risk for Existing Users:

   • There should be no impact on existing users. I haven't changed any data structures, and the module remains inactive until the settings are configured.

2. Extra Steps for Developers:

   • The only additional dependency is the Supabase lib. Other than that, no extra build steps are required. Instructions for setting up the necessary tables in Supabase are included in the README. Everything should be self-explanatory.

3. Future Maintenance:

   • Short answer: "Maybe."
   • Long answer: I really like this project. The only thing preventing me from using it more is the desire for a backend database rather than just a sync. My preference is to have a switch (environment flag) that changes the entire app storage from local browser to backend API with database storage, suitable for "user-hosted" scenarios only. If this is out of scope for the project but you are open to updating the message and folder data structures to be more "sync-friendly," then I would be willing to maintain this tool/module/feature, but really keen to discuss best way to implement a back end API with Database.

Please let me know if you need any further information.

[enricoros]

Thanks for the detailed answers @joelmnz, and also for the good quality code; it's evident you know your way and caught up on all the subsystems and implied frameworks of the app.

I really want your sync in, and I'm surprised how efficient you were with the code (the updated timestamp worked well there).

I'm working on this branch to add full multipart support (i.e. multimodal input and output to all providers, which meshes well with features such as screen share, and will enable useful workflows).
Code in progress is here, and it's dangerous, do not run it. I had to change DConversation (little) and DMessage (a LOT) and everywhere:
main...feature-multipart
I see an issue where the user may be having different version of DMessage/DConversation (saved as a json, or shared, or synced to Supabase) and should have overall a better handling of this.

Regarding your comment about updating the Conv / Message data structures, do you have suggestions/advice? I really welcome those. And as I'm working on this, maybe we can get 2 birds with one stone.
Please let me know of any feedback on the data structures as I'm overhauling all of that right now.

[joelmnz]

Hi @enricoros , thanks for that, new multipart sounds interesting.

Schema changes are an interesting challenge, I will need to do an update to the Supabase/DB sql schema to match the new structure and maybe have a "version" check in my sync to see if the running instance & the servers SQL schema match, I'll wait till you finish the schema changes, just let me know.

Schema change wish list

If we could

• remove the conversationIds field from DFolder
• and add folderId to DConversation (migration needed)

That would be super helpful for syncing, the standard sync for Conversations would already have folderId's, then I could just sync the Folders list to the server. Unsure on how to handle Folder renames and deletions, might need to find a hook in the UI to use for those events so server = always up to date list of folders.

Nice to Have

I tried to add a "Sync" button above the "Manage Modules" sidebar below the "GitHub" link but started getting too fiddley, would be really cool to have there, less clicks to Sync.

Also, if you could suggest an event/hook location for "Sync On Conversation Changed", I want to add a "Sync Auto/Manual" toggle as it will probably get annoying to try to remember to sync manually each time.

[enricoros]

Thanks @joelmnz , yes, lets' wait until I finish the change - I need to review and test lots of lines of code and I'm being very careful.

Question: do you think we could have multi-tenancy? So many times people are asking on discord about a sync solution for big-AGI.com, and with your patch + login (e.g. I have a Clerk branch) we could support multiple users. Just a thought.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
big-agi-open	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 3, 2024 8:34am
big-agi-open-next	❌ Failed (Inspect)			Jul 3, 2024 8:34am

[joelmnz]

Hi @enricoros , my original plan was multi user via a "sync key" or passphrase. It's a very easy change, I'll update my PR.

When you say + login, does that mean you have done user login and have a userId?

Either way, userId/syncKey it will still work and easy to do.

It's not secure or private, just more of data separation via a key value. The owner of the Supabase DB could always find anyone's keys. I'll add a note to the settings page to inform users.

[enricoros]

@joelmnz correct, I have a userId which is persistent. Using Clerk.com as the user authentication provider, deployment managers can setup their logins (Google, GitHub, SAML, ...) and have protected routes and userIds.

Actually - even Supabase can be configured for Authentication, by enabling it and getting the Google OAuth keys for instance. If we used Supabase auth, then people would be able to register, login, and sync across devices, and it would all be protected end to end. Maybe the Supabase auth is the way to go?

[joelmnz]

@enricoros , yes good point, Supabase has Auth and eventually I would like to use a DB only back end (e.g. no Sync local browser db).

I haven't used Supabase much, I'll have a look and try using their Auth (unless you are familiar with how it works). If that gives us a key to segregate data by then that's great.

[enricoros]

Yes! Please look into it. I've just looked into Supabase for the first time on the weekend, after you led the way. But I gotta focus on the multipart branch, which is a mess. If you can solve for both auth and sync, it's a Great addition!

[enricoros]

@joelmnz to give you an update, I'm making great progress in the mutlimoal / multipart/ function call branch. That will be Big AGI 2.0. and I think we should have Supabase sync, which includes user log in, and possibly files storage (images work great now).
What's your feel?

[joelmnz]

That's some good work @enricoros , looking forward to v2. I've got user login working, need to tidy up the error handling but it works.

Will try to get pr updated over the coming week

[enricoros]

Oh that's really great. Then we shall release it in v2. Users will be very impressed.

[joelmnz]

@enricoros PR ready for review, please make change/fix suggestions as require.

Also, I wasn't sure how to tell users how to setup Supabase, I have a README.md file in the modules folder, maybe once this PR goes in we can put a link to that file from the "Supabase Tool Settings"?

Another question is around Folders, would it be possible to change folders so the "FolderId" is on the Conversation? this will make sync much easier.

NOTE: This first beta of Sync does not handle deletes, its add or updates only for now, will need to hook into the Delete conversation for that. Hopefully that can be in beta-2 of this module.

[joelmnz]

@enricoros I wasn't able to get the supabase auth UI working nicely, I think these might be able to be removed, thoughts?

[enricoros]

I'll take a look at the default components provided by the package, I shall be able to integrate this. Worst case we drop this package and code a manual UI.

[enricoros]

Thanks so much for this work. I'll get to it and comment as soon as the development branch gets to a good state (few days).

[enricoros]

Still open and under eval.

[noahkiss]

@joelmnz

you mention that your initial motivation for this integration was to enable sync for self-hosters.

I see Supabase can be self-hosted. Is this also true for the Auth portion?

I’m very interested in backend storage so that my state isn’t linked to the current browser, but I don’t want that to come at the cost of a third party web connection for storing my chats.

Apologies for my ignorance, just looking for some clarification. Thanks!

[joelmnz]

Hi @noahkiss , yes Supabase is easy to host with docker, the auth part works as well.

Ideally I would like to add a module that can hook into all "data" events and move all to database rather than browser storage. Either with Supabase, AppWrite or something like Postgress etc or even a simple SQLite db.

Current the framework doesn't have the hooks to support swapping data storage locations.

@enricoros when you finished v2 hopefully we could talk about having a generic Storage interface allowing for saving and retrieving (paged and searchable) the conversations from storage of things like Supabase or SQLite.

[noahkiss]

@joelmnz did you happen to try the Remote-Storage recommendation from the other thread? Would that make it easier to switch from localStorage / indexedDB for a single-user cross-browser replication? The remote storage repo has a server which runs redis, and I've modified it (successfully, I believe) to run SQLite. You should be able to pull it from my container repository here and it should be configured to run with sqlite and no auth.

[enricoros]

@joelmnz I'd be all ears for a storage layer, defining the APIs is the most important thing, then any implementation is possible. So much has happened in Big-AGI 2, time to re-focus on this. Would love your ideas on a storage layer or APIs - could you help with some?

[joelmnz]

@noahkiss I had a quick look but decided it wasn't what I wanted, I didn't want to "sync" or use the localDB for what I envision my large history of chat and prompts so didn't take it any further. Ideally I would like to use a database of some kind for security, large storage and hopefully in the future my AI Agent(s) can use my chats to draw knowledge and info from (e.g. Supabase / PostgresSql can store vectors, that way I can vector index all my chats).

[joelmnz]

@enricoros yep I would be keen to help or at least mock up some ideas, it would be a big change and its close to Christmas holidays so that might be a next year thing ;)

[enricoros]

Thanks for the follow up @joelmnz! Definitely would love to hear the requirements before or after the Christmas vacations