-
Notifications
You must be signed in to change notification settings - Fork 408
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The future of Hive #246
Comments
Rewriting in Rust sounds so interesting. Better performance and less memory usage are attractive and welcome. However, I'm seriously worried about compatibility of Box. Can boxes for v1.x still be used in v2 as well? It's dubious as to what other users actually meant in the polls. For me, only breaking changes of APIs are acceptable, not of Box. I have to decide whether to leave Hive and choose another package for the app I'm currently developing if there is a risk that I need to go through all the trouble to port old boxes to new ones on my own sometime in the future. Having said that, your plan is exciting too all the same. I look forward to seeing the first release of the new major version. |
Yes I agree. It is bad that old boxes will not be compatible and existing apps in production cannot upgrade to the new version without loosing their data. Hive is very young and I still think it is the right path. For future breaking changes there will be auto migration. Unfortunately that is not possible for this change because we switch the backend. |
A noob question: how will you make Rust to work with Dart? |
Using Dart FFI |
But DART FFI is only for C language, not Rust, right? It will require the user to use FFI or are you planning to provide a working interface already in Dart? |
Rust does provide C interop...
The user will only use Dart and does not even notice the Rust backend. |
You should give https://vlang.io/ a shot 😜 |
V is interesting but in my opinion, there are multiple reasons why it is not a good idea to use V at the moment:
|
What initially excited me about Hive is that it's a pure Dart library without external dependencies, so it runs everywhere. |
Yes, that was the goal but it turned out that most users don't want a database that is basically an in-memory KV-store. The problem with Dart is that it is kind of slow, its objects are rather memory hungry and it misses essential features to implement a more advanced database.
I thought the same thing but the list of candidates is short. In fact, I didn't find a single database that is suitable for mobile devices and our requirements. Also, to my knowledge, there is no database that is built as a counterpart to IndexedDB. It is not trivial to write a database that works exactly the same in the browser. IndexedDB is very different from most other databases.
As I said, I don't think there exists a single cross-platform database that also works in the browser and I don't think existing databases can be easily used with Dart and still have great performance. Realm, for example, will never work with Dart because it relies on proxy objects. So I'm writing basically writing an abstraction around IndexedDB and LMDB in Rust which can be compiled to a binary or WASM. And then there will be the Dart wrapper around this "backend". It should be easily possible to use only the Rust side for example with React native. Edit: I already have a fully working prototype of the LMDB part of the wrapper and not much Rust code is required. The performance is exceptional. If you have an alternative approach that allows us to have a "real" database which also works in the browser, I'd love to discuss it. Edit2: Another advantage of this approach is that breaking changes of the binary format will no longer be required and bugs that corrupt the database will not happen anymore because the storing of the data will be handled by LMDB and IndexedDB respectively. Edit3: Like most other databases, Noria, the one you linked, is for backends and thus not really suitable for mobile devices. |
Okay, I see. I was really expecting more lightweight Rust databases to exist. |
There is one topic where I still need input: Since we are rebuilding Hive anyway, I'd like to make it ready for synchronization from the beginning. What do you guys think about CouchDB as a backend? |
Syncing with CouchDB, is exactly what I am looking for my next project. PouchDB, implementation in dart will be great solution. |
Used CouchDB a long time ago, and while the conflict resolution mechanism was neat, queries were a pain. That might have changed, or not. I thought one of the major drivers of this Hive rewrite was query support. |
Yes, the queries you use with should be more or less independent of CouchDB. CouchDB is just an idea and nothing I have decided yet. I just want to figure this out before the first stable release of the new version has been released. It would be great if someone knows a backend which fits our use case. |
Unsolicited advice / thought: I realize you're planning for the future ahead of time by factoring in sync support, but it's a complex I also say this because I'm not using CouchDB and while I'd love conflict resolution, I haven't found any really easy way to sync CouchDB changes to Postgres. This seems to be a problem for many who may use another database. Limited research, but I just wanted to throw it out there given that my anecdotal evidence suggests that more people are using Postgres/MySQL/etc and if CouchDB support for those is weak, it may not be worth the additional effort upfront. All that said, killer job with Hive thus far. Excited to see what's next. |
Thanks for your opinion. Yes it would be cool to have a conflict resolution which is independent of the backend database but I have to do a lot of research because I have no idea how to do it 😆 |
I think it would be prudent to create a second project> Hive ffi <or something like that. I have 9 applications in production using Hive, and it makes me very afraid to think that users with 300mb/500mb of data on Hive, may lose everything after a library update. |
I don't think it will be possible to automatically migrate the data because the two models are not entirely compatible. But I will maintain a branch that contains the current version so you can just continue to use it. |
@leisim In your vision, will the new Hive still allow to Create adapter manually? I'm not using code generation In my project, I'm just defining my own |
when this version can be release and we can use that? 💃 💃 💃 |
@algodave I don't think it will be possible in the same way as it is currently because in order to query your data, Hive needs to understand its structure. Probably there will still be adapters that map objects to
@MahdiPishguy It probably still takes another month until I have the first test version. |
I've been working on a mission-critical project with Hive, and a major pull was that it's completely written in Dart, so I may need to fork as well. If you're interested, I can keep you posted. |
Might I request you create a new library? A complete rewrite, with different behavior, and large api changes is not a new version but a new library. Going down this path means there will be numerous forks of Hive 1, and people would just be better served if you started a new project and let Hive continue to evolve. |
@xylobol @jonataslaw I am one of those who would be interested in a fork |
@leisim any news? |
@leisim I'm also interested in the progress on this. |
@yringler Yes I'm quite happy with the progress. According to my tests and benchmarks, Isar will probably solve all problems people have with Hive currently. |
@leisim Can we expect stable version by this month? |
@dgandhi17 stable probably not because that implies that it has been battle tested. You can expect a beta version within the next few weeks. |
A noob opinion: Advantages ( I thought)
Disadvantages ( I also thought)
Feel free to correct me and point if this is possible |
Casual observer here, but I wonder whether Dart compiler improvements are having/will have any impact. The new compiler in SDK 2.12 contains memory and performance optimisations when code is compiled with sound null safety. Some other enhancements on the roadmap: |
Humble Hive user here. Is it possible to make data posts or hive files have max age, stale ... options like we find in caching properties? |
I only need a db solution for my dart server project, have you thought about making a pure dart wrapper around LMDB as a side project to avoid bloat by including indexdb etc. I'm using Redis now, but i suspect LMDB would be faster. |
Maybe some benchmarks for |
@erf Dart is very good at tree shaking unused code so the indexeddb wrapper will never be included in your mobile app build. Redis is an in-memory database so it's likely faster than anything else. I'm not sure benchmarks are very useful here because the databases are completely different. That being said I expect Isar to be faster than all other databases currently available for Flutter. Currently I have more important tasks (unit tests 🙄) than writing a benchmark but maybe someone from the community will write one. |
Hello, is it already possible to use custom queries? or sort by date as you said at the beginning of the post? |
Hi friends! As is suggested here, maybe this package is helpful: https://github.com/fzyzcjy/flutter_rust_bridge |
I have to say that if this makes hive harder to deploy then I would be against it. Hive performance is OK but its missing decent indexing and of course memory usage is a problem. These could be solved by hive implement its own disk based btree indexes in dart. Use meta data to define the indexes and as you have suggested the 'where' methods to access the indexes. I found this implementation of a btree in java which would port easily across to dart. I would suggest that this would also be much less work than the proposed rust path. If I had to do additional work to ship with hive (i.e. deploying a binary) then I wouldn't have chosen hive. FWIW: I build and deploy cli tooling that needs to work across linux/macos/windows. |
@bsutton Tge additional work to deploy the Isar binaries is to add: dependencies:
isar_flutter_libs: any Dart misses mmap and threads and is therefore not a good language for a database. Please read the first post for a more detailed explanation. |
Just want to add that, from what I see is, there's no interest in implementing proper threading support to dart, the team rather prefers to improve their Isolate implementation which does limits what could be done with threads, lacks shared memory and does involve serialization to move static typed data between isolates |
@leisim I'm not actually doing flutter Dev but rather server side. Does the flutter lib work with for non flutter apps? Are all the noted platforms supported? |
@bsutton I don't recommend using any embedded database on the server. You're almost always better off using a dedicated database. |
From my use case an embedded server is the correct answer.
This is a cli script not a server.
S. Brett Sutton
Noojee Contact Solutions
03 8320 8100
…On Sun, 12 Dec 2021 at 00:14, Simon Leier ***@***.***> wrote:
I don't recommend using any embedded database on the server. You're almost
always better off using a dedicated database.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#246 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG32OEILNC6WQKB6SSHLFLUQNFERANCNFSM4K56YKVA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
Your work is hugely appreciated Simon. I think Isar complements Hive. Hive could be used instead of shared preferences for simple stuff and Isar could be used for large queryable data. EDIT: One might ask, why not use
|
Fully agree! It's not intended to be a replacement. |
Isar is stable now. Thanks everyone for your help and participation! |
TLDR: Hive 2.0 will be rewritten in Rust to get amazing performance, multithreaded queries, read and write transactions, and super low memory usage. The code will work 1:1 in the browser.
Situation
I have thought a long time how to correctly implement queries in Hive and I have come to the conclusion that it is not possible with the current architecture.
I have reviewed many projects on GitHub which use Hive and most of them have to create their own suboptimal workaround for queries.
Apart from queries, Hive has another problem: Dart objects use much RAM. Since Hive currently keeps at least the keys in RAM, you can hit the OS limit of mobile devices quite fast.
I also created polls some time ago on the Hive documentation page and there were two very strong takeaways:
Idea
So here is what I have come up with:
I will completely rewrite Hive in Rust. I will use the strengths of the old implementation (like auto migration) and fix the issues.
On the VM, Hive will use LMDB as backend and on the Browser IndexedDB. The VM implementation will provide the same features as IndexedDB to allow easy code sharing.
The two main goals of Hive will stay the same: Simplicity and Performance.
I have a small prototype and the performance is amazing. LMDB has to be some kind of black magic.
Sample
Here is how it is going to work:
The model definition is very similar to current models:
Hive will then generate extension methods so you can write the following query:
where()
vsfilter()
The difference between
where()
andfilter()
is thatwhere()
uses an index andfilter()
needs to test each of the resulting elements. Normally a database figures out when to use an index itself but Hive will provide you the option to customize.There are multiple reasons for that:
Things to figure out
Blocking Issues (pls upvote)
Other issues
For existing apps using Hive 1.x:
I will continue to keep a Hive 1.x branch for bugfixes etc.
What do you think?
The text was updated successfully, but these errors were encountered: