You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Yes, would make a lot of sense. Right now all files are also parsed/split every time even if they're not changed since the last commit also. So both these changes would be nice to improve performance.
For the embedding part of the ingestion I've also been thinking about ways to detect if there are significant changes and only embed chunks again if there are any. Haven't figured out how though...
I really appreciate the notebook for efficiently building the index by using the previously persisted index.
However, this can still be slow as the splitting process still needs to run from scratch for every instance (correct me if I'm wrong).
Do you think the splitting process can also be incremental and utilizing previously split data? I think tree-sitter is incremental in parsing the AST.
The text was updated successfully, but these errors were encountered: