Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor: clean up codebase, simplify where possible #189

Merged
merged 46 commits into from
Jan 2, 2025

Conversation

z3z1ma
Copy link
Owner

@z3z1ma z3z1ma commented Dec 28, 2024

This will be a WIP for a bit. dbt has evolved enough since this tool was created that we can probably simplify some things. Furthermore, certain experimental things can be removed. I still like the workbench. Suffice to say there are interesting streamlit applications we have not explored like an interactive documentation input flow for progenitors, etc.
But otherwise a lot we can kill. We can also remove core interface folding it into osmosis.py and leverage whatever dbt core has available now in its pythonic interface -- and be more strict on what dbt version(s) are supported with a focus on newer versions.
Also includes cleanups with py 3.9 being min version.

@z3z1ma z3z1ma changed the title chore!: start refactor Refactor: clean up codebase, simplify where possible Dec 28, 2024
@syou6162
Copy link
Contributor

Also includes cleanups with py 3.9 being min version.

I'm sorry if there is a conflict between the changes made to #190 and what you are trying to do with this pull request 🙏🙏🙏 .

@z3z1ma z3z1ma force-pushed the refactor/clean-up-codebase branch from 2695252 to ba430de Compare December 29, 2024 09:16
@z3z1ma z3z1ma force-pushed the refactor/clean-up-codebase branch from febe2df to 06890b3 Compare December 29, 2024 23:35
@z3z1ma z3z1ma force-pushed the refactor/clean-up-codebase branch from 655152d to 87157a3 Compare December 30, 2024 05:49
@z3z1ma z3z1ma self-assigned this Dec 30, 2024
@z3z1ma z3z1ma added the refactoring Refactoring label Dec 30, 2024
@z3z1ma
Copy link
Owner Author

z3z1ma commented Dec 30, 2024

@syou6162 there is some more to do, but at least you can see the significant changes here which will simplify maintainability and even extensibility for LLM integrations, user defined plugins / column fuzzing logic, etc...

My approach in such a refactor as this is to collapse code into one file as much as possible such that we can centrally reason on the abstractions and code paths. From there, I can re-break out the functions into different namespaces/modules (if really needed) once the codebase refactor is done. The tests will be updated as appropriate soon too and hopefully things reveal themselves as easier to test as well which would evidence we move in the right direction.

@z3z1ma z3z1ma force-pushed the refactor/clean-up-codebase branch from 70fb173 to 7e501d4 Compare December 30, 2024 23:26
@z3z1ma z3z1ma force-pushed the refactor/clean-up-codebase branch from 7e501d4 to e9c040a Compare December 30, 2024 23:27
@z3z1ma z3z1ma force-pushed the refactor/clean-up-codebase branch from 5c632e4 to 77c99af Compare December 31, 2024 18:22
@z3z1ma z3z1ma force-pushed the refactor/clean-up-codebase branch from a780930 to 5b31abf Compare December 31, 2024 22:36
@z3z1ma z3z1ma force-pushed the refactor/clean-up-codebase branch from 6ffb027 to 30acd2e Compare January 1, 2025 21:01
chore: only use get node yaml if needed otherwise stick to node itself
@z3z1ma z3z1ma force-pushed the refactor/clean-up-codebase branch from 30acd2e to a2b7afc Compare January 1, 2025 21:24
@z3z1ma z3z1ma force-pushed the refactor/clean-up-codebase branch from 471d200 to 1d265f0 Compare January 2, 2025 08:47
@z3z1ma z3z1ma merged commit 86bac36 into main Jan 2, 2025
13 checks passed
z3z1ma added a commit that referenced this pull request Jan 2, 2025
refactor!: clean up codebase, simplify where possible

chore: fix workbench req txt

fix: get release working
z3z1ma added a commit that referenced this pull request Jan 2, 2025
refactor!: clean up codebase, simplify where possible

chore: fix workbench req txt

fix: get release working
z3z1ma added a commit that referenced this pull request Jan 2, 2025
refactor!: clean up codebase, simplify where possible

chore: fix workbench req txt

fix: get release working
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
refactoring Refactoring
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants