Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixed Llama 2 to 3.2 NBs #388

Merged
merged 9 commits into from
Oct 6, 2024
Merged

fixed Llama 2 to 3.2 NBs #388

merged 9 commits into from
Oct 6, 2024

Conversation

d-kleine
Copy link
Contributor

@d-kleine d-kleine commented Oct 6, 2024

  • fixed RoPE formula (-2)
  • updated requirements (safetensors missing, ipywidgets for HF progress bar)
  • removed redundant code lines
  • removed duplicated imports
  • fixed local_dir for llama 3.1 and 3.2 files

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@d-kleine
Copy link
Contributor Author

d-kleine commented Oct 6, 2024

About RoPe, I cannot follow the code implementation compared to the formula. This also applies to converting-gpt-to-llama2.ipynb where I think it would be beneficial to add the RoPE formula too.

@d-kleine d-kleine marked this pull request as ready for review October 6, 2024 10:10
@rasbt
Copy link
Owner

rasbt commented Oct 6, 2024

Thanks for these refinements!

Regarding

About RoPe, I cannot follow the code implementation compared to the formula. This also applies to converting-gpt-to-llama2.ipynb where I think it would be beneficial to add the RoPE formula too.

Yeah, the formula is only shown to highlight the "theta" and "base" aspects but the one formula is not enough to follow the implementation of the complete RoPE method. I was originally thinking of explaining RoPE here, but this would be better for a separate article because it's a whole topic (or even paper) in itself.

@rasbt rasbt merged commit a0c0c76 into rasbt:main Oct 6, 2024
8 checks passed
@d-kleine
Copy link
Contributor Author

d-kleine commented Oct 6, 2024

Alright, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants