Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Quantizing and running inference on bloom-176B required some changes #21

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Quantizing and running inference on bloom-176B required some changes #21

wants to merge 1 commit into from

Conversation

barsuna
Copy link

@barsuna barsuna commented Apr 2, 2023

  • Most issues are due to fact that embedding layer 250880x14336 is too large to fit into signed integer
  • Above affects the main, quantize, and also ggml code
  • 2nd issue is that main seems to estimate amount of necessary memory on the low side
  • Above is not fixed, i have just added 5GB for weights and doubled the size of context used for model evaluation Being very far away from proficiency in C++, these changes need to be civilized by someone experienced with ggml and c++

@barsuna
Quantizing and running inference on bloom-176B required some changes
2d0e478 
- Most issues are due to fact that embedding layer 250880x14336 is too large to fit into signed integer
- Above affects the main, quantize, and also ggml code
- 2nd issue is that main seems to estimate amount of necessary memory on the low side
- Above is not fixed, i have just added 5GB for weights and doubled the size of context used for model evaluation
Being very far away from proficiency in C++, these changes need to be civilized by someone experienced with ggml and c++
@akumaburn akumaburn mentioned this pull request Apr 11, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@barsuna