Code

Optim - added quantization code.#968

Code

Open

sgsharma2000 wants to merge 70 commits intometa-llama:mainmeta-llama/llama:mainfrom gtamer2:optimgtamer2/hpml_llama:optimCopy head branch name to clipboard

+971-466

Commits

Commits on Dec 13, 2023

• Add inference benchmark, delete extra files

  gtamer2

  committed
  236b696

• more deletions

  gtamer2

  committed
  ed186f5

• Merge pull request #1 from gtamer2/add_benchmark

  Show description for 64ab5a8
  gtamer2

  authored
  64ab5a8

• Changes

  gtamer2

  committed
  62f242d

• import fire

  gtamer2

  committed
  257962f

• git ignore

  gtamer2

  committed
  a89488a

• fixes

  gtamer2

  committed
  5a84145

• Push changes

  gtamer2

  committed
  cf2b652

• comment out omdel.to

  gtamer2

  committed
  47400da

• print x for debugging

  gtamer2

  committed
  4c4d439

• inference_benchmamrk

  gtamer2

  committed
  2071539

• inference_benchmamrk

  gtamer2

  committed
  58eee14

• Try large batch

  gtamer2

  committed
  86ca1c7

• Get workign for 1 batch

  gtamer2

  committed
  f146895

• Get workign for 1 batch

  gtamer2

  committed
  fd8a6cf

Commits on Dec 14, 2023

• Add torch profiler

  gtamer2

  committed
  04cb363

• Indent

  gtamer2

  committed
  40abad2

• Indent

  gtamer2

  committed
  d6336fc

• Profiel cpu and cuda

  gtamer2

  committed
  f80cf22

• move profiler down

  gtamer2

  committed
  8436d96

• Try using simplified llama

  gtamer2

  committed
  60bc868

• revert

  gtamer2

  committed
  84416a4

• Move outside

  gtamer2

  committed
  7dfe9e9

• one more yolo

  gtamer2

  committed
  e5f0408

• rerecord mem

  gtamer2

  committed
  d7da6e1

• Merge pull request #2 from gtamer2/benchmark2

  Show description for b988f0e
  gtamer2

  authored
  b988f0e

• added torch.jit.script to model in generation.py

  

  Guru Sharma

  committed
  431ebab

• Script to run benchmarks

  gtamer2

  committed
  67d5a14

• Fix missing param

  gtamer2

  committed
  dcda8a1

• tried changing .jit.script to be around the llama object call in inference benchmark

  

  Guru Sharma

  committed
  816b895

• Empty cuda cache

  gtamer2

  committed
  078aa9c

• changed to trace from script

  

  Guru Sharma

  committed
  47aaa29

• .trace on generation.py

  

  Guru Sharma

  committed
  e8f2d14

• get rid of trace entirely to see if it is causing issue

  

  Guru Sharma

  committed
  c18ee45

• Add torchscript python script

  gtamer2

  committed
  a79d31a

• reset

  gtamer2

  committed
  680d2a6

• remove bard hallucination

  gtamer2

  committed
  aeab877

• Merge pull request #3 from gtamer2/new_benchmarks

  Show description for 93d16e2
  gtamer2

  authored
  93d16e2

• Merge branch 'main' of https://github.com/gtamer2/hpml_llama into torchscript

  gtamer2

  committed
  10188e4

• revisions

  gtamer2

  committed
  c588e99

• revisions

  gtamer2

  committed
  714dd19

• revisions

  gtamer2

  committed
  0abe652

• Add benchmarks

  gtamer2

  committed
  5534bbf

• get quantization working

  

  Guru Sharma

  committed
  44feb11

• Merge pull request #5 from gtamer2/torchscript

  Show description for ae535a2
  gtamer2

  authored
  ae535a2

• add some more quantization lines

  

  Guru Sharma

  committed
  8e34266

• got rid of fuse_model()

  

  Guru Sharma

  committed
  94c70fb

• adding convert to quantization model

  

  Guru Sharma

  committed
  3920ee5

• quantize script

  gtamer2

  committed
  adb0190

Commits on Dec 15, 2023

• fire launch

  gtamer2

  committed
  d71ffa0

• Inplace

  gtamer2

  committed
  748bc32

• access the transformer

  gtamer2

  committed
  775475a

• New quant logic

  gtamer2

  committed
  165cd54

• rge branch 'main' of https://github.com/gtamer2/hpml_llama into optim

  gtamer2

  committed
  005d8c5

• fix quant sample inf

  gtamer2

  committed
  c995e43

• Fix args

  gtamer2

  committed
  575c197

• move h=quant(h) to after firist layer. It was operating on tokens whicch are Half, not float

  gtamer2

  committed
  787efd5

• added a sample for pruning model

  

  Guru Sharma

  committed
  d0b8b39

• maybe it prunes now?

  

  Guru Sharma

  committed
  fab7b44

• added attention

  

  Guru Sharma

  committed
  1a85f36

• add torch.nn.parameter to this

  

  Guru Sharma

  committed
  0bf73c4

• added some changes for including torch.nn.Parameter into the pruning, created my own attribute

  

  Guru Sharma

  committed
  ac4fb51

• get rid of quantization modifications in model.py

  

  Guru Sharma

  committed
  caf7482

• checking sparsity

  

  Guru Sharma

  committed
  b790b37

• trying to find sparsity again

  

  Guru Sharma

  committed
  1c13854

• we got rid of sparsity

  

  Guru Sharma

  committed
  bf0ebe0

• giving sparsity another try

  

  Guru Sharma

  committed
  cb43151

• trying newer things with prune_model.py

  

  Guru Sharma

  committed
  6779933

• got rid of print statements, left them until after it was done

  

  Guru Sharma

  committed
  660e231

• trying to make sure layer is being modified in place

  

  Guru Sharma

  committed
  651f8fb