Gemma-7b&&Gemma-2b

[intellinjun]

Type of Change

feature or bug fix or documentation or others
API changed or not

Description

detail description
Issues: xxx

Expected Behavior & Potential Risk

the expected behavior that triggered by this PR

How has this PR been tested?

how to reproduce the test (including hardware information)

Dependency Change?

any library dependency introduced or removed

[intellinjun]

f32 inference result

[intellinjun]

gemma_2b inference result

[intellinjun]

<style> </style>

model_name	model_type	piqa
gemma_2b	weight_dtype f32	77.09
gemma_2b	w4cint8g32	75.68
gemma_2b	pytorch	77.3
gemma_7b	weight_dtype f32	80.25
gemma_7b	w4cint8g32	78.56
gemma_7b	pytorch	81.2

[intellinjun]

<style> </style>

model_name model_type piqa
gemma_2b weight_dtype f32 77.09
gemma_2b w4cint8g32 75.68
gemma_2b pytorch 77.3
gemma_7b weight_dtype f32 80.25
gemma_7b w4cint8g32 78.56
gemma_7b pytorch 81.2

here is gemma7b&2b accuray,the inference result of gemma7b is bad when quantize config is weight_dtype int4 compute dtype int8 group_size 128, but gemma2b's result looks good.

[intellinjun]

https://huggingface.co/google/gemma-7b-it/discussions/38
Here are discussions about gemma7b gguf model's bad output.

[a32543254]

LGTM

[intellinjun]

https://inteltf-jenk.sh.intel.com/job/neural_speed_extension/103/artifact/report.html

gemma-2b performance, looks so bad.
@luoyu-intel , I profiled the gemma-2b performance, and find the main cost is on the innerproduct, gemma-2b has two innerproduct, lm_head.weight[2048, 256000] and token_embd.weight[256000,2048](use fp32 cause if gemma-7b use q40 quant this weight will get bad result), can you have solution to fix it?

[intellinjun]

new gemma2 performance, @luoyu-intel ,thanks.
https://inteltf-jenk.sh.intel.com/job/neural_speed_extension/107/artifact/report.html