小白提问，如何在综合演示模型中开启int8精度的量化？谢谢

[starkshow]

您好！
我的显存不足，只有10GB，因此想要运行INT8精度的模型，我发现在ChatGLM3/composite_demo/client.py文件下有如下代码似乎可以开启量化：

class HFClient(Client):
    def __init__(self, model_path: str, tokenizer_path: str, pt_checkpoint: str = None):
        self.model_path = model_path
        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_path, trust_remote_code=True)

        if pt_checkpoint is not None and os.path.exists(pt_checkpoint):
            config = AutoConfig.from_pretrained(
                model_path,
                trust_remote_code=True,
                pre_seq_len=PRE_SEQ_LEN
            )
            self.model = AutoModel.from_pretrained(
                model_path,
                trust_remote_code=True,
                config=config,
                device_map="auto"
            ).eval()
            prefix_state_dict = torch.load(os.path.join(pt_checkpoint, "pytorch_model.bin"))
            new_prefix_state_dict = {}
            for k, v in prefix_state_dict.items():
                if k.startswith("transformer.prefix_encoder."):
                    new_prefix_state_dict[k[len("transformer.prefix_encoder."):]] = v
            print("Loaded from pt checkpoints", new_prefix_state_dict.keys())
            self.model.transformer.prefix_encoder.load_state_dict(new_prefix_state_dict)
        else:
            self.model = (
                AutoModel.from_pretrained(
                    MODEL_PATH,
                    trust_remote_code=True,
                    device_map="auto"
                ).eval())
            # plus .quantized() if you want to use quantized model

注释部分有提示 # plus .quantized() if you want to use quantized model，本人纯小白一枚，请问大神.quantized()我加在哪里能开启量化？

[robotkeys]

第141行的 eval() 替换为quantized(8)
136 self.model = AutoModel.from_pretrained(
137 model_path,
138 trust_remote_code=True,
139 config=config,
140 device_map="auto"
141 #).eval()
142 ).quantized(8)

[PING-An32]

In client.py
line 136

self.model = AutoModel.from_pretrained(
                model_path,
                trust_remote_code=True,
                config=config,
                device_map="auto").eval()

change into ↓

self.model = AutoModel.from_pretrained(
                model_path,
                trust_remote_code=True,
                config=config,
                ).quantize(bits=8, device="cuda").cuda().eval()

line 151
self.model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True,device_map="auto").eval()
change into ↓
self.model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True).quantize(bits=8, device="cuda").cuda().eval()