进行推理过程中无法吃满GPU

[lostmaniac]

环境说明

1. RTX4090 * 2
2. CUDA Version: 12.2
3. Driver Version: 535.129.03
4. 硬盘：SSD，测速380M/S
5. python == 3.10.12
6. git仓库分支 main
7. git提交版本 de97149
8. 使用文件： ./openai_api.py
9. 调用API为fastgpt

cuda监控：

Thu Dec  7 08:58:46 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        Off | 00000000:3B:00.0 Off |                  Off |
| 37%   45C    P2              95W / 450W |   8157MiB / 24564MiB |     27%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off | 00000000:D8:00.0 Off |                  Off |
| 39%   46C    P2             100W / 450W |   8061MiB / 24564MiB |     48%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A     74071      C   python                                     8148MiB |
|    1   N/A  N/A     74071      C   python                                     8052MiB |
+---------------------------------------------------------------------------------------+

调整过的文件：openai_api.py

    tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH, trust_remote_code=True)
    if 'cuda' in DEVICE:  # AMD, NVIDIA GPU can use Half Precision
        # model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True).to(DEVICE).eval()
        embeddings_model = SentenceTransformer('./m3e-base', device='cpu')

        # Multi-GPU support, use the following two lines instead of the above line, num gpus to your actual number of graphics cards
        from utils import load_model_on_gpus
        model = load_model_on_gpus(MODEL_PATH, num_gpus=2)

    else:  # CPU, Intel GPU and other GPU can use Float16 Precision Only
        model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True).float().to(DEVICE).eval()
    uvicorn.run(app, host='0.0.0.0', port=8000, workers=1)

生成过程中使用率始终为40%-55%左右的使用率

请问需要怎么调整能够吃满GPU，让速度更快一点。感谢

[zRzRzRzRzRzRzR]

两卡的效率本来就很低了，建议是单卡。另外，这是默认的hf加载方式，没有做任何推理优化，你可以使用vllm等推理优化框架尝试

[lostmaniac]

感谢