Update model.py to enable CUDA when available

[FNGarvin]

Shell out to nvidia-smi for NVidia detection, add device or gpu args to conman as appropriate for docker vs podman. Import subprocess (to shell out for nvidia-smi) and shutil (to duplicate available() functionality and test whether we're using podman or docker)

Summary by Sourcery

Enhancements:

• Enable CUDA support in container setup by detecting NVidia GPUs using nvidia-smi and configuring container arguments accordingly.

[sourcery-ai]

Reviewer's Guide by Sourcery

This PR adds CUDA GPU support to the container management system by detecting NVIDIA GPUs and configuring the appropriate container runtime arguments for both Docker and Podman environments. The implementation shells out to nvidia-smi for GPU detection and handles container-specific GPU flags differently based on the container runtime being used.

Sequence diagram for GPU detection and container setup

sequenceDiagram
    participant User
    participant Model as model.py
    participant Subprocess
    participant Shutil

    User->>Model: Call setup_container(args)
    Model->>Model: get_gpu()
    Model->>Subprocess: Run nvidia-smi
    alt NVIDIA GPU detected
        Subprocess-->>Model: Return CUDA_VISIBLE_DEVICES, gpu_count
        Model->>Shutil: Check if Podman is available
        alt Podman available
            Model->>Model: Add --device nvidia.com/gpu=all to conman_args
        else Docker available
            Model->>Model: Add --gpus all to conman_args
        end
    else No NVIDIA GPU
        Model->>Model: Return None, None
    end
    Model-->>User: Return conman_args

Loading

File-Level Changes

Change	Details	Files
Added NVIDIA GPU detection using nvidia-smi command	Added subprocess call to 'nvidia-smi -L' to detect NVIDIA GPUs Returns CUDA_VISIBLE_DEVICES and count of available GPUs if nvidia-smi succeeds Implements graceful fallback if nvidia-smi command fails	ramalama/model.py
Implemented container runtime-specific GPU configuration	Added detection for Podman vs Docker runtime Configured Podman-specific GPU flags using --device nvidia.com/gpu=all Configured Docker-specific GPU flags using --gpus all	ramalama/model.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey @FNGarvin - I've reviewed your changes - here's some feedback:

Overall Comments:

• The empty catch block in get_gpu() silently swallows all exceptions. Consider logging errors or handling specific exceptions to avoid masking real issues.
• Consider extracting the podman/docker detection logic into a shared utility function to avoid the noted code duplication with common.py

Here's what I looked at during the review

• 🟡 General issues: 1 issue found
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

suggestion: Redundant error checking with subprocess.run()

Using check=True with manual returncode checking is redundant as check=True will raise CalledProcessError on non-zero exit codes.

        check_output = subprocess.run(['nvidia-smi', '-L'], capture_output=True, check=True)

[rhatdan]

Use run_cmd

[bmahabirbu]

We can go a step further and query the nvidia-smi command itself to get more info! For example doing this command nvidia-smi --query-gpu=index,memory.total --format=csv,noheader,nounits | sort -t, -k2 -nr | head -n 1 Can get us the largest vram GPU and id formatted as "id, vram-in-mb".

We can do something like this

try:
        command = ['nvidia-smi', '--query-gpu=index,memory.total', '--format=csv,noheader,nounits']
        output = run_cmd(command)
        gpus = output.stdout.strip().split('\n')
        gpus_sorted = sorted(gpus, key=lambda x: int(x.split(',')[1]), reverse=True)
        return "CUDA_VISIBLE_DEVICES",  gpus_sorted[0][0]
except Exception: {} #fall through
    return None, None

[FNGarvin]

I think that's a fabulous idea. My inclination in general is to avoid shelling out, but I don't think we're going to find a better or more lightweight way to test for the presence of NVidia and CUDA. Probably why all the NVidia Container Toolkit docs seem to use it as a sanity check for installs.

[bmahabirbu]

Same I'd rather not shell out myself but doing it this way avoids complications with different systems. if we can assume a system has an nvidia GPU then most likely there will be drivers installed with nvidia-smi as well.

Right now the vulkan backed for llama.cpp doesn't have all the functionality like cuda and hip blas does. But later down the line id like to switch to vulkan and use the vulkan SDK to query GPU data since it supports amd nvidia and intel graphics.

[rhatdan]

Thanks @FNGarvin. I found some issues.

[FNGarvin]

Superceded by #490 etc