Skip to content

Run Your Own Model with llama.cpp on Open OnDemand

When you need a specific model, a private endpoint, or control over how a model is served, you can run your own LLM on a slice of an NVIDIA A100 GPU. The llama.cpp AI Chatbot app in Open OnDemand does this from a web form: you choose the resources and the model, and it writes and submits the Slurm job for you. No terminal is required.

Time needed: about 5 minutes plus queue time. Prerequisites: an active Hopper account and, off campus, the Mason VPN.

Step 1 · Open the app

  1. Go to ondemand.orc.gmu.edu and sign in with your NetID.
  2. Open Interactive Apps and find the Servers section.
  3. Click llama.cpp AI Chatbot.

Open OnDemand Servers section with the llama.cpp AI Chatbot app

Tip

The VSCode Code-Server app on the same page gives you VS Code in a browser tab, which is an alternative to setting up Remote-SSH.

Step 2 · Fill in the job form

Partition, GPU type, and number of GPUs fields

Field Recommended value Why
Partition GPU Gives you a GPU without preemption. The Contrib partitions also have GPUs, but they need a QoS parameter or your job can be preempted (stopped) by the node's owners.
GPU type 40GB (3/7) A slice of an A100 with 40 GB of GPU memory, enough for every model in the list. For a full A100, submit a Slurm script instead (see Running GPU Jobs).
Number of GPUs 1 One slice is enough to serve any model offered.
Number of cores 8 The GPU does the heavy lifting, so a few CPU cores is plenty.
Time limit 1–12 hours Ask only for the time you need; 12 hours is the maximum. The session ends when the time is up.
Reservation, CUDA, CUDNN leave blank Unless ORC has told you otherwise.

Step 3 · Choose a model

Model dropdown in the llama.cpp AI Chatbot form

Model Quantization Weights size Notes
Qwen3 8B Q4_K_M 5.84 GB Fast, with lots of headroom. Start here.
Qwen3.6 35B-A3B Q4_K_M 20.0 GB Mixture of experts: large, but only about 3B parameters are active per token, so it runs faster than its size suggests.
Gemma 4-31B-it UD-Q4_K_XL 23.5 GB The largest model offered. Keep an eye on context length.

You can also select choose later and pick the model once the session starts.

Why the weights size isn't the whole story

The size shown is the model file. While the model runs, it also needs memory for the KV cache, which grows with the length of the conversation (the context). The weights and the KV cache share the same 40 GB of GPU memory, so leave headroom, especially with the larger models and long prompts. If the server runs out of memory, use a smaller model or a shorter context.

Q4_K_M and UD-Q4_K_XL are 4-bit quantizations: roughly a quarter of the full-precision size, with a small cost in quality.

Step 4 · Launch and connect

  1. Click Launch. The session appears under My Interactive Sessions as Queued, then Starting.
  2. When it shows Running, click the connect button on the session card to open the chat interface.
  3. Chat with the model. It runs entirely inside your own job; your prompts are not sent to the ORC gateway.

Step 5 · End the session when you're done

A running session holds a GPU slice that other researchers are waiting for. When you finish, click Delete on the session card rather than leaving it to time out.

Gateway or your own server?

Use the ORC gateway when… Run your own server when…
You want an assistant in your editor now, with no setup. You need a particular model or quantization.
qwen-3.8-27B does the job. The work shouldn't leave your own job.
You work from your laptop as often as from Hopper. You are benchmarking or changing serving parameters.
You don't want to hold a GPU reservation. You can work within the 12-hour limit.

Troubleshooting

Problem Fix
The session stays Queued for a long time All GPU slices are busy. Wait, try a shorter time limit, or use the ORC gateway in the meantime.
The session ends early on a Contrib partition It was preempted. Use the GPU partition.
The model fails to load or crashes on long prompts Out of GPU memory. Pick a smaller model or shorten the context.
Replies are slow Try Qwen3 8B, or Qwen3.6 35B-A3B, which is fast for its size.
The session ended while you were working The time limit was reached. Start a new session with a longer limit, up to 12 hours.

See Also