Get started

Try it on a laptop in a few minutes, then run the master on the machine that stays on and add your other machines one by one.

Try it on a laptop

A demo fleet runs a master and two node agents on one machine, with simulated models in place of real ones. Nothing touches Docker or a GPU, and everything listens on loopback only.

You need the .NET 10 SDK (10.0.100 or a later feature band) and PowerShell 7 or Bash.

git clone https://github.com/samykabu/MirbadLLM.git
cd MirbadLLM
pwsh tools/dev-multinode/start.ps1 -Reset   # Windows, or PowerShell 7 anywhere
tools/dev-multinode/start.sh --reset        # Linux and macOS

Open http://127.0.0.1:5080/. The script builds everything, enrolls two demo machines, adds a model on each and sends chat requests through the endpoint, so the Overview fills with live numbers. Stop it with pwsh tools/dev-multinode/stop.ps1 or tools/dev-multinode/stop.sh.

Run the master

Pick the machine that stays on. The master runs there with that machine’s own agent, as two containers. Any Linux PC with an NVIDIA GPU works, from an old gaming tower to a DGX Spark.

You need Docker Engine with the NVIDIA Container Toolkit, and a vLLM image the machine can run.

git clone https://github.com/samykabu/MirbadLLM.git && cd MirbadLLM
sudo install -d -o 1654 -g 1654 /srv/mirbadllm/data /srv/mirbadllm/backups   # 1654 = the "app" user of the .NET image
sudo install -d /srv/mirbadllm/models /srv/mirbadllm/cache
cp deploy/.env.example deploy/.env
# In deploy/.env set at least:
#   DOCKER_GID=<output of: stat -c %g /var/run/docker.sock>
#   MIRBADLLM_VLLM_IMAGE=<a vLLM image available on this machine>
#   MIRBADLLM_ADMIN_PASSWORD=<admin password for the UI>
docker compose -f deploy/compose.multinode.yaml --env-file deploy/.env up -d --build

Open http://<host>:8080/, sign in, and add a model on Models.

Note

The compose file treats the machine as one unified memory pool, as on a DGX Spark. On a PC with discrete NVIDIA GPUs, join the PC as a node with the Linux agent installer instead, so memory is checked per GPU. The setup guide covers both.

Add your machines

1. On the master, open Nodes › Add node and copy the one-line command for Windows, Linux or macOS.
2. Run it on the gaming PC, the laptop or the Mac. It installs the agent as a service and enrolls it.
3. Back in the browser, compare the machine’s key fingerprint and approve it.

From then on, the machine’s models appear on the same endpoint as everything else.

Machine	Needs	Runtime
Windows 11 PC	an NVIDIA GPU	llama.cpp as a native process
Linux PC	an NVIDIA GPU, Docker	vLLM in Docker
Mac	Apple Silicon	llama.cpp with Metal, or MLX

Details per machine: Linux, Windows, Mac.

Call a model

curl http://<host>:8080/v1/chat/completions \
  -H "Authorization: Bearer $MIRBADLLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"<served-name>","messages":[{"role":"user","content":"Hello"}],"max_tokens":128}'

Any OpenAI SDK works the same way:

from openai import OpenAI

client = OpenAI(base_url="http://<host>:8080/v1", api_key="<your MirbadLLM API key>")
reply = client.chat.completions.create(
    model="<served-name>",
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)

Until the first API key exists, /v1 is open and the UI says so in its header. Create a key on Access, or set MIRBADLLM_REQUIRE_API_KEY=true to require one from the start.