← redevops.io
Bring your own model · self-hosted inference · redevops.io

Run it on your own models

Point the whole stack at a model on hardware you control — Ollama, a local vLLM, or another machine via NVIDIA PAIR. No cloud dependency.
This page is about the model layer, not the install. Stand the stack up first — the core stack (runtimes + Projects control plane) and, on top of it, the business apps — both take a single OpenAI-compatible endpoint. This page is how you make that endpoint one you host.

Why local. The stack talks to one OpenAI-compatible model endpoint — nothing about it requires a vendor's cloud. Run the model on a laptop for evaluation, on a workstation with a GPU for real work, or across a few trusted machines that share one database and one model. Your data and your prompts stay on hardware you control; the only thing that leaves the box is what you deliberately wire to an external tool.

Install the stack first. The per-OS (Windows · macOS · Ubuntu) and cloud steps live on the install pages — Core install for the runtimes + control plane, then Deploy the business apps. Both read the model endpoint from REDEVOPS_LLM_BASE_URL in .env. Come back here to make that endpoint a model on your own hardware.

Point it at your own model

The stack needs one OpenAI-compatible endpoint — that is the whole contract, and anything that speaks that API works. Set it once in the stack's .env:

# .env — the one line that matters for AI
REDEVOPS_LLM_BASE_URL=http://host.docker.internal:11434/v1   # a local Ollama, for example

Here are the local options, cheapest-effort first.

Ollama or LM Studio — a model on this machine

Ollama is the simplest: install it, pull a model, and it serves an OpenAI-compatible API on :11434.

ollama pull qwen2.5:7b-instruct        # ~5 GB; runs on CPU or any recent GPU
ollama serve                           # serves http://localhost:11434/v1
# then in the stack's .env (containers reach the host via host.docker.internal):
REDEVOPS_LLM_BASE_URL=http://host.docker.internal:11434/v1

On Ubuntu without Docker Desktop, host.docker.internal isn't automatic — either run the stack with --add-host=host.docker.internal:host-gateway (the compose bundles already do this) or point REDEVOPS_LLM_BASE_URL at your host's LAN IP.

NVIDIA PAIR — models on another machine you own

Have a spare box with a GPU? NVIDIA Personal AI Router (PAIR) routes inference across trusted local machines — the workstation serves the model, the laptop runs the stack, and PAIR is the fabric between them. ReDevOps ships a thin provider adapter over PAIR, so the stack sees it as just another endpoint. Set REDEVOPS_LLM_BASE_URL to the PAIR router's OpenAI-compatible URL and every agent's inference lands on the GPU box — nothing leaves your network.

PAIR is an optional inference fabric; the Runtime does not depend on it and stays provider-agnostic. Where a request runs — a local device via PAIR, or a cloud provider — is a Mission-policy decision, gated and recorded like any other capability. NVIDIA does not endorse or certify ReDevOps.

A local vLLM — a bigger model on a local GPU

For a larger or quantized model on a workstation GPU, serve it with vLLM (also OpenAI-compatible) and point the stack at it:

vllm serve Qwen/Qwen2.5-7B-Instruct --port 8001
REDEVOPS_LLM_BASE_URL=http://host.docker.internal:8001/v1

Set nothing — a guided default to start

If you leave REDEVOPS_LLM_BASE_URL empty, onboarding walks you through a free-tier option so you can see the stack work before you commit hardware — then swap in a local endpoint above whenever you're ready. Resolution order is deliberate: your own endpoint → PAIR on your network → a model on this machine → a guided fallback. The stack never silently phones a cloud you didn't choose.

Sizing a local model: a small 7–8B model runs on ~8 GB of RAM (CPU) or any recent GPU; larger models want a GPU with 16 GB+ of VRAM. If this machine can't host a model, point the endpoint at another box (PAIR) or a free tier and the stack runs the same.

Multiple instances, one database and one model

Local doesn't mean single-machine. Several installs can act as one system: point them at a shared database and a shared model endpoint, and they coordinate through the same durable event log — one identity, one evidence trail, one replayable history across all of them.

# .env on every instance that should join the same system
POSTGRES_DSN=postgresql://redevops:...@db-host:5432/agentic
REDEVOPS_LLM_BASE_URL=http://gpu-host:8001/v1

Verify the model

  1. Model reachable: from a shell, curl $REDEVOPS_LLM_BASE_URL/models lists your model. If this fails, nothing else will — fix the endpoint first.
  2. Services healthy: docker compose ps shows every service healthy/running.
  3. Cockpit live: open the console from your install (the core install and apps install pages give the port), launch the sample mission, and watch it plan → gate on your approval → execute → leave evidence.

Troubleshooting the model layer

SymptomCause & fix
Agents error on every requestThe model endpoint is unreachable. curl $REDEVOPS_LLM_BASE_URL/models from the host; if a container can't reach a host model, use host.docker.internal (Docker Desktop) or --add-host=host.docker.internal:host-gateway (Ubuntu).
GPU not used by a local modelOn Ubuntu, install the NVIDIA Container Toolkit; on macOS, run the model on the host (Ollama/LM Studio) — Docker can't pass the Apple GPU into a container.
Out of memoryThe model, not the stack, is the hog. Use a smaller/quantized model, or move it to another box via PAIR and keep only the stack local.
Slow model calls on WindowsRun the stack inside WSL2 with the repo on the Linux filesystem (~/agentic-os-stack), not /mnt/c/… — bind-mount performance across the Windows filesystem is much slower.

Questions, or stuck on a step? Ask in #agentic-apps on the ReDevOps Slack, or email info@redevops.io — the team and community help with deploys, connectors, sources and governance there.

redevops.io · Run ReDevOps on your own models. Your data and prompts stay on hardware you control.  ·  deploy the apps  ·  #agentic-apps on Slack  ·  info@redevops.io