Run AI on hardware you own —
without losing reliability.

One endpoint, two execution paths

Your application keeps calling an OpenAI-style chat/completions API with one key. Fallbakit decides — per request — whether to serve from your local runtime or a cloud provider, and handles health checks, fallback, and usage tracking behind the scenes.

Before you start

You'll want the following ready. Everything except the account is optional depending on how far you go.

• A Fallbakit account — sign in at fallbakit.com.
• A local model runtime for local-first routing — Ollama (easiest), oMLX, or vLLM. Ollama listens on :11434; oMLX and vLLM on :8000.
• Your app's language toolchain — Python 3.9+ or Node.js 18+ for the SDKs.
• (Optional) A cloud provider key — OpenAI, Gemini, Anthropic, or OpenRouter — for BYOK fallback.
• (Optional) Docker — if you'd rather run the tunnel agent as a container or on a server.

Pull a local model (Ollama)

# install Ollama from https://ollama.com, then pull a small model
ollama pull llama3.2

# confirm it serves on :11434
curl http://localhost:11434/api/tags

Local-first in five minutes

The fast path: create an app key, connect one local runner with the CLI, then send a request. The CLI does the runner setup and launches the agent for you.

1. Install the CLI and sign in

   curl -fsSL https://fallbakit.com/install.sh | sh
   fallbakit login

   login opens your browser for device-flow auth and stores a cli_… token in ~/.fallbakit.

2. Create a runner and launch the agent

   fallbakit runner create --runtime ollama --launch

   This registers a runner, downloads the tunnel agent, and connects it to your local Ollama on :11434. Leave it running — it's your local-first path.

3. Scaffold an app + API key into your project

   cd my-project
   fallbakit app init

   Detects Python/Node, creates an application and an or_… key, and writes FALLBAKIT_API_KEY + FALLBAKIT_BASE_URL into a git-ignored .env.

4. Send your first request
   Python Node.js curl

   import os
   from fallbakit import Fallbakit
   
   client = Fallbakit(api_key=os.environ["FALLBAKIT_API_KEY"])
   
   response = client.chat.completions.create(
       model="llama3.2",
       messages=[{"role": "user", "content": "Say hello from my own hardware."}],
   )
   print(response["choices"][0]["message"]["content"])

   import { Fallbakit } from "@fallbakit/sdk";
   
   const client = new Fallbakit({ apiKey: process.env.FALLBAKIT_API_KEY });
   
   const response = await client.chat.completions.create({
     model: "llama3.2",
     messages: [{ role: "user", content: "Say hello from my own hardware." }],
   });
   console.log(response.choices[0].message.content);

   curl https://api.fallbakit.com/v1/chat/completions \
     -H "Authorization: Bearer $FALLBAKIT_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"model":"llama3.2","messages":[{"role":"user","content":"Say hello from my own hardware."}]}'

Set up in the dashboard

Prefer clicking to typing? Everything the CLI does, you can do in the dashboard at fallbakit.com. This is also where API keys, runner status, provider credentials, billing, and usage live.

1. Create an application

   Dashboard → Applications → New application. An application groups keys, requests, and usage for one app or service.

2. Copy the application API key

   The key is prefixed or_… and shown once — store it now. This is what your app and the SDKs use. Set it as an environment variable rather than hard-coding it:

   export FALLBAKIT_API_KEY=or_your_generated_api_key

3. Create a runner (for local-first routing)

   Dashboard → Runners → New runner. Pick the runtime (Ollama / oMLX / vLLM). You'll get a rr_… runner key and a FALLBAKIT_RUNNER_ID. These belong to the tunnel agent, not your app.

4. Add a cloud provider (for fallback)

   Dashboard → Providers → Add provider. Paste your own OpenAI / Gemini / Anthropic / OpenRouter key (BYOK). Fallbakit never resells tokens — the provider bills you directly.

5. Generate a CLI token (optional, for CI)

   Dashboard → Settings → CLI tokens for a cli_… token you can pass to fallbakit login --token on servers where a browser isn't available.

The fallbakit CLI

The CLI manages runners, applications, keys, and insights from your terminal. It talks to the dashboard management API with your cli_… token.

Install

curl -fsSL https://fallbakit.com/install.sh | sh
# installs to ~/.fallbakit/bin — add it to your PATH if needed

brew install --cask fallbakit/homebrew-tap/fallbakit

go install github.com/fallbakit/cli/cmd/fallbakit-cli@latest

Authenticate

fallbakit login                     # opens a browser (device flow)
fallbakit login --token cli_xxx     # non-interactive, for CI / servers
fallbakit whoami                    # show the signed-in user + account + plan
fallbakit logout                    # remove stored credentials

Runners

fallbakit runner create --runtime ollama --launch     # create + connect in one step
fallbakit runner create --name "GPU box" --runtime vllm --target docker \
  --local-url http://localhost:8000
fallbakit runner up <runner-id>      # launch the agent for an existing runner
fallbakit runner list                # alias: ls
fallbakit runner status              # live status of every runner
fallbakit runner status <id> --watch # diagnose one runner, refreshing every 3s
fallbakit runner rotate <id>         # rotate its rr_ key
fallbakit runner rm <id>             # delete it

runner create flags: --name, --runtime (ollama|omlx|vllm), --target (binary|docker), --local-url, --launch. It saves credentials to ~/.fallbakit/runners/<id>.json and prints the runner env block.

Applications & keys

fallbakit app init                   # detect project, create app + or_ key, write .env
fallbakit app init --name billing-api
fallbakit app init --no-files        # mint the key only, don't touch project files
fallbakit app list                   # alias: ls
fallbakit app enable  <app-id>
fallbakit app disable <app-id>
fallbakit app key create --app <id> --name "prod key"
fallbakit app key rm <key-id> --app <id>

Insights

fallbakit requests                          # last 20 requests, default app, 24h
fallbakit requests --limit 50 --range 7d
fallbakit requests --app billing-api --watch
fallbakit usage --range 30d                 # totals: local vs cloud, cost, savings
fallbakit usage --app billing-api --json

--range accepts 24h, 7d, or 30d. Add --json for raw output, --watch to refresh live.

Config

fallbakit config path                                  # ~/.fallbakit
fallbakit config get                                   # resolved dashboard + API URLs
fallbakit config set apiBaseUrl https://api.fallbakit.com
fallbakit version

Connect a local runner (tunnel agent)

The open-source fallbakit-agent runs next to your model and opens an outbound WebSocket tunnel to Fallbakit. Your runtime is never exposed to the internet — no inbound ports, no public IP. If you ran fallbakit runner create --launch, the CLI already did this; the manual setup below is for servers and containers.

Install the agent

# download the archive for your OS/arch from the releases page, then:
tar -xzf fallbakit-agent_<version>_<os>_<arch>.tar.gz
sha256sum -c checksums.txt --ignore-missing
sudo mv fallbakit-agent /usr/local/bin/
fallbakit-agent -h

docker pull ghcr.io/fallbakit/fallbakit-agent:latest

git clone https://github.com/Fallbakit/tunnel.git
cd tunnel
go build -trimpath -ldflags="-s -w" -o fallbakit-agent ./cmd/fallbakit-agent
./fallbakit-agent -h

Run it

Set the runner credentials from the dashboard, pick your runtime, and start. Pick the tab for your local runtime:

export FALLBAKIT_RUNNER_ID=runner_from_dashboard
export FALLBAKIT_RUNNER_API_KEY=rr_from_dashboard
export FALLBAKIT_BASE_URL=https://api.fallbakit.com
export FALLBAKIT_LOCAL_PROVIDER=ollama
export FALLBAKIT_LOCAL_BASE_URL=http://localhost:11434

fallbakit-agent

export FALLBAKIT_RUNNER_ID=runner_from_dashboard
export FALLBAKIT_RUNNER_API_KEY=rr_from_dashboard
export FALLBAKIT_BASE_URL=https://api.fallbakit.com
export FALLBAKIT_LOCAL_PROVIDER=omlx
export FALLBAKIT_LOCAL_BASE_URL=http://localhost:8000

fallbakit-agent

export FALLBAKIT_RUNNER_ID=runner_from_dashboard
export FALLBAKIT_RUNNER_API_KEY=rr_from_dashboard
export FALLBAKIT_BASE_URL=https://api.fallbakit.com
export FALLBAKIT_LOCAL_PROVIDER=vllm
export FALLBAKIT_LOCAL_BASE_URL=http://localhost:8000
export FALLBAKIT_LOCAL_API_KEY=optional_vllm_api_key   # only if vLLM was started with --api-key

fallbakit-agent

Configuration reference

Precedence is CLI flags → environment → YAML file → defaults. Point at a YAML file with -config <path> or FALLBAKIT_AGENT_CONFIG. Durations use Go syntax (1s, 500ms, 2m).

YAML config file

# agent.yaml — run with: fallbakit-agent -config agent.yaml
api_key: "rr_from_dashboard"
runner_id: "runner_from_dashboard"
base_url: "https://api.fallbakit.com"
local_provider: "ollama"          # ollama, omlx, or vllm
local_base_url: "http://localhost:11434"
local_api_key: ""                 # forwarded only agent -> local runtime
agent_id: "workstation-01"
metrics_addr: ":9093"
log_format: "json"
min_backoff: "1s"
max_backoff: "30s"
connect_timeout: "10s"
local_timeout: "3s"

Local runtimes at a glance

Add cloud fallback (BYOK)

Fallback keeps you reliable when local can't serve — the runtime is offline, the model is missing, or the request times out. You bring your own provider key; the provider bills you directly.

1. Add a provider in the dashboard

   Dashboard → Providers → Add provider, then paste your own key for OpenAI, Gemini, Anthropic, or OpenRouter.

2. Choose fallback behavior

   Fallback is on by default. You can steer it per request (see Routing controls) — force local, disable fallback, or pin a specific fallback provider and model.

3. Set spend limits (optional)

   Cap BYOK spend in the dashboard so runaway fallback can't surprise your cloud bill. Estimated provider cost and estimated local savings both show up in usage.

Call the API from your app

Use the Fallbakit SDK, or point the official OpenAI SDK at Fallbakit — the endpoint is OpenAI-compatible. Either way you pass one or_… key and hit POST /v1/chat/completions.

Install & first call

pip install fallbakit

import os
from fallbakit import Fallbakit

client = Fallbakit(api_key=os.environ["FALLBAKIT_API_KEY"])

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Write a tiny launch checklist."}],
)
print(response["choices"][0]["message"]["content"])

npm install @fallbakit/sdk

import { Fallbakit } from "@fallbakit/sdk";

const client = new Fallbakit({ apiKey: process.env.FALLBAKIT_API_KEY });

const response = await client.chat.completions.create({
  model: "llama3.2",
  messages: [{ role: "user", content: "Write a tiny launch checklist." }],
});
console.log(response.choices[0].message.content);

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["FALLBAKIT_API_KEY"],
    base_url="https://api.fallbakit.com/v1",
)

completion = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Write a tiny launch checklist."}],
)
print(completion.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.FALLBAKIT_API_KEY,
  baseURL: "https://api.fallbakit.com/v1",
});

curl https://api.fallbakit.com/v1/chat/completions \
  -H "Authorization: Bearer $FALLBAKIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [{ "role": "user", "content": "Write a tiny launch checklist." }]
  }'

Streaming

for chunk in client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Stream a short answer."}],
    stream=True,
):
    delta = chunk["choices"][0].get("delta", {})
    print(delta.get("content", ""), end="", flush=True)
print()

const stream = await client.chat.completions.create({
  model: "llama3.2",
  messages: [{ role: "user", content: "Stream a short answer." }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

Timeouts & error handling

from fallbakit import Fallbakit, FallbakitAPIError, FallbakitError

client = Fallbakit(api_key=os.environ["FALLBAKIT_API_KEY"], timeout=30)  # seconds, client-wide

try:
    client.chat.completions.create(
        model="llama3.2",
        messages=[{"role": "user", "content": "Hello"}],
        timeout=10,  # per-request override
    )
except FallbakitAPIError as error:
    print(error.status_code, error.code, error)
except FallbakitError as error:
    print("client error:", error)

import { Fallbakit, FallbakitAPIError, FallbakitError } from "@fallbakit/sdk";

const client = new Fallbakit({ apiKey: process.env.FALLBAKIT_API_KEY, timeoutMs: 30_000 });

try {
  await client.chat.completions.create({
    model: "llama3.2",
    messages: [{ role: "user", content: "Hello" }],
    timeoutMs: 10_000, // per-request override
  });
} catch (error) {
  if (error instanceof FallbakitAPIError) {
    console.error(error.status, error.code, error.message);
  } else if (error instanceof FallbakitError) {
    console.error("client error:", error.message);
  } else {
    throw error;
  }
}

Routing & fallback controls

By default Fallbakit routes local-first and falls back to cloud when needed. Override that per request. Python uses snake_case keyword arguments; Node uses camelCase params.

# Pin the fallback target and keep local-first
client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Summarize local-first routing."}],
    fallback_provider="openai",
    fallback_model="gpt-4o-mini",
)

# Sensitive prompt — never leave local hardware
client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Redact this PII..."}],
    local_model_only=True,
)

// Grouped form (Node-only): everything under `fallbakit`
await client.chat.completions.create({
  model: "llama3.2",
  messages: [{ role: "user", content: "Summarize local-first routing." }],
  fallbakit: {
    fallbackProvider: "gemini",
    fallbackModel: "gemini/gemini-1.5-flash",
  },
});

await client.chat.completions.create({
  model: "llama3.2",
  messages: [{ role: "user", content: "Summarize local-first routing." }],
  extra_body: {
    fallbakit: { fallbackProvider: "openai", fallbackModel: "gpt-4o-mini" },
  },
});

See where every request ran

Fallbakit tracks each route decision, local-vs-cloud split, estimated provider cost, estimated local savings, latency, and full request history. Read it from the CLI or the dashboard.

fallbakit requests --range 7d        # TIME · ROUTE · MODEL · STATUS · LATENCY · TOKENS · COST
fallbakit usage --range 30d          # totals: requests, local, cloud, errors, spend, savings

Diagnostic response headers help when debugging a single call:

Run the agent on a server

For always-on local-first routing, run the tunnel agent as a container or service next to your model. It restarts and reconnects on its own with exponential backoff.

docker run --rm \
  -e FALLBAKIT_RUNNER_ID=runner_from_dashboard \
  -e FALLBAKIT_RUNNER_API_KEY=rr_from_dashboard \
  -e FALLBAKIT_BASE_URL=https://api.fallbakit.com \
  -e FALLBAKIT_LOCAL_PROVIDER=ollama \
  -e FALLBAKIT_LOCAL_BASE_URL=http://host.docker.internal:11434 \
  -e FALLBAKIT_METRICS_ADDR=:9093 \
  -p 9093:9093 \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/fallbakit/fallbakit-agent:latest

services:
  fallbakit-agent:
    image: ghcr.io/fallbakit/fallbakit-agent:latest
    environment:
      FALLBAKIT_RUNNER_ID: ${FALLBAKIT_RUNNER_ID:?set FALLBAKIT_RUNNER_ID}
      FALLBAKIT_RUNNER_API_KEY: ${FALLBAKIT_RUNNER_API_KEY:?set FALLBAKIT_RUNNER_API_KEY}
      FALLBAKIT_BASE_URL: ${FALLBAKIT_BASE_URL:-https://api.fallbakit.com}
      FALLBAKIT_LOCAL_PROVIDER: ${FALLBAKIT_LOCAL_PROVIDER:-ollama}
      FALLBAKIT_LOCAL_BASE_URL: ${FALLBAKIT_LOCAL_BASE_URL:-http://host.docker.internal:11434}
      FALLBAKIT_METRICS_ADDR: ":9093"
      FALLBAKIT_LOG_FORMAT: json
    ports:
      - "9093:9093"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped

export FALLBAKIT_RUNNER_ID=runner_from_dashboard
export FALLBAKIT_RUNNER_API_KEY=rr_from_dashboard
docker compose up -d

# /etc/fallbakit/agent.env
FALLBAKIT_RUNNER_ID=runner_from_dashboard
FALLBAKIT_RUNNER_API_KEY=rr_from_dashboard
FALLBAKIT_BASE_URL=https://api.fallbakit.com
FALLBAKIT_LOCAL_PROVIDER=ollama
FALLBAKIT_LOCAL_BASE_URL=http://localhost:11434
FALLBAKIT_METRICS_ADDR=:9093
FALLBAKIT_LOG_FORMAT=json

# /etc/systemd/system/fallbakit-agent.service
[Unit]
Description=Fallbakit Tunnel Agent
After=network-online.target
Wants=network-online.target

[Service]
EnvironmentFile=/etc/fallbakit/agent.env
ExecStart=/usr/local/bin/fallbakit-agent
Restart=always
RestartSec=2
DynamicUser=yes
NoNewPrivileges=yes

[Install]
WantedBy=multi-user.target

sudo chmod 600 /etc/fallbakit/agent.env
sudo systemctl daemon-reload
sudo systemctl enable --now fallbakit-agent
journalctl -u fallbakit-agent -f

# Secret + ConfigMap + Deployment (metrics on :9093).
# Liveness -> /healthz, readiness -> /readyz on the metrics port.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fallbakit-agent
spec:
  replicas: 1
  selector:
    matchLabels: { app.kubernetes.io/name: fallbakit-agent }
  template:
    metadata:
      labels: { app.kubernetes.io/name: fallbakit-agent }
    spec:
      containers:
        - name: agent
          image: ghcr.io/fallbakit/fallbakit-agent:latest
          envFrom:
            - configMapRef: { name: fallbakit-agent }
          env:
            - name: FALLBAKIT_RUNNER_API_KEY
              valueFrom:
                secretKeyRef: { name: fallbakit-agent, key: runner-api-key }
          ports:
            - { name: metrics, containerPort: 9093 }
          readinessProbe:
            httpGet: { path: /readyz, port: metrics }
          livenessProbe:
            httpGet: { path: /healthz, port: metrics }

kubectl apply -f fallbakit-agent.yaml
kubectl rollout status deployment/fallbakit-agent

Health endpoints

Served only when FALLBAKIT_METRICS_ADDR is set (e.g. :9093):

Self-hosted (Standalone edition)

Security-driven and regulated teams can license Fallbakit Standalone — the same engine, self-hosted entirely inside your own network.

Troubleshooting

Quick reference

Endpoints & URLs

Environment variables

Ports