Goethe user guide
Run and share your models on your own infrastructure in three steps, and run them fast.
Goethe turns a Linux machine into your own API, compatible with OpenAI, Anthropic and LiteLLM clients. Install it, add a model, and point your apps, your team or your other machines at one address. Your prompts, your data and your models stay on hardware you control.
Contents
- What you can use it for
- Quick install
- API compatibility
- CLI
- Add models
- Load and run
- Share the API
- Hardware requirements
- Software requirements
- Where it runs
- Supported models
- Settings
- Troubleshooting
What you can use it for
- Easy to serve. Every model you load is served at the same URL; a request picks its model by name.
- Fast. Use the custom kernel attachments to accelerate and optimise model capability.
- A private API for your team. One key per person or app; nothing reaches the API until you allow it.
- Home or office over a VPN. Serve from a GPU box and use it from your laptop through your VPN.
- Your own apps and tools. Point OpenAI, Anthropic or LiteLLM clients, Open WebUI, coding assistants and similar tools at hardware you control.
- Many models, many GPUs. Compare models by name on the same address, split large models across GPUs, or run on cloud GPUs when you need them.
Quick install
curl -fsSL https://meshapplied.com/install.sh | bash
goethe login
goethe start
goethe login links the machine to the Goethe platform, which also unlocks optimised kernels; it is optional. goethe start starts Goethe and walks you through three steps: install a model, serve it through the API, and configure who can reach it. You can stop after any step and continue later with goethe start. If you prefer plain commands:
# 1. Start Goethe and load a model
goethe up
goethe install zai-org/GLM-5.3-Flash --load
# 2. Share it
goethe api configure # this machine only, your VPN (recommended), or a public address
goethe api manage # add, scope or delete API keys in a menu
goethe api # the address to give your apps
goethe status tells you whether Goethe is running, with the API address and your keys. Check the machine with goethe doctor: each WARN or FAIL line names its fix. See the hardware and software requirements before you install.
API compatibility
The API is at https://<address>:<port>/v1. It speaks three schemas at once, so OpenAI, Anthropic and LiteLLM clients all work against the same address and key:
- OpenAI: chat, completions, responses, embeddings, rerank and models.
- Anthropic:
POST /v1/messagesand its token count. Errors on these routes use Anthropic's format. - LiteLLM proxy: every route also works without
/v1, plus/model/info,/model_group/info,/health/livelinessand/health/readiness.
| Endpoint | Purpose |
|---|---|
POST /v1/chat/completions |
Chat, with streaming, tools and reasoning |
POST /v1/completions |
Plain text completion |
POST /v1/responses |
OpenAI Responses API |
POST /v1/messages |
Anthropic Messages API, with streaming |
POST /v1/messages/count_tokens |
Anthropic token count |
POST /v1/embeddings |
Embeddings, for embedding models |
POST /v1/rerank |
Reranking, for reranker models |
GET /v1/models, GET /v1/models/{id} |
Models your key can use |
GET /model/info, GET /model_group/info |
Models in LiteLLM's format |
GET /health |
Up check, no key needed |
Send the key in whichever header your client uses: Authorization: Bearer <key> (OpenAI), x-api-key (Anthropic), x-litellm-api-key (LiteLLM) or api-key (Azure style).
The API uses HTTPS with a certificate Goethe creates for itself. Clients either skip verification (curl -k, verify=False) or trust <goethe folder>/nginx/certs/server.crt.
curl
curl -k https://<address>:8443/v1/chat/completions \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"model":"zai-org/GLM-5.3-Flash","messages":[{"role":"user","content":"Hello!"}],"max_tokens":128}'
OpenAI SDK
import httpx
from openai import OpenAI
client = OpenAI(
base_url="https://<address>:8443/v1",
api_key="goethe_sk_...",
http_client=httpx.Client(verify=False), # or verify="/path/to/server.crt"
)
stream = client.chat.completions.create(
model="zai-org/GLM-5.3-Flash",
messages=[{"role": "user", "content": "Explain mixture-of-experts in one paragraph."}],
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="", flush=True)
Anthropic SDK
import httpx
from anthropic import Anthropic
client = Anthropic(
base_url="https://<address>:8443",
api_key="goethe_sk_...",
http_client=httpx.Client(verify=False),
)
reply = client.messages.create(
model="zai-org/GLM-5.3-Flash",
max_tokens=256,
messages=[{"role": "user", "content": "Hello!"}],
)
print(reply.content[0].text)
LiteLLM
import litellm
reply = litellm.completion(
model="litellm_proxy/zai-org/GLM-5.3-Flash",
api_base="https://<address>:8443",
api_key="goethe_sk_...",
messages=[{"role": "user", "content": "Hello!"}],
ssl_verify=False,
)
Or add Goethe to a LiteLLM proxy's config.yaml as an OpenAI-compatible backend (model: openai/zai-org/GLM-5.3-Flash, api_base: https://<address>:8443/v1).
Other tools
Open WebUI, LangChain, LlamaIndex, coding assistants and any tool that takes a custom OpenAI or Anthropic endpoint work the same way: set the base URL to https://<address>:<port>/v1 (Anthropic clients: without /v1), your Goethe key, and a model name from goethe api. If a tool can't skip certificate checks, add Goethe's certificate to the system's trusted certificates.
Limits
- Each client address can send up to 20 requests per second.
- Each key has its own per-minute limit, set with
--rpm. - Request bodies can be up to 10 MB.
- When a model unloads, requests in progress finish first; new requests get
503with aRetry-Afterheader.
CLI
The model is a plain argument (goethe load qwen 27b: an id, an alias, or words from its name, with a menu when several match); every other value goes after its flag (goethe gpu --allow gpu0). API settings are under goethe api, model housekeeping under goethe models. Listing commands accept --json. Exit codes: 0 success, 1 error, 2 wrong usage, 3 a confirmation was needed without a terminal (add --yes). The command reference lists every command and option.
Everyday
goethe up # start; also starts at boot from now on
goethe status # running or not, API address and keys
goethe down # stop; won't start at boot until the next up
goethe ps # what is running
goethe api # address, keys and loaded models
goethe logs zai-org/GLM-5.3-Flash -f
goethe logs --daemon
goethe doctor
goethe hw profile
Models that were loaded when Goethe stopped come back when it starts again, including after a reboot. goethe service restore --off turns that off.
Start and stop
| Command | What it does |
|---|---|
goethe start |
Start, and guided setup: model, serve, API access |
goethe up |
Start Goethe and the API, and start at boot (--no-boot to skip) |
goethe status |
Running state, address and keys |
goethe down |
Stop Goethe and its models, and stop starting at boot |
goethe service |
enable, disable, status, restore --on/--off |
Models
| Command | What it does |
|---|---|
goethe models recommend |
Catalog models that fit your GPUs |
goethe install … |
Download from Hugging Face, or register a local folder |
goethe models scan |
Add folders you placed in the models directory |
goethe models |
Installed and catalog models, and whether each fits |
goethe models config … |
Default context, concurrency and GPUs for a model |
goethe remove … |
Unload and delete (or unregister a folder) |
Running
| Command | What it does |
|---|---|
goethe load … |
Load and wait until ready (--gpu, --tp, --ctx, --max-conc, --replica) |
goethe unload … |
Stop a model (--all, --force) |
goethe ps |
Running instances, GPUs and speed |
goethe logs [model] |
--daemon, --gateway, -f, -n |
Sharing
| Command | What it does |
|---|---|
goethe api configure |
Local, VPN or public access |
goethe api |
Address, authentication, models, keys and a curl example |
goethe api manage |
Add, scope, show or delete API keys in a menu |
goethe api keys |
--create, --revoke, --rpm, --expires, --models, --admin |
goethe api allowlist |
--add / --remove client addresses |
goethe models alias |
Short model names for clients (--set, --remove) |
goethe api trace |
Record requests and replies as training traces (on, off, --key); see traces.md |
goethe api learn |
Let a session keep what compaction would drop (on, off, --model); see learn.md |
goethe api gateway |
Where the API listens (--bind 127.0.0.1, 0.0.0.0 or an IP) |
goethe api domain |
DNS record for your own hostname (--check <name>) |
goethe serve --public 5h --key <name> |
Open the API on a public address for a while for chosen keys (shutdown closes it) |
GPUs and system
| Command | What it does |
|---|---|
goethe gpu |
Memory, processes and bandwidth use, live until q; allow/deny (--deny, --allow) |
goethe hw profile |
GPUs, CPU, memory, disk and cloud provider |
goethe doctor |
Setup check with a fix for each problem |
goethe update |
Install the latest Goethe (--check to only look) |
goethe uninstall |
Remove Goethe and its models (--keep-models to keep them) |
goethe completion install |
Tab completion for bash, zsh or fish: commands, models, keys, GPUs |
goethe |
The home screen (on a terminal) |
goethe login |
Optional: link this node to the Goethe platform to see its health there (--authkey without a browser) |
goethe logout |
Unlink it again |
Add models
goethe models recommend # catalog models that fit your GPUs
goethe install zai-org/GLM-5.3-Flash # download it
goethe install zai-org/GLM-5.3-Flash --load # download and load in one step
The model id is the Hugging Face repo name. Any supported model on Hugging Face works, not only the catalog. Useful flags: --load, --ctx, --max-conc, --trust-remote-code (only for models you trust) and --hf-token.
Optimised kernels: some models have a kernel pack: the optimised kernels for every supported GPU microarchitecture, with the model's quantizations and configs (NVIDIA CUDA for now; other platforms such as AMD ROCm later). Packs are available to nodes logged in to the Goethe platform (goethe login), and goethe install uses them automatically. A node that is not logged in is asked to log in, or continues with SGLang. goethe models marks the models that have optimised kernels for your machine.
Gated or private models: accept the license on the model's Hugging Face page, create a read token at huggingface.co/settings/tokens, then run goethe install <org>/<model> --hf-token and type the token when asked. It is used for that download only and never stored. In goethe start, choose "A private or gated model".
Models you already have: move the folder into Goethe's models folder and run goethe models scan; it checks each folder and asks before adding it. A folder named <org>__<model> gets the id <org>/<model>. To register a folder where it is:
goethe install /data/models/my-finetune --name me/my-finetune
List and remove:
goethe models # installed and catalog models, and whether each fits
goethe remove zai-org/GLM-5.3-Flash # unload it and delete the files Goethe downloaded
For a folder you registered in place, remove only unregisters it.
Load and run
goethe load zai-org/GLM-5.3-Flash
Goethe picks the GPUs, sizes the memory and waits until the model answers:
RUNNING: zai-org/GLM-5.3-Flash instance inst-... on GPU 0 port 30001 (45.2 s)
The first load on a new machine takes a few extra minutes; later loads are faster. goethe load shows how the GPU memory is used, and goethe hw pick-gpus --model <model> shows the same without loading.
goethe load zai-org/GLM-5.3-Flash --gpu gpu1 # a specific GPU
goethe load zai-org/GLM-5.3-Flash --tp 2 # split across 2 GPUs Goethe picks
goethe load zai-org/GLM-5.3-Flash --gpu gpu0 gpu1 # split across GPUs 0 and 1
goethe load zai-org/GLM-5.3-Flash --ctx 65536 --max-conc 32 # longer context, more parallel requests
goethe load zai-org/GLM-5.3-Flash --replica # another copy, for more traffic
goethe load zai-org/GLM-5.3-Flash -- --chunked-prefill-size 4096 # extra engine options after --
goethe models config zai-org/GLM-5.3-Flash --ctx 65536 --max-conc 32 # make settings the default
Model names are forgiving: zai-org/glm-5.3-flash and zai-org__GLM-5.3-Flash mean the same model, and a mistyped name gets a suggestion.
Load as many models as your GPUs can hold. They share one API address, each request picks its model by name, and goethe ps shows what is running.
goethe unload zai-org/GLM-5.3-Flash # finish requests in progress, then stop
goethe unload --all # every model
goethe unload zai-org/GLM-5.3-Flash --force # stop now
Sharing a GPU with other software: if a GPU is already used by other programs, Goethe lists them and asks before loading next to them. It never stops processes it did not start. Add --yes in scripts. To keep Goethe off a GPU, use goethe gpu --deny gpu0; goethe gpu --allow gpu0 allows it again.
Share the API
goethe api configure
- Local only: programs on this machine.
- VPN or private network (recommended): connect your VPN first, then pick it from the list. Any VPN with a network interface works, and so does a cloud VM's private network. Goethe lists tunnels and interfaces on private addresses, leaves out container bridges and public network cards, and shows the range each one reaches, taken from the system's routes. When a VPN routes each device separately, the private block its address is in (such as
100.64.0.0/10or10.0.0.0/8) is used, so devices added later can connect too; you can narrow it to specific addresses. Default routes are never allowed. The gateway follows the interface: a new address after a reconnect or a reboot is picked up within a few seconds, and while the interface is down only this machine can connect (goethe statussays so). The allowlist itself only changes when you rungoethe api configureagain;goethe doctorwarns if it no longer lets the network's clients in. A VPN without an interface (userspace mode) can't be listened on directly: use local mode and the VPN's own port forwarding. - Public network: any address that can reach this machine. You open the port in your firewall yourself.
It then asks whether to restrict access to specific IP addresses, and offers to create the first key. In scripts: goethe api configure --mode vpn --yes, or --mode public --allow 203.0.113.7 --yes.
Keys
goethe api manage # full-screen: create, scope, limit or delete keys
goethe api keys --create my-laptop # shows the key once
goethe api keys --create ci --rpm 600 --expires 30d
goethe api keys --create chat-only --models zai-org/GLM-5.3-Flash
goethe api keys # keys, limits and today's usage
goethe api keys --revoke <key-id> # takes effect immediately
KEY=$(goethe api keys --create ci --raw) # only the key, for scripts
Every request needs a valid key; without one the API answers 401. A key can expire (--expires 30d or a date), have its own rate limit (--rpm, default 120 per minute, 0 for unlimited), be limited to some models (--models), or be an admin key that can read the health endpoints (--admin).
Allowed addresses
On top of keys, only allowed client addresses can connect. goethe api configure sets this up; to change it by hand:
goethe api allowlist
goethe api allowlist --add 192.168.1.0/24 --description "office LAN"
goethe api allowlist --add 203.0.113.7
goethe api allowlist --remove 192.168.1.0/24
Share for a while on a public address
To give someone outside your network access for a limited time, without changing how the API is normally reached:
goethe serve --public 5h --key my-app # prints the URL, the key and the certificate to send
goethe serve --public 1d # later: the same keys again
goethe serve --public shutdown # close early
The public address exists only while it is open, accepts only the keys you chose (never admin keys) and closes by itself when the time runs out. Keys stay the same every time. Details and the full list of security precautions: goethe serve --public.
Address, aliases and your own domain
goethe api # the address, models, keys and a curl command
goethe models alias --set chat --model zai-org/GLM-5.3-Flash # clients can send "model": "chat"
goethe api domain # the DNS record for your own hostname
goethe api domain --check api.example.com # check that the name points at the API
Change the model behind an alias at any time and every client follows. For a hostname such as api.example.com, add the DNS A record goethe api domain shows; clients then use https://api.example.com:8443/v1 with the same certificate settings as for the IP address.
Hardware requirements
Goethe runs on NVIDIA GPUs from the Ampere generation onwards:
| Architecture | Compute capability | GPUs |
|---|---|---|
| Blackwell | 12.0, 12.1 | GeForce RTX 5090, RTX PRO 6000 Blackwell, DGX Spark (GB10) |
| Blackwell data center | 10.0, 10.3 | B200, GB200, B300, GB300 |
| Hopper | 9.0 | H100, H200, GH200 |
| Ada | 8.9 | GeForce RTX 4090, RTX 6000 Ada, L40S, L40, L4 |
| Ampere | 8.0, 8.6 | A100, A30, A40, A10, RTX A6000, GeForce RTX 3090 |
Older GPUs (Volta, Turing and earlier) are not supported; Goethe names them and skips them. A machine can mix generations; each model runs on GPUs of one generation. Large models can be split across GPUs with --tp, and mixture-of-experts models can also use --ep. 4-bit NVFP4 models run on every supported GPU.
Goethe ships its own CUDA 13 runtime, so the machine only needs the NVIDIA driver:
| NVIDIA driver | CUDA it offers | Goethe runs on |
|---|---|---|
| 580 or newer | 13.0 and newer | Every supported GPU |
| 535 to 579 | 12.2 to 12.9 | Data-center GPUs (A100, H100, H200, B200, L40S, L4 and similar), through CUDA forward compatibility |
| Older than 535 | Not supported; the installer offers to update the driver |
GeForce and TITAN cards need driver 580 or newer. Check the driver with nvidia-smi. Enough GPU memory for the model is the other requirement; goethe models recommend lists what fits.
Software requirements
- Linux on x86_64 or ARM64 with glibc 2.30 or newer, for example Ubuntu 20.04+ or Debian 11+.
- Disk space for your models, plus about 2 GB for Goethe itself and about 8 GB more for SGLang, which installs once, with the first model that needs it.
- No CUDA toolkit: Goethe brings its own CUDA runtime and compiler.
The installer checks the machine first and installs missing pieces, such as a C++ compiler or the NVIDIA driver, after asking. GOETHE_INSTALL_DEPS=1 and GOETHE_INSTALL_DRIVER=1 answer yes in advance, =0 no. Inside a container it never touches the driver, which belongs to the host. A GPU problem never stops the install; goethe doctor names the fix afterwards:
What goethe doctor finds |
Fix it names |
|---|---|
| No NVIDIA GPU visible | Attach the GPU to the VM (PCI passthrough or vGPU), or start the container with --gpus all |
| A GPU but no driver loaded | The driver package for your distribution, then a reboot |
| A driver too old for this GPU | The same, to update it |
| The proprietary module with a Blackwell GPU | The open kernel module, which Blackwell needs |
| A GPU the driver did not start | A full power cycle, power cables, Above 4G decoding in the BIOS |
A driver but no /dev/nvidia* |
Pass the devices into the container, or sudo nvidia-modprobe -u -c=0 on the host |
| No C++ compiler | sudo apt install build-essential or sudo dnf install gcc-c++ |
After installing, open a new terminal if goethe is not found yet. Goethe keeps its files in /workspace/.goethe on cloud machines with a persistent volume, /var/lib/goethe as root, or ~/.goethe otherwise.
goethe update --check # is there a new version?
goethe update # install it; loaded models keep serving
goethe uninstall # remove Goethe and its models; add --keep-models to keep them
Where it runs
| Environment | Examples |
|---|---|
| Your own hardware | Workstations and servers with one or more NVIDIA GPUs, including DGX Spark |
| Cloud GPU virtual machines | AWS, Google Cloud, Azure, Oracle Cloud, Lambda, CoreWeave, Paperspace |
| GPU containers | RunPod, Vast.ai, and Docker with the NVIDIA Container Toolkit |
Goethe recognises the cloud provider or container it runs in, keeps its files on the persistent volume when there is one, and shows the address to use from outside. In a container, the GPU and its driver come from the host.
Supported models
Goethe serves models in the Hugging Face format (config.json plus .safetensors weights), across more than 200 architectures:
| Family | Examples |
|---|---|
| GLM | GLM-4, GLM-4.5, GLM-5.2, GLM-5.3 Flash |
| Qwen | Qwen 2.5, Qwen 3, Qwen3-Next, Qwen 3.5, Qwen3-VL |
| DeepSeek | DeepSeek V3, R1, V4 |
| Llama | Llama 3.x, Llama 4 |
| Mistral | Mistral, Mixtral, Mistral Small, Pixtral |
| Gemma | Gemma 2, 3 and 4 |
| Others | GPT-OSS, Kimi, MiniMax, Nemotron, Phi, Granite, Command R, OLMo, InternLM, InternVL, MiMo, EXAONE |
| Embeddings | BERT-style and LLM-based embedding and reranker models |
Dense, mixture-of-experts, hybrid (linear attention and Mamba) and vision-language models are supported. goethe install and goethe models scan say straight away if an architecture is not.
| Format | Typical use |
|---|---|
| BF16 / FP16 | Original weights |
| NVFP4 | 4-bit, about a quarter of BF16's size; every supported GPU |
| FP8 | 8-bit; native on Hopper and Blackwell |
| MXFP4 | GPT-OSS models |
| AWQ, GPTQ | 4-bit community quantizations |
| INT8 (W8A8) | 8-bit integer |
| Mixed precision | NVFP4 and FP8 in one model |
The format is detected from the model files. .bin (PyTorch pickle) weights are not supported; use the safetensors release of the model. GGUF models run only with optimised kernels (goethe-engine, logged-in nodes); goethe install says before downloading anything when a model is not supported.
Settings
Optional. Set them in your shell, or in goethe.env inside Goethe's folder so the background service picks them up.
| Variable | What it changes |
|---|---|
GOETHE_HOME |
Where Goethe keeps its files |
GOETHE_PUBLIC_PORT |
API port (default 8443; another is used if it's taken) |
GOETHE_NO_AUTOSTART=1 |
Never set up start at boot |
GOETHE_NO_COMPLETION=1 |
Don't turn on Tab completion at install |
GOETHE_PUBLIC_IP |
Public IP shown by goethe api configure in public mode |
GOETHE_MAX_BODY_MB |
Largest request size in MB (default 10) |
GOETHE_GATEWAY_RATE_PER_IP |
Requests per second one client address may send (default 100, bursts up to twice that) |
GOETHE_GATEWAY_CONN_PER_IP |
Open API connections one client address may hold (default 512) |
GOETHE_PLATFORM_URL |
Platform goethe login links to (default https://platform.meshapplied.com) |
GOETHE_PUBLIC_IP_URL |
Where goethe api configure, goethe api domain and goethe doctor look up the public IP (default: the platform's /api/ip; a plain-text or {"ip": ...} answer both work) |
GOETHE_PUBLIC_IP_LOOKUP=0 |
Never look up the public IP (set GOETHE_PUBLIC_IP to name it) |
NO_COLOR=1 |
Turn off colored output |
Troubleshooting
goethe: command not found. Open a new terminal, or run export PATH="$HOME/.local/bin:$PATH".
goethed is not running. Run goethe up; goethe status shows whether it is running.
Unknown model. Use an id from goethe models.
A load fails. Goethe prints the reason and the end of the model's log; goethe logs <model> -n 200 shows more. Common fixes: a smaller context (--ctx), fewer parallel requests (--max-conc), more GPUs (--tp 2), or a quantized version of the model; --trust-remote-code when the model needs its own code; goethe doctor for driver and GPU problems.
The first request is slow. The first request after a load can take a while; later requests are fast.
Requests fail:
| Answer | Meaning | Fix |
|---|---|---|
| No connection | Wrong address or port, http instead of https, or a firewall |
Use the address from goethe api |
403 |
Your address isn't allowed | goethe api allowlist --add <your-ip> |
401 |
Missing or wrong key | Send the key, for example Authorization: Bearer <key> |
404 Model Not Found |
Unknown model, or the key isn't allowed that model | Use a name from goethe api |
429 |
Too many requests | Slow down, or raise the key's --rpm |
503 |
The model isn't ready, or is unloading | Wait, or check goethe ps |
Anything else. Run goethe doctor, then goethe logs --daemon.