Self-hosting a Hermes model on a VPS with Ollama

Self-hosting an open language model answers three concrete needs: keeping data on your own machine, having a fixed cost rather than a metered one, and depending on no provider for availability. Nous Research's Hermes models suit the exercise well, they are fine-tunes of open models, tuned for instruction following and tool calling, and distributed as open weights.

Let's start with the part most tutorials skip: what you can realistically expect from a VPS with no GPU.

1. Size it before you install it

Two constraints decide everything: RAM has to hold the model's weights, and throughput depends on memory bandwidth, not CPU clock speed. Rough figures for Q4_K_M quantization, the most common:

3B, about 2 GB of weights, 4 GB of usable RAM. Comfortable, but limited at reasoning.
8B, about 4.7 GB of weights, 6 to 8 GB of usable RAM. The sweet spot on a VPS.
14B, about 9 GB of weights, 12 GB of usable RAM. Markedly slower on CPU.
70B and beyond, out of reach without a GPU. Don't waste your time.

On throughput, expect a few tokens per second for an 8B on 4 vCPUs, roughly a thirty-second to one-minute answer. That is perfectly usable for background processing, classification, extraction, or a lightly-used internal assistant. It is not usable for a consumer-facing chat interface. If you need speed and reasoning capability, a remote API stays cheaper; self-hosting earns its place on privacy and cost predictability.

Context costs memory on top of the weights. The attention cache grows with conversation length: on an 8B, budget 1 to 2 GB extra for 8,000 tokens of context. Size the VM on weights + cache + system, not on weights alone.

2. Install Ollama

Ollama wraps llama.cpp in an HTTP API and a model manager. It is the shortest path from a bare VPS to a model that answers.

bash

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama --no-pager

The installer creates a systemd service and a dedicated user. By default the daemon listens on 127.0.0.1:11434 only, which is exactly right: Ollama's API has no authentication and must never be exposed directly.

3. Pull a Hermes model

The Ollama library carries the Hermes 3 generation under the name hermes3, in several sizes:

bash

ollama pull hermes3:8b
ollama list
ollama run hermes3:8b "Summarize what a KVM VPS is in three points."

For a newer generation or a specific quantization, Ollama can pull a GGUF straight from Hugging Face, useful for models Nous Research publishes before they land in the library:

bash

# syntax: hf.co/<organization>/<repo>:<quantization>
ollama pull hf.co/NousResearch/Hermes-4-14B-GGUF:Q4_K_M

Check the exact repository name and the available quantizations on the model's page before starting the download: filenames vary between releases, and a pull that fails after several gigabytes is irritating.

4. Tune the daemon

Three environment variables actually change something. Set them in a systemd override, not in the unit file the package ships.

bash

sudo systemctl edit ollama.service

systemd

[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MODELS=/var/lib/ollama/models"

bash

sudo systemctl daemon-reload
sudo systemctl restart ollama

OLLAMA_KEEP_ALIVE, how long the model stays in memory after a request. The default is short: spaced-out calls then pay the full reload of the weights from disk each time. On a lightly-used service that must answer quickly, lengthen it; on a memory-tight VM, shorten it.
OLLAMA_NUM_PARALLEL, how many requests are handled concurrently. On CPU, going above 1 buys you nothing: the requests share the same cores and each one slows down accordingly.
OLLAMA_MODELS, where the weights live. Models are large; point this at the volume that has the room.

5. Use the API

Ollama exposes both a native API and an OpenAI-compatible one, which lets most existing libraries connect unmodified.

bash

# native API
curl -s http://127.0.0.1:11434/api/chat -d '{
  "model": "hermes3:8b",
  "stream": false,
  "messages": [{"role": "user", "content": "Hello"}]
}' | jq -r '.message.content'

# OpenAI-compatible API, same host
curl -s http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "hermes3:8b", "messages": [{"role": "user", "content": "Hello"}]}' \
  | jq -r '.choices[0].message.content'

Measure real throughput before building anything on top of it. The native API's response carries the counters that matter:

bash

curl -s http://127.0.0.1:11434/api/chat -d '{
  "model": "hermes3:8b", "stream": false,
  "messages": [{"role": "user", "content": "Write a paragraph about IPv6 routing."}]
}' | jq '{
  tokens: .eval_count,
  seconds: (.eval_duration / 1000000000),
  tokens_per_s: (.eval_count / (.eval_duration / 1000000000))
}'

eval_count is the number of tokens produced and eval_duration the time spent producing them, in nanoseconds. The ratio is your real throughput, the number to weigh against your expectations before going further.

6. Expose the service, properly

Ollama's API has no authentication, no rate limiting, and no quota. Publishing port 11434 to the internet hands your CPU to the first scanner that finds it. The right build is an Nginx proxy that terminates TLS and demands a token.

bash

sudo apt install -y apache2-utils
sudo htpasswd -c /etc/nginx/ollama.htpasswd agent

nginx

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name llm.example.com;

    auth_basic           "llm";
    auth_basic_user_file /etc/nginx/ollama.htpasswd;

    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_buffering off;          # required for streamed responses
        proxy_read_timeout 600s;      # a long generation outlives the default
    }
}

The two lines that matter are the last two. Without proxy_buffering off, Nginx accumulates the whole response before forwarding it and streaming becomes pointless; without a raised proxy_read_timeout, a long response is cut off mid-sentence by the default 60-second timeout.

Add the certificate with Certbot, open port 443 in UFW, and leave 11434 closed from outside. A check from another machine confirms the build:

bash

nmap -Pn -p 11434,443 llm.example.com

7. When it's worth it, and when it isn't

After a few weeks of running it, the question resolves simply. Self-hosting wins when the data must not leave, when volume is steady and predictable, or when the task is simple, classification, extraction, rewriting, short summaries. It loses when you need the reasoning capability of large models, when traffic is bursty, or when latency is visible to a user.

Nothing forces a choice: routing simple tasks to your local Hermes and hard ones to a remote API is a common build, and the first usually absorbs most of the volume.

Checklist

RAM sized on weights + attention cache + system.
Real throughput measured with eval_count and eval_duration before building on it.
Daemon bound to 127.0.0.1, never exposed directly.
OLLAMA_KEEP_ALIVE matched to the real call frequency.
Nginx proxy with TLS, authentication, buffering off, and a raised timeout.
Port 11434 verified closed from outside.
A snapshot taken once the model is downloaded, it saves downloading it again.