AI Middleware

The AI middleware is an OpenAI-compatible gateway that sits between the application and your LLM providers. Instead of talking to Azure OpenAI, Anthropic, Ollama or IBM watsonx directly, the application sends standard OpenAI chat-completion requests to the middleware, which translates them for the configured provider.

Three reasons this matters operationally:

• The provider credentials live in exactly one place — the middleware’s configuration directory. They never reach the application container, the database, or a browser.
• The provider can be changed without touching the application. Switching from a cloud model to an on-premise watsonx CPD deployment is a change in one YAML file.
• Documents are prepared and cached, not re-sent. Processing a rulemap presents the same documents to the model over and over. The middleware converts them once and serves them from its cache for every following call, which saves both processing time and tokens. This is what the persistent data/aimiddleware/ directory holds.

Where it runs

The middleware runs as the aimiddleware service in the same Compose project as the application, listening on port 8000.

Caller	Address
The application	http://aimiddleware:8000/rmsaimiddleware/… over the Compose network — see Connecting the application
Host, for troubleshooting	http://127.0.0.1:8000

Configuration files

The middleware reads two files from /app/config, which the Compose file mounts read-only from ./config:

File	Contents
config/credentials.yaml	Provider endpoints and keys, CORS, HTTP proxy, MLflow
config/api_keys.yaml	The API keys that clients must present to call the middleware

Both paths are fixed by the image through the environment variables RMS_CREDENTIALS_PATH and RMS_API_KEYS_PATH. Templates for both files ship as .example next to them.

Never copy these files between customers

credentials.yaml holds your LLM provider keys and api_keys.yaml controls who may use them. Both are per-installation and must be generated fresh — a copied file means another installation’s keys are live in yours.

credentials.yaml — providers

Each provider is one top-level block. The block name follows the convention provider_api_model:

# Azure OpenAI, chat completions
azure_completions_gpt4o:
  api_key: "…"
  endpoint: "https://your-resource.openai.azure.com"
  api_version: "2024-02-01"
  model_name: "azure/your-deployment-name"
  multimodal: true

# An on-premise OpenAI-compatible endpoint (vLLM, llama.cpp, gateway route)
ollama_completions_internal:
  endpoint: "https://internal-vllm.example.net/v1"
  model_name: "ollama/internal-model"
  api_key: "…"
  omit_model_param: true      # endpoint already points at a fixed model

model_name is the identifier clients use in the model field of a request. See credentials.yaml.example for the full list of supported providers (Anthropic, IBM watsonx, watsonx CPD, Ollama) and their required fields.

credentials.yaml — CORS

Only relevant if browsers call the middleware directly. Restrict it before going live:

cors:
  allowed_origins:
    - "https://<prefix>.<your-domain>"
  allow_credentials: true
  allow_methods: ["GET", "POST"]
  allow_headers: ["*"]

The shipped example contains allowed_origins: ["*"], which is a development default and must not stay in production.

credentials.yaml — network proxy

If the middleware’s outbound LLM calls have to pass through a proxy, configure it here.

proxy:
  enabled: true
  http_proxy: "http://proxy.company.com:8080"
  https_proxy: "http://proxy.company.com:8080"
  no_proxy: "localhost,127.0.0.1,.internal.company.com"
  ssl_verify: true
  ca_cert_path: "/etc/ssl/certs/company-ca-bundle.crt"

If your proxy performs TLS inspection, ca_cert_path must point at its CA bundle inside the container. Mount it in, for example by adding a volume line to the aimiddleware service.

credentials.yaml — MLflow

MLflow records traces of the LLM calls. It is a separate service and is not part of this Compose setup, so switch it off explicitly:

mlflow:
  enabled: false

Leaving the section out does not disable it

The default is enabled: true with a tracking URI of http://localhost:5000. Without an MLflow server there, the middleware retries the connection on every start and does not begin serving for several minutes.

api_keys.yaml — inbound authentication

This file lists the keys that callers must present. The application sends its key on its own; you only need one by hand for the checks further down.

api_keys:
  - "…"   # application
  - "…"   # operations / monitoring

Generate one key per caller so a single key can be revoked without affecting the others:

openssl rand -hex 32

Revoking a key means deleting the line and restarting the service:

docker compose restart aimiddleware

The health and model-listing endpoints are reachable without a key, everything else requires one.

Connecting the application

The application is told where to find the middleware by URL, and that URL has to carry the path prefix /rmsaimiddleware/. The container serves that prefix itself through the ROOT_PATH variable in the Compose file, so inside the Compose network the address is:

http://aimiddleware:8000/rmsaimiddleware/v1/chat/completions

No reverse proxy is needed for this — the middleware only has to be published if something outside the Docker network has to reach it as well.

The URL itself is set per rulemap in the tree configuration, together with your Rulemapping Solutions contact. It is not part of the installation.

Persistent data

/app/data holds attachments and the file cache and must survive a restart. The Compose file binds it to ./data/aimiddleware, which has to be writable by the container user:

sudo chown -R 6000:6000 ./data/aimiddleware

Verifying the installation

# 1. Is the service healthy?
docker compose ps aimiddleware

# 2. Health endpoint (no API key required)
curl -fsS http://127.0.0.1:8000/health

# 3. Are the configured models loaded? (no key needed)
curl -fsS http://127.0.0.1:8000/v1/models

# 4. End-to-end: does a provider actually answer?
curl -fsS -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer <your key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"azure/your-deployment-name",
       "messages":[{"role":"user","content":"ping"}]}'

If step 3 returns an empty model list, credentials.yaml was not read — check docker compose logs aimiddleware for the resolved configuration path.

Security checklist

• config/credentials.yaml and config/api_keys.yaml generated for this installation, not copied from another one
• File permissions 0600, owner is the operator account
• Neither file in version control
• Port 8000 not reachable from untrusted networks
• cors.allowed_origins restricted to your own domain
• One API key per caller, documented who holds which