Local Open AI Models from rented intelligence to sovereign infrastructure

The strategic shift

The most important question in artificial intelligence is no longer whether a closed commercial API gives a slightly better answer on a benchmark. The real question is who controls the infrastructure, the data, the cost curve, the audit trail and the right to adapt the system. Open models, especially open-weight models, have changed the economics of AI. Universities, public agencies, municipalities, research centres and small companies can now run capable language models on infrastructure they own, rather than depending entirely on remote proprietary platforms.

There is, however, an important distinction. A fully open AI model should expose not only weights, but also the code, training process, data documentation, evaluation methods and enough artefacts for serious inspection and reuse. An open-weight model lets users download and run the weights, but may still keep training data and key pipeline details private. For public interest deployments, fully open models such as OLMo and Apertus should be preferred whenever possible. For operational deployments, strong open-weight families such as Qwen, DeepSeek, Llama, Gemma, Mistral, GLM, Kimi and gpt-oss can also be useful, provided their licence, safety profile and governance limits are documented.

What a 4× Radeon RX 7900 XTX Linux server can actually do

A Linux server with 4× AMD Radeon RX 7900 XTX provides 96 GB of aggregate GPU memory. This does not turn the system into a single seamless 96 GB accelerator. The cards communicate through PCIe, and performance depends on tensor parallelism, quantization, context length, KV cache size, batching strategy and the inference engine. Still, this is already enough for a serious local AI service.

Small models in the 7B to 14B range are the easiest operational choice. Qwen, Gemma, Llama, OLMo and smaller Mistral-class models can support chat, document summarisation, classification, retrieval assistance, educational tools, citizen-service assistants and internal helpdesks. With vLLM or SGLang, continuous batching and short prompts, such a server can support many lightweight concurrent sessions. It is not a frontier lab, but it is more than enough for a university department, a municipality, a research group or a controlled internal public-service deployment.

The most attractive production tier is the 24B to 35B range. Models such as Devstral, OLMo 32B, Qwen3.5-35B-A3B and DeepSeek-R1-Distill-Qwen-32B offer a strong balance between capability and cost. With 4-bit quantization, they fit comfortably on 4× RX 7900 XTX while leaving room for KV cache. This is the practical sweet spot for many public-sector and research workloads: legal and administrative retrieval, code assistance, technical support, Greek-language fine-tunes, structured drafting, and RAG over official corpora. In realistic interactive use, the same server can serve roughly tens of active sessions, depending on the length of input and output.

The 70B tier is possible, but less forgiving. Apertus 70B, Llama 70B, some Qwen 72B variants and larger multimodal models require careful quantization, context discipline and lower concurrency. They should be treated as heavier expert models for selected users or more demanding tasks, not as the default model for mass service. Models above 100B, including gpt-oss-120B, may fit under specific quantized formats, but leave limited memory for long context and concurrent users. Very large frontier MoE systems, such as full DeepSeek-V3/R1, GLM-5.2 or Kimi K2.6, are not a realistic everyday target for a 4× RX 7900 XTX machine. They belong either on larger clusters or should be accessed through smaller distilled variants.

What changes with 8× Radeon RX 7900 XTX

An 8× RX 7900 XTX system doubles the aggregate memory to 192 GB. This makes 30B to 35B models much more comfortable, improves the experience of 70B deployments and makes some 120B quantized models more plausible for limited-concurrency reasoning workloads. It still does not replace enterprise GPU clusters with high-bandwidth interconnects and data-centre accelerators. The lesson is simple: 4× cards are enough for a strong local service, 8× cards are enough for a serious institutional node, but full frontier MoE inference remains a data-centre problem.

The open software stack is the real advantage

The hardware matters, but the stack matters more. A practical AMD-based local AI server can run Ubuntu Server or Rocky Linux, ROCm for AMD acceleration, PyTorch on ROCm, vLLM or SGLang for production inference, llama.cpp for GGUF and lighter deployments, LiteLLM as an OpenAI-compatible gateway, Open WebUI as the user interface, Keycloak for identity, OpenSearch or Qdrant for retrieval, PostgreSQL for metadata and Grafana or Superset for monitoring.

This stack reduces vendor lock-in. Applications call a stable API while models can be changed behind the gateway. Data can remain local. Logs can be retained for audit. RAG can force answers to be grounded in official documents. Costs can be measured per request. Model Cards, Dataset Datasheets and service dashboards can make the system inspectable. A ministry, university or municipality can therefore treat AI not as a subscription product, but as a governed digital commons.

From local inference to democratic AI capacity

Local open models are not just a cheaper way to imitate commercial chatbots. They are a way to build public AI capacity. A municipality can provide assistance without sending sensitive citizen queries to a third-party cloud. A university can give students and researchers access to AI tools under clear academic rules. A ministry can run RAG over laws, circulars, procurement data and administrative procedures without surrendering institutional knowledge to opaque platforms.

The right architecture is hybrid and layered: many small and medium models for routine tasks, one stronger model for complex analysis, strict RAG for legal and administrative answers, human responsibility for final decisions, and full technical documentation for every critical workflow. In this model, a low-cost AMD server is not a replacement for national supercomputers. It is the missing middle layer between laptops and hyperscale data centres. It turns AI from a rented service into a public, auditable and reusable infrastructure.

Sources for the article:

1. AMD, Radeon RX 7900 XTX Specifications: AMD’s official product page documents the 24 GB GDDR6 memory capacity of each RX 7900 XTX, which is the basis for 96 GB in a 4-card server and 192 GB in an 8-card server: https://www.amd.com/en/products/graphics/desktops/radeon/7000-series/amd-radeon-rx-7900xtx.html,
2. AMD ROCm, Linux System Requirements: AMD’s ROCm documentation lists the RX 7900 XTX as a supported Radeon GPU for compute workloads on Linux, using the RDNA3 gfx1100 target: https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html.
3. AMD ROCm, vLLM Inference Documentation: AMD documents vLLM inference workflows on ROCm, showing the production relevance of the AMD open acceleration stack for LLM serving:https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/benchmark-docker/vllm.html,
4. vLLM, ROCm Installation Guide: vLLM’s ROCm installation documentation describes AMD GPU support for Linux deployments and is directly relevant to local LLM serving: https://docs.vllm.ai/en/v0.6.5/getting_started/amd-installation.html,
5. SGLang, AMD GPU Platform Documentation: SGLang’s AMD GPU documentation explains how to run the inference runtime on AMD GPUs, useful for structured generation, agents and high-throughput serving: https://sgl-project.github.io/platforms/amd_gpu.html,
6. Open Source Initiative, Open Source AI Definition 1.0: The OSI definition is essential for distinguishing fully open AI systems from models that merely release weights: https://opensource.org/ai/open-source-ai-definition,
7. Allen Institute for AI, OLMo: OLMo is one of the most important fully open language model families, with 7B and 32B variants suitable for research, auditability and public-interest deployments: https://allenai.org/olmo,
8. ETH Zurich, EPFL and CSCS, Apertus: Apertus is presented as a fully open, transparent and multilingual model family, relevant to European AI sovereignty and multilingual public infrastructure: https://ethz.ch/en/news-and-events/eth-news/news/2025/09/press-release-apertus-a-fully-open-transparent-multilingual-language-model.html,
9. Z.ai, GLM-5.2: Z.ai’s GLM-5.2 release is relevant as an example of a very large open-weight frontier model that is important strategically, but generally beyond the practical limits of a 4× RX 7900 XTX production node: https://z.ai/blog/glm-5.2.