When we started building our SaaS product, the OpenAI API felt like magic. We could ship AI features in a weekend. But as our user base grew, our monthly API bill grew even faster. At one point, it was one of our biggest costs after salaries.
So we tried something many founders are now testing in 2026: moving our high-volume inference work to open-weight 70B models running on a compact, unified-memory workstation in our own office.
Quick answer: We cut our OpenAI API bill by about 85% by moving our predictable, high-volume tasks (summaries, classification, extraction, and internal RAG) to a quantized 70B open-weight model running locally. We kept the API only for the hardest tasks. The key was a mini workstation with a very large pool of unified memory, which can hold a 70B model that normal consumer GPUs cannot.
Here is the full story: why we moved, the hardware details, our homelab stack, and the real ROI math.
Why SaaS Founders Are Migrating Inference Workloads Off the Cloud in 2026
We are not the only ones doing this. In founder communities, we see the same pattern again and again. The reasons are simple:
1. API costs scale with usage, not with revenue
Every new user means more tokens. If your pricing is flat, your margins shrink as people use your product more. That is the opposite of what you want.
2. Open-weight models got good enough
Open 70B-class models now handle many everyday tasks well, such as summarizing, tagging, extracting data into JSON, and answering questions over documents. For these tasks, the gap to top closed models is often small.
3. Privacy and customer trust
Some of our B2B customers asked where their data goes. Being able to say “this runs on our own hardware” made sales calls easier.
4. Predictable costs
A workstation is a one-time cost plus electricity. That is much easier to plan for than a bill that jumps every time a customer runs a big batch job.
We did not move everything. We split our workload into two groups:
| Summaries, tagging, classification | Local 70B model |
| Structured data extraction | Local 70B model |
| Internal RAG over our docs | Local 70B model |
| Complex reasoning and edge cases | Still on the API |
In our case, the local group was most of our token volume. That is where the savings came from.
The VRAM Bottleneck: How 128GB Unified LPDDR5X Memory Changes Local LLM Quantization
The biggest problem with running large models locally is memory. A 70B model has 70 billion parameters. How much memory it needs depends on quantization, which means storing each parameter with fewer bits:
| FP16 (16-bit) | ~140 GB |
| Q8 (8-bit) | ~70–75 GB |
| Q4 (4-bit) | ~40 GB |
Most consumer graphics cards have 8 GB to 32 GB of VRAM. Even a 4-bit 70B model does not fit on one of them. You end up splitting the model across several GPUs, which is expensive, loud, and power-hungry.
Unified memory changes this. On these systems, the CPU and the integrated GPU share one large pool of fast LPDDR5X memory. Once a system has around 128 GB of unified memory, things open up: a 70B model fits at 8-bit quality, with room left over for the context window (the KV cache) and other services. More memory beyond that gives space for longer contexts or running two models side by side.
We tested with both Ollama (easy to start) and vLLM (better for serving many requests). A few honest notes from our tests:
- Memory bandwidth decides token speed. For a large dense model, generation speed depends mostly on how fast the system can read the model’s weights from memory.
- Dense 70B models are usable, not instant. For batch jobs like summaries and tagging, this was more than fine. For live chat, we used smaller models or the API.
- Mixture-of-Experts (MoE) models are much faster. They only use part of their parameters for each token, so they generate tokens much faster on the same hardware.
- Batching is your friend. vLLM can serve many requests together, which raises total throughput a lot.
The chip we used is the ryzen ai max+ pro 495. It has 16 Zen 5 CPU cores, a Radeon 8065S integrated GPU with 40 compute units, and an NPU rated at 55 TOPS. In the workstation we chose, it is paired with 192 GB of LPDDR5X-8533 memory, and up to 160 GB can be assigned to graphics. That means the 128 GB threshold is cleared with plenty of room to spare.
Our Production Homelab Stack: Clustering via Dual 10GbE and USB4 v2 (80Gbps)
We wanted a setup that was quiet enough for the office, used little power, and could grow later. Here is what we run:
Hardware
- One main inference node with large unified memory
- One small NAS for model files, logs, and backups
- A 10GbE network switch
Networking
Our main node has dual 10GbE ports. We use one for serving API requests to our app servers and one for storage traffic to the NAS. It also has USB4 v2 ports at 80 Gbps, which can be used for very fast direct links between nodes or for fast external storage. When we add a second node, we plan to split models across nodes or run different models on each.
Software
- Proxmox as the base system, so we can run separate VMs and containers
- vLLM for production serving with an OpenAI-compatible API
- Ollama for quick tests and new model trials
- A small router service that sends each request to the local model or to the OpenAI API, based on task type
Because vLLM offers an OpenAI-compatible endpoint, switching most of our code was mostly a matter of changing the base URL and the model name.
The machine at the center of this stack is the MINISFORUM ms-s1 max. It is a compact desktop or rack-friendly node with dual 10GbE, USB4 v2, a PCIe slot for expansion, and two M.2 slots for storage. For a small team, it gave us data-center-style features in a box that sits quietly on a shelf.
ROI Breakdown: Cloud GPU Rental vs. Owning Compact Compute Nodes
Here is how we thought about the money. Your numbers will be different, so use this as a template, not a promise.
Option A: Keep everything on the API
- Cost scales with every token.
- No hardware to manage.
- Best for low volume or very complex tasks.
Option B: Rent cloud GPUs
- You pay by the hour for GPUs with enough memory for 70B models.
- Costs are lower than the API at high volume, but running 24/7 still adds up every month.
- You still send data to a third party.
Option C: Own a compact compute node
- One-time hardware cost.
- Electricity for a low-power mini workstation is small compared with a GPU server.
- Data stays in-house.
Our simple ROI formula:
Payback period (months) = Hardware cost ÷ (Monthly API savings − Monthly power cost)
Here is the simple math behind our 85% number. About 85% of our monthly token volume was routine work: summaries, tagging, extraction, and internal RAG. We moved all of that to the local 70B model. The remaining 15% of tasks, which needed the strongest reasoning, stayed on the API. Since the API bill follows token volume, the bill dropped by about the same share, and the hardware paid for itself within a few months.
| Upfront cost | None | None | One-time |
| Monthly cost | High, grows with use | Medium to high | Low (power) |
| Data privacy | Third party | Third party | In-house |
| Setup effort | Very low | Medium | Medium |
| Best for | Low volume, hard tasks | Short bursts | Steady high volume |
If you want to compare different compute nodes before you decide, it is worth browsing an ai workstation lineup to see the options for memory, networking, and expansion.
Lessons We Learned
- Measure first. Log your token usage by task type before you buy anything.
- Do not move everything. Keep the hardest tasks on the best API model.
- Test quality on your own data. Run a side-by-side test before switching.
- Use an OpenAI-compatible server. It makes switching easy.
- Plan for growth. Choose hardware with fast networking so you can add nodes later.
Frequently Asked Questions
Can you run a 70B LLM on a Mini PC?
Yes, if it has enough unified memory. A 4-bit 70B model needs about 40 GB, and an 8-bit version needs about 70–75 GB, so systems with 128 GB or more of unified memory can run them.
Is a local LLM cheaper than the OpenAI API?
For steady, high-volume tasks, it often is. For low volume or very complex tasks, the API can still be cheaper and better.
What software do you need to run local LLMs?
Ollama is the easiest way to start. vLLM is better for production serving because it handles many requests at once and offers an OpenAI-compatible API.
Is local LLM speed good enough for production?
For batch tasks and internal tools, yes. For real-time chat with large dense models, you may want smaller models, MoE models, or a hybrid setup with the API.
Final Thoughts
Running local LLMs is no longer only for hobbyists. With unified-memory workstations, a small team can run 70B-class models in-house, protect customer data, and turn a fast-growing API bill into a predictable one-time cost. For us, it was one of the best infrastructure decisions we made this year.
The post How We Cut Our OpenAI API Bill by 85% Running 70B Local LLMs on a Unified-Memory Mini Workstation appeared first on The Hype Magazine.

1 week ago
17
