Most AI features in business software still depend on cloud APIs. Your CRM sends a prompt to a remote server, waits for a response, and pipes it back into your workflow. It works, but it comes with trade-offs: latency, recurring API costs, and your customer data leaving your network every single time.
That’s starting to change. The latest GPUs now make it possible to run large language models locally, and some businesses are pulling AI inference back on-premises. The software on top, from CRMs to sales platforms, is adapting to meet them there. We’ll get into what this looks like in practice, where the savings show up, and what the hardware demands actually are.
What “Locally Run” Actually Means Here
When people talk about running AI locally, they mean hosting a large language model on your own hardware instead of calling out to OpenAI, Anthropic, or Google’s APIs. Open-weight models like Llama 3 and Mistral have made this realistic. A company with the right GPU setup can run inference on-site, keeping prompts and outputs entirely within its own infrastructure.
For business software, this opens up a few doors. A CRM that uses AI to score leads, summarise call transcripts, or draft follow-up emails can do all of that without sending data to a third-party endpoint. The model sits on a local server, the data never leaves, and the response comes back faster because it doesn’t travel across the internet.
The Privacy Argument Is Stronger Than You’d Think
Privacy is the most obvious reason to run models locally, but it goes deeper than “we don’t want our data in the cloud.” In regulated industries like finance or healthcare, sending customer records to an external API can create real compliance headaches. Running models on-premises removes that variable. If a sales team wants to use AI to analyse deal pipelines or flag at-risk accounts, the entire process stays internal. No API logs on a third-party server, no ambiguity about where the data went.
This matters more as AI features get baked into everyday tools. It’s one thing to send a single prompt to ChatGPT. It’s another to have your CRM automatically processing thousands of customer interactions a day through a cloud model. The volume changes the risk profile.
Where GPU Hardware Fits In
This is where it gets relevant for anyone who cares about what’s inside the box. Running a 7B model for light summarisation is one thing. Running a 70B model that handles complex reasoning across large datasets is another, and it needs serious hardware.
NVIDIA’s RTX 4090 and RTX 5090 can handle smaller models for single-user or small-team setups. For production workloads, businesses are looking at A100s, H100s, or B200 GPUs. The bottleneck is usually VRAM. A 70B model in FP16 needs around 140 GB, so you’re either running multi-GPU setups or using quantised versions. Quantisation has improved enough that a 4-bit 70B model can run on a pair of 24 GB consumer cards, and for most business tasks like email drafting or meeting summaries, the quality difference is negligible.
How CRMs and Sales Tools Are Adapting
The CRM market has been racing to add AI features. Salesforce has Agentforce, HubSpot has its AI agents, and newer platforms like Attio are building AI into the data layer itself. Most still rely on cloud inference, but the architecture is shifting.
Some CRMs now support connecting to local model endpoints through standard APIs. If you’re running an LLM on your own server with an OpenAI-compatible wrapper like Ollama or vLLM, you can point certain CRM features at your local endpoint instead of a cloud one. The CRM doesn’t care where the model lives, as long as it gets a response in the right format. GTM Tools tracks which CRMs and sales platforms support AI features natively, which will prove very useful when weighing up built-in cloud AI against your own infrastructure.
The Cost Equation Isn’t Simple
Cloud API pricing looks cheap at first. A few pence per thousand tokens adds up slowly when you’re testing. But once AI is embedded across a team’s daily workflow, processing thousands of calls, emails, and records, those costs scale fast.
Running locally flips the model. You pay upfront for hardware, electricity, and maintenance, but after that, inference is essentially free. For a company already investing in GPUs for data analysis, rendering, or engineering simulations, adding a local LLM is a marginal cost. The break-even depends on volume. A five-person sales team probably won’t justify the investment. A 200-person operation will likely save money within the first year.
Hardware Decisions Will Shape Software Choices
For years, the assumption has been that business tools live in the cloud and the local machine is just a thin client. Local AI flips that. The GPU in your server room becomes a factor in which CRM features you can use, how fast they’ll run, and how much they’ll cost.
For anyone already building or speccing out systems with high-end GPUs, this is one more reason the investment pays off beyond gaming or content creation. The same card that handles real-time ray tracing can run the AI model behind your sales pipeline. As open-weight models keep improving, the gap between local and cloud inference will only get narrower.