A few months ago, I was testing a new AI prompt in WindowSill and realized I'd just sent a draft of a sensitive email to OpenAI's servers without thinking twice. The email wasn't mine. It was from a user testing my app and they shared some details about their workflow in their company. Their words, their context, their problem, processed on a server neither of us controlled.
That moment didn't sit right with me. I'd built WindowSill as a tool that works with whatever is on your screen: text in emails, documents, chat messages, notes. People paste private things into text fields. If my app is going to touch that text with AI, I owe them a way to keep it on their machine.
So I built local LLM support. First Ollama, then Microsoft AI Foundry Local, then LM Studio. Three local providers, each with their own quirks. And the honest answer is: it cost me more than I expected.
What "local-first AI" means (and doesn't mean)
Local-first AI means the language model runs on your computer. Your text goes from your app to a model loaded in your RAM or VRAM. No internet request. No API call. No server logs. The prompt and the response never leave your machine.
What it doesn't mean: offline-only, or anti-cloud. WindowSill still supports cloud providers like OpenAI, Anthropic, Google, Mistral, and others through bring-your-own-key. Local-first is about giving users the option to keep everything local, not forcing them into it.
The distinction matters because I'm not making a religious argument against cloud AI. Cloud models are faster, smarter, and easier to set up. The case for local is narrower and more specific: some text shouldn't travel over a network, and some users (and their IT departments) have good reasons to keep it that way.
The privacy decision that shaped everything
WindowSill's AI features work inline. You select text in any app, pick a prompt (rewrite, fix grammar, translate, change tone), and the result replaces your selection. That means the app sees the raw text people are editing, not a sanitized version they chose to paste into a chat window.
That's a different threat model than ChatGPT. When someone opens ChatGPT, they make a conscious decision to share something with a cloud service. When someone selects text in Outlook and hits "fix grammar" through WindowSill, the expectation is that it's a local tool doing local work. The text they selected might be a medical note, a legal draft, an HR document, a message to their therapist.
I decided early that WindowSill would never store your prompt content on our servers. When you bring your own API key, your text goes directly to that provider's API. If you use the built-in WindowSill AI service, your prompts pass through our server as a relay to OpenAI, but we don't store them. We track usage statistics (which provider was used, model name, token counts) so we can understand what's popular and debug issues. But the actual text? We don't keep it. And if you chose a local model, it never leaves your hardware.
This wasn't the easiest path. A managed cloud service would have been simpler to build, simpler to support, and simpler to monetize. But the inline nature of the tool made privacy non-negotiable. If you're going to intercept text in every app on someone's desktop, you'd better have clean hands.
What it actually cost me
Let me be honest about the tradeoffs, because the "privacy is a feature" story usually skips the hard parts.
Engineering time
Supporting one cloud API is straightforward. You send a request, you get a response. Supporting three local inference runtimes, each with their own API shape, model format, discovery mechanism, and failure modes, is a different kind of work.
Ollama uses its own REST API. LM Studio exposes an OpenAI-compatible endpoint. Foundry Local uses Microsoft's SDK with a local service manager that needs to be started, sometimes restarted, and occasionally coaxed into cooperation. Each provider has a settings page, a model selection flow, an endpoint configuration, and an allow-list for organizations that want to restrict which models their employees can use.
I built an abstraction layer (a common IModelClient interface) so the rest of the app doesn't care which provider is active. But each implementation has its own edge cases. Ollama needs to check which models are running vs. downloaded. LM Studio needs to query /v1/models and handle cases where the server isn't started. Foundry Local needs to manage a background service, load models explicitly, and retry when the service crashes.
I'd estimate the local provider work took three to four times longer than adding cloud BYOK support for the same number of providers.
Setup friction for users
Cloud AI works out of the box if you have an API key. Paste it in, pick a model, done. Local AI asks a lot more of the user: install a second application, download a model (which can be several gigabytes), make sure the local server is running, configure the endpoint in WindowSill, and then hope your hardware can run the model at a reasonable speed.
I've answered more support questions about local AI setup than any other feature. "Ollama isn't detected." "LM Studio shows no models." "The response is empty." "It's been generating for two minutes." Most of these trace back to the local server not running, or the user picking a model too large for their hardware.
Model quality
This is the tradeoff nobody in the "run AI locally" community likes to talk about. Local models are worse than cloud models for writing tasks. Not a little worse. Noticeably worse.
A 7B or 8B parameter model running on a laptop can fix basic grammar and do simple rewrites. It can translate short text reasonably well. But ask it to adjust the tone of a nuanced paragraph, or rewrite a technical document for a non-technical audience, and the quality gap becomes obvious. Cloud models like GPT-4o or Claude Sonnet (and let's not even talk about Claude Opus) handle those tasks with an ease that local models can't match today.
I'm not saying local models are bad. For straightforward prompts (fix grammar, make shorter, translate to French), they're good enough. For complex writing tasks, the gap is real, and pretending otherwise would be dishonest.
Speed
A fast cloud API returns a response in one to three seconds for most writing prompts. A local model on a laptop without a GPU can take fifteen to thirty seconds for the same task. With a decent NVIDIA GPU, you can get that down to three to eight seconds, which is workable. But not everyone has a GPU, and the people most likely to care about privacy (enterprise users, healthcare professionals, legal workers) aren't always on high-end hardware.
Streaming helps. WindowSill supports streaming responses from local providers, so you see tokens appearing as they're generated instead of waiting for the full response. It doesn't make it faster, but it makes it feel less broken.
What users actually care about
After supporting local AI for a while, here's what I've learned about what drives adoption:
Privacy is the lead reason, but not the way I expected. Individual users who care about personal privacy exist, but the bigger driver is organizational policy. Companies in healthcare, legal, finance, and government have rules about where data can go. "Our policy says no cloud AI" is a conversation-ender for cloud-only tools. Local AI turns that into "yes, the data never leaves the machine, or company network."
Cost is the second reason. API keys have usage costs. If you're running hundreds of AI rewrites a day, those tokens add up. Local models cost nothing per request after the initial setup. For heavy users and organizations with many seats, the math starts to favor local.
Control is the third reason. Some users want to pick a specific model, fine-tuned for their domain, running on their network, with no dependency on a third party's uptime or pricing changes. Local gives them that.
Quality is not the lead concern for local users. This surprised me. People who choose local AI have already accepted the quality tradeoff. They're not comparing a local 8B model to GPT-4o and complaining that it's worse. They know it's worse. They chose local anyway because privacy or cost mattered more for their use case.
Why I think local AI is the future (but not yet)
I believe desktop apps will eventually default to local AI for routine tasks. Grammar checking, quick rewrites, simple translation, these don't need a 200-billion-parameter model running in a data center. A well-optimized small model on local hardware should handle them fine.
But the hardware isn't there yet for most users. NPUs (neural processing units) are shipping in new laptops, but the software ecosystem around them is still young. Model optimization has improved a lot (quantization, distillation, speculative decoding), but a local model that matches cloud quality for complex tasks is still a few hardware generations away.
The play right now is to support both. Give users the choice. Let the privacy-conscious and cost-sensitive run local. Let the quality-maximizers use cloud. Don't force anyone into one camp.
That's what WindowSill does. You pick a provider, you pick a model, and the app doesn't care whether it's running on your GPU or on a server in Virginia. The same prompts work everywhere. The same UI, the same keyboard shortcuts, the same inline workflow.
How to try it yourself
If you want to set up local AI in WindowSill, I've written step-by-step guides for each provider:
- How to use WindowSill with Ollama (command-line, lightweight)
- How to use WindowSill with LM Studio (visual model browser, GUI-first)
- How to use WindowSill with Microsoft AI Foundry Local (Microsoft ecosystem, NPU support)
Each one takes about ten minutes to set up. You'll need WindowSill and one of the local runtimes.
Frequently Asked Questions
Does local AI in WindowSill work without an internet connection?
Yes. Once you've downloaded a model and installed the local runtime (Ollama, LM Studio, or Foundry Local), everything runs on your hardware. No internet required for inference.
Which local provider should I choose?
Ollama if you're comfortable with the command line and want the lightest setup. LM Studio if you prefer a graphical interface for browsing and managing models. Foundry Local if your organization standardizes on Microsoft tooling and you want NPU support.
Do I need a GPU for local AI?
No, but it helps. All three providers run on CPU. A dedicated NVIDIA GPU with 6+ GB of VRAM will make responses three to five times faster.
Can my IT admin restrict which AI providers employees use?
Yes. WindowSill supports organization-level AI configuration. Admins can restrict providers to an allow-list, lock specific models per prompt, and manage settings remotely.
Is local AI free?
The local runtimes (Ollama, LM Studio, Foundry Local) are free. The models are free. WindowSill's AI integration features require a WindowSill+ license, but there's no per-token cost for local usage.
I built local AI support because I think tools that touch your private text should give you a private option. It wasn't easy, it wasn't cheap to build, and the results aren't as good as cloud models yet. But for the people who need it, nothing else will do.
Download WindowSill and try it yourself.