It's only been a few years since ChatGPT took the world by storm. What was once an uncanny novelty that brought sci-fi to real life as an omniscient personal helper is now a regular part of our internet experience. From search-engine integration to advanced code autocompletes, this is a tool that we will come to rely on — for better and for worse. But where does this future take us? I foresee two big issues with the current landscape of generative-AI distribution. One is that many of these models are running on the cloud, by large, billion-dollar companies. For many, this raises serious privacy concerns, since these models have access to so much valuable data — people's health concerns, job resumes, coding projects, etc. Another concern that doesn't get talked about as much is that these models are burning cash like crazy. If you know how late capitalism works — especially the big-tech blitzscaling strategy — then you know this is a common tactic to gain market share before the eventual scaling back of features behind increasingly expensive paywalls. This will make these companies a lot of money but force many users to use lower-quality models than their peers who can afford the best models and features being offered.
There is a saving grace here, though: locally run LLMs. Shortly after ChatGPT was released, developers quickly realized that they could create very good, lighter-weight models (think fewer parameters, less space, less compute) by using ChatGPT and other heavyweight models to train their own in a process often called "distillation." Thus began the trend of lighter-weight open-source models that are free for people to download — and some are even commercially viable for developers to build apps from. Leveraging local LLMs has been gaining traction for a while, but there are still few truly polished, commercially viable apps for most casual users. And what happens when larger models get put behind paywalls? I think this creates a great opportunity for the development of better local AI platforms that won't be extremely expensive to host in the cloud. Over time, these local models should get even better at running on-device, solving more advanced problems while using less compute. So let's take a look at the current landscape of private, local generative AI.
Today, several pioneering apps demonstrate what on-device LLMs can do, though most are still in early stages of UX and model polish. Apps like LLMFarm leverage Metal acceleration to run quantized models (LLaMA, Falcon, MPT) locally on modern iPhones, offering a menu of open-source checkpoints for users to choose from. Private LLM packages a standalone chatbot experience with simple prompt templates, but its model options and context length remain limited. And On-Device AI: LLM & Voice Memo integrates speech-to-text with a small, distilled model for short reply generation entirely offline. These apps prove feasibility but often trade off fluency, context window, or speed to fit within RAM and battery budgets.
Under the hood, developer toolchains have matured rapidly. llama.cpp popularized 4-bit and 8-bit quantization pipelines plus a lightweight C library that runs on CPU/Metal; Swift wrappers now make integration into Xcode projects straightforward. MLC LLM extends this by compiling PyTorch/ONNX models into ANE- and Metal-optimized kernels, with an iOS SDK that handles memory-mapping and streaming inference. Apple's Core ML remains a go-to for those preferring a first-party solution — coremltools can ingest a quantized model and output an .mlmodel that taps the Neural Engine automatically. Community efforts are also standardizing on-device embedding indexes for local retrieval, enabling apps to search user documents without exposing data externally.
As both models and hardware evolve, the bar for on-device intelligence will keep rising. Upcoming smaller architectures (e.g., mixture-of-experts or sparse-attention designs) promise to deliver higher accuracy per parameter, while next-gen Apple silicon will boost on-chip RAM and Neural Engine throughput. We'll likely see more adaptive model loading, where an app dynamically swaps in larger sub-models only when needed, and federated fine-tuning that personalizes an LLM on your device without sharing raw data. Over the next 1–2 years, expect local LLMs to handle multi-turn dialogues with longer contexts, integrate seamlessly with on-device APIs (calendar, photos, documents), and even support private voice assistants that never connect to the cloud. For developers, this means a widening scope of privacy-first features, new UX paradigms around streaming inference, and business models that blend paid model-packs with one-time app purchases — all fueling a robust edge-AI ecosystem outside the server farms.
I'm still early in my research, but it's clear there's enormous room for growth over the next few years. Developers who build intuitive, user-friendly apps leveraging local LLMs will gain a major advantage as the AI sector expands. This article represents my first deep dive into the topic — stay tuned for more insights as I explore building my own on-device LLM application and document my progress every step of the way.