On-Device AI: What Changes When the Model Lives on Your Phone
More AI features now run on the phone or laptop itself instead of in a data centre. That trade affects privacy, speed, battery and what the feature can do.
Noor Castellanos Editor, AI and Software
For the first wave of AI features, everything happened somewhere else. Your words went to a server, a large model did the work, and the answer came back. The newest phones and laptops increasingly do some of that work themselves, on a part of the chip built for it.
Why companies are moving models onto devices
Running a model locally means no round trip to a server, so simple tasks like transcribing a voice note, cleaning up a photo or suggesting a reply can happen instantly and work offline. It also costs the company less, because it is not paying for the server time.
The bigger selling point is privacy. If the recording or the photo never leaves your device, it cannot be logged, leaked or used for training on the other end.
If the photo never leaves the phone, there is nothing on the other end to leak.
The trade-offs
A model small enough to fit in a phone's memory is far smaller than the ones running in data centres, so it is better at narrow tasks than open-ended conversation. That is why many features quietly hand harder requests to the cloud. Good implementations tell you when that happens; others do not, so it is worth reading the settings.
Local models also use the chip and memory hard while they work. Expect some extra battery drain and warmth during heavy use, and expect older devices to miss out on features entirely because they lack the memory or the dedicated hardware.
What to look for when buying
If AI features matter to you, look at memory first. Manufacturers increasingly reserve their local AI tools for models with more RAM, and memory cannot be upgraded later on a phone. A device with a recent chip but the minimum memory can end up locked out of next year's features.
The settings worth checking
Most systems now have a page that lists which AI features are on and whether they can use online processing. Spend two minutes there. Turning off online processing keeps everything local, at the cost of the features that need a bigger model.
Why memory matters more than the chip
A model's size is measured in parameters, the internal numbers it learned during training, and every one of them has to sit in memory while the model runs. A small on-device model can easily need several gigabytes on its own, on top of the apps you already have open. That is the real reason last year's phone with a perfectly fast processor may not get this year's AI features.
Manufacturers squeeze models down by storing each number with less precision, a technique called quantisation. It shrinks the model dramatically with a modest loss in quality, which is usually invisible for tasks like summarising a notification but more noticeable for longer writing.
Battery, heat and the hidden cost
Dedicated AI hardware, usually called a neural processing unit, does this kind of maths far more efficiently than the main processor, which is why a short request barely registers on the battery. Sustained use is different. Transcribing a long meeting or generating a stream of image edits keeps the chip busy, and you will feel the warmth through the case.
Phones manage this by slowing down when they get hot, so a long local task may take longer on a warm afternoon than on a cold morning. It is normal behaviour, not a fault.
Last year's fast phone can miss this year's AI features for one reason: memory.
What stays private, and what does not
On-device processing is a genuine privacy improvement, but it is not a blanket guarantee. Many assistants decide request by request whether to answer locally or online. Some use a privacy-preserving cloud service for the harder ones; others send them to a regular server. The settings page and the company's privacy documentation are the only reliable way to tell which you are getting.
It is also worth remembering that the result of a local task can still be shared. A summary written on the device is private until the app syncs it to a cloud account, at which point the usual rules for that account apply.
On-device AI will not replace the big cloud models any time soon. It is better understood as a fast, private first pass, with the cloud as the backup, and the more you know about where that line sits, the better you can decide what to share.
psst, there is a letter