I spent a good chunk of today testing a local model against a simple automation task. The idea was to have it extract specific fields from an invoice PDF and output structured data. On paper, this is exactly the sort of thing local LLMs should be good at. No need for a cloud API, no data leaving the machine, no recurring cost.
The reality was a mess. The model hallucinated two fields, misread a date format, and took 45 seconds to do what GPT-4o does in three. I could probably tune it, run a smaller quantised version, or switch to a different architecture. But that's the problem. The setup time to make a local model reliable for one task is longer than just using an API for a hundred tasks.
This is where the conversation around local LLMs gets stuck. Everyone talks about privacy and cost. Those are real concerns for specific use cases. But for most automation work, the value is in speed and accuracy. If a local model is slower and less accurate, it doesn't matter that it's free. You're paying in time and frustration.
What actually works is using local LLMs for things where latency doesn't matter. Batch processing overnight. Prototyping a prompt structure before paying for API calls. Or running a model that only needs to be right 80% of the time, like summarising internal chat logs where a miss isn't critical.
But for anything where you need reliability and speed, the cloud models still win. The local stuff is getting better. It's just not there yet for real automation.
Paul