Why I'm Obsessed With Local AI, and Why You Should Be Too

Three months ago, I handed a client's detailed financial records to a SaaS invoice platform. Within a week, they got a phishing email targeted specifically at their business name and industry vertical. Coincidence? Maybe. But it was enough to make me think differently about where sensitive data flows.

Then I stumbled on BillCraft AI, a project that does something I've been wanting to see more of in the indie dev space: it refuses to ship data to someone else's servers. No monthly SaaS fees, no cloud lock-in, just your receipts, your computer, and a local AI model running quietly in the background. It sounds simple until you realize how radical that actually is.

The Problem Nobody Talks About Loudly Enough

Freelancers and small business owners face a genuinely annoying problem that most venture-backed companies have decided isn't worth solving cheaply. Manual invoice creation and receipt digitization is tedious work, I've done it myself, but the existing "solutions" come with hidden costs that go beyond the $50/month subscription.

When you upload financial documents to mainstream accounting software, you're trusting a company with information that could expose your profit margins, client relationships, and banking infrastructure. Most of these platforms are legitimate, sure. But they're also honeypots. Breaches happen. Worse, your data lives in someone else's database indefinitely, subject to terms of service that can change.

BillCraft AI flips this on its head by running everything locally, OCR extraction, invoice reading, even the AI model that enhances descriptions. Your financial data never leaves your machine.

The Technical Approach That Actually Matters

What impressed me about this project isn't just the philosophy; it's the engineering discipline required to execute it. The creator chose Qwen 2.5 7B, an open-weight model from Alibaba, and runs it through Ollama on localhost. For OCR, they paired RapidOCR with PyMuPDF for spatial text extraction. The backend validates all math server-side before saving to SQLite.

This is the opposite of throwing everything at Claude API and hoping it works. It's intentional architecture. You're running inference locally (with Cloudflare Workers AI as a fallback for demos), handling document parsing with proper PDF extraction libraries, and validating results before persistence.

The React 19 + Vite frontend stays responsive because it's not waiting on API calls for every keystroke. Calculations happen instantly. That matters more than people realize when you're actually working, not just demoing.

My Take: Local-First Is the Future, But It Requires Discipline

I'm genuinely bullish on this direction. We've normalized cloud-everything for so long that we forgot what it means to have compute resources on your own machine. But, and this is important, local-first only works if you're willing to be opinionated about architecture.

The tradeoffs are real. Running Ollama locally means you need reasonable hardware. Your users need to understand they're installing a ~7GB model. The developer experience is different. You can't just spin up a quick endpoint and call it from JavaScript; you need to understand ONNX runtimes, model quantization, and inference pipelines.

What I'd do differently: I'd be more aggressive about auto-downloading the model as part of onboarding, with progress transparency. Most people will tolerate a 5-minute first-run setup if they understand what's happening and why it matters for their privacy.

A Practical Reality Check

Here's the code pattern I'd use for local inference fallback:

async function extractInvoiceData(pdfText) {
 try {
 // Try local Ollama first
 const response = await fetch('http://localhost:11434/api/generate', {
 method: 'POST',
 body: JSON.stringify({
 model: 'qwen2.5:7b',
 prompt: `Extract vendor, amount, date from: ${pdfText}`,
 stream: false
 })
 });
 return await response.json();
 } catch (error) {
 // Fall back to Cloudflare Workers AI
 console.log('Local inference unavailable, using edge fallback');
 return await callCloudflareWorkers(pdfText);
 }
}

This pattern gives you local-first privacy with graceful degradation for edge cases. That's production-grade thinking.

What's Missing (And What I'm Curious About)

The project doesn't deeply address what happens when users want to sync data across devices or collaborate with an accountant. Local-first is powerful, but it creates friction in multi-device workflows. I'd love to see encrypted sync layers explored, something like local-first with optional E2E-encrypted cloud backup.

Also, the math validation is server-side, which means you still need a backend. Is that running on the user's machine too, or do they need to deploy their own FastAPI instance? Clarity on this operational model would help others replicate it.

What About You?

Are you building something that handles sensitive data and still believes cloud APIs are the default? I'd push you to spend a weekend exploring what local inference actually looks like. The barrier to entry is lower than it's ever been.

Source: This post was inspired by "BillCraft AI: Invoice Generator + AI-Powered Invoice Reader" by Dev.to. Read the original article