PrivateCore → SiliconOracle — On-Device Language Models for Personal AI

01 — Initial hypothesisThe bet: the model never leaves the device

The baseline question behind PrivateCore was simple: what does a personal AI look like if the model never leaves the device? This was not a philosophical position — it was an engineering bet. Apple Silicon iPhones from the iPhone 15 generation carry a unified-memory architecture, 8 GB of DRAM, a 16-core Neural Engine, and a GPU capable of mixed-precision matrix multiply at meaningful bandwidth. The 2023 wave of open-weight models in the 2B–7B range was being actively quantized to 4-bit, collapsing from multi-gigabyte bloat into a footprint a phone could actually hold. The question was whether quality-per-watt was good enough for the task.

The system

PrivateCore was built as a full-lifecycle personal knowledge-management substrate, entirely on-device:

Read access to the Photos library (a pointer model — asset IDs only, never copies), Calendar events, and HealthKit daily aggregates.
An on-device embedding index (MiniLM-L6-v2, 384 dimensions, served via sqlite-vec) for semantic search.
An on-device vision pipeline (Qwen2.5-VL 2B, 4-bit) for automatic photo captioning.
An on-device text-generation pipeline (Ministral 3B, 4-bit) for narrative.
Derived artifacts: encyclopedia-style wiki articles, force-directed concept graphs, and 9:16 vertical videos combining photos, motion, music and narration.

PrivateCore system architecture, a layered on-device stack with a hard cloud boundary at the top — **Figure 1** — The on-device stack. Nothing crosses the cloud boundary at the top.

The ML stack

The inference stack is MLX Swift, Apple's own framework for Apple Silicon unified memory. The choice was deliberate: MLX runs directly against Metal, eliminating the CoreML compilation and weight-conversion overhead of ONNX or llama.cpp paths; the same weights run identically on the developer's Mac Mini M4 and on the iPhone 15 Pro target — critical for fast iteration; and MLX's lazy graph execution fits a streaming-generation UX.

Model	Task	Quantization	RAM footprint
Qwen2.5-VL 2B	Vision captioning	Q4	~1.5 GB
Ministral 3B	Text generation	Q4	~2.2 GB

With iOS overhead (~2.0 GB) and app storage (~0.3 GB), total memory pressure on an 8 GB device reaches ~6 GB when both models are warm — near the jetsam ceiling for a foreground app. This forced a sequential hot-swap architecture: the two models cannot coexist in memory. ModelRouter enforces exclusive GPU access through MLXQueue, a custom actor that serializes acquire()/release(). Once the vision pipeline enters text generation, incoming vision requests are queued, not rejected — they wait. A hardcoded 20 MB MLX cache limit prevents the allocator from ballooning and triggering an out-of-memory kill before model weights are evicted.

02 — The generation pipelineHow a day becomes a wiki

The daily wiki generator in PrivateCore operated as a single coherent pass: gather sources (photos, events, cards) → even-sample four photos spread across the day to avoid over-weighting burst sessions → pre-fill captions for any uncaptioned sampled photo via the vision model, persisting each caption for reuse → assemble a context block from captions, event titles, geocoded location, and health aggregates → make a single structured-JSON completion from Ministral 3B → persist the article and its source links → guard with a SHA-256 hash of the source IDs so unchanged inputs skip regeneration.

The prompt follows a structured-output pattern familiar from Karpathy's "LLMs as structured programs" framing: we don't ask for prose and then parse it — we specify the exact JSON schema and enforce it through the system prompt with a concrete example. Small models (2–3B) struggle with unconstrained generation at length but reliably produce valid JSON when the schema is short (fewer than three nesting levels, fewer than ten keys).

Monolithic wiki generation pipeline — **Figure 2** — One pass, one large JSON completion. The cache gate skips unchanged days.

The embedding layer

Every piece of text entering the system — captions, card content, wiki leads, event notes — passes through MiniLM-L6-v2, a 22 MB CoreML model producing 384-dimensional sentence embeddings stored in a sqlite-vec virtual table with cosine-similarity lookup. The result is a semantic memory layer that answers "show me everything around the time I visited the mountains" without a date match — the embedding of "alpine hiking" lands near captions like "rocky trail, snow-capped peaks, pine forest." MiniLM is the one model that is always loaded; at 22 MB it fits comfortably and never competes for the GPU, since it runs entirely on the Neural Engine via CoreML.

03 — Performance findingsWhat worked, and what was hard

What worked

Q4 quantization quality for personal narrative is sufficient. 4-bit Ministral 3B degrades factual recall and long-range coherence versus FP16 — but personal narrative has a much lower coherence bar than knowledge retrieval. The model never needs to recall that the Battle of Hastings was in 1066; it constructs evocative prose from captions and timestamps handed to it in-context. At that task it performs surprisingly well.
Qwen2.5-VL 2B is an excellent captioner at Q4. For standard iPhone photography — people, places, food, landscapes — captions are accurate and detailed enough to enrich generation. Failure modes are predictable: low light, heavily cropped faces, abstract art.
Structured JSON from small models works if the schema is small. A full daily wiki (lead + three sections + infobox) runs ~600–900 tokens at temperature 0.6 and stays on-format unless pushed into a very long section.
The pointer model for Photos is correct. The library is append-only, assets are immutable, and the identifier is stable. Working entirely with pointers keeps storage overhead near zero.
Local retrieval is fast. Approximate-nearest-neighbour queries over 50,000 embeddings complete in under 50 ms on A17 Pro. Hybrid semantic + full-text search covers both fuzzy concepts ("beach vacation") and exact terms ("Hotel Bellevue") in one path.

What was hard

Memory pressure is a constant ceiling. The hot-swap solves coexistence but adds latency: unload the vision model, wait ~500 ms for page reclaim, load the text model, initialize — 3–8 seconds before the first token streams. Tolerable on a 15 Pro; worse on a 6 GB device.
Thermal throttling disrupts long pipelines. Generating a wiki right after heavy ingestion runs hot. At a "serious" thermal state, quality degrades as clocks step down; at "critical" we abort. The "process 20, pause one second" pacing throughout the background pipelines is pragmatic thermal management, not preference.
Monolithic calls fail silently under thermal pressure. One call for the full article means that if the device escalates to critical mid-generation, the in-flight inference is cancelled and nothing is saved. This single failure mode is the direct motivation for the incremental approach built later.
Small models are coherence-limited for multi-entity wikis. Single-day wikis (bounded ~1,500 input tokens) are handled well. Trips spanning multiple days and locations strain the window; the narrative through-line breaks past ~2,000 tokens. Weekly wikis are viable only with aggressive per-day summarization before synthesis.
Background generation on iOS is constrained by design. The short background-refresh window and the charging/no-network processing window are each insufficient for a full pipeline. Background generation is therefore best-effort, not reliable delivery — users who don't open the app daily find the queue grows stale.
Factual diary output is harder than it looks. The most common failure is the model inventing plausible-but-wrong details — events that didn't happen, people who weren't there. For a memoir app where correctness is the contract, that is a critical failure, not a tolerable side effect. Grounding with explicit captions reduces invention but does not eliminate it.

Memory budget on an 8 GB iPhone 15 Pro — **Figure 3** — Only one large model fits at a time. The hot-swap is forced by the 8 GB ceiling.

04 — The pivotFrom archivist to oracle

The findings pointed at a fundamental tension in PrivateCore's value proposition:

The app promised factual, accurate personal memory. The models delivered creative, plausible personal narrative.

These are not the same thing. A personal knowledge-management system has a strong correctness contract — you trust it to retrieve facts about your life accurately. A 3B-parameter, 4-bit-quantized model running under thermal pressure on a phone is not a reliable factual archivist. It is an excellent stylist given facts it was handed.

The insight that unlocked the pivot came from watching users: they were most delighted not when the wiki accurately summarized their day, but when it transformed the day into something more — more dramatic, more poetic, more theatrically significant. The generative leap was the feature. Factual accuracy was the expectation they had to manage around.

This is a known property of small models. As Karpathy has articulated it, LLMs at inference time are essentially autocomplete engines shaped by RLHF toward instruction following. At 2–3B parameters they have limited factual memory but substantial stylistic fluency. They know how Shakespeare sounds. They know how a hardboiled detective narrates. They can sustain a persona across a few hundred tokens far more reliably than they can hold factual precision across a multi-entity prompt.

The pivot was to lean into what the models are actually good at: take a photo, extract visual context with the vision model, and render it through a strongly-voiced stylistic persona. Deliberately non-factual output. The tendency to invent becomes a feature, because the output was never meant to be a transcript — it is an interpretation, an augmentation, a play.

The pivot, a 2x2 of model reliability versus user expectation of accuracy — **Figure 4** — Same models, different contract. The pivot moves the product into the quadrant the models can actually serve.

What changed

PrivateCore was forked into SiliconOracle in late May 2026. The fork is intentionally permanent — the two codebases share a common ancestor but have divergent contracts.

	PrivateCore	SiliconOracle
Output contract	Factual personal record	Deliberate non-factual interpretation
Failure mode	Hallucination (bad)	Creative invention (feature)
User contract	"This is what happened"	"This is how the oracle sees it"
Scope	Full PKM (photos, calendar, health, people, places, trips)	Photos only
Graph	Semantic concept map	Oracle cosmology (motif "birth chart")
Style system	2 generic presets	4 strongly-voiced personas
Thermal recovery	Abort + error	Partial save + resume
Generation	Monolithic (one call)	Incremental (one call per section)

The features removed from SiliconOracle — people, trip detection, places, contacts, health, dashboard — are not degradations. They are deliberate scope reductions. Each carried the same correctness contract PrivateCore struggled to deliver. SiliconOracle makes no claim about the world that could be falsified.

05 — SiliconOracleArchitecture and current state

SiliconOracle is a focused iOS app for oracle-register interpretation of personal photos. The user selects a photo, chooses a voice (Shakespeare, Sherlock, Diva, Manga), and receives a short stylized "reading" rendered as a shareable 9:16 card. Daily and weekly wikis persist in the oracle register with the same four styles. The ML stack is inherited from PrivateCore — same two models, same MLX Swift runtime, same serialized GPU access. The improvements are in generation architecture and production readiness.

Incremental wiki generation

SiliconOracle replaces the monolithic call with incremental, per-section generation: a short "lead" call produces the opening paragraph; each subsequent call produces exactly one section body; after each call the partial article is saved; and a stream-state object updates the UI in real time — "Captioning photos… → Writing overview… → Writing timeline…" rather than a spinner.

Thermal recovery is possible. If a critical state is hit between sections, the pending context is saved and resumption is scheduled when the device cools. Partial wikis surface with a "still forming" banner.
Perceived latency improves. The first section appears in 2–4 seconds; users see progress, not a blank screen.
Each call is shorter. A 600-token monolithic response becomes roughly five 120-token section responses — more reliable from small models and less sensitive to thermal clock-down.
Cache is finer-grained. A retry regenerates only the failed section, not the whole article.

Incremental generation architecture with a thermal-state monitor — **Figure 5** — Decomposition buys thermal recovery, visible progress, and per-section retries.

The oracle persona system

The four voices are not cosmetic themes applied afterward — they are system-prompt personas that condition the entire pass from the first token: Shakespeare (early-modern English, dramatic soliloquy), Sherlock (forensic first-person deduction), Diva (pop-tabloid energy, breathless emphasis), Manga (inner monologue, weight-of-moment, destiny vocabulary). Each defines a prompt prefix, section titles, a visual theme, and a card-specific prompt.

At 3B parameters, Ministral cannot reliably recall what the Roman Forum is — but it can sustain early-modern English verse across 600 tokens without breaking register.

The persona acts as a strong prior that pulls the token distribution toward a recognizable stylistic attractor, leaving the model free to invent within those bounds rather than hallucinate against a factual baseline.

Persona as a token distribution prior — **Figure 6** — Conditioning trades factual precision for stylistic reliability — the correct trade for small models.

The cosmology graph, scheduling, downloads

PrivateCore's semantic concept map is reimagined as a cosmological birth chart: a dark star-field with concept nodes in orbital rings and a slow precession. The underlying data is identical — frequency-weighted keyword extraction from captions and journal text — but the metaphor shifts from information architecture to divination, with Day / Week / Full scopes mapping concept frequency to orbital radius and node size.

Background work is centralized in a job queue with typed entries (captioning, embedding, OCR), priority levels, retry with exponential backoff, and explicit thermal gating. A five-second grace period after foreground return precedes any GPU-bound job — rapid background-to-foreground transitions destabilize the Metal command queue and cause silent failures, a behaviour discovered empirically. Vision jobs are excluded from background processing entirely, since iOS revokes GPU access in the background; only the CPU-bound embedding and OCR jobs run off-screen.

The app ships with no model weights — the bundle is ~100 MB over-the-air, comfortably under Apple's 200 MB cellular threshold. Models download on first launch over a background URL session that survives suspension, with per-file progress, SHA-256 checksum verification, and resumable downloads persisted to disk so they survive a process kill.

The accuracy safety system

A deliberately non-factual posture required an explicit safety system — not for technical accuracy, but for psychological risk. An oracle that diagnoses mental state, offers life advice, or makes future-facing predictions about a person can be harmful in ways a wrong historical date cannot. The context builder enforces five constraints entirely through the prompt: tense restriction (present/past only, no "you will…"), register matching (heavy journal entries get grave, never ironic, prose), no mockery, no counsel, and no stamped date/location headers. Enforcing through the prompt rather than a regex filter is deliberate — filters produce false positives on innocent phrasing like "what will linger" in a past-tense reflection.

Human-in-the-loop assuranceA human sampling protocol gates the journal-aware oracle path before any external beta: at least ten oracle outputs are reviewed, including at least two drawn from heavy journal entries (grief, loss, anxiety), each walked through a five-row tonal-safety checklist. No automated test substitutes for human judgment on tonal safety.

Monetization

SiliconOracle monetizes via a soft tip-gate: after every few generations a skippable sheet offers a one-time unlock. Generation is never blocked — the gate is always skippable — and sharing is always free and never gated. The interval is A/B tested through remote config (aggressive / control / lenient cohorts).

06 — Current statusTwo codebases, one foundation

PrivateCore remains the engineering foundation — the substrate from which SiliconOracle was extracted. It is, user-facing, named Lifelore.

Complete & functional (PrivateCore / Lifelore)

Full ML inference runtime (serialized GPU access, Metal engine, model router, MLX Swift)
Vision captioning pipeline (Qwen2.5-VL 2B Q4)
Daily / weekly / trip / person / place wiki generation
Full SQLite data model + vector store + full-text index
Force-directed, filterable concept graph
9:16 video assembly (motion, music, narration)
PDF export, share extension, semantic search

In progress & planned

UI polish, PDF export, onboarding, and freemium (StoreKit 2) work
An infrastructure-hardening merge from SiliconOracle: deadlock-safe GPU queue, model-download recovery, caption checkpointing for interrupted sessions, generation telemetry
Incremental wiki generation (pending a schema change and SQLite migration) and thermal-resume, best executed as a dedicated sprint
The cosmology graph views (pending a design decision on branding vs. the richer concept taxonomy)

SiliconOracle's near-term priority is its first external beta: a single-photo "Character Reading" feature and cosmology-graph scope navigation are the headline features, gated behind completion of the human sampling protocol for the journal-aware path.

07 — Architectural lessonsWhat generalizes

Match the task to the model's actual strengths

The most general lesson: a 3B-parameter Q4 model is not a smaller GPT-4. It is a different kind of tool — strong short-range coherence (under ~500 tokens), reliable structured output for small schemas, excellent stylistic fluency for well-represented registers; weak factual recall, poor cross-document coherence, rapid degradation under thermal pressure. Designing for these actual characteristics — rather than papering over the limitations — is what made SiliconOracle viable. The oracle register converts the model's primary failure mode (hallucination) into its primary value proposition (interpretation).

Strong personas outperform weak prompts at small scale

"Write a diary entry about my day" produces generic, slightly uncanny prose. "Thou art the Bard; speak of this day as Hamlet might" produces something genuinely entertaining. The persona is a strong distributional prior, pulling generation toward a well-represented basin in the training distribution. It is not a trick — instruction-tuned small models saw far more Shakespeare-style exercises than "personal diary" writing in fine-tuning, so persona conditioning is genuinely more reliable.

Incremental beats monolithic in thermal environments

A 600-token monolithic completion is thermally risky on a phone under load. Five 120-token completions are individually safer, and a failure at step three preserves steps one and two. The per-call overhead is negligible against inference cost. For any LLM pipeline in a thermally-constrained environment, incremental is simply the more robust architecture.

The pointer model is the correct abstraction; sequential GPU access is non-negotiable

Identifiers are stable, the library is append-only, and the system manages asset lifecycle correctly — any approach that copies pixel data into app storage is fighting the platform. And concurrent Metal command buffers from two MLX model instances on iPhone produce crashes that are hard to reproduce and impossible to debug from logs. A serialized GPU queue with explicit acquire/release is the minimum viable discipline; any future multi-model architecture (e.g. a small draft model for speculative decoding) must reason explicitly about GPU ownership.

Actor-based GPU serialization timeline — **Figure 7** — One consumer holds the GPU at any time. The inset shows what happens without serialization.

08 — What comes nextThe research horizon

The sequential hot-swap constraint is the current ceiling on user-experience quality. One path forward — once Apple Silicon reaches 12 GB+ on iPhone — is persistent loading of both models simultaneously. At today's memory configurations, careful pipelining (begin loading the text model while the vision model finishes captioning) can cut the swap penalty from 3–8 seconds to 1–2.

The deeper open question is whether a small, fast draft model (e.g. a 135M-parameter speculative draft for the 3B target) could accelerate time-to-first-token enough to meaningfully change the perceived latency of on-device generation for personal AI. That is on the research horizon — and it is exactly the kind of question that decides whether private, on-device intelligence becomes the default rather than the exception.