{"updated": "2026-10-25T12:20:00Z", "project": "Ninja Cortex", "kick": "Build log 001 · a private AI analyst over our own threat graph", "tagline": "Facts stay in the graph. A small model reads the dossier and answers only from what it read. Runs on four CPU cores we own. Every number here is measured, and the misses are listed.", "meta": [{"k": "started", "v": "23 Oct 2026"}, {"k": "GPU spend", "v": "< $30"}, {"k": "reader", "v": "3B + our adapter"}, {"k": "serving", "v": "4 CPU cores, 6-bit"}, {"k": "status", "v": "Round 3 in progress"}], "phase": "Round 3 · v3 training", "now": "v2 landed at 11:32 UTC: +10 points on unseen phrasings (76.6%), the yes/no denials fixed, no regression on the templated set, but the hand-written 100 fell from 91 to 85 because the new answer formats bled into ordinary questions. v1 stays the serving adapter. v3 is training now with the corrected recipe: half templated pairs, half paraphrased, positives cut to 2.5%, open-question refusals in.", "progress": {"label": "v3 adapter, LoRA on Qwen2.5-3B, 58,823 pairs", "done": 820, "total": 1839, "unit": "steps", "eta": "evals ~13:30 UTC"}, "headline": [{"label": "Hand-written analyst questions", "value": "91 / 100", "tone": "good", "bars": [{"l": "base model", "v": 86, "t": "86", "c": "var(--dim)"}, {"l": "v1 adapter", "v": 91, "t": "91", "c": "var(--good)"}, {"l": "v2 adapter", "v": 85, "t": "85", "c": "var(--amber)"}, {"l": "v1, 4-bit CPU", "v": 85, "t": "85", "c": "var(--dim)"}, {"l": "v1, 6-bit CPU", "v": 87, "t": "87", "c": "var(--cyan)"}], "note": "25 real dossiers, expectations checked against the source first. 6-bit now serves: +2 over 4-bit and faster per question."}, {"label": "Held-out facts, unseen phrasings", "value": "76.6%", "tone": "good", "bars": [{"l": "base model", "v": 45.3, "t": "45.3%", "c": "var(--dim)"}, {"l": "v1 adapter", "v": 66.8, "t": "66.8%", "c": "var(--cyan)"}, {"l": "v2 adapter", "v": 76.6, "t": "76.6%", "c": "var(--good)"}], "note": "563 probes asked in forms never used in training. v2 generalises further; v1 still serves (see hand-written)."}, {"label": "Dossier in context vs closed book", "value": "76.8%", "tone": "amber", "bars": [{"l": "closed book", "v": 32.5, "t": "32.5%", "c": "var(--dim)"}, {"l": "line removed", "v": 45.5, "t": "45.5%", "c": "var(--amber)"}, {"l": "dossier", "v": 76.8, "t": "76.8%", "c": "var(--good)"}], "note": "Same 3B model, 938 four-way probes, chance 25%. This is why retrieval is the architecture."}, {"label": "Invented answers, fact present", "value": "0.0%", "tone": "good", "bars": [{"l": "base model", "v": 0.7, "t": "0.7%", "c": "var(--dim)"}, {"l": "+ adapter", "v": 0, "t": "0.0%", "c": "var(--good)"}], "note": "400 held-out pairs. A capitalised token in the answer that is in neither dossier nor question counts as invented."}, {"label": "Blind open questions refused", "value": "0.4%", "tone": "bad", "bars": [{"l": "base model", "v": 8.6, "t": "8.6%", "c": "var(--dim)"}, {"l": "+ adapter", "v": 0.4, "t": "0.4%", "c": "var(--bad)"}], "note": "Should be high. The answering line was removed; the only right answer is a refusal. Training-set gap, fix training now (v3)."}, {"label": "Serving", "value": "4 cores", "tone": "", "bars": [{"l": "tokens/s", "v": 18.9, "t": "18.9", "c": "var(--cyan)"}, {"l": "model size", "v": 24, "t": "2.4 GB", "c": "var(--cyan)"}, {"l": "per question", "v": 5.5, "t": "~5.5 s", "c": "var(--cyan)"}], "note": "6-bit (Q6_K), llama.cpp, no GPU in production. Promoted 11:55 UTC after scoring 87/100 vs 85 for 4-bit, at 5.5 s a question on the hand set."}], "closed": ["Making the weights the database. From-scratch models at 34M, 110M and 320M parameters recalled their own training facts at 54%, 46% and 43%. Bigger was worse, with lower training loss and better English. The weights learn the dialect; the facts stay in the graph and the model reads."], "scaling": {"label": "Seen-fact recall by model size (the closed line)", "points": [{"l": "34M", "v": 53.7}, {"l": "110M", "v": 46.3}, {"l": "320M", "v": 43.2}], "note": "Each model asked for facts it was trained on. Chance is 25%. Bigger recalled less."}, "next": ["v3 adapter (training now): refuse open questions the dossier cannot answer; keep the hand-100 score.", "Grow the hand-written set from 100 to 200, then 300, and publish the harness.", "Multi-dossier questions (a crew's victims by sector) and temporal questions (first seen, what changed).", "Compare bases at the same running cost: Qwen3-4B, Qwen3-30B-A3B, Foundation-Sec-8B.", "Dossier caching per entity on the CPU server so repeat questions answer in a second."], "limits": ["Knows nothing the graph does not know; will say so when the dossier is silent (once v3 lands).", "Not a general assistant; one entity per question today.", "Weak sense of time; temporal questions near chance.", "The hand-written set is 100 questions: enough to find failure classes, not a decimal place.", "Nothing it writes reaches a published page without a human-grade gate."], "principles": ["The graph is the memory; the model reads.", "Every answer shows the dossier it was read from.", "Nothing leaves the network; no third-party model in the answer path.", "Cast-iron or do not publish: leak-filtered probes, checked expectations, failures listed with examples.", "Rent GPUs to train, serve on what you own."]}