Hamza Ahmad An AI analyst that measures its own accuracy and switches itself off
Most AI analysis tools let the model do everything and can't tell you whether they work. I built one where code gathers and checks the data, the model only reasons over clean inputs, every output is tracked to its outcome, and the system rewrites its own lessons each week. It ran unattended for weeks and paused itself when it crossed its loss limit, exactly as designed.
How it works
- Data in code, not in the prompt. A Python service pulls from 9 sources (prices, news, calendars, macro data), caches everything and compares two price feeds within 0.15%. A third source breaks ties. If they still disagree, the instrument is skipped and logged. The model never sees conflicting data.
- Rules in code. Indicators are computed in Python. A plain IF node blocks analysis around high-impact events and earnings. That is a rule, not a judgment call.
- Structured output. Claude returns a strict JSON signal with an explicit invalidation condition. "No trade" is a normal answer: about 90% of outputs were exactly that.
- Validate and retry. The validator finds the right content block, checks the schema and, on failure, sends the model its own answer plus the exact error. Two retries, then a logged failure.
- Outcomes and post-mortems. A nightly job resolves every signal with no AI. Post-mortems run through the Batch API at half price, judged against the news available at the time to avoid hindsight bias.
- Self-rewritten memory. Each week a larger model consolidates the lessons file under a token cap, retires lessons that new evidence contradicts, and commits the change to git.
- Kill switch. If drawdown passes 10%, the system deactivates its own signal workflow through the n8n API and alerts me. The alert fires even if the deactivation call fails.
Cost engineering
- Model routing per stage: a mid-size model for signals and post-mortems, the largest model only for the weekly memory rewrite, and no model at all for outcome checks and the kill switch.
- Prompt caching: the static block (instructions and lessons) always goes first, dynamic data after. Calls were serialized after I found parallel calls each wrote their own cache.
- Batch API for post-mortems, at 50% off.
- Every call logged with input, output, cache-read and cache-write tokens. Budget model: about $19 a month with caching and batching against about $33 without, with an alarm at $30.
Debugging stories
Thinking blocks broke parsing
The validator assumed the first content block was the answer, so every retry failed. Fixed by locating the text block properly.
A guardrail leak in production
When a tiebreak failed, conflicting prices slipped into the prompt through an error message. The model correctly refused every trade. Fixed with clean-record-or-raise.
Rate limits at scale
Growing the watchlist from 5 to 19 instruments needed 13-second throttling, retry with backoff on 429s and degrade-not-fail on secondary sources.
An honest self-audit
After the kill switch fired, I documented the real flaws: no open-position guard and no holiday skip caused stacked duplicate signals.