Skip to content
H
All work

Case study

macOS Troubleshooting Agent

An agentic RAG system for macOS troubleshooting — a branching LangGraph, a domain knowledge base built from 9 scrapers, and a blind-judge evaluation harness that blocks its own deploys on quality regressions.

Role
Solo — design, build, deploy, evaluate
Timeline
Stack
PythonLangGraphFastAPIQdrantLlama 3.3 70BPostgreSQLDockerPrometheusGrafanaOracle Cloud
Links
LiveSource
Agent graph: intake branches to knowledge-base retrieval and a confidence-gated web search, merges, then a decide node branches to answer, clarify, or request a diagnostic.A LangGraph state machine, not a linear chain. Intake feeds two parallel retrieval paths — the knowledge base always, and a web search only when retrieval confidence falls below a threshold. Both merge into smart_merge, which feeds the decide node. Decide is the branch point: it can answer directly, ask a clarifying question, or request a diagnostic from the user.intakekb_retrieveweb_searchconfidence < τsmart_mergedecideanswerclarifyrequest diagnostic

Problem

macOS troubleshooting knowledge is scattered across forums, Stack Overflow, and man pages. Generic assistants guess at fixes, or lead with "have you tried restarting it" without ever asking what a user has already ruled out.

Context

A solo project — Python, LangGraph, FastAPI, Qdrant, Llama 3.3 70B, PostgreSQL, Docker, and Oracle Cloud, at roughly 17,500 lines of code, and at macos-agent-hari.duckdns.org.

Role

Solo: design, build, deploy, and evaluate.

Exploration

The write path is pulling from Apple Developer Forums, Ask Different, Stack Overflow, Reddit, WWDC, man pages, dev docs, and GitHub — cleaned, tagged, and chunked one problem per document. The open question wasn't how to retrieve; it was when to stop trusting the knowledge base and go outside it.

Decisions

  • intake → {kb_retrieve, web_search} → smart_merge → decide, where decide branches three ways: answer directly, ask a clarifying question, or request a diagnostic from the user. That three-way branch isn't expressible as a straight-line RAG pipeline.
  • Confidence-gated web fallback. A Tavily search scoped to apple.com fires only when knowledge-base retrieval confidence is low — RAG-first, not search-first.
  • Difficulty-tier triage. Every knowledge-base entry carries a difficulty tier, so the agent can infer what a user has likely already tried instead of leading with the easiest step every time.
  • Stateful multi-turn memory. macOS version, chip, and already-tried steps accumulate into a running case file across turns, persisted via langgraph-checkpoint-postgres so it instead of living in process memory.
  • Cross-provider LLM failover , so a single provider outage doesn't take the agent down mid-conversation.

Implementation

Production concerns that don't usually show up in a portfolio project: Google OAuth into session JWTs, , per-turn token and cost accounting with weekly and monthly spend caps, rate limiting via slowapi, Alembic migrations run on boot, Prometheus and Grafana dashboards with structured logs shipped through Loki, GitHub Actions CI, and deploy/rollback/recover scripts backed by a smoke test. cover auth, credits, guardrails, and the async Postgres checkpointer.

Challenges

The hardest part wasn't the agent graph — it was trusting the agent's own opinion of itself.

Eval gate: a candidate build runs an eval suite, is scored blind by two judge models, compared against a committed baseline, then a gate either deploys it or blocks the deploy.regression_gate.py in the macos-agent repo. A candidate build runs the eval suite; two judge models score it blind, without seeing each other's scores; the result is compared against a committed baseline.json. If the score falls below baseline, the gate blocks the deploy outright rather than merely warning.candidateeval suitejudge Ablindjudge Bblindscore vsbaseline.jsondeployBLOCKEDscore below baseline
regression_gate.py: two judge models score a candidate build blind, against a committed baseline. A score below baseline blocks the deploy outright.

Two judge models score every candidate build blind — without seeing each other's scores — across correctness, actionability, grounding, and depth-triage, and the result is checked against a committed baseline.json. If a change makes the agent measurably worse, .

Results

This hasn't shipped to real users, so — the honest results are the eval's, not a usage dashboard.

Eval harness results — both benchmarks
 AgentGPT-4o
Multi-turn resolution (n=4)2 / 41 / 4
Avg. turns to resolve (n=4)2.253.25
Single-shot head-to-head (n=6)1 win / 5 losses5 wins / 1 loss

Judged blind — judges never saw each other's scores. A second judge model scored the single-shot set 0 wins out of 3 for the agent. Both sample sizes are small: treat this as a signal, not a verdict.

What the eval told me

I built the judge panel expecting it to confirm the agent was good. It didn't.

On single-shot questions — one prompt, one answer, no follow-up — . A 70B model with retrieval does not out-recall a frontier model, and my own eval said so plainly.

On multi-turn diagnosis, which is what the product actually does, the numbers inverted: . Four cases, run interactively — treat that as a signal, not a result.

The split is the finding. The agent's advantage isn't knowing more; it's asking for the one diagnostic it's missing instead of guessing, and remembering what you already tried. Building the harness is what lets me say that with a straight face — and it's why regression_gate.py blocks a deploy when scores fall below the committed baseline. I'd rather have a red build than a quieter, worse agent.