Notes from the field.
Inspiration and learned experience while building, shared to be used.
Explore by question
- 01 · Evaluate
How do you know an agent is working?
Learn what correct behavior means and how to test it.
Read the method: What is an LLM evaluation harness? Graders, protocols and closed loops for AI agents → - 02 · Set boundaries
Where should an agent's responsibility end?
Separate prediction, recommendation and authority.
Read the method: A wrong forecast is absorbed. A wrong action reaches a guest. →Applied inAetherix
- 03 · Learn
Whose truth should an agent learn from?
Distinguish approval, prediction and observed truth.
Read the method: Whose truth is the ground truth? →
All entries
3 pieces- October 11, 2026Product
Whose truth is the ground truth?
A forecast is a claim. A manager's "yes" is a preference. Real covers are a fact, the next day. The plan you did not choose is never seen. Which of these your product learns from decides what it becomes, and what it is worth.
- July 31, 2026Evaluation
What is an LLM evaluation harness? Graders, protocols and closed loops for AI agents
What a grader is, what a harness is, what a protocol decides, and why the difference matters the first time a system is allowed to block a release. Built from two products, with the gaps named.
- April 20, 2026Agentic AI
A wrong forecast is absorbed. A wrong action reaches a guest.
Prophet is a statistical forecasting library. Claude is a large language model. Keeping them as two components, rather than letting one model both predict and act, is the decision that shaped everything after it.
Discuss an evaluation protocol
These notes document real systems and real failure modes. Want to challenge an evaluation harness, or compare notes on agent observability?