Notes from the workshop.

LLM-as-a-judge lets you evaluate agent output at scale, but it has real failure modes. When to trust it, where it breaks, and how to make the judge reliable.

Data isolation for any AI SaaS touching your code is what decides adoption. The spectrum of approaches, the leak surface beyond the database, and what to ask.

AI coding agents cost real money and real review time. A straight cost-benefit framework for where they pay off, where they do not, and how to measure it.

The real question in AI agent architecture is who holds the control flow: your code or the model. Here is the spectrum, the trade-offs, and where each one fits.

Human in the loop AI is becoming a checkbox. For agents that ship code, an approval must be a real, enforced decision a named person can be accountable for.

An LLM token cost calculation for real agentic coding work: the 200k-in, 30k-out unit, why output dominates, and how iterations quietly multiply the bill.

An agent workflow buried in if-statements and retries is impossible to reason about. Model it as an explicit state machine and the chaos becomes legible.

AI accountability in software development cannot transfer to the tool. When an agent ships a bug, a named human owns the merge and the deploy. Here is why.

A repo-grounded coding agent re-sends a huge stable context on every call. Prompt caching cuts that input cost by roughly 90 percent. Here is how to earn it.

RAG for code helps an agent when retrieval is precise and hurts when it adds noise. Learn when to reach for it, when to skip it, and what to use instead.

A hallucinated file path is the easiest LLM code error to catch and one of the most damaging to miss. Here is how to check references before you trust them.

LLM agents retry constantly, and a retry that is not safe becomes a double charge or a duplicate deploy. How idempotent AI agents make retries safe by design.

Living documentation keeps docs in step with the code. See why drift happens, how detection works, and how current docs keep an agent grounded, not misled.

Prompt injection turns content an agent reads into commands. Defense in depth for AI agents: treat input as data, least privilege, allowlists, and human gates.

Build an LLM eval harness for a coding agent: deterministic graders plus judges, step and run scoring, and blocked-vs-failed, so quality stops being a guess.

Pricing an AI product is harder than SaaS because inference is a real cost on every call. A practical framework for pricing that covers cost and tracks value.

Code tells an agent what your system does. Tests tell it what the system is supposed to do. That is why tests are the better ground truth to read first.

For agents acting on real systems, the AI audit trail is the product. What a good record captures, and why reconstructing any decision has to stay possible.

Code impact analysis before you edit: map the blast radius with references, call graphs, and tests so surprises surface early, not later in production.

Least privilege for AI agents means minimum access per task, not standing keys. Scoped, short-lived credentials and boundaries that bound the blast radius.

Prompt engineering gets the attention, but context engineering decides whether a coding agent is reliable: what to put in the window, and what to leave out.

A grounded way to forecast what an AI coding agent costs per feature: the public model prices, the sequence of runs, and the four levers that move the bill most.

Context for AI coding agents matters more than clever prompts. Feed an agent the right slice of your codebase so it stops writing confident, wrong code.

Gating everything trains rubber-stamping; gating nothing is reckless. A practical framework for deciding where human oversight of AI decisions actually belongs.

Text rots; code is the territory. How the Architect agents build a living knowledge base and why every plan starts there.