Dynamic context

Context - not raw intelligence - has emerged as the primary bottleneck in building effective LLM-based agents. The primary concern of context engineering and agent design is to ensure that at every step, the agent is in the goldilocks zone of having just enough context to make the right decision, no more, no less. Stale instructions need to be removed; edge cases, irrelevant data and future steps need not be shown, so that the most pressing and important set of context is the only thing visible to the agent. Show too much and we risk pushing it off track. Hide appropriate instructions and it hallucinates the answer.

Because not all context is important all the time, it is often necessary for context to be dynamically injected and removed given the state of a task - just as humans dynamically retrieve and forget information as we work.

Pruning and compression

This pillar focuses on removing stale context (such as old tool calls and instructions from previous steps) and compressing context over time (for long running tasks, keeping the facts without the bloat). Anthropic is a primary innovator in this area, introducing tool result clearing and thinking block clearing in their agent API, which automatically remove context from old blocks when context exceeds a token threshold.

The challenge is knowing what to prune and when. We have experimented with having our customer support agent periodically forget the results of its knowledge search tool - but it's unclear when this should happen to avoid refetching information and randomly causing hallucination due to lost context. More recent agent designs offload tool results into off-context stores like the file system and replace the tool result with a placeholder message, allowing the agent to quickly refetch it if required. To my knowledge the best technique for when to prune is simply when a token threshold is met where we believe performance will start degrading. Some experiments are looking at employing LLMs to prune unproductive reasoning branches or relevance based signals.

Compaction is another common technique, primarily used in coding agents that run for longer and accumulate vast amounts of context from exploration and reasoning. Common approaches involve extracting facts from the current conversation, or using a prompt to summarise the conversation and restarting the thread. It is not clear whether customer experience use cases actually run into scenarios where compaction is required, given the brevity of interactions; but as the complexity of tasks CX agents need to solve grows, this will become something to look out for.

Experiments worth running

Validate the need for compaction. Monitor the distribution of token usage across different ticket types and complexity levels. If there's a long tail of conversations exceeding performance thresholds, compaction becomes relevant. Track the correlation between token count and response quality degradation.

Offload pruned tool results. Removing old knowledge search calls and results comes at the risk of pruning too early - causing the agent to hallucinate due to lost context, or to fetch the information again. Instead, keep an out-of-context but retrievable store of tool results, and replace each result with an identifier the agent can use to recall it if needed. Claude Code, for example, uses the file system both for old tool calls and large tool outputs.

Prune on task completion. If we can detect a goal shift, or the agent explicitly indicates it has completed a step or task, it may be reasonable to prune tool calls and reasoning traces that are no longer relevant. Many workflows start with an authentication step whose details stop mattering once the customer is authenticated, so most of that context can be compacted or pruned.

Prune results from every tool. Knowledge search is the obvious candidate because it's called many times and returns large outputs, but the same is true of many business API tools. These can be pruned to remove unnecessary tokens, most likely by offloading into an out-of-context store rather than removing them outright, so nothing useful is lost by accident.

Explicit context request

Explicit context request is when the agent is given the ability to expand its own context. A knowledge search tool is an example of this. The idea is to combine reasoning with information retrieval to ensure that we load the appropriate context at the right time. Techniques in this space rely on the model's reasoning to let the agent control both the timing and the selection of instructions.

Claude Skills is the most prominent example of explicit context request - instructions can be requested by the agent based on its own reasoning. Many coding agents take this a step further with the "files are all you need" paradigm, where most context - previous conversations, memory, knowledge base, tool results, instructions - is stored as files and folders, allowing the agent to dynamically request whatever is available through the single interface of the file system. There are many interesting things about this approach:

1. Context can be dynamically requested.
2. Many different types of context reduce to a single retrieval mechanism (bash).
3. There's a natural preview of what is available for the agent to reason about (what do I have access to, the top 5 most popular articles, how many conversations have I had).

Context extends beyond instructions: dynamic tool search is another common pattern for reducing prompt size. The key thing to note is that the number of tools is not the issue. It is the instructions that must ship alongside each tool explaining how to use it - these can quickly fill up the context window.

Experiments worth running

Deeply nested skills. Skills can go further with more progressive disclosure. A workflow might offer one layer of expansion into its full set of instructions today, but those instructions could be nested in multiple layers for the different branches the agent might explore - much like how a well structured Claude Code skill points to further files only when they're needed.

An agent filesystem. Offload many types of context into a filesystem the agent can fetch from. It doesn't actually have to be a filesystem, but coding agents show that frontier models are quite good at using bash to explore files. The main thing is to offer everything the model might want, immediately, but out of context and ready to be requested.

Trigger based context injection

When we cannot rely on the model to know or correctly reason about requesting context, trigger based context injection provides additional instructions at the appropriate time. This is useful for:

1. Safety: triggers can be evaluated independently of the agent to produce an unbiased outcome.
2. Edge cases: instructions for long tail scenarios can be numerous and quickly bloat the context window if they're available at all times. Edge cases are also often not obvious.

Guardrails are an example of safety and edge case handling, with triggers evaluated independently to inject steering instructions that guide the agent at the right time. Instructions on how to interpret a tool result can also be returned alongside the tool call, and similarly for how to handle errors when they occur. Triggers can also be combined with explicit context request to nudge the agent: for example, detecting the need for a skill and injecting instructions guiding the agent to read that skill.

Trigger-based injection can misfire: false positives inject noise that distracts the agent, while false negatives miss the moment entirely. When trigger confidence is low, the fallback is to nudge the agent toward explicit context request rather than injecting directly.

Experiments worth running

Workflow micro-guidance. Similar to guardrails but at the workflow level: detect workflow or topic specific edge cases and inject a steering prompt when they're detected.

Context from where the conversation happens. If we know where the chat widget is installed, we can have specific instructions for each location - putting them all in the prompt would quickly bloat it. For example, if the customer is on the orders page, nudge the agent to ask about their recent purchase.

Authentication gates. Unlock instructions only when certain conditions are met. For example, authentication must happen before the agent sees the tools or instructions for cancelling an account or changing booking dates.