↓ Skip to main content
  1. Archive/

CS329Z Lecture 2: The Language Model Beneath the Agent

Author
AI Assistant
Curated for Pradithya Aria Pura · AI-written summary of the source material
Stanford CS329Z: Engineering AI Agents - This article is part of a series.
Part : This Article

An agent builder does not need to train a foundation model, but does need to understand what the model is doing. CS329Z’s second lecture connects the machinery of language models to choices that show up in a deployed agent: sampling, long context, latency, structured outputs, and the contents of the prompt.

Key insights
#

  1. Sampling settings change output diversity, but the best setting depends on the task and budget.
  2. Model architecture shapes an agent’s usable context and inference cost; nominal context length is only part of the story.
  3. Training stages explain where a capability comes from, while prefill, decode, and test-time scaling explain what it costs to use.
  4. Structured output and context engineering turn a raw language model into a dependable component of an agent.

From token probabilities to behavior
#

Diagram of multiple attention heads operating over the same input sequence

Source: CS329Z Lecture 2, PDF page 45.

A language model predicts a distribution over the next token. Temperature changes the sharpness of that distribution; top-k and top-p restrict the choices under consideration. Those are behavior controls, not repairs for a weak task definition. A low-temperature model can still choose the wrong tool if the relevant instructions or evidence are missing.

The lecture also discusses beam search, which keeps several candidate sequences as generation proceeds. It can be useful when you can afford additional search, but it costs time and memory. For code or mathematics, predictable outputs often favor lower diversity. For creative generation, sampling a wider range of possibilities can be useful. There is no universally best decoding recipe; use a task-level evaluation to choose one.

The lecture walks through decoder-only attention: each token can use earlier tokens when predicting the next one. Attention Is All You Need introduced the Transformer as an alternative to recurrent sequence models, making training far more parallel. But full attention grows quadratically with sequence length. The assigned linear attention paper shows how a different formulation can process long autoregressive sequences with recurrent state and linear scaling. The slides stress the tradeoff: efficiency and memory use improve, while quality on some long-context tasks may suffer. Hybrid designs combine global and linear attention.

For an agent, this is more than architecture trivia. Tool outputs and histories can become very long. The useful question is not merely how many tokens a model accepts, but whether it can use the right evidence at the right point and what each additional token costs.

The lecture’s attention walkthrough builds queries, keys, and values from token representations, scores relevant earlier tokens, normalizes those scores, and combines the values. Multiple heads learn separate projections. Full attention allows each token to directly reference prior tokens, but its sequence-length cost and growing KV cache matter for long-running agents. Linear attention instead maintains a compressed state with more favorable scaling. The slides emphasize that compression can make some long-range retrieval harder, which motivates hybrid designs with occasional global attention.

Even without an architectural context cap, a model only learns to use positions and patterns it saw during training. A huge advertised window is not proof that it will recall a fact buried in a long tool trace. Long-context evaluation should test the specific retrieval and reasoning behavior the application needs.

Training and inference have different bottlenecks
#

Hybrid model architecture combining linear attention and global attention

Source: CS329Z Lecture 2, PDF page 88.

The lecture traces pretraining, domain-oriented midtraining, instruction tuning, preference learning, and reinforcement learning with verifiable rewards. Each stage shapes what the model can do before an application developer writes a prompt. Agent-specific training can use complete trajectories, but collecting good trajectories and rewards is hard.

Pretraining teaches broad next-token prediction. Midtraining changes the data mix toward domains such as code or specialized text. Post-training teaches instruction-following and preference-aligned behavior. The slides are careful about human feedback: raters can disagree or reward confident-sounding answers, creating incentives for sycophancy. Verifiable rewards, such as unit tests, exact matches, or a checked system state, give a stronger signal for math, code, and some agent tasks—but the verifier itself must be designed carefully.

At inference time, the slides separate prefill, which processes the input context, from decode, which emits tokens sequentially. This distinction makes long tool histories and verbose reasoning operational costs. The lecture also covers speculative decoding and test-time scaling: more samples or more reasoning can improve a hard answer, especially when a verifier can recognize success, but they spend latency and compute. A reliable agent should spend that budget selectively.

Speculative decoding illustrates a systems optimization: a cheaper draft model proposes tokens and a target model verifies them. Test-time scaling is a different choice: spend more model work to search for a better answer. Repeated sampling is particularly useful when a test can identify success; without a verifier, self-consistency is a weaker but sometimes useful aggregation strategy. The right budget should follow the cost of a wrong decision, not be fixed uniformly for every tool call.

Structured output and context are part of the interface
#

An agent’s model output often needs to become a typed tool call rather than free-form prose. JSON schemas and constrained decoding can prevent malformed syntax, although valid syntax does not guarantee a correct action. DSPy signatures are presented as another way to specify the inputs and outputs of a model call.

The context is broader than the system prompt. The slides list message history, memories, tool results, and attached files as additional inputs. Adding every available item seems safe but can increase distraction and make relevant evidence harder to use. A small, well-described tool set may be more effective than a long catalog of barely relevant functions.

The final section shifts from prompt wording to context engineering: which instructions, tool descriptions, retrieved facts, and history should be visible for this decision? The lecture warns about context rot and proposes empirical testing of context choices. Compaction has a sharp tradeoff: keep too much and the history remains noisy; discard too much and the agent forgets a commitment or crucial observation. Anthropic’s assigned guide makes the same point: effective context is the entire token state presented to the model, not just a system prompt.

My practical takeaway is to benchmark a model with the actual agent context shape: long histories, tool schemas, retrieval snippets, and constrained outputs. A model leaderboard score on short questions is a poor substitute.

Further reading
#

Stanford CS329Z: Engineering AI Agents - This article is part of a series.
Part : This Article

Related