Cutting LLM agent token costs by 71% with S3 offloading and paging
AWS engineers reduced latency by 80% and token usage by 71% in Amazon Bedrock agents by offloading large JSON payloads to S3 and implementing on-demand result paging.
Amazon Web Services engineers have published a technical deep-dive detailing how they optimized a production AI agent for Flightscape, an airline operations platform. By implementing three specific strategies for handling large JSON inputs, the team reduced token consumption by up to 71% and cut average latency by up to 80%. The work, published on November 17, 2026, offers a blueprint for developers building agentic systems that struggle with context window limits and high inference costs.
What happened
The Flightscape team built a Training Schedule Optimization agent using LangGraph and Amazon Bedrock AgentCore. This agent helps airlines verify crew training assignments against complex business rules, processing inputs that can exceed 20 megabytes. The system integrates over 40 tools and handles massive structured data sets, including optimization requests, responses, and validation reports. While functionally robust, the initial implementation suffered from severe performance bottlenecks. Average request latencies exceeded 15 seconds, and the cost of running the agent was prohibitive due to excessive token usage.
The core issue was architectural: every tool invocation injected its full payload into the large language model’s context window. As conversations extended over multiple turns, these large JSON objects compounded, filling the context window before the agent could complete its reasoning. To solve this, the team re-architected how the agent manages state and retrieves data. They moved away from keeping heavy artifacts inline within the conversational state and instead adopted a hybrid approach using Amazon Simple Storage Service (S3) for storage and lightweight references for session management.
How it works
The optimization relies on three composable strategies that separate heavy data from the active reasoning context. First, the team implemented a dehydration and hydration pattern for session state. Instead of storing raw JSON payloads in Amazon Bedrock AgentCore memory, the agent stores only content-addressed references. When a turn begins, the hydrate_state() function restores lightweight session data. At the end of a turn, dehydrate_state() strips heavy payloads, uploads them to S3, and saves only the SHA-256-based reference keys. This ensures that identical payloads are not uploaded twice, leveraging content addressing for idempotency.
Second, the team introduced on-demand paging for large list results. When a tool returns a list exceeding a specific sample size, the full result is stored in S3. The model receives only a compact summary containing the total count, a small sample, and a result handle. If the agent needs more data, it calls a get_result_page_tool with the handle, offset, and limit to retrieve specific slices. This prevents the context window from being flooded with unused data.
Third, for oversized dictionary outputs, the team used descriptor-based compaction. Rather than sending nested dictionaries verbatim, the agent generates a structural summary. Lists are represented by their count and a sample, while nested dictionaries are summarized by key counts. Scalar values remain inline. This allows the model to understand the structure of the data without consuming tokens on the full content, fetching details only when necessary via the result handle.
Key details
- The optimization reduced token consumption by up to 71% and cut average latency by up to 80% on a representative 23 MB payload.
- The solution uses Amazon Bedrock AgentCore memory for lightweight state persistence and Amazon S3 for heavy artifact storage.
- Content addressing via SHA-256 hashes ensures that identical payloads reuse existing S3 objects, avoiding redundant uploads.
- Tool results larger than the sample threshold return a handle and summary, requiring explicit paging calls to retrieve full data.
- Dictionary compaction preserves structural cues like field types and counts while omitting verbose nested content.
- The architecture is built with LangGraph, Python 3.12+, and boto3, running on Amazon Bedrock with Guardrails for response validation.
Why it matters
For engineers building production agents, context window management is often the difference between a viable product and a costly failure. As agents gain access to more tools and larger datasets, the naive approach of passing full payloads leads to exponential cost growth and latency spikes. This case study demonstrates that treating the LLM context as a scarce resource, rather than an infinite buffer, is essential for scalability. By decoupling data storage from reasoning context, developers can maintain high-fidelity access to large datasets without paying for every byte in every turn.
The techniques described here are framework-agnostic in principle, though implemented with LangGraph and Bedrock. The concept of hydrating and dehydrating state can be applied to any orchestration framework that supports external state stores. Similarly, result paging and descriptor compaction address universal challenges in structured data processing. These patterns allow teams to build agents that handle enterprise-scale inputs while keeping inference costs predictable and response times low.
What you can do
- Audit your agent’s state structure to identify heavy payloads currently stored inline within conversational memory.
- Implement a content-addressed storage layer using S3 or similar object storage to offload large JSON artifacts.
- Replace direct tool returns with summary objects that include result handles for oversized lists or dictionaries.
- Build paging tools that allow the model to request specific slices of data using offsets and limits when needed.
- Use descriptor-based compaction for nested dictionaries to preserve structural understanding without full content injection.
- Test your agent with realistic large payloads to measure latency improvements and token savings after refactoring.



