⭐️This is Day 23 of the JAPAN AI Advent Calendar 2025⭐️

This is Kui Xiong, tech lead of the Agent Core team at JAPAN AI, where we build our agentic framework that enables other teams to build powerful agents for our customers.

LLMs power an agent's ability to complete tasks, but only when we provide them with accurate and comprehensive context. Without proper context management, even the most advanced models fail.

Introduction

With the progress of fundamental LLM and the prevalence of agents, the complexity of tasks that agents complete continues to increase. Correspondingly, the amount of contextual information that agents need to process is also growing rapidly. In this situation, we face:

• Growing context gradually becomes noise, unable to be effectively utilized by LLMs or even misleading LLMs into making incorrect judgments
• Massive context gradually exceeds LLM context window limits, causing tasks to fail in the middle
• The number of tools corresponding to task completion keeps growing, ineffectively occupying context

Although for over the past 3 years, we see model hallucination rates decreasing and model context handling capacity increasing over time, just as in the traditional computer industry where we still need an advanced operating system to maximize chip capabilities, to fully leverage LLM performance, we need an effective system to manage context and accurately send it to LLMs for analysis and processing when needed. This is what we call Context Engineering when it comes to managing the context for each user's request before involving LLM.

Nowadays, we have implemented a sophisticated system to compose the context for LLM both cost friendly and efficiently, from which I would like to highlight the following:

• Our memory extraction/retrieval system to provide LLMs with long-term important information, along with windowed user-assistant message history and dynamic context injection system to compose the final message list for LLM
• Multi-agent architecture to reduce context occupation from tool call results with isolated context for sub-agent to complete independent task
• Both user command driven and auto detection driven context compaction methods to prevent context overflow
• Intelligent tool selection and dynamic tool activation to reduce tool token consumption and still remain accessibility to necessary tools

How We Manage Context

LLM context generally consists of two parts, as we can see in below illustration:

• Messages: system instructions, user instructions, assistant responses and tool execution results
• Available tool list: tool descriptions and parameter descriptions