I'm Amry Fitra Amanah, an Engineer on the Agent Core team at JAPAN AI株式会社, where we build frameworks for AI Agent Services. Our team develops the underlying platform that enables users to implement their own AI solutions. Working on this infrastructure gives our team visibility into how diverse organizations use the framework to solve different problems. Through analyzing usage patterns, token consumption data, and user feedback across our platform, we've observed that successful AI implementations aren't about using the most powerful models—they're about matching capabilities to actual needs.
As part of our release of JAPAN AI's super agent platform—we faced a significant engineering challenge: how do we integrate with the rapidly growing Model Context Protocol (MCP) ecosystem while maintaining cost efficiency and multi-tenant isolation?
The MCP standard has fostered a thriving ecosystem of specialized servers developed by the open-source community and various organizations. These servers cover diverse use cases—from GitHub and Gmail integrations to Slack, Notion, and enterprise-specific workflows. The value proposition is clear: by integrating with MCP, we unlock immediate access to this expanding toolkit for our users.
However, the standard MCP deployment model presents a fundamental problem. Each MCP server instance is tied to a specific user's authentication token. For example, a GitHub MCP server requires a user's personal access token to operate. This creates a strict 1:1 mapping between user+token and server instance—if we kept all instances running continuously for every user, our infrastructure costs would skyrocket.
This article details how we built JAI MCP Cluster: a Kubernetes-native infrastructure that dynamically provisions per-user MCP servers with scale-to-zero capabilities, achieving production cold starts under 16 seconds with sub-200ms median response times for warm requests—acceptable latency for agentic workflows where LLM reasoning already takes several seconds.
We built JAI MCP Cluster to solve the cost problem of per-user MCP servers. Using Kubernetes + Knative, we achieve:
JAI MCP Cluster is our self-hosted MCP server orchestration platform built on Kubernetes and Knative. It solves the core tension between per-user isolation requirements and cost efficiency through intelligent autoscaling and dynamic provisioning.