Repository navigation
NB Bank: Building a Full-stack Cloud-native AI Service System Based on Kubernetes and HAMi #141
Description
Name
Kubernetes and HAMi Deployment at NB Bank
Organization
Bank of Ningbo Co., Ltd. (“NB Bank”), DaoCloud
Team(s)
Bank of Ningbo Co., Ltd. (“NB Bank”) is a leading regional city commercial bank with a strong presence in local inclusive finance, retail banking, corporate banking, and intelligent risk management. The bank continues to advance its digital and intelligent transformation. In recent years, NB Bank has steadily deployed AI capabilities including private deployment of financial large language models, scenario-specific fine-tuning, and intelligent inference services. It has built a diverse portfolio of model assets encompassing large-scale general-purpose models, long-context business models, lightweight question-answering models, and financial-domain-specific models. At the same time, the bank supports multiple types of heterogeneous computing hardware, including general-purpose GPUs and domestic AI accelerators, and has gradually established a scalable, operationalized in-house AI ecosystem.
Compared with large banks, city commercial banks typically focus their AI strategies on lightweight implementation, steady iteration, cost reduction, and efficiency improvement rather than the accumulation of very large clusters. They place greater emphasis on reusing existing computing resources, standardizing model-service governance, and accurately matching technology to business scenarios. This cloud-native AI initiative is a core project in NB Bank’s effort to standardize and consolidate its AI infrastructure.
Overview and Goals
Built on the open source Kubernetes foundation, NB Bank developed a lightweight cloud-native AI support system tailored to a city commercial bank, centered on two core capabilities: heterogeneous computing-resource governance with HAMi and inference-service traffic governance with Higress. The system directly addresses production challenges including fragmented heterogeneous computing deployment, uneven resource utilization, mismatched multi-model invocation, and rigid scheduling across multi-chip inference clusters.
This initiative moves away from a capital-intensive, large-scale expansion model. Instead, it uses open source cloud-native technologies to unlock the value of existing computing resources and model assets, enabling lightweight, standardized, and efficient implementation of financial AI scenarios. It provides a practical, replicable benchmark aligned with industry realities for regional city commercial banks and other small and mid-sized banks seeking to scale AI services through the open source ecosystem.
Projects
- Kubernetes: A Unified Cloud-Native Foundation for Financial AI
- HAMi: The Core of Unified Heterogeneous Computing-Resource Governance
- Higress: A Governance Gateway Designed for Financial AI Inference Services
- Kueue: A Technology Reserve for Future Evolution
Planned Evolution
- Continue optimizing HAMi scheduling strategies for heterogeneous computing resources, deepen unified adaptation across domestic accelerators and general-purpose GPUs, and further improve fine-grained computing-resource utilization;
- Continue evolving Higress AI inference scheduling, optimize multi-model routing strategies and weighted traffic scheduling across clusters, and improve intelligent scheduling accuracy for complex business scenarios;
- Strengthen integration between the large-model service platform and the two core scheduling systems to enable intelligent end-to-end scheduling of computing resources, models, and services;
- Introduce Kueue batch-workload queue scheduling in a lightweight manner and, when computing resources are shared across more departments and batch training reaches greater scale, gradually deploy queue-priority and quota-management capabilities;
- Continue documenting implementation experience with heterogeneous AI clusters in a city commercial bank, contribute lessons back to the open source community, and promote scenario-specific optimization of HAMi and Higress for small and mid-sized financial institutions.
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
rochaporto commentedon Oct 18, 2026
@Giannaliu929 do you have availability in the coming days for a quick call? Would love to help push this one through.
rochaporto commentedon Nov 1, 2026
@Giannaliu929 let me know if there's a chance to have a sync on this during the week
Giannaliu929 commentedon Nov 2, 2026
Hi, we've completed the reference architecture documentation. Could you please review it? If you have any questions or feedback, feel free to reach out to us directly. Thanks！
Bank of Ningbo: Cloud-native AI Resource Management and Inference Service Governance with Kubernetes, HAMi and Higress
Organization or Team Name
Bank of Ningbo Co., Ltd. ("Bank of Ningbo"), Shanghai DaoCloud Network Technology Co., Ltd.（"DaoCloud"）
Organization Website
https://www\.nbcb\.com\.cn/
https://www\.daocloud\.io/
Industries
Financial services (city commercial bank)
Relevant CNCF Projects
Kubernetes、HAMi、Higress
Describe your organisation
Bank of Ningbo is a regional city commercial bank with a strong focus on local inclusive finance, retail banking, corporate banking, and intelligent risk management. It continues to advance its digital and AI transformation. In recent years, the Bank has progressively deployed financial LLMs in its private environment, carried out scenario-specific fine-tuning, and introduced intelligent inference services. Its model portfolio now includes large general-purpose models, long-context business models, lightweight question-answering models, and finance-specific models. The Bank also uses heterogeneous compute resources, including general-purpose GPUs and domestically developed accelerators, as AI applications increasingly enter routine production use.
Compared with large banks, city commercial banks generally prioritize lightweight implementation, incremental iteration, and practical results in AI initiatives. Rather than pursuing very large clusters, they focus on making better use of existing compute capacity, standardizing model services, and tailoring them to business scenarios. This project represents a phased effort by the Bank to standardize and consolidate its AI infrastructure.
Key Challenges Before Implementation
As AI moved from pilots into everyday banking services, Bank of Ningbo found itself operating multiple model types across general-purpose GPUs and domestic NPUs. Hardware-specific deployment stacks, whole-card allocation, and independently exposed model services created resource islands. Some clusters were overloaded while others sat underused, and teams had to manually match each workload to a model, accelerator, and endpoint.
Can you expand on why you are using those projects/services?
|Kubernetes: A Unified Cloud-native Foundation|Kubernetes fits the Bank's financial private-cloud architecture. It provides essential capabilities such as container orchestration, resource scheduling, and cluster self-healing, supports AI service deployment across multiple chip types and models, and establishes a standardized foundation for resource management and inference-service governance.|
|HAMi: Unified Management and Resource Scheduling of Heterogeneous Accelerators|HAMi integrates natively with Kubernetes without requiring a redesign of the existing technology stack. Its main value is bringing previously adapted heterogeneous accelerators into the Kubernetes resource management system, providing a relatively consistent resource view, resource request mechanism, and scheduling process while reducing the complexity of managing and configuring different hardware types separately. Drivers, runtimes, model images, and operator adaptations for different chips remain the responsibility of the corresponding hardware platforms. Whole-device or fine-grained resource allocation is selected according to each device's capabilities.|
|Higress: Unified Inference-Service Entry Point and Traffic Governance|Compared with a general-purpose API gateway, Higress provides inference-oriented capabilities including a unified entry point, request-attribute-based routing, backend selection based on traffic weights and health status, removal of unhealthy backends, and basic observability. It is a traffic-governance component at the inference-service layer. It neither replaces underlying compute scheduling nor performs inference-engine performance optimization.|
|Kueue: A Technology Option for Future Development|The Bank currently manages training and inference resources in separate pools, with relatively limited resource contention among tenants and no large-scale requirement for queueing batch jobs. Kueue is being retained as a technology option for future use when compute sharing across departments and batch-training workloads expand; it is not part of the implemented outcomes covered by this case study.|
Brief overview of your architecture and any potential goals you are trying to achieve with it?
Architecture Design
Using Kubernetes as the foundation, the Bank has established a layered architecture centered on "resource management + inference-service governance." From bottom to top, it comprises four layers: heterogeneous compute infrastructure and resource pools, the financial Kubernetes foundation, the LLM service platform, and intelligent business applications. The design follows three principles: (1) separation of concerns, allowing compute, platform, and service layers to evolve independently; (2) unified management of different resource pools through multiple Kubernetes clusters to reduce repetitive configuration; and (3) incremental extension, introducing components such as Kueue and Spiderpool as required by specific scenarios.
Overall Solution
Heterogeneous Compute Infrastructure and Resource Pools
General-purpose GPUs, domestically developed NPUs, and other accelerators, together with supporting high-speed networks and distributed storage, are deployed in the Bank's financial private cloud to form the heterogeneous compute infrastructure. Resources are divided by chip type into a GPU compute pool and a domestic accelerator pool, each managed by an independent Kubernetes cluster. This prevents devices from different vendors from interfering with one another in terms of drivers, runtimes, and operational processes. The layer does not alter chip-specific drivers or runtime environments; instead, it reduces the complexity of managing and configuring diverse resources at the resource-management layer.
Financial Kubernetes Foundation and Compute Resource Management
The financial Kubernetes foundation manages GPU and domestic accelerator pools through a multi-cluster approach, providing a consistent container execution environment for model services and platform components. On this foundation, components including HAMi, Kueue, and Spiderpool are introduced or planned for resource scheduling, queue and quota management, and networking, respectively:
HAMi (heterogeneous compute management and scheduling): Registers accelerators with adapted device plugins and runtime environments as resources that Kubernetes can recognize and request, creating a relatively consistent resource view, request mechanism, and scheduling process. Whole-device allocation or fine-grained partitioning is selected according to hardware capabilities, reducing the complexity of managing and configuring different hardware types.
Kueue (queue and quota management): A planned component for a later phase. When compute sharing across departments and batch-training workloads expand, it will provide queue priorities and quota controls.
Spiderpool (network management): Provides IP address allocation and network orchestration for containers in clusters as a supporting foundation component.
Together with Kubernetes-native scheduling, these three components support standardized resource request templates, scheduling constraints, and deployment workflows, reducing repetitive configuration when the same model is deployed across different clusters.
LLM Service Platform and Inference-Service Governance
The LLM service platform consists of a model management foundation layer and an inference-service governance layer. The foundation layer handles end-to-end model lifecycle management, model tagging, classification and grouping, access control, and a model marketplace, providing a unified view of model assets and a request entry point for different business units. Higress provides the inference-service governance layer, including a unified entry point, input-attribute-based model routing, traffic-weight allocation, health checks and removal of unhealthy backends, operational observability, and security controls. Its responsibilities remain distinct from those of the Bank's existing API gateway and security platforms.
（1）Tiered Multi-Model Routing Based on Input Attributes
To support mixed deployments of large general-purpose models, long-context business models, lightweight question-answering models, and finance-specific models, Higress routes requests to appropriate model services using preconfigured policies. Routing inputs include business origin, API type and path, caller identity, business tags, context length, request priority, SLA tier, the requested model, and backend service status. More complex workloads are routed to larger models, while routine Q&A and lightweight scenarios are routed to smaller models to avoid unnecessary consumption of high-end compute resources.
（2）Logical Pooling of Inference Services Across Chip Types
HAMi allocates resources at deployment time, while Higress schedules traffic at runtime. Once models of the same specification are deployed separately on GPU and domestic accelerator clusters, Higress provides a relatively unified service entry point and distributes new requests among different backends based on preset weights, business rules, and health status. This enables a degree of shared use of cross-chip inference services and basic failover capabilities.
Requests are distributed across inference clusters through preconfigured traffic weights, business-tier policies, service health checks, and threshold-based triggers. When a primary cluster becomes unhealthy or reaches a preset load threshold, eligible new requests are redirected to a standby cluster, providing basic traffic overflow and failover.
Intelligent Business Application Layer
Internal AI applications—including intelligent risk management, intelligent customer service, operational auditing, text parsing, and intelligent marketing—invoke model services through the unified entry point. Business systems do not need to be aware of the underlying chip types or cluster distribution. Combined with model tags and access controls, this enables business units to request model capabilities as needed and gradually reuse models, compute resources, and service capabilities.
What has worked well?
Preliminary Outcomes
The Bank has made initial progress in the following areas of unified resource management and inference-service governance.
Resource Management
The scope of unified management of heterogeneous resources has expanded gradually, with adapted accelerators onboarded through a consistent process.
Duplicate resource configuration and service onboarding work has been reduced.
The applicability of whole-device and fine-grained resource allocation has been validated in selected scenarios.
Inference-Service Governance
Initial tiered multi-model routing capabilities have been established, based on input attributes and preconfigured policies.
Cross-cluster traffic switching mechanisms, including health-check-based backend removal and threshold-triggered diversion, have been validated in selected scenarios.
A technical foundation has been established for future improvements in model invocation costs and service reliability.
Core Value
Clear Layering and Well-Defined Component Responsibilities
The two-layer architecture—"HAMi for underlying resource management + Higress for upper-layer inference-service governance"—matches the current implementation scope, with clearly defined responsibilities that prevent the components' capabilities from being conflated.
Designed for Mixed Heterogeneous Environments at Small and Midsize Banks
Unlike architectures built around a single compute type and standardized models, this solution addresses the coexistence of multiple chip types and mixed deployments of models with different specifications—a common situation among city commercial banks. Starting with resource management and unified service entry points, it provides a lightweight approach with manageable investment requirements and offers a reference for similar institutions.
Pragmatic Reuse of Open Source Components
HAMi focuses on resource management and scheduling, while Higress focuses on inference-service traffic governance. Management of both components is incorporated into the Bank's existing security management framework. The solution is based on open source components configured and integrated for the Bank's deployment and operational needs, without overstating what those components can do.
A Lightweight Path Focused on Optimizing Existing Resources
Rather than relying solely on hardware expansion, the Bank uses resource management and service governance to improve utilization of its existing compute resources and model assets. This offers small and midsize banks a practical implementation path with manageable investment needs and incremental rollout options.
What's next for your architecture? What are you looking to do next?
Expand HAMi coverage: Progressively onboard additional accelerator models once their device plugins and runtime environments have been adapted, while continuing to refine allocation policies for whole devices and partitioned resources.
Strengthen inference-service governance: Improve input-attribute-based routing policies and establish mechanisms for adjusting traffic weights periodically and in response to operating conditions. In the future, lightweight classification or intent recognition components may be introduced, with their outputs used as routing tags by the AI gateway, gradually enabling semantic model selection for AI agent scenarios.
Explore real-time dynamic traffic weighting: Once inference-engine metrics, service discovery, monitoring systems, and weight-control mechanisms are in place, investigate dynamic traffic allocation based on real-time latency, concurrency levels, and node health.
Introduce Kueue: Add queue priority and quota management when compute sharing across departments and batch-training workloads reach sufficient scale.
Consolidate lessons learned: Continue documenting experience with heterogeneous AI clusters in city commercial banks and contribute scenario-specific requirements back to the HAMi and Higress communities.
mrbobbytables commentedon Nov 2, 2026
thanks for the update :)
Here is a link to the PR: cncf/endusers#2092
rochaporto commentedon Nov 2, 2026
Closing as this is now merged, many thanks!
https://endusers.cncf.io/architectures/bank-of-ningbo/
Add Bank of Ningbo reference architecture