Skip to content

Update mixer deployment configuration #1906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
juliawu wants to merge 19 commits into
base: master
Choose a base branch
from
Draft

Update mixer deployment configuration #1906

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
f7a1a8f
add cpuRequests, set maxSurge to 1, and add pod anti-affinity
juliawu May 20, 2026
010915d
add cpu limits
juliawu May 20, 2026
f6e419a
implement max surge suggestion and remove affinity
juliawu May 20, 2026
04ef31c
add cpuLimit if statement
juliawu May 21, 2026
4cc78fa
Make esp configurable and bump observation esp memory up to 500M
juliawu May 21, 2026
ddd4047
Right-size only cpu limits and requests
juliawu May 21, 2026
e567cb1
update cpu request and limits
juliawu May 22, 2026
7ea1f1f
up esp memory limit
juliawu May 22, 2026
c49e5db
match maxSurge to maxUnavailable and add startupProbe to esp
juliawu May 26, 2026
babac98
remove cpuLimits except for observation
juliawu May 26, 2026
c7972c8
Merge branch 'master' of https://github.com/datacommonsorg/mixer into…
juliawu May 26, 2026
2a45792
bump memory on mixer container in observation pods too, use scaler re…
juliawu May 26, 2026
c8653af
Bump up node resources to match observation pods
juliawu May 27, 2026
498c7f9
undo maxSurge changes
juliawu May 27, 2026
c96be8a
Add gomemlimit, adjust cpuRequests, remove esp startupProbe
juliawu May 27, 2026
d1449ac
Switch to binary instead of decimal suffixes for memory values
juliawu May 27, 2026
1ab4435
rightsize per env
juliawu May 27, 2026
e89ba27
return to basic configuration
juliawu May 27, 2026
7fd86ab
remove gomemlimit
juliawu May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions deploy/helm_charts/mixer/templates/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ spec:
type: RollingUpdate
rollingUpdate:
# maximum number of Pods that can be created over the desired number of Pods
# 25% of default pods (+1 in case default < 4)
maxSurge: {{ $group.maxSurge | default (div $group.replicas 4 | add 1) }}
# Set to 1 to reduce resource overhead during deployment
maxSurge: 1

@gemini-code-assist gemini-code-assist Bot May 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Hardcoding maxSurge to 1 removes the ability to customize the rollout strategy per service group and environment. This change ignores existing configurations such as maxSurge: 0 in deploy/helm_charts/envs/mixer_staging.yaml. Additionally, for large deployments (e.g., the recon group with 90 replicas in production), a maxSurge of 1 will significantly slow down the rollout process as it only creates one new pod at a time. It is recommended to use 1 as a default while still allowing overrides from values.yaml.

      # Set to 1 to reduce resource overhead during deployment, but allow overrides.
      maxSurge: {{ $group.maxSurge | default 1 }}
References
  1. For mixer container deployments, configuration keys can be added to the base values.yaml file to provide default values that are then overridden by environment-specific files.

# Maximum number of pods that can be unavailable during the update process
# 25% of default pods (+1 in case default < 4)
maxUnavailable: {{ div $group.replicas 4 | add 1 }}
Expand All @@ -60,6 +60,24 @@ spec:
nodeSelector:
cloud.google.com/gke-nodepool: {{ $group.nodePool }}
{{- end }}
{{- if $group.enableAntiAffinity }}
# Soft pod anti-affinity at the node level.
# Encourages the scheduler to place each replica of this service on a distinct node.
# This is a soft rule, so if there are not enough nodes, the scheduler will schedule pods
# on the same node rather than letting them get stuck in a Pending state.
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: service
operator: In
values:
- {{ include "mixer.fullname" $ }}-{{ $serviceName }}
topologyKey: "kubernetes.io/hostname"
{{- end}}
serviceAccountName: {{ $.Values.serviceAccount.name }}
volumes:
- name: schema-mapping
Expand Down
6 changes: 6 additions & 0 deletions deploy/helm_charts/mixer/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -436,6 +436,7 @@ serviceGroups:
resources:
memoryRequest: "8G"
memoryLimit: "8G"
cpuRequest: "100m"
cacheSVG: true
# If the name of the recon service is changed here,
# please also change the name in the deployment.yaml template.
Expand All @@ -448,6 +449,7 @@ serviceGroups:
resources:
memoryRequest: "2G"
memoryLimit: "2G"
cpuRequest: "100m"
observation:
urlPaths:
- "/bulk/stats"
Expand All @@ -459,6 +461,8 @@ serviceGroups:
resources:
memoryRequest: "4G"
memoryLimit: "8G"
cpuRequest: "1"
enableAntiAffinity: true

@gemini-code-assist gemini-code-assist Bot May 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Pod anti-affinity is currently only enabled for the observation group. Given that other service groups like recon and node also have a high number of replicas in production (90 and 60 respectively), it is recommended to enable anti-affinity for them as well. This ensures that replicas are spread across different nodes, improving the service's resilience to node failures.

References
  1. Configuration keys for mixer deployments can be added to the base values.yaml file to establish default behaviors across environments.

node:
urlPaths:
- "/node/*"
Expand All @@ -473,6 +477,7 @@ serviceGroups:
resources:
memoryRequest: "8G"
memoryLimit: "8G"
cpuRequest: "500m"
cacheSVG: true
default:
urlPaths:
Expand All @@ -481,3 +486,4 @@ serviceGroups:
resources:
memoryRequest: "2G"
memoryLimit: "2G"
cpuRequest: "100m"
Loading