BKCI-Builder-Mutator基于Kind集群的端到端测试指南
本文档基于 macOS (Apple Silicon) 环境实测验证,记录从零搭建 Kind 集群到完成全部测试用例的完整流程。
一、前置准备
1.1 工具要求
kind version # >= 0.20.0
kubectl version # >= 1.28
helm version # >= 3.12
docker version # Docker Desktop 运行中
go version # >= 1.22
macOS 安装:
brew install kind kubectl helm go
1.2 项目源码
cd /path/to/bkci-builder-mutator
1.3 关于 MySQL 依赖
BKCI Builder Mutator 的 MySQL resolver 是可选组件。当环境变量 BKCI_MYSQL_HOST 为空时,webhook 正常启动但跳过 PipelineRef 匹配,只有 namespace 匹配生效。
本测试指南使用无 MySQL 模式运行,覆盖 namespace 匹配、资源变更、DryRun、Max 策略等核心功能。PipelineRef + MySQL 的集成测试需要额外部署 MySQL 实例,不在本指南范围内。
二、创建 Kind 集群
2.1 集群配置
创建 kind-config.yaml:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
两个 worker 节点用于验证 HA 部署(2 replicas 分布到不同节点)。
2.2 创建集群
kind create cluster --name bkcimutator-test --config kind-config.yaml
2.3 导出 kubeconfig
Kind 生成的 kubeconfig 可能存在权限问题,建议导出到独立文件:
kind get kubeconfig --name bkcimutator-test > /tmp/bkcimutator-kubeconfig
export KUBECONFIG=/tmp/bkcimutator-kubeconfig
后续所有
kubectl/helm命令均依赖此环境变量,请确保每个终端窗口都已 export。
验证集群就绪:
kubectl get nodes
# NAME STATUS ROLES AGE VERSION
# bkcimutator-test-control-plane Ready control-plane 30s v1.31.2
# bkcimutator-test-worker Ready <none> 20s v1.31.2
# bkcimutator-test-worker2 Ready <none> 20s v1.31.2
三、构建镜像并加载到 Kind
Kind 集群使用独立的容器运行时,不能直接访问宿主机 Docker 镜像,需要将镜像显式加载进去。
3.1 构建 Docker 镜像
cd /path/to/bkci-builder-mutator
# 构建机 / Linux amd64(默认)
docker build -t bkci-builder-mutator:test .
# 本地 Apple Silicon Mac
docker build --build-arg GOARCH=arm64 -t bkci-builder-mutator:test .
3.2 加载到 Kind 集群
kind load docker-image bkci-builder-mutator:test --name bkcimutator-test
验证镜像已加载:
docker exec -it bkcimutator-test-worker crictl images | grep bkcimutator
四、部署 BKCI Builder Mutator
4.1 Helm 安装
helm upgrade --install bkci-builder-mutator chart/ \
--namespace bkcimutator-system \
--create-namespace \
--set image.repository=bkci-builder-mutator \
--set image.tag=test \
--set image.pullPolicy=Never \
--set bkci.mysql.host=""
pullPolicy=Never确保使用 Kind 中预加载的本地镜像,而不是尝试从远程仓库拉取。
bkci.mysql.host=""启用无 MySQL 模式,PipelineRef 规则不生效,但 namespace 匹配正常工作。
4.2 验证部署
# 检查 Pod 状态(默认 2 副本)
kubectl get pods -n bkcimutator-system
# NAME READY STATUS RESTARTS AGE
# bkci-builder-mutator-xxxxx 1/1 Running 0 30s
# bkci-builder-mutator-yyyyy 1/1 Running 0 30s
# 检查 webhook 配置
kubectl get mutatingwebhookconfigurations
# NAME WEBHOOKS AGE
# bkci-builder-mutator 1 30s
# 检查 CRD 已注册
kubectl get crd bkcibuildermutationrules.bkcimutator.io
# 检查 TLS 证书 Secret 已生成
kubectl get secret -n bkcimutator-system | grep tls
# bkci-builder-mutator-tls kubernetes.io/tls 3 30s
如果 Pod 未就绪,排查:
kubectl logs -n bkcimutator-system -l app.kubernetes.io/name=bkci-builder-mutator
kubectl get events -n bkcimutator-system --sort-by='.lastTimestamp'
4.3 确认无 MySQL 模式生效
kubectl logs -n bkcimutator-system -l app.kubernetes.io/name=bkci-builder-mutator | grep "BKCI_MYSQL_HOST"
# {"level":"info",...,"msg":"BKCI_MYSQL_HOST not set, running without MySQL resolver (PipelineRef rules disabled)"}
五、创建测试 namespace 和规则
5.1 创建目标 namespace
kubectl create namespace test-bkci
kubectl label namespace test-bkci bkcimutator.io/enabled=true
验证:
kubectl get namespace test-bkci --show-labels
# NAME STATUS AGE LABELS
# test-bkci Active 5s bkcimutator.io/enabled=true,...
5.2 创建测试用 BKCIBuilderMutationRule
cat <<'EOF' | kubectl apply -f -
apiVersion: bkcimutator.io/v1alpha1
kind: BKCIBuilderMutationRule
metadata:
name: test-memory-fix
spec:
description: "测试规则:将匹配 namespace 的 Pod memory request 提升到 4Gi"
match:
namespaces: ["test-bkci"]
mode: Enforce
mutations:
resources:
requests:
memory: "4Gi"
cpu: "500m"
limits:
memory: "8Gi"
cpu: "2"
strategy: Replace
annotations:
bkcimutator.io/mutated: "true"
bkcimutator.io/rule: "test-memory-fix"
EOF
验证规则创建:
kubectl get bkcibuildermutationrules
# NAME MODE AGE
# test-memory-fix Enforce 5s
六、测试用例
6.1 测试一:匹配的 Pod 被正确 mutate
创建一个位于 test-bkci namespace 的 Pod(资源原始值:memory 1Gi / cpu 100m):
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: test-builder-match
namespace: test-bkci
spec:
containers:
- name: worker
image: busybox
command: ["sleep", "3600"]
resources:
requests:
memory: "1Gi"
cpu: "100m"
limits:
memory: "2Gi"
cpu: "1"
EOF
验证资源被修改:
# 检查 resource 被 Replace 为规则定义的值
kubectl get pod test-builder-match -n test-bkci -o jsonpath='{.spec.containers[0].resources}' | jq .
# {
# "limits": { "cpu": "2", "memory": "8Gi" },
# "requests": { "cpu": "500m", "memory": "4Gi" }
# }
# 检查审计注解
kubectl get pod test-builder-match -n test-bkci -o jsonpath='{.metadata.annotations}' | jq .
# {
# "bkcimutator.io/mutated": "true",
# "bkcimutator.io/rule": "test-memory-fix",
# "bkcimutator.io/rules": "test-memory-fix",
# ...
# }
预期结果:
- memory request: 1Gi → 4Gi
- memory limit: 2Gi → 8Gi
- cpu request: 100m → 500m
- cpu limit: 1 → 2
- 新增审计注解
bkcimutator.io/mutated=true、bkcimutator.io/rules=test-memory-fix
6.2 测试二:非目标 namespace 不受影响
kubectl create namespace test-other
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: test-other-ns
namespace: test-other
spec:
containers:
- name: worker
image: busybox
command: ["sleep", "3600"]
resources:
requests:
memory: "1Gi"
EOF
kubectl get pod test-other-ns -n test-other -o jsonpath='{.spec.containers[0].resources.requests.memory}'
# 1Gi ← namespace 未启用 bkcimutator.io/enabled label,webhook 未拦截
kubectl get pod test-other-ns -n test-other -o jsonpath='{.metadata.annotations.bkcimutator\.io/mutated}'
# (空,无此注解)
6.3 测试三:DryRun 模式
# 将规则切换为 DryRun
kubectl patch bkcibuildermutationrule test-memory-fix --type=merge -p '{"spec":{"mode":"DryRun"}}'
# 创建新 Pod
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: test-dryrun
namespace: test-bkci
spec:
containers:
- name: worker
image: busybox
command: ["sleep", "3600"]
resources:
requests:
memory: "1Gi"
EOF
# 资源不应被修改
kubectl get pod test-dryrun -n test-bkci -o jsonpath='{.spec.containers[0].resources.requests.memory}'
# 1Gi ← DryRun 不实际 patch 资源
# 审计注解仍存在,但规则名带 (dry-run) 后缀
kubectl get pod test-dryrun -n test-bkci -o jsonpath='{.metadata.annotations.bkcimutator\.io/rules}'
# test-memory-fix (dry-run)
# webhook 日志中可以看到 patchCount=2(仅审计注解,无资源 patch)
kubectl logs -n bkcimutator-system -l app.kubernetes.io/name=bkci-builder-mutator | grep "test-dryrun"
测试完恢复为 Enforce:
kubectl patch bkcibuildermutationrule test-memory-fix --type=merge -p '{"spec":{"mode":"Enforce"}}'
6.4 测试四:Resource 策略 — Max
先删除 Replace 规则(避免两个规则同时生效干扰测试结果),创建 Max 规则:
kubectl delete bkcibuildermutationrule test-memory-fix
cat <<'EOF' | kubectl apply -f -
apiVersion: bkcimutator.io/v1alpha1
kind: BKCIBuilderMutationRule
metadata:
name: test-max-strategy
spec:
match:
namespaces: ["test-bkci"]
mode: Enforce
mutations:
resources:
requests:
memory: "4Gi"
strategy: Max
EOF
场景 A:Pod 原始 request 低于 4Gi → 被提升
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: test-max-low
namespace: test-bkci
spec:
containers:
- name: app
image: busybox
command: ["sleep", "3600"]
resources:
requests:
memory: "1Gi"
EOF
kubectl get pod test-max-low -n test-bkci -o jsonpath='{.spec.containers[0].resources.requests.memory}'
# 4Gi ← max(1Gi, 4Gi) = 4Gi
场景 B:Pod 原始 request 高于 4Gi → 保持不变
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: test-max-high
namespace: test-bkci
spec:
containers:
- name: app
image: busybox
command: ["sleep", "3600"]
resources:
requests:
memory: "8Gi"
EOF
kubectl get pod test-max-high -n test-bkci -o jsonpath='{.spec.containers[0].resources.requests.memory}'
# 8Gi ← max(8Gi, 4Gi) = 8Gi,保持原值
测试完恢复 Replace 规则:
kubectl delete bkcibuildermutationrule test-max-strategy
cat <<'EOF' | kubectl apply -f -
apiVersion: bkcimutator.io/v1alpha1
kind: BKCIBuilderMutationRule
metadata:
name: test-memory-fix
spec:
match:
namespaces: ["test-bkci"]
mode: Enforce
mutations:
resources:
requests:
memory: "4Gi"
cpu: "500m"
limits:
memory: "8Gi"
cpu: "2"
strategy: Replace
annotations:
bkcimutator.io/mutated: "true"
bkcimutator.io/rule: "test-memory-fix"
EOF
6.5 测试五:Webhook 高可用
# 删掉一个副本,验证另一个继续工作
POD=$(kubectl get pod -n bkcimutator-system -l app.kubernetes.io/name=bkci-builder-mutator -o jsonpath='{.items[0].metadata.name}')
kubectl delete pod $POD -n bkcimutator-system
# 立即创建新 Pod 测试
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: test-ha
namespace: test-bkci
spec:
containers:
- name: worker
image: busybox
command: ["sleep", "3600"]
resources:
requests:
memory: "1Gi"
EOF
kubectl get pod test-ha -n test-bkci -o jsonpath='{.spec.containers[0].resources.requests.memory}'
# 4Gi ← 另一个副本正常工作
6.6 测试六:Prometheus 指标
# 端口转发到 metrics 端口
kubectl port-forward -n bkcimutator-system svc/bkci-builder-mutator 8080:8080 &
# 查看指标
curl -s localhost:8080/metrics | grep bkcimutator
# bkcimutator_requests_total{mode="enforce",namespace="test-bkci",rule="test-memory-fix"} 3
# bkcimutator_request_duration_seconds_count{namespace="test-bkci"} 3
# bkcimutator_rules_matched_total{namespace="test-bkci",rule="test-memory-fix"} 3
# bkcimutator_patches_applied_total{namespace="test-bkci",rule="test-memory-fix"} 3
# 结束端口转发
kill %1
七、检查日志
7.1 Webhook 日志
kubectl logs -n bkcimutator-system -l app.kubernetes.io/name=bkci-builder-mutator
示例输出:
{
"level": "info",
"msg": "mutation applied",
"pod": "test-builder-match",
"namespace": "test-bkci",
"matchedRules": ["test-memory-fix"],
"patchCount": 8,
"dryRun": false
}
7.2 DryRun 日志
{
"level": "info",
"msg": "mutation applied",
"pod": "test-dryrun",
"namespace": "test-bkci",
"matchedRules": ["test-memory-fix (dry-run)"],
"patchCount": 2,
"dryRun": false
}
注意 DryRun 模式下 patchCount=2(仅审计注解),而 Enforce 模式下 patchCount=8(资源 + 审计注解)。
八、排障指南
8.1 TLS 证书问题
如果日志中出现 tls: bad certificate,通常是因为 Secret 和 WebhookConfiguration 的 CA 不匹配(Helm chart 升级时可能发生)。解决方案:
helm upgrade --install bkci-builder-mutator chart/ \
... \
--set tls.regenerate=true
kubectl rollout restart deployment/bkci-builder-mutator -n bkcimutator-system
8.2 Pod 创建未被拦截
- 检查 namespace 是否有 label:
kubectl get namespace <ns> --show-labels | grep bkcimutator - 检查 webhook failurePolicy:
kubectl get mutatingwebhookconfigurations bkci-builder-mutator -o yaml | grep failurePolicy - 如果是
Ignore(默认),webhook 故障时 Pod 会正常创建但不被修改
8.3 规则匹配但资源未变
- 检查规则 mode 是否为
DryRun:kubectl get bkcibuildermutationrule <name> -o jsonpath='{.spec.mode}' - 检查 strategy:
Max策略在 Pod 原始值已经更大时不会修改
九、清理
# 删除测试 Pod
kubectl delete pod --all -n test-bkci
kubectl delete pod --all -n test-other
# 删除测试规则
kubectl delete bkcibuildermutationrule --all
# 卸载 webhook
helm uninstall bkci-builder-mutator -n bkcimutator-system
# 删除 namespace
kubectl delete namespace test-bkci test-other bkcimutator-system
# 删除 Kind 集群
kind delete cluster --name bkcimutator-test
十、测试 Checklist
| # | 测试项 | 预期结果 | 通过 |
|---|---|---|---|
| 1 | 匹配的 Pod 资源被修改 | memory: 1Gi → 4Gi, cpu: 100m → 500m | ✅ |
| 2 | 匹配的 Pod 有审计注解 | bkcimutator.io/mutated=true, bkcimutator.io/rules=test-memory-fix | ✅ |
| 3 | 非目标 namespace 不受影响 | memory 保持 1Gi,无审计注解 | ✅ |
| 4 | DryRun 模式不修改资源 | memory 保持 1Gi,审计注解有 (dry-run) 后缀 | ✅ |
| 5 | Max 策略:低值被提升 | 1Gi → 4Gi | ✅ |
| 6 | Max 策略:高值保持 | 8Gi 不变 | ✅ |
| 7 | 删掉一个副本仍正常 | mutation 继续生效 | ✅ |
| 8 | Prometheus 指标存在 | bkcimutator_* 有数据 | ✅ |