BKCI-Builder-Mutator基于Kind集群的端到端测试指南

本文档基于 macOS (Apple Silicon) 环境实测验证,记录从零搭建 Kind 集群到完成全部测试用例的完整流程。

一、前置准备

1.1 工具要求

kind version        # >= 0.20.0
kubectl version     # >= 1.28
helm version        # >= 3.12
docker version      # Docker Desktop 运行中
go version          # >= 1.22

macOS 安装:

brew install kind kubectl helm go

1.2 项目源码

cd /path/to/bkci-builder-mutator

1.3 关于 MySQL 依赖

BKCI Builder Mutator 的 MySQL resolver 是可选组件。当环境变量 BKCI_MYSQL_HOST 为空时,webhook 正常启动但跳过 PipelineRef 匹配,只有 namespace 匹配生效。

本测试指南使用无 MySQL 模式运行,覆盖 namespace 匹配、资源变更、DryRun、Max 策略等核心功能。PipelineRef + MySQL 的集成测试需要额外部署 MySQL 实例,不在本指南范围内。

二、创建 Kind 集群

2.1 集群配置

创建 kind-config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

两个 worker 节点用于验证 HA 部署(2 replicas 分布到不同节点)。

2.2 创建集群

kind create cluster --name bkcimutator-test --config kind-config.yaml

2.3 导出 kubeconfig

Kind 生成的 kubeconfig 可能存在权限问题,建议导出到独立文件:

kind get kubeconfig --name bkcimutator-test > /tmp/bkcimutator-kubeconfig
export KUBECONFIG=/tmp/bkcimutator-kubeconfig

后续所有 kubectl / helm 命令均依赖此环境变量,请确保每个终端窗口都已 export。

验证集群就绪:

kubectl get nodes
# NAME                             STATUS   ROLES           AGE   VERSION
# bkcimutator-test-control-plane   Ready    control-plane   30s   v1.31.2
# bkcimutator-test-worker          Ready    <none>          20s   v1.31.2
# bkcimutator-test-worker2         Ready    <none>          20s   v1.31.2

三、构建镜像并加载到 Kind

Kind 集群使用独立的容器运行时,不能直接访问宿主机 Docker 镜像,需要将镜像显式加载进去。

3.1 构建 Docker 镜像

cd /path/to/bkci-builder-mutator

# 构建机 / Linux amd64(默认)
docker build -t bkci-builder-mutator:test .

# 本地 Apple Silicon Mac
docker build --build-arg GOARCH=arm64 -t bkci-builder-mutator:test .

3.2 加载到 Kind 集群

kind load docker-image bkci-builder-mutator:test --name bkcimutator-test

验证镜像已加载:

docker exec -it bkcimutator-test-worker crictl images | grep bkcimutator

四、部署 BKCI Builder Mutator

4.1 Helm 安装

helm upgrade --install bkci-builder-mutator chart/ \
  --namespace bkcimutator-system \
  --create-namespace \
  --set image.repository=bkci-builder-mutator \
  --set image.tag=test \
  --set image.pullPolicy=Never \
  --set bkci.mysql.host=""

pullPolicy=Never 确保使用 Kind 中预加载的本地镜像,而不是尝试从远程仓库拉取。

bkci.mysql.host="" 启用无 MySQL 模式,PipelineRef 规则不生效,但 namespace 匹配正常工作。

4.2 验证部署

# 检查 Pod 状态(默认 2 副本)
kubectl get pods -n bkcimutator-system
# NAME                                    READY   STATUS    RESTARTS   AGE
# bkci-builder-mutator-xxxxx              1/1     Running   0          30s
# bkci-builder-mutator-yyyyy              1/1     Running   0          30s

# 检查 webhook 配置
kubectl get mutatingwebhookconfigurations
# NAME                   WEBHOOKS   AGE
# bkci-builder-mutator   1          30s

# 检查 CRD 已注册
kubectl get crd bkcibuildermutationrules.bkcimutator.io

# 检查 TLS 证书 Secret 已生成
kubectl get secret -n bkcimutator-system | grep tls
# bkci-builder-mutator-tls   kubernetes.io/tls   3      30s

如果 Pod 未就绪,排查:

kubectl logs -n bkcimutator-system -l app.kubernetes.io/name=bkci-builder-mutator
kubectl get events -n bkcimutator-system --sort-by='.lastTimestamp'

4.3 确认无 MySQL 模式生效

kubectl logs -n bkcimutator-system -l app.kubernetes.io/name=bkci-builder-mutator | grep "BKCI_MYSQL_HOST"
# {"level":"info",...,"msg":"BKCI_MYSQL_HOST not set, running without MySQL resolver (PipelineRef rules disabled)"}

五、创建测试 namespace 和规则

5.1 创建目标 namespace

kubectl create namespace test-bkci
kubectl label namespace test-bkci bkcimutator.io/enabled=true

验证:

kubectl get namespace test-bkci --show-labels
# NAME        STATUS   AGE   LABELS
# test-bkci   Active   5s    bkcimutator.io/enabled=true,...

5.2 创建测试用 BKCIBuilderMutationRule

cat <<'EOF' | kubectl apply -f -
apiVersion: bkcimutator.io/v1alpha1
kind: BKCIBuilderMutationRule
metadata:
  name: test-memory-fix
spec:
  description: "测试规则:将匹配 namespace 的 Pod memory request 提升到 4Gi"

  match:
    namespaces: ["test-bkci"]

  mode: Enforce

  mutations:
    resources:
      requests:
        memory: "4Gi"
        cpu: "500m"
      limits:
        memory: "8Gi"
        cpu: "2"
      strategy: Replace
    annotations:
      bkcimutator.io/mutated: "true"
      bkcimutator.io/rule: "test-memory-fix"
EOF

验证规则创建:

kubectl get bkcibuildermutationrules
# NAME              MODE      AGE
# test-memory-fix   Enforce   5s

六、测试用例

6.1 测试一:匹配的 Pod 被正确 mutate

创建一个位于 test-bkci namespace 的 Pod(资源原始值:memory 1Gi / cpu 100m):

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-builder-match
  namespace: test-bkci
spec:
  containers:
    - name: worker
      image: busybox
      command: ["sleep", "3600"]
      resources:
        requests:
          memory: "1Gi"
          cpu: "100m"
        limits:
          memory: "2Gi"
          cpu: "1"
EOF

验证资源被修改:

# 检查 resource 被 Replace 为规则定义的值
kubectl get pod test-builder-match -n test-bkci -o jsonpath='{.spec.containers[0].resources}' | jq .
# {
#   "limits": { "cpu": "2", "memory": "8Gi" },
#   "requests": { "cpu": "500m", "memory": "4Gi" }
# }

# 检查审计注解
kubectl get pod test-builder-match -n test-bkci -o jsonpath='{.metadata.annotations}' | jq .
# {
#   "bkcimutator.io/mutated": "true",
#   "bkcimutator.io/rule": "test-memory-fix",
#   "bkcimutator.io/rules": "test-memory-fix",
#   ...
# }

预期结果:

6.2 测试二:非目标 namespace 不受影响

kubectl create namespace test-other

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-other-ns
  namespace: test-other
spec:
  containers:
    - name: worker
      image: busybox
      command: ["sleep", "3600"]
      resources:
        requests:
          memory: "1Gi"
EOF
kubectl get pod test-other-ns -n test-other -o jsonpath='{.spec.containers[0].resources.requests.memory}'
# 1Gi  ← namespace 未启用 bkcimutator.io/enabled label,webhook 未拦截

kubectl get pod test-other-ns -n test-other -o jsonpath='{.metadata.annotations.bkcimutator\.io/mutated}'
# (空,无此注解)

6.3 测试三:DryRun 模式

# 将规则切换为 DryRun
kubectl patch bkcibuildermutationrule test-memory-fix --type=merge -p '{"spec":{"mode":"DryRun"}}'

# 创建新 Pod
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-dryrun
  namespace: test-bkci
spec:
  containers:
    - name: worker
      image: busybox
      command: ["sleep", "3600"]
      resources:
        requests:
          memory: "1Gi"
EOF
# 资源不应被修改
kubectl get pod test-dryrun -n test-bkci -o jsonpath='{.spec.containers[0].resources.requests.memory}'
# 1Gi  ← DryRun 不实际 patch 资源

# 审计注解仍存在,但规则名带 (dry-run) 后缀
kubectl get pod test-dryrun -n test-bkci -o jsonpath='{.metadata.annotations.bkcimutator\.io/rules}'
# test-memory-fix (dry-run)

# webhook 日志中可以看到 patchCount=2(仅审计注解,无资源 patch)
kubectl logs -n bkcimutator-system -l app.kubernetes.io/name=bkci-builder-mutator | grep "test-dryrun"

测试完恢复为 Enforce:

kubectl patch bkcibuildermutationrule test-memory-fix --type=merge -p '{"spec":{"mode":"Enforce"}}'

6.4 测试四:Resource 策略 — Max

先删除 Replace 规则(避免两个规则同时生效干扰测试结果),创建 Max 规则:

kubectl delete bkcibuildermutationrule test-memory-fix

cat <<'EOF' | kubectl apply -f -
apiVersion: bkcimutator.io/v1alpha1
kind: BKCIBuilderMutationRule
metadata:
  name: test-max-strategy
spec:
  match:
    namespaces: ["test-bkci"]
  mode: Enforce
  mutations:
    resources:
      requests:
        memory: "4Gi"
      strategy: Max
EOF

场景 A:Pod 原始 request 低于 4Gi → 被提升

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-max-low
  namespace: test-bkci
spec:
  containers:
    - name: app
      image: busybox
      command: ["sleep", "3600"]
      resources:
        requests:
          memory: "1Gi"
EOF

kubectl get pod test-max-low -n test-bkci -o jsonpath='{.spec.containers[0].resources.requests.memory}'
# 4Gi  ← max(1Gi, 4Gi) = 4Gi

场景 B:Pod 原始 request 高于 4Gi → 保持不变

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-max-high
  namespace: test-bkci
spec:
  containers:
    - name: app
      image: busybox
      command: ["sleep", "3600"]
      resources:
        requests:
          memory: "8Gi"
EOF

kubectl get pod test-max-high -n test-bkci -o jsonpath='{.spec.containers[0].resources.requests.memory}'
# 8Gi  ← max(8Gi, 4Gi) = 8Gi,保持原值

测试完恢复 Replace 规则:

kubectl delete bkcibuildermutationrule test-max-strategy

cat <<'EOF' | kubectl apply -f -
apiVersion: bkcimutator.io/v1alpha1
kind: BKCIBuilderMutationRule
metadata:
  name: test-memory-fix
spec:
  match:
    namespaces: ["test-bkci"]
  mode: Enforce
  mutations:
    resources:
      requests:
        memory: "4Gi"
        cpu: "500m"
      limits:
        memory: "8Gi"
        cpu: "2"
      strategy: Replace
    annotations:
      bkcimutator.io/mutated: "true"
      bkcimutator.io/rule: "test-memory-fix"
EOF

6.5 测试五:Webhook 高可用

# 删掉一个副本,验证另一个继续工作
POD=$(kubectl get pod -n bkcimutator-system -l app.kubernetes.io/name=bkci-builder-mutator -o jsonpath='{.items[0].metadata.name}')
kubectl delete pod $POD -n bkcimutator-system

# 立即创建新 Pod 测试
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-ha
  namespace: test-bkci
spec:
  containers:
    - name: worker
      image: busybox
      command: ["sleep", "3600"]
      resources:
        requests:
          memory: "1Gi"
EOF

kubectl get pod test-ha -n test-bkci -o jsonpath='{.spec.containers[0].resources.requests.memory}'
# 4Gi  ← 另一个副本正常工作

6.6 测试六:Prometheus 指标

# 端口转发到 metrics 端口
kubectl port-forward -n bkcimutator-system svc/bkci-builder-mutator 8080:8080 &

# 查看指标
curl -s localhost:8080/metrics | grep bkcimutator
# bkcimutator_requests_total{mode="enforce",namespace="test-bkci",rule="test-memory-fix"} 3
# bkcimutator_request_duration_seconds_count{namespace="test-bkci"} 3
# bkcimutator_rules_matched_total{namespace="test-bkci",rule="test-memory-fix"} 3
# bkcimutator_patches_applied_total{namespace="test-bkci",rule="test-memory-fix"} 3

# 结束端口转发
kill %1

七、检查日志

7.1 Webhook 日志

kubectl logs -n bkcimutator-system -l app.kubernetes.io/name=bkci-builder-mutator

示例输出:

{
  "level": "info",
  "msg": "mutation applied",
  "pod": "test-builder-match",
  "namespace": "test-bkci",
  "matchedRules": ["test-memory-fix"],
  "patchCount": 8,
  "dryRun": false
}

7.2 DryRun 日志

{
  "level": "info",
  "msg": "mutation applied",
  "pod": "test-dryrun",
  "namespace": "test-bkci",
  "matchedRules": ["test-memory-fix (dry-run)"],
  "patchCount": 2,
  "dryRun": false
}

注意 DryRun 模式下 patchCount=2(仅审计注解),而 Enforce 模式下 patchCount=8(资源 + 审计注解)。

八、排障指南

8.1 TLS 证书问题

如果日志中出现 tls: bad certificate,通常是因为 Secret 和 WebhookConfiguration 的 CA 不匹配(Helm chart 升级时可能发生)。解决方案:

helm upgrade --install bkci-builder-mutator chart/ \
  ... \
  --set tls.regenerate=true

kubectl rollout restart deployment/bkci-builder-mutator -n bkcimutator-system

8.2 Pod 创建未被拦截

  1. 检查 namespace 是否有 label:kubectl get namespace <ns> --show-labels | grep bkcimutator
  2. 检查 webhook failurePolicy:kubectl get mutatingwebhookconfigurations bkci-builder-mutator -o yaml | grep failurePolicy
  3. 如果是 Ignore(默认),webhook 故障时 Pod 会正常创建但不被修改

8.3 规则匹配但资源未变

  1. 检查规则 mode 是否为 DryRun:kubectl get bkcibuildermutationrule <name> -o jsonpath='{.spec.mode}'
  2. 检查 strategy:Max 策略在 Pod 原始值已经更大时不会修改

九、清理

# 删除测试 Pod
kubectl delete pod --all -n test-bkci
kubectl delete pod --all -n test-other

# 删除测试规则
kubectl delete bkcibuildermutationrule --all

# 卸载 webhook
helm uninstall bkci-builder-mutator -n bkcimutator-system

# 删除 namespace
kubectl delete namespace test-bkci test-other bkcimutator-system

# 删除 Kind 集群
kind delete cluster --name bkcimutator-test

十、测试 Checklist

# 测试项 预期结果 通过
1 匹配的 Pod 资源被修改 memory: 1Gi → 4Gi, cpu: 100m → 500m ✅
2 匹配的 Pod 有审计注解 bkcimutator.io/mutated=true, bkcimutator.io/rules=test-memory-fix ✅
3 非目标 namespace 不受影响 memory 保持 1Gi,无审计注解 ✅
4 DryRun 模式不修改资源 memory 保持 1Gi,审计注解有 (dry-run) 后缀 ✅
5 Max 策略:低值被提升 1Gi → 4Gi ✅
6 Max 策略:高值保持 8Gi 不变 ✅
7 删掉一个副本仍正常 mutation 继续生效 ✅
8 Prometheus 指标存在 bkcimutator_* 有数据 ✅