蓝盾「Docker公共构建机」缓存清理
背景
在使用蓝盾「Docker公共构建机」一段时间后,我们发现构建镜像偶发性超时。排查后发现是由于集群的 Node 节点的磁盘满了,本文会介绍如何清理构建缓存。
我们发现构建镜像偶发性超时,排查发现是上了 Docker-in-Docker 构建镜像之后发生的,而且发生频率越来越高,进一步排查发现是由于 Pod 会通过 hostPath 挂载工作目录和日志目录,由于构建任务过多导致 Node 节点磁盘打满。
排查过程
事件分析
通过 Pod 事件可以发现是由于 Node 节点磁盘打满,导致 Pod 被驱逐,构建任务失败。
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Evicted 20m kubelet The node was low on resource: ephemeral-storage. Container build1753761077695-ivcpmoxg was using 1580320Ki, which exceeds its request of 0.
Normal NodeHasNoDiskPressure 3m (x32 over 6d5h) kubelet Node 10.10.32.2 status is now: NodeHasNoDiskPressure
Pod yaml:
volumes:
- hostPath:
path: /data/landun/workspace/build1753761077695-ivcpmoxg
type: ""
name: data-volume
- hostPath:
path: /data/landun/logs/build1753761077695-ivcpmoxg
type: ""
name: logs-volume
Pod 通过 hostPath 挂载工作目录和日志目录做缓存,当同一流水线任务重复执行时能够加速。
dispatch-k8s-manager 模块的配置文件:
dispatch:
volume:
builderConfigMap:
name: dispatch-kubernetes-builder
items:
- key: initsh.properties
path: init.sh
- key: sleepsh.properties
path: sleep.sh
hostPath:
dataHostDir: /data/landun/workspace
logsHostDir: /data/landun/logs
cfs:
path: /data/cfs
volumeMount:
dataPath: /data/landun/workspace
logPath: /data/logs
builderConfigMapPath: /data/landun/config
cfs:
path: /data/bkdevops/apps
readOnly: true
源码分析
// getBuilderVolumeAndMount 获取一些构建机的常规的被挂载到pod上的volume和mount
func getBuilderVolumeAndMount(
workloadName string,
nFSs []types.NFS,
) (volumes []corev1.Volume, volumeMounts []corev1.VolumeMount) {
volumes = getBuilderPodVolume(workloadName)
volumeMounts = getBuilderPodVolumeMount()
...
return volumes, volumeMounts
}
// getBuilderPodVolume 获取一些构建机的常规的被挂载到pod上的volume,包括配置configmap和data目录hostpath
func getBuilderPodVolume(workloadName string) []corev1.Volume {
dataHostPath := filepath.Join(config.Config.Dispatch.Volume.HostPath.DataHostDir, workloadName)
logHostPath := filepath.Join(config.Config.Dispatch.Volume.HostPath.LogsHostDir, workloadName)
...
return ...
}
通过源码分析可以发现 hostPath 是通过配置文件的 dataHostDir/logsHostDir 加上 workloadName 拼接而成的,没办法通过配置文件控制不使用 hostPath,于是我们通过定时任务来清理该缓存。
解决方案
参考 bk-applog-bkapp-filebeat 的日志清理方案,通过 DaemonSet 实现蓝盾挂载工作目录定时清理操作。
DaemonSet 配置:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: bk-ci-builder-cleaner
namespace: blueking
labels:
app: bk-ci-builder
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
app: bk-ci-builder
template:
metadata:
labels:
app: bk-ci-builder
name: bk-ci-builder-cleaner
spec:
hostPID: true
restartPolicy: Always
containers:
- name: batch-delete-files
image: bk-ci-builder-cleaner:v1
imagePullPolicy: IfNotPresent
command:
- bash
args:
- -c
- while true; do ./delete_files.sh; sleep 21600; done;
resources:
requests:
cpu: 25m
memory: 32Mi
limits:
cpu: 2560m
memory: 256Mi
volumeMounts:
- mountPath: /data/devops/workspace
name: data-volume
- mountPath: /data/devops/logs
name: logs-volume
volumes:
- name: data-volume
hostPath:
path: /data/landun/workspace
type: DirectoryOrCreate
- name: logs-volume
hostPath:
path: /data/landun/logs
type: DirectoryOrCreate
缓存清理脚本 delete_files.sh:
#!/usr/bin/env bash
set -euo pipefail
# --------- 可配置参数 ---------
ROOT_DIRS=("/data/devops/workspace" "/data/devops/logs")
RETENTION_DAYS=7
LOG_FILE="/tmp/delete_build_dirs.log"
# -----------------------------
log() {
printf '%s [%s] %s\n' "$(date '+%F %T')" "$1" "$2" | tee -a "$LOG_FILE"
}
cutoff_date=$(date -d "$RETENTION_DAYS days ago" +%F)
log INFO "==== 开始检查并删除 $RETENTION_DAYS 天未更新的 build* 目录 ===="
for root in "${ROOT_DIRS[@]}"; do
[[ -d $root ]] || { log WARN "目录不存在: $root"; continue; }
for dir in "$root"/build*; do
[[ -d $dir ]] || continue
if ! find "$dir" -type f -newermt "$cutoff_date" -print -quit | grep -q .; then
log DELETE "$dir"
rm -rf "$dir"
else
log SKIP "$dir"
fi
done
done
log INFO "==== 清理完成,日志: $LOG_FILE ===="