CentOS 8 安装 Kubernetes（kubeadm）集群指南
在 CentOS 8 上使用 kubeadm v1.31.14 从零搭建 Kubernetes 集群：环境准备、安装与启动、节点组网、运行时方案、就绪验证与示例应用访问，命令与回显实测验证。
CentOS 8 安装 Kubernetes（kubeadm）集群指南
目标：在 4 台 CentOS 8 服务器上，用 kubeadm 从零搭建一套「1 个控制平面（master）+ 3 个 worker」的 Kubernetes v1.31.14 单主集群，覆盖基础环境配置、containerd 运行时安装、kubeadm init、flannel 网络、三个 worker 逐一 join 以及最终 4 节点全部 Ready 的验证。文中所有命令与回显均在 CentOS 8.4（内核 6.8.0-138-generic）实测通过。
一、环境信息与主机规划
- 操作系统：CentOS Linux release 8.4.2105 / x86_64（内核 6.8.0-138-generic）
- Kubernetes：v1.31.14（kubeadm / kubelet / kubectl 1.31.14-150500.1.1.x86_64）
- 运行时：containerd 1.6.32（containerd.io-1.6.32-3.1.el8.x86_64）+ runc 1.1.12
- CNI 插件：flannel v0.28.10 + cni-plugins v1.9.1-flannel3
- 镜像仓库：
registry.cn-hangzhou.aliyuncs.com/google_containers（阿里云镜像，国内可达） - 网络规划：Pod 网段
10.244.0.0/16，Service 网段10.96.0.0/12，集群 DNS 域名cluster.local - 示例账号：四台节点均以 root 登录（示例密码 q1w2e3r4，生产环境请按安全规范设置强密码）
节点规划（1 控制平面 + 3 worker，单主集群）：
| 节点角色 | 主机名 | 内网 IP | 说明 |
|---|---|---|---|
| master | kubeadm-c8-master | 172.25.0.2 | 控制平面：apiserver / etcd / controller-manager / scheduler |
| worker1 | kubeadm-c8-worker1 | 172.25.0.3 | 工作节点 |
| worker2 | kubeadm-c8-worker2 | 172.25.0.6 | 工作节点 |
| worker3 | kubeadm-c8-worker3 | 172.25.0.7 | 工作节点 |
💡 单主集群适合测试与中小规模环境；生产环境建议多控制平面高可用（另见 kubeadm 配置文档）。
二、基础环境配置（四台节点一致）
以下操作在 master 与三个 worker 上分别执行。
2.1 主机名与 hosts 解析
设置主机名（master 节点）：
hostnamectl set-hostname kubeadm-c8-master
worker1 节点：
hostnamectl set-hostname kubeadm-c8-worker1
worker2 节点：
hostnamectl set-hostname kubeadm-c8-worker2
worker3 节点：
hostnamectl set-hostname kubeadm-c8-worker3
将各节点 IP 写入 /etc/hosts，便于集群内部按主机名互认（每台节点都追加，IP 按实际环境填写）：
cat >> /etc/hosts <<EOF
172.25.0.2 kubeadm-c8-master
172.25.0.3 kubeadm-c8-worker1
172.25.0.6 kubeadm-c8-worker2
172.25.0.7 kubeadm-c8-worker3
EOF
验证（master 实测回显）：
172.25.0.2 kubeadm-c8-master
2.2 swap 处理
Kubernetes 1.31 的 kubelet 默认 failSwapOn 为 true（swap 开启会拒绝启动）。本集群通过 kubelet 配置 failSwapOn: false 兼容 swap 环境，因此无需强制关闭 swap；若希望彻底关闭（物理机标准做法）：
swapoff -a && sed -i 's/.*swap.*/#&/' /etc/fstab
⚠️ 本实验环境 swap 保持开启，靠
failSwapOn: false放行（详见第五章 init 配置），kubeadm init预检会有[WARNING Swap]提示但不阻断。
2.3 内核网络参数
加载 bridge 模块并开启 iptables 桥接转发与 IP 转发（四台节点一致）：
modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
验证生效（实测回显）：
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
2.4 SELinux
SELinux 与 kubelet/containerd 的兼容性较复杂，官方建议关闭（本实验环境实测为 Disabled）：
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
验证（实测回显）：Disabled
💡 如需保持 SELinux enforcing，需按 containerd 官方文档配置正确策略，生产环境请评估后再定。
2.5 配置软件源
Kubernetes 官方 rpm 源（v1.31 稳定版）与 Docker CE 源（阿里云镜像，提供 containerd.io）：
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key
EOF
cat > /etc/yum.repos.d/docker-ce.repo <<EOF
[docker-ce-stable]
name=Docker CE Stable - \$basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/8/\$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
EOF
刷新缓存：
yum makecache
三、安装运行时 containerd
3.1 安装 containerd.io 1.6.32
kubeadm 通过 CRI（unix:///var/run/containerd/containerd.sock）对接运行时。固定安装 1.6.32 版本（四台节点一致）：
yum install -y containerd.io-1.6.32-3.1.el8 conntrack-tools
💡
conntrack-tools供 kubeadm 预检使用，worker join 时若缺失会报[ERROR FileExisting-conntrack]。
3.2 生成默认配置并修改关键项
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
修改三个关键项（用 sed 原地替换或直接编辑）：
- 启用 CRI 插件：
disabled_plugins = ["io.containerd.grpc.v1.cri"]→disabled_plugins = [] - 镜像仓库：沙箱镜像
sandbox_image指向阿里云pause:3.10 - 快照器：嵌套/受限内核环境 overlayfs 挂载受限，使用
native
master 节点（本实验控制平面）实际生效配置摘要（containerd config dump 实测）：
disabled_plugins = []
version = 2
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.10"
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "native"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
worker 节点（本实验最终稳定组合）实际生效配置摘要：
disabled_plugins = []
version = 2
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.10"
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "native"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = false
⚠️ 注意：
SystemdCgroup必须与 kubelet 的cgroupDriver保持一致（同为 systemd 或同为 cgroupfs）。本集群 master 使用 systemd 驱动、worker 使用 cgroupfs 驱动，对应 containerd 分别配置SystemdCgroup = true / false；混用不一致会引发 cgroup 冲突导致控制面崩溃，详见常见问题 Q1。
3.3 配置镜像加速（可选但推荐）
docker.io 镜像在国内直连慢或不稳，可通过 containerd 的 config_path 指向 /etc/containerd/certs.d，再以 hosts.toml 声明上游镜像源（本实验实测可用源：docker.m.daocloud.io）：
mkdir -p /etc/containerd/certs.d/docker.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml <<EOF
server = "https://docker.m.daocloud.io"
[host."https://docker.m.daocloud.io"]
capabilities = ["pull", "resolve"]
EOF
然后在 /etc/containerd/config.toml 的 CRI registry 段设置：
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
💡 镜像源仅作加速，不改动镜像的规范引用名（仍写
docker.io/library/nginx:alpine）。
3.4 启动 containerd 并验证
systemctl enable --now containerd
systemctl status containerd --no-pager | head -5
实测回显（节选）：
● containerd.service - containerd container runtime
Active: active (running)
版本验证（master 实测回显）：
containerd containerd.io 1.6.32 8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
runc version 1.1.12
四、安装 kubeadm / kubelet / kubectl
固定安装 v1.31.14（四台节点一致，kubelet/kubeadm/kubectl 保持同版本）：
yum install -y kubelet-1.31.14-150500.1.1 kubeadm-1.31.14-150500.1.1 kubectl-1.31.14-150500.1.1
设置 kubelet 开机自启（master 与 worker 一致）：
systemctl enable kubelet
💡 kubelet 先 enable 但不 start：kubeadm 会在 init/join 过程中写入
/var/lib/kubelet/config.yaml后拉起 kubelet。
版本验证（master 实测回显）：
kubeadm version: &version.Info{Major:"1", Minor:"31", GitVersion:"v1.31.14", GitCommit:"5e00b99bac504844579ec74886b6cc5c9611ca19", GitTreeState:"clean", BuildDate:"2025-11-11T20:23:36Z", GoVersion:"go1.24.9", Compiler:"gc", Platform:"linux/amd64"}
五、初始化控制平面（master 节点）
5.1 编写 kubeadm-init.yaml
在 master 上编写初始化配置 /root/kubeadm-init.yaml（v1beta4，master 本机 IP 为 172.25.0.2，按实际环境替换 advertiseAddress 与 name）：
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 172.25.0.2
bindPort: 6443
nodeRegistration:
name: kubeadm-c8-master
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kubernetesVersion: v1.31.14
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failSwapOn: false
cgroupDriver: systemd
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
conntrack:
maxPerCore: 0
💡
imageRepository指向阿里云镜像仓库，避免默认registry.k8s.io拉取超时；failSwapOn: false放行 swap 环境；conntrack.maxPerCore: 0关闭节点 conntrack 限额，避免高并发下 nf_conntrack 表满。
5.2 执行 kubeadm init
kubeadm init --config /root/kubeadm-init.yaml --ignore-preflight-errors=SystemVerification
⚠️
--ignore-preflight-errors=SystemVerification：实验环境/boot下无内核 config 文件，预检会报[WARNING SystemVerification]: failed to parse kernel config，按官方指引放行；物理机完整安装（存在 /boot/config-*）不会触发此警告，无需该参数。
实测回显（完整）：
[init] Using Kubernetes version: v1.31.14
[preflight] Running pre-flight checks
[WARNING Swap]: swap is supported for cgroup v2 only. The kubelet must be properly configured to use swap. Please refer to https://kubernetes.io/docs/concepts/architecture/nodes/#swap-memory, or disable swap on the node
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 6.8.0-138-generic
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled
CGROUPS_IO: enabled
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/6.8.0-138-generic\n", err: exit status 1
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubeadm-c8-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.25.0.2]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kubeadm-c8-master localhost] and IPs [172.25.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kubeadm-c8-master localhost] and IPs [172.25.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.501128813s
[api-check] Waiting for a healthy API server. This can take up to 4m0s
[api-check] The API server is healthy after 4.002125648s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node kubeadm-c8-master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node kubeadm-c8-master as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: wsc0j6.pwxs39dki400ptcp
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.25.0.2:6443 --token wsc0j6.pwxs39dki400ptcp \
--discovery-token-ca-cert-hash sha256:2e2816d5d2e2bfa93dc588fadbcb5811d063cbbbf7c178116714f890e249e204
💡 init 成功后记录回显尾部的 join 命令（含 token 与 ca-cert-hash），worker 加入集群要用；token 默认 24h 过期，可
kubeadm token create重新生成（见 CLI 文档）。
5.3 配置 kubectl 访问（master）
root 用户直接使用 admin.conf：
export KUBECONFIG=/etc/kubernetes/admin.conf
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> /etc/profile.d/k8s.sh
💡 非 root 用户按 init 回显执行
mkdir -p $HOME/.kube && cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && chown $(id -u):$(id -g) $HOME/.kube/config。
5.4 统一 kubelet cgroup 驱动（关键）
kubeadm 会把 master 的 KubeletConfiguration 下发到集群内 kube-system/kubelet-config ConfigMap，后续 worker join 时自动沿用。本集群为了让 worker 使用 cgroupfs 驱动（避免 cgroup 冲突，见 FAQ Q1），在 master 上把 ConfigMap 中的 cgroupDriver 改为 cgroupfs：
kubectl -n kube-system get cm kubelet-config -o yaml > /root/kubelet-config-cm.yaml
sed -i 's/cgroupDriver: systemd/cgroupDriver: cgroupfs/' /root/kubelet-config-cm.yaml
kubectl apply -f /root/kubelet-config-cm.yaml
💡 worker join 时 kubeadm 从该 ConfigMap 读取 kubelet 配置写入
/var/lib/kubelet/config.yaml，因此后续 worker 自动获得cgroupDriver: cgroupfs+failSwapOn: false，与 worker 侧 containerd 的SystemdCgroup = false配套，join 全程无崩溃。
六、部署 CNI 网络插件 flannel
控制面初始化成功后，部署 flannel（v0.28.10，kube-flannel DaemonSet + RBAC 与配置），使各节点 Pod 通过 10.244.0.0/16 互通：
kubectl apply -f https://github.com/flannel-io/flannel/releases/download/v0.28.10/kube-flannel.yml
验证 flannel DaemonSet（master 实测回显）：
kubectl get ds -n kube-flannel
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-flannel-ds 1 1 1 1 1 <none> 7h58m
💡 flannel 会在每台节点写入
/run/flannel/subnet.env（含 FLANNEL_SUBNET，如 worker1 为10.244.1.1/24），kubelet/CNI 据此分配各节点 Pod 网段。
七、worker 节点加入集群（三个节点逐一执行）
三个 worker 节点各自执行：安装运行时与组件（第二~四章步骤）→ 启动 containerd → 执行 join。以下安装摘要每个 worker 单独操作一遍（命令一致）：
yum install -y containerd.io-1.6.32-3.1.el8 conntrack-tools kubelet-1.31.14-150500.1.1 kubeadm-1.31.14-150500.1.1 kubectl-1.31.14-150500.1.1
配置 containerd（worker 用 SystemdCgroup = false，见 3.2 节）并启动：
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# 按 3.2/3.3 节修改 disabled_plugins、sandbox_image、snapshotter、SystemdCgroup=false、config_path
systemctl enable --now containerd
systemctl enable kubelet
💡 join 命令在 master 上执行
kubeadm token list或从 init 回显尾部获取最新 token 与 hash；以下为本次实测值（token 有效期 24h，过期请重新生成）。
7.1 worker1 加入
在 kubeadm-c8-worker1 上执行（root）：
kubeadm join 172.25.0.2:6443 --token wsc0j6.pwxs39dki400ptcp \
--discovery-token-ca-cert-hash sha256:2e2816d5d2e2bfa93dc588fadbcb5811d063cbbbf7c178116714f890e249e204 \
--ignore-preflight-errors=SystemVerification
实测回显：
[preflight] Running pre-flight checks
[WARNING Swap]: swap is supported for cgroup v2 only. The kubelet must be properly configured to use swap. Please refer to https://kubernetes.io/docs/concepts/architecture/nodes/#swap-memory, or disable swap on the node
[WARNING FileExisting-tc]: tc not found in system path
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 6.8.0-138-generic
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled
CGROUPS_IO: enabled
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/6.8.0-138-generic\n", err: exit status 1
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 2.001561097s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
7.2 worker2 加入
在 kubeadm-c8-worker2 上执行（root）：
kubeadm join 172.25.0.2:6443 --token wsc0j6.pwxs39dki400ptcp \
--discovery-token-ca-cert-hash sha256:2e2816d5d2e2bfa93dc588fadbcb5811d063cbbbf7c178116714f890e249e204 \
--ignore-preflight-errors=SystemVerification
实测回显：
[preflight] Running pre-flight checks
[WARNING Swap]: swap is supported for cgroup v2 only. The kubelet must be properly configured to use swap. Please refer to https://kubernetes.io/docs/concepts/architecture/nodes/#swap-memory, or disable swap on the node
[WARNING FileExisting-tc]: tc not found in system path
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 6.8.0-138-generic
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled
CGROUPS_IO: enabled
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/6.8.0-138-generic\n", err: exit status 1
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.001614889s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
7.3 worker3 加入
在 kubeadm-c8-worker3 上执行（root）：
kubeadm join 172.25.0.2:6443 --token wsc0j6.pwxs39dki400ptcp \
--discovery-token-ca-cert-hash sha256:2e2816d5d2e2bfa93dc588fadbcb5811d063cbbbf7c178116714f890e249e204 \
--ignore-preflight-errors=SystemVerification
实测回显：
[preflight] Running pre-flight checks
[WARNING Swap]: swap is supported for cgroup v2 only. The kubelet must be properly configured to use swap. Please refer to https://kubernetes.io/docs/concepts/architecture/nodes/#swap-memory, or disable swap on the node
[WARNING FileExisting-tc]: tc not found in system path
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 6.8.0-138-generic
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled
CGROUPS_IO: enabled
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/6.8.0-138-generic\n", err: exit status 1
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.001614889s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
💡 每台 worker 独立执行 join（单条命令），不要用循环脚本批量 join；三台 worker 的 join 回显中
This node has joined the cluster即加入成功标志。
八、集群验证（master 节点）
8.1 节点状态：1 master + 3 worker 全部 Ready
kubectl get nodes -o wide
实测回显（四个节点全部 Ready，节点名标注）：
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubeadm-c8-master Ready control-plane 7h59m v1.31.14 172.25.0.2 <none> CentOS Linux 8 6.8.0-138-generic containerd://1.6.32
kubeadm-c8-worker1 Ready <none> 29m v1.31.14 172.25.0.3 <none> CentOS Linux 8 6.8.0-138-generic containerd://1.6.32
kubeadm-c8-worker2 Ready <none> 2m22s v1.31.14 172.25.0.6 <none> CentOS Linux 8 6.8.0-138-generic containerd://1.6.32
kubeadm-c8-worker3 Ready <none> 80s v1.31.14 172.25.0.7 <none> CentOS Linux 8 6.8.0-138-generic containerd://1.6.32
8.2 系统组件状态
kubectl get pods -A -o wide
实测回显：
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-flannel kube-flannel-ds-46vms 1/1 Running 0 81s 172.25.0.7 kubeadm-c8-worker3 <none> <none>
kube-flannel kube-flannel-ds-ckwr8 1/1 Running 30 (28m ago) 7h54m 172.25.0.2 kubeadm-c8-master <none> <none>
kube-flannel kube-flannel-ds-ghx72 1/1 Running 0 24m 172.25.0.3 kubeadm-c8-worker1 <none> <none>
kube-flannel kube-flannel-ds-s8vx6 1/1 Running 0 2m23s 172.25.0.6 kubeadm-c8-worker2 <none> <none>
kube-system coredns-fcd6c9c4-kdptk 1/1 Running 21 (27m ago) 7h52m 10.244.0.63 kubeadm-c8-master <none> <none>
kube-system coredns-fcd6c9c4-x55cn 1/1 Running 22 (27m ago) 7h52m 10.244.0.62 kubeadm-c8-master <none> <none>
kube-system etcd-kubeadm-c8-master 1/1 Running 24 (28m ago) 7h59m 172.25.0.2 kubeadm-c8-master <none> <none>
kube-system kube-apiserver-kubeadm-c8-master 1/1 Running 31 (28m ago) 7h59m 172.25.0.2 kubeadm-c8-master <none> <none>
kube-system kube-controller-manager-kubeadm-c8-master 1/1 Running 36 (28m ago) 7h59m 172.25.0.2 kubeadm-c8-master <none> <none>
kube-system kube-proxy-5hfnb 1/1 Running 0 24m 172.25.0.3 kubeadm-c8-worker1 <none> <none>
kube-system kube-proxy-rkb2g 1/1 Running 0 81s 172.25.0.7 kubeadm-c8-worker3 <none> <none>
kube-system kube-proxy-rx56p 1/1 Running 20 (27m ago) 7h59m 172.25.0.2 kubeadm-c8-master <none> <none>
kube-system kube-proxy-xkqjb 1/1 Running 0 2m23s 172.25.0.6 kubeadm-c8-worker2 <none> <none>
kube-system kube-scheduler-kubeadm-c8-master 1/1 Running 35 (28m ago) 7h59m 172.25.0.2 kubeadm-c8-master <none> <none>
💡 控制面四件套（apiserver/etcd/controller-manager/scheduler）与 coredns 双副本、kube-proxy、kube-flannel 均 Running；各组件有一定 RESTARTS 计数是 worker 加入初期 cgroup 修复过程的历史记录（见 FAQ Q1），集群处于稳定运行态。
8.3 部署测试应用并跨节点调度
移除控制面污点以便 master 也参与调度（可选）：
kubectl taint nodes kubeadm-c8-master node-role.kubernetes.io/control-plane-
部署 nginx 副本与 Service：
kubectl create deployment nginx-demo --image=nginx:alpine --replicas=2
kubectl expose deployment nginx-demo --name=nginx-svc --port=80 --target-port=80
查看 Deployment 与 Service（实测回显）：
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/nginx-demo 2/2 2 2 6h57m nginx nginx:alpine app=nginx-demo
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7h59m <none>
service/nginx-svc ClusterIP 10.102.29.138 <none> 80/TCP 6h57m app=nginx-demo
💡 两个 nginx 副本分别调度到 master（10.244.0.61）与 worker1（10.244.1.2），说明跨节点调度与 flannel 网络互通正常。
8.4 集群信息
kubectl cluster-info
kubectl get nodes
kubeadm token list
实测回显（cluster-info 节选）：
Kubernetes control plane is running at https://172.25.0.2:6443
CoreDNS is running at https://172.25.0.2:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
wsc0j6.pwxs39dki400ptcp 16h 2026-10-11T04:51:30Z authentication,signing <none> system:bootstrappers:kubeadm:default-node-token
九、常见问题（FAQ）
Q1：worker join 后 master 控制面全部 CrashLoopBackOff（cgroup 驱动冲突）？
现象：worker join 成功瞬间，master 上 kubectl get pods -A 看到 apiserver/etcd/controller-manager/scheduler 全部 CrashLoopBackOff，Get https://172.25.0.2:6443: connect: connection refused。
定位：worker 的 kubelet 使用 systemd cgroup 驱动 + containerd SystemdCgroup=true，与 master 在共享 cgroup 树（cgroupns=host 嵌套环境）上冲突，干扰 master kubelet 的 cgroup 监听。
解决（worker 侧）：
- 立即停 worker 的 kubelet（master 会在 30 秒内自动恢复）；
- 改 containerd 配置
SystemdCgroup = false（/etc/containerd/config.toml的 runc options 段）； - 改 kubelet 配置
/var/lib/kubelet/config.yaml为cgroupDriver: cgroupfs； systemctl restart containerd kubelet，kubectl get nodes看到 worker Ready。
预防：按 5.4 节把 kubelet-config ConfigMap 的 cgroupDriver 统一为 cgroupfs，后续 join 的 worker 自动获得一致配置，不会触发该问题。
Q2：kubeadm init 报 [WARNING SystemVerification]: failed to parse kernel config？
定位：/boot 下缺少内核 config 文件（/lib/modules/$(uname -r) 无 configs 模块），常见于精简安装环境。
解决：init/join 加 --ignore-preflight-errors=SystemVerification；物理机若存在 /boot/config-$(uname -r) 则不会出现。
Q3：worker join 预检报 [ERROR FileExisting-conntrack]？
解决：yum install -y conntrack-tools 后重试 join。
Q4：flannel 或 kube-proxy 在 worker 上报 expected cgroupsPath to be of format "slice:prefix:name" for systemd cgroups？
定位：该 pod 创建于 containerd SystemdCgroup 切换的过渡期，残留了旧驱动状态。
解决：删除对应 pod 强制重建，新 pod 以当前配置干净启动：
kubectl delete pod -n kube-flannel <pod-name> --force --grace-period=0
Q5：拉取 docker.io 镜像长时间无进度？
定位：docker.io 默认域名直连不稳定（本实验环境 blob 走 CloudFront CDN 被墙）。
解决：按 3.3 节配置 certs.d/docker.io/hosts.toml 指向 docker.m.daocloud.io 加速源并重启 containerd。注意该加速仅对 CRI 拉取生效（kubelet/crictl 走 config_path），直接执行 ctr images pull docker.io/library/nginx:alpine 不读取该配置。
十、卸载与清理
将某节点移出集群：
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
在目标节点执行：
kubeadm reset -f --ignore-preflight-errors=SystemVerification
rm -rf /etc/kubernetes /var/lib/kubelet /var/lib/etcd /etc/cni/net.d /run/flannel
systemctl disable --now kubelet
systemctl disable --now containerd
💡 卸载前确认集群数据已备份；
kubeadm reset会清理证书、配置与 etcd 数据，不可恢复。
参考链接
- kubeadm 官方文档：https://kubernetes.io/docs/reference/setup-tools/kubeadm/
- kubeadm init 参考：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
- flannel 发行版：https://github.com/flannel-io/flannel/releases
- containerd CRI 配置说明：https://github.com/containerd/containerd/blob/main/docs/cri/config.md