【K8s笔记】实操，纯手动创建一个集群

参考了项目kubernetes-the-hard-way

前置

¶需要

4台虚拟机，其中一台jumpbox，一台是master node，两台worker node，配置如下

• jumpbox - 1H1G
• master node - 4H4G
• worker node - 4H4G

其中，实际的CPU为Intel(R) Xeon(R) E5-2666 v3 @ 2.90 GHz，内存为DDR3 1666Mhz ECC，四台机器的虚拟硬盘文件均放在HDD上面

¶大致步骤

• 设置SSH秘钥，DNS映射，PKI服务，TLS证书
• 创建k8s配置文件，包括kubelet，kube-proxy，kube-controller-manager，kube-scheduler，管理员配置文件，然后分发这些配置文件
• 生成加密秘钥和加密配置文件，用来加密Secret等数据
• 在master node上配置etcd
• 在master node上配置control plane
• 在worker node上配置runc，containred，kubelet，kube-proxy和网络工具
• 在master node上配置kubectl

¶一些apiGroup的关键字

apiGroup 名称	说明	典型资源
""（空字符串）	核心组（core group）	pods, services, nodes
apps	应用组	deployments, daemonsets
batch	批处理组	jobs, cronjobs
rbac.authorization.k8s.io	RBAC 相关	roles, rolebindings
apiextensions.k8s.io	CRD 相关	customresourcedefinitions
networking.k8s.io	网络相关	networkpolicies, ingress
policy	安全策略相关	podsecuritypolicies
storage.k8s.io	存储相关	storageclasses
…	还有很多，详见官方文档

¶一些apiVersion关键字

格式一般是{API_GROUP}/{VERSION}

apiVersion	典型资源 (kind)	说明
v1	Pod, Service, ConfigMap	核心资源
apps/v1	Deployment, StatefulSet	应用控制器
batch/v1	Job, CronJob	批处理任务
rbac.authorization.k8s.io/v1	Role, ClusterRole	RBAC 权限
networking.k8s.io/v1	Ingress, NetworkPolicy	网络相关
storage.k8s.io/v1	StorageClass, CSIDriver	存储相关
apiextensions.k8s.io/v1	CustomResourceDefinition	自定义资源
admissionregistration.k8s.io/v1	MutatingWebhookConfiguration	准入控制
policy/v1	PodDisruptionBudget	策略相关
autoscaling/v2	HorizontalPodAutoscaler	自动扩缩容

配置文件

先把各种的配置文件给写好吧

¶10-bridge.conf

这玩意儿是给worker node里面的网桥用的，每一个pod都会由cni插件包中的bridge插件根据里面的子网掩码分配一个IP，并连接上网桥，使其能够互相通信

{
  "cniVersion": "1.0.0",
  "name": "bridge",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "ranges": [
      [{"subnet": "SUBNET"}]
    ],
    "routes": [{"dst": "0.0.0.0/0"}]
  }
}

这里面的host-local表示网络由worker node自己管理，还有其他例如dhcp。static以及其他第三方的网络路由工具

¶99-loopback.conf

很容易忘记！但是至关重要！

{
  "cniVersion": "1.1.0",
  "name": "lo",
  "type": "loopback"
}

¶containerd-config.toml

这是workder node运行时环境的配置文件

version = 2

[plugins."io.containerd.grpc.v1.cri"]
  [plugins."io.containerd.grpc.v1.cri".containerd]
    snapshotter = "overlayfs"
    default_runtime_name = "runc"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/opt/cni/bin"
  conf_dir = "/etc/cni/net.d"

需要配置的东西

• containerd本体 - 快照的实现方式overlayfs，容器的运行时runc
• runc - 运行时的版本
• cgroup - 管理方式
• cni - 二进制文件和配置文件

¶encryption-config.yaml

这里要配置秘钥，用于加密master node中的敏感信息

kind: EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}

¶kube-apiserver-to-kubelet.yaml

这个配置文件用于master node访问worker node，配置权限。上半部分是定义system:kube-apiserver-to-kubelet这个角色的权利，下半部分是赋予具体的用户以定义的这个角色的权利。

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes

上半部分

• system:kube-apiserver-to-kubelet，这里面的system是约定俗称的，用于标注角色性质，其实无实意
• 配置管理proxy, stats, log, spec, metrcs的用户
• rbac.authorization.kubernetes.io/autoupdate: "true" - 跟随k8s更新而自动更新到符合新版本的资源
• kubernetes.io/bootstrapping: rbac-defaults - k8s启动的时候将这个资源创建为rbca的默认资源
• verbs - 允许该用户做的对应操作的东西

下半部分

• roleRef.apiGroup填写的应该是roleRef.kind所在的apiGroup
• User - 关键字，代表用户

¶kube-proxy-config.yaml

这个文件配置的就是kubeproxy了，主要配置的是proxy分配的IP段，所以比较简单。

kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  kubeconfig: "/var/lib/kube-proxy/kubeconfig"
mode: "iptables"
clusterCIDR: "10.200.0.0/16"

需要配置的东西

• k8s本身的配置文件，用来与api server通信。这个配置文件可以由kubectl来手动生成
• 可分配的IP段

¶kube-scheduler.yaml

这玩意儿丢在master node上面

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
leaderElection:
  leaderElect: true

• clientConnect - 同上面的kube-proxy
• leaderElection - 确保在多调度器存在的时候只使用一个调度器，建议始终开启

¶kubelet-config.yaml

这个就是配置kubelet的了，需要配置的东西还挺多的

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: "0.0.0.0"
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/var/lib/kubelet/ca.crt"
authorization:
  mode: Webhook
cgroupDriver: systemd
containerRuntimeEndpoint: "unix:///var/run/containerd/containerd.sock"
enableServer: true
failSwapOn: false
maxPods: 16
memorySwap:
  swapBehavior: NoSwap
port: 10250
resolvConf: "/etc/resolv.conf"
registerNode: true
runtimeRequestTimeout: "15m"
tlsCertFile: "/var/lib/kubelet/kubelet.crt"
tlsPrivateKeyFile: "/var/lib/kubelet/kubelet.key"

需要配置的内容

• address - 监听地址，只能填一个IP地址，不能用CIDR
• anthentication - 鉴权外部访问，包括匿名访问，webhook访问，以及authentication.x509证书
• authorization - 授权机制，一般都是webhook

以及其他配置项，剩下的应该都看得懂了

¶下载需要使用的二进制文件

• kubectl - https://kubernetes.io/releases/download/
• kube-apiserver - https://kubernetes.io/releases/download/
• kube-controller-manager - https://kubernetes.io/releases/download/
• kube-scheduler - https://kubernetes.io/releases/download/
• kube-proxy - https://kubernetes.io/releases/download/
• kubelet - https://kubernetes.io/releases/download/
• crictl - https://github.com/kubernetes-sigs/cri-tools
• runc - https://github.com/opencontainers/runc
• cni-plugins - https://github.com/containernetworking/plugins
• containerd - https://github.com/containerd/containerd
• etcd - https://github.com/etcd-io/etcd

配置堡垒机

这一步就是配置堡垒机和节点之间的通信，包括hosts和ssh，确保各个机器之间的连接畅通。

¶配置节点信息

新建一个machine.txt文件，里面写好ip，本地域名，主机名，子网网段，我的是这样的

192.168.6.12 master.k8s.local master
192.168.6.13 node0.k8s.local node0 10.6.0.0/24
192.168.6.14 node1.k8s.local node1 10.6.1.0/24

然后我们在堡垒机上执行ssh-keygen，创建ssh秘钥，并分发到各个节点上面。

root@debian:~# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:JxJqqUZzZbwRopku4kJwTspMgkdULbrQKZSsFGy+vfw root@debian
The key's randomart image is:
+---[RSA 3072]----+
|oo=.o..          |
|.O +.o..         |
|XoBo .*          |
|XO+  = +         |
|+*B.= o S .      |
|++.*   . o       |
|..+ .            |
|.. o             |
|    .E           |
+----[SHA256]-----+

然后分发到各个节点，用ssh-copy-id，但是加点循环

while read IP FQDN HOST SUBNET; do
	ssh-copy-id root@${IP}
done < machine.txt

然后我们就要设置各个节点的主机名了，用sed命令写一下/etc/hosts文件，然后用hostnamectl设置主机名，然后重启服务。同样是写一个脚本

while read IP FQDN HOST SUBNET; do
    CMD="sed -i 's/^127.0.1.1.*/127.0.1.1\t${FQDN} ${HOST}/' /etc/hosts"
    ssh -n root@${IP} "$CMD"
    ssh -n root@${IP} hostnamectl set-hostname ${HOST}
    ssh -n root@${IP} systemctl restart systemd-hostnamed
done < machine.txt

然后就是把这些节点信息写到堡垒机的本地hosts了，可以直接nano拷进去，也可以写脚本

echo "" >> /etc/hosts
echo "# k8s node config" >> /etc/hosts
while read IP FQDN HOST SUBNET; do
    ENTRY="${IP} ${FQDN} ${HOST}"
    echo $ENTRY >> /etc/hosts
done < machine.txt

还要写入节点内

for NODE in node0 node1; do
  while read IP FQDN HOST SUBNET; do
    ENTRY="${IP} ${FQDN} ${HOST}"
    ssh -n root@${NODE} "echo '${ENTRY}' >> /etc/hosts"
  done < machine.txt
done

¶配置CA认证服务

先来生成根CA证书和私钥，openssl genrsa -out ca.key 4096来生成私钥。然后来写根证书的配置文件ca-root.conf。

[req]
distinguished_name = req_distinguished_name
prompt             = no
x509_extensions    = ca_x509_extensions

[ca_x509_extensions]
basicConstraints = CA:TRUE
keyUsage         = cRLSign, keyCertSign

[req_distinguished_name]
C   = CN
ST  = Hubei
L   = Wuhan
CN  = CA

然后我们就有了ca.key以及ca.crt这两个文件了。然后我们再来配置节点的证书，需要证书的有这些服务admin, node0, node1, kube-proxy, kube-scheduler, kube-controller-manager, kube-api-server, service-accounts。同样，先来写配置文件，可以写在一个配置文件ca-node.conf里面

[admin]
distinguished_name = admin_distinguished_name
prompt             = no
req_extensions     = default_req_extensions

[admin_distinguished_name]
CN = admin
O  = system:masters

# Service accounts
[service-accounts]
distinguished_name = service-accounts_distinguished_name
prompt             = no
req_extensions     = default_req_extensions

[service-accounts_distinguished_name]
CN = service-accounts

# node0
[node0]
distinguished_name = node-0_distinguished_name
prompt             = no
req_extensions     = node-0_req_extensions

[node-0_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth, serverAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client
nsComment            = "Node0 Certificate"
subjectAltName       = DNS:node0, IP:127.0.0.1
subjectKeyIdentifier = hash

[node-0_distinguished_name]
CN = system:node:node0
O  = system:nodes
C  = CN
ST = Hubei
L  = Wuhan

# node1
[node1]
distinguished_name = node-1_distinguished_name
prompt             = no
req_extensions     = node-1_req_extensions

[node-1_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth, serverAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client
nsComment            = "Node1 Certificate"
subjectAltName       = DNS:node1, IP:127.0.0.1
subjectKeyIdentifier = hash

[node-1_distinguished_name]
CN = system:node:node1
O  = system:nodes
C  = CN
ST = Hubei
L  = Wuhan


# Proxy
[kube-proxy]
distinguished_name = kube-proxy_distinguished_name
prompt             = no
req_extensions     = kube-proxy_req_extensions

[kube-proxy_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth, serverAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client
nsComment            = "Kube Proxy Certificate"
subjectAltName       = DNS:kube-proxy, IP:127.0.0.1
subjectKeyIdentifier = hash

[kube-proxy_distinguished_name]
CN = system:kube-proxy
O  = system:node-proxier
C  = CN
ST = Hubei
L  = Wuhan


# Controller Manager
[kube-controller-manager]
distinguished_name = kube-controller-manager_distinguished_name
prompt             = no
req_extensions     = kube-controller-manager_req_extensions

[kube-controller-manager_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth, serverAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client
nsComment            = "Kube Controller Manager Certificate"
subjectAltName       = DNS:kube-controller-manager, IP:127.0.0.1
subjectKeyIdentifier = hash

[kube-controller-manager_distinguished_name]
CN = system:kube-controller-manager
O  = system:kube-controller-manager
C  = CN
ST = Hubei
L  = Wuhan


# Scheduler
[kube-scheduler]
distinguished_name = kube-scheduler_distinguished_name
prompt             = no
req_extensions     = kube-scheduler_req_extensions

[kube-scheduler_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth, serverAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client
nsComment            = "Kube Scheduler Certificate"
subjectAltName       = DNS:kube-scheduler, IP:127.0.0.1
subjectKeyIdentifier = hash

[kube-scheduler_distinguished_name]
CN = system:kube-scheduler
O  = system:system:kube-scheduler
C  = CN
ST = Hubei
L  = Wuhan


# API Server
[kube-api-server]
distinguished_name = kube-api-server_distinguished_name
prompt             = no
req_extensions     = kube-api-server_req_extensions

[kube-api-server_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth, serverAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client, server
nsComment            = "Kube API Server Certificate"
subjectAltName       = @kube-api-server_alt_names
subjectKeyIdentifier = hash

[kube-api-server_alt_names]
IP.0  = 127.0.0.1
IP.1  = 10.6.0.1
DNS.0 = kubernetes
DNS.1 = kubernetes.default
DNS.2 = kubernetes.default.svc
DNS.3 = kubernetes.default.svc.cluster
DNS.4 = kubernetes.svc.cluster.local
DNS.5 = master.k8s.local
DNS.6 = api-server.k8s.local

[kube-api-server_distinguished_name]
CN = kubernetes
C  = CN
ST = Hubei
L  = Wuhan


[default_req_extensions]
basicConstraints     = CA:FALSE
extendedKeyUsage     = clientAuth
keyUsage             = critical, digitalSignature, keyEncipherment
nsCertType           = client
nsComment            = "Admin Client Certificate"
subjectKeyIdentifier = hash

由于需要的证书对的服务太多了，老样子，还是先写脚本

certs=(
  "admin" "node0" "node1"
  "kube-proxy" "kube-scheduler"
  "kube-controller-manager"
  "kube-api-server"
  "service-accounts"
)
for i in ${certs[*]}; do
  openssl genrsa -out "${i}.key" 4096

  openssl req -new -key "${i}.key" -sha256 \
    -config "ca-node.conf" -section ${i} \
    -out "${i}.csr"

  openssl x509 -req -days 3653 -in "${i}.csr" \
    -copy_extensions copyall \
    -sha256 -CA "ca.crt" \
    -CAkey "ca.key" \
    -CAcreateserial \
    -out "${i}.crt"
done

然后就是分发证书啦，先分发master node的证书，要把根CA, admin, api-server, service account的证书直接弄到上面去，admin, kube-controller-manager, kube-scheduler的证书等下集成到配置文件里面再传上去。我们用scp命令通过ssh来传文件。

scp \
  ca.key ca.crt \
  kube-api-server.key kube-api-server.crt \
  service-accounts.key service-accounts.crt \
  root@master:~/

然后再来传worker node的，先只传根CA证书，剩下的服务证书集成到配置文件里面传

for host in node0 node1; do
  ssh root@${host} mkdir /var/lib/kubelet/

  scp ca.crt root@${host}:/var/lib/kubelet/

  scp ${host}.crt \
    root@${host}:/var/lib/kubelet/kubelet.crt

  scp ${host}.key \
    root@${host}:/var/lib/kubelet/kubelet.key
done

¶整理二进制文件

最后来整理一下下载好的二进制文件，我整理好的二进制文件是这样的

.
├── client
│   ├── etcdctl
│   └── kubectl
├── cni-plugins
│   ├── bandwidth
│   ├── bridge
│   ├── dhcp
│   ├── dummy
│   ├── firewall
│   ├── host-device
│   ├── host-local
│   ├── ipvlan
│   ├── LICENSE
│   ├── loopback
│   ├── macvlan
│   ├── portmap
│   ├── ptp
│   ├── README.md
│   ├── sbr
│   ├── static
│   ├── tap
│   ├── tuning
│   ├── vlan
│   └── vrf
├── controller
│   ├── etcd
│   ├── kube-apiserver
│   ├── kube-controller-manager
│   └── kube-scheduler
└── worker
    ├── containerd
    ├── containerd-shim-runc-v2
    ├── containerd-stress
    ├── crictl
    ├── ctr
    ├── kubelet
    ├── kube-proxy
    └── runc

5 directories, 34 files

其中，cni-plugins里面的二进制文件就是项目下载的压缩文件里面解压出来的。containerd开头的文件以及ctr是containred项目下载的压缩包解压出来的。先给所有的二进制文件赋予可执行权限。堡垒机上要用的主要是kubectl，所以复制到/usr/local/bin下面。

生成配置文件

也是在堡垒机上完成，因为理论上堡垒机是唯一能够与节点通信的机器。
首先来生成kubelet的配置文件，给worker node用的，使用一下命令来生成配置文件：

• kubectl config set-cluster - 将节点加入到集群当中
• kubectl config set-credentials - 将节点证书写入到配置文件中
• kubectl config set-context - 将节点加入到context当中
• kubectl config use-context - 设置节点使用context

这四个命令全部写入一个worker node的配置文件，合起来的脚本是这样的

for host in node0 node1; do
  kubectl config set-cluster k8s \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://master.k8s.local:6443 \
    --kubeconfig=${host}.kubeconfig

  kubectl config set-credentials system:node:${host} \
    --client-certificate=${host}.crt \
    --client-key=${host}.key \
    --embed-certs=true \
    --kubeconfig=${host}.kubeconfig

  kubectl config set-context default \
    --cluster=k8s \
    --user=system:node:${host} \
    --kubeconfig=${host}.kubeconfig

  kubectl config use-context default \
    --kubeconfig=${host}.kubeconfig
done

然后来生成kube-proxy的配置文件，是给全体node用的，所以只有一个kubeconfig，命令和上面差不多

kubectl config set-cluster k8s \
  --certificate-authority=ca.crt \
  --embed-certs=true \
  --server=https://master.k8s.local:6443 \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials system:kube-proxy \
  --client-certificate=kube-proxy.crt \
  --client-key=kube-proxy.key \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=k8s \
  --user=system:kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default \
  --kubeconfig=kube-proxy.kubeconfig

下面来生成kube-controller-manager，只给master node用，所以也只有一个

kubectl config set-cluster k8s \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://master.k8s.local:6443 \
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
--client-certificate=kube-controller-manager.crt \
--client-key=kube-controller-manager.key \
--embed-certs=true \
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context default \
--cluster=k8s \
--user=system:kube-controller-manager \
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context default \
--kubeconfig=kube-controller-manager.kubeconfig

然后来生成kube-scheduler，同上

kubectl config set-cluster k8s \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://master.k8s.local:6443 \
--kubeconfig=kube-scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
--client-certificate=kube-scheduler.crt \
--client-key=kube-scheduler.key \
--embed-certs=true \
--kubeconfig=kube-scheduler.kubeconfig

kubectl config set-context default \
--cluster=k8s \
--user=system:kube-scheduler \
--kubeconfig=kube-scheduler.kubeconfig

kubectl config use-context default \
--kubeconfig=kube-scheduler.kubeconfig

然后生成admin的配置文件，只在master node上用，所以api server是127.0.0.1自己

kubectl config set-cluster k8s \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://127.0.0.1:6443 \
--kubeconfig=admin.kubeconfig

kubectl config set-credentials admin \
--client-certificate=admin.crt \
--client-key=admin.key \
--embed-certs=true \
--kubeconfig=admin.kubeconfig

kubectl config set-context default \
--cluster=k8s \
--user=admin \
--kubeconfig=admin.kubeconfig

kubectl config use-context default \
--kubeconfig=admin.kubeconfig

最后来生成加密配置文件吧，抽出提前准备好的encryption-config.yaml文件，先来创建一个随机秘钥export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)，然后把秘钥弄到文件里头envsubst < encryption-config.yaml > encryption-config-real.yaml，然后替换掉源文件rm encryption-config.yaml && mv encryption-config-real.yaml encryption-config.yaml。
然后就来分发了，kube-proxy和kubelet配置文件丢给worker node：

for host in node0 node1; do
  ssh root@${host} "mkdir -p /var/lib/{kube-proxy,kubelet}"

  scp kube-proxy.kubeconfig \
    root@${host}:/var/lib/kube-proxy/kubeconfig \

  scp ${host}.kubeconfig \
    root@${host}:/var/lib/kubelet/kubeconfig
done

然后kube-controller-manager、kube-scheduler、admin还有encryption-config.yaml是给master node用的，丢过去

scp admin.kubeconfig \
  kube-controller-manager.kubeconfig \
  kube-scheduler.kubeconfig \
  encryption-config.yaml \
  root@master:~/

启动etcd

接下来启动etcd，etcd是跑在master node上面的，这次只跑单个etcd实例罢。先把systemd的服务文件给写好etcd.service：

[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name controller \
  --initial-advertise-peer-urls http://127.0.0.1:2380 \
  --listen-peer-urls http://127.0.0.1:2380 \
  --listen-client-urls http://127.0.0.1:2379 \
  --advertise-client-urls http://127.0.0.1:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster controller=http://127.0.0.1:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

然后把文件弄到master node上面去

scp \
  downloads/controller/etcd \
  downloads/client/etcdctl \
  etcd.service \
  root@master:~/

然后进master node的ssh，直接在堡垒机上ssh root@master，然后来配置

mv etcd etcdctl /usr/local/bin/
mkdir -p /etc/etcd /var/lib/etcd
chmod 700 /var/lib/etcd
cp ca.crt kube-api-server.key kube-api-server.crt \
  /etc/etcd/
mv etcd.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

最后来验证一下跑起来没etcdctl member list，输出6702b0a34e2cfd39, started, controller, http://127.0.0.1:2380, http://127.0.0.1:2379, false。

启动control plane

现在来把master node上面的东西启动好吧，需要启动api server, controller manager还有scheduler。先把这仨的systemd服务文件给写好

• kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
  --allow-privileged=true \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/audit.log \
  --authorization-mode=Node,RBAC \
  --bind-address=0.0.0.0 \
  --client-ca-file=/var/lib/kubernetes/ca.crt \
  --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --etcd-servers=http://127.0.0.1:2379 \
  --event-ttl=1h \
  --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.crt \
  --kubelet-client-certificate=/var/lib/kubernetes/kube-api-server.crt \
  --kubelet-client-key=/var/lib/kubernetes/kube-api-server.key \
  --runtime-config='api/all=true' \
  --service-account-key-file=/var/lib/kubernetes/service-accounts.crt \
  --service-account-signing-key-file=/var/lib/kubernetes/service-accounts.key \
  --service-account-issuer=https://master.k8s.local:6443 \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/var/lib/kubernetes/kube-api-server.crt \
  --tls-private-key-file=/var/lib/kubernetes/kube-api-server.key \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

• kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --bind-address=0.0.0.0 \
  --cluster-cidr=10.6.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.crt \
  --cluster-signing-key-file=/var/lib/kubernetes/ca.key \
  --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \
  --root-ca-file=/var/lib/kubernetes/ca.crt \
  --service-account-private-key-file=/var/lib/kubernetes/service-accounts.key \
  --service-cluster-ip-range=10.7.0.0/24 \
  --use-service-account-credentials=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

• kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --config=/etc/kubernetes/config/kube-scheduler.yaml \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

加上前面准备好的kube-scheduler.yaml和kube-apiserver-to-kubelet.yaml，还有对应的二进制文件加上kubectl，全部丢到master node上面去

scp \
  downloads/controller/kube-apiserver \
  downloads/controller/kube-controller-manager \
  downloads/controller/kube-scheduler \
  downloads/client/kubectl \
  kube-apiserver.service \
  kube-controller-manager.service \
  kube-scheduler.service \
  kube-scheduler.yaml \
  kube-apiserver-to-kubelet.yaml \
  root@master:~/

然后同样连上ssh，继续配置

mkdir -p /etc/kubernetes/config
mv kube-apiserver \
  kube-controller-manager \
  kube-scheduler kubectl \
  /usr/local/bin/
mkdir -p /var/lib/kubernetes/
mv ca.crt ca.key \
  kube-api-server.key kube-api-server.crt \
  service-accounts.key service-accounts.crt \
  encryption-config.yaml \
  /var/lib/kubernetes/
mv kube-apiserver.service \
  /etc/systemd/system/kube-apiserver.service
mv kube-controller-manager.kubeconfig /var/lib/kubernetes/
mv kube-controller-manager.service /etc/systemd/system/
mv kube-scheduler.kubeconfig /var/lib/kubernetes/
mv kube-scheduler.yaml /etc/kubernetes/config/
mv kube-scheduler.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable kube-apiserver \
  kube-controller-manager kube-scheduler
systemctl start kube-apiserver \
  kube-controller-manager kube-scheduler
kubectl apply -f kube-apiserver-to-kubelet.yaml \
  --kubeconfig admin.kubeconfig

还要记得把swap给关了，不同发行版的情况不同，不过可以先用swapoff -a临时关闭（重启之后会恢复）。
等一会儿，然后来验证一下跑起来没kubectl cluster-info --kubeconfig admin.kubeconfig，输出Kubernetes control plane is running at https://127.0.0.1:6443
再回到堡垒机，看看证书部署好没curl --cacert ca.crt https://master.k8s.local:6443/version
输出

{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.3",
  "gitCommit": "32cc146f75aad04beaaa245a7157eb35063a9f99",
  "gitTreeState": "clean",
  "buildDate": "2025-03-11T19:52:21Z",
  "goVersion": "go1.23.6",
  "compiler": "gc",
  "platform": "linux/amd64"
}

启动worker node

这里需要安装runc，cni插件，containerd，kubelet，kube-proxy。先把所有的systemd的服务文件给准备好

• containred.service（确保满足高并发需求）

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target

[Service]
ExecStartPre=/sbin/modprobe overlay
ExecStart=/bin/containerd
Restart=always
RestartSec=5
Delegate=yes
KillMode=process
OOMScoreAdjust=-999
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity

[Install]
WantedBy=multi-user.target

• kubelet.service

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \
  --config=/var/lib/kubelet/kubelet-config.yaml \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

• kube-proxy.service

[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --config=/var/lib/kube-proxy/kube-proxy-config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

然后就是掏出前面准备好的10-bridge.conf, 99-loopback.conf, kubelet-config.yaml, containred-config.toml, kube-proxy-config.yaml。对于每个节点，需要把网桥的子网网段给填好，然后丢到节点上面去。合起来的脚本就是这样的：

for HOST in node0 node1; do
  SUBNET=$(grep ${HOST} machine.txt | cut -d " " -f 4)
  sed "s|SUBNET|$SUBNET|g" \
    10-bridge-template.conf > 10-bridge.conf

  sed "s|SUBNET|$SUBNET|g" \
    kubelet-config-template.yaml > kubelet-config.yaml

  scp 10-bridge.conf kubelet-config.yaml \
    downloads/worker/* \
    downloads/client/kubectl \
    99-loopback.conf \
    containerd-config.toml \
    kube-proxy-config.yaml \
    containerd.service \
    kubelet.service \
    kube-proxy.service \
  root@${HOST}:~/

  scp \
    downloads/cni-plugins/* \
    root@${HOST}:~/cni-plugins/
done

然后就是进每个节点来执行部署命令了，不过太麻烦了还是用循环来搞。

for HOST in node0 node1; do
  ssh -n root@${HOST} '{
    apt-get update
    apt-get -y install socat conntrack ipset kmod
    swapoff -a
    mkdir -p \
      /etc/cni/net.d \
      /opt/cni/bin \
      /var/lib/kubelet \
      /var/lib/kube-proxy \
      /var/lib/kubernetes \
      /var/run/kubernetes
    mv crictl kube-proxy kubelet runc \
      /usr/local/bin/
    mv ctr /bin
    mv containerd containerd-shim-runc-v2 containerd-stress /bin/
    mv cni-plugins/* /opt/cni/bin/
    mv 10-bridge.conf 99-loopback.conf /etc/cni/net.d/
    modprobe br-netfilter
    echo "br-netfilter" >> /etc/modules-load.d/modules.conf
    echo "net.bridge.bridge-nf-call-iptables = 1" \
      >> /etc/sysctl.d/kubernetes.conf
    echo "net.bridge.bridge-nf-call-ip6tables = 1" \
      >> /etc/sysctl.d/kubernetes.conf
    sysctl -p /etc/sysctl.d/kubernetes.conf
    mkdir -p /etc/containerd/
    mv containerd-config.toml /etc/containerd/config.toml
    mv containerd.service /etc/systemd/system/
    mv kubelet-config.yaml /var/lib/kubelet/
    mv kubelet.service /etc/systemd/system/
    mv kube-proxy-config.yaml /var/lib/kube-proxy/
    mv kube-proxy.service /etc/systemd/system/
    systemctl daemon-reload
    systemctl enable containerd kubelet kube-proxy
    systemctl start containerd kubelet kube-proxy
  }'
done

唔里面配置网络的那里需要解释一下，首先是把br-netfilter添加进内核模块中，然后添加的net.bridge.bridge-nf-call-iptables = 1和net.bridge.bridge-nf-call-ip6tables = 1让流经网桥的流量都可以通过iptables来处理。然后就是立即应用了。这样就启用了网桥的防火墙策略，让网桥的流量正常通过防火墙。

root@master:~# kubectl get nodes   --kubeconfig admin.kubeconfig
NAME    STATUS   ROLES    AGE   VERSION
node0   Ready    <none>   8h    v1.32.3
node1   Ready    <none>   8h    v1.32.3

非常好！

配置堡垒机访问权限

接下来配置堡垒机的kubelet访问权限，首先是生成admin的鉴权文件admin.key，用kueblet命令生成，需要使用到之前的根CA证书，和admin自己的证书

kubectl config set-cluster k8s \
  --certificate-authority=ca.crt \
  --embed-certs=true \
  --server=https://master.k8s.local:6443

kubectl config set-credentials admin \
  --client-certificate=admin.crt \
  --client-key=admin.key

kubectl config set-context k8s \
  --cluster=k8s \
  --user=admin

kubectl config use-context k8s

然后直接试一下能不能get到节点

root@debian:~# kubectl get nodes
NAME    STATUS   ROLES    AGE   VERSION
node0   Ready    <none>   8h    v1.32.3
node1   Ready    <none>   8h    v1.32.3

很好！

配置pod之间的路由通信

因为每个pod会分配一个单独的IP，所以需要添加一个路由让pod子段内的IP能够相互通信，master对于worker的跨节点的pod通信

NODE0_IP=$(grep node0 machine.txt | cut -d " " -f 1)
NODE0_SUBNET=$(grep node0 machine.txt | cut -d " " -f 4)
NODE1_IP=$(grep node1 machine.txt | cut -d " " -f 1)
NODE1_SUBNET=$(grep node1 machine.txt | cut -d " " -f 4)

ssh root@master "
  ip route add '${NODE0_SUBNET}' via '${NODE0_IP}'
  ip route add '${NODE1_SUBNET}' via '${NODE1_IP}'
"

ssh root@node0 "
  ip route add '${NODE0_SUBNET}' via '${NODE0_IP}'
"

ssh root@node1 "
  ip route add '${NODE1_SUBNET}' via '${NODE1_IP}'
"

然后看一下master的路由验证一下

root@debian:~# ssh root@master ip route
default via 192.168.6.1 dev enp1s0 
10.6.0.0/24 via 192.168.6.13 dev enp1s0 
10.6.1.0/24 via 192.168.6.14 dev enp1s0 
192.168.6.0/24 dev enp1s0 proto kernel scope link src 192.168.6.12 

没问题！

测试 & 清理

这里就直接参照github原版来吧
https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/12-smoke-test.md
https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/13-cleanup.md

完结！

啊你以为会有结尾么~