在 Ubuntu 20.04 中安装 k8s 集群

在本文中，我们使用三台云主机来搭建一个基于 Ubuntu 20.04 的 kubernetes 的集群.

准备主机

在云平台申请三台云主机, 在申请时选择使用 Ubuntu 20.04 作为主机的操作系统。完成后得到三台云主机的内网地址分别是:

Master - 192.168.11.16
Node-01 - 192.168.11.2
Node-02 - 192.168.11.6

设定ubunt 20.04 的更新源为国内源

参考为 Ubuntu 20.04 设置国内源

关闭交换分区

为三台机器关闭交换分区。
分别在三台机器中执行以下步骤

临时关闭

1	sudo swapoff -a

永久关闭需要编辑 fstab 文件

1	sudo vi /etc/fstab #注释掉 swap 一行

验证, 执行

free -h

可以看到， Swap 的数量为 0

1
2
3

              total        used        free      shared  buff/cache   available
Mem:          3.8Gi       215Mi       1.9Gi       2.0Mi       1.7Gi       3.3Gi
Swap:            0B          0B          0B

更改内核参数

顺序执行以下命令

1	sudo modprobe overlay

1	sudo modprobe br_netfilter

sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

1	sudo sysctl --system

验证，执行：

1	lsmod \| grep br_netfilter

可以看到系统显示下面的内容

1 2	br_netfilter 28672 0 bridge 176128 1 br_netfilter

安装 docker

安装证书

1	curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg \| sudo apt-key add -

设置镜像为阿里云

1	sudo add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"

更新并安装

1	sudo apt update

1	sudo apt install -y docker.io

修改 docker 配置文件

sudo tee /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

重启 docker

1	sudo systemctl restart docker

设置 docker 开机启动

1	sudo systemctl enable docker

将当前用户加上到 docker 组，这样以后执行 docker 命令就不需要输入 sudo. 假设当前用户名为 stu, 执行：

1	sudo usermod -aG docker stu

注意，要上上面的命令起效，需要重新登陆

验证安装, 可以执行一些常用的 docker 命令，看是否正常, 如：

1	docker images

1	docker --version

改变主机名

使用下面的命令更改主机。

安装文章开头的列表来进行IP来进行修改
Master - 192.168.11.16
Node-01 - 192.168.11.2
Node-02 - 192.168.11.6

如在 192.168.11.16 执行：

1	sudo hostnamectl set-hostname "Master"

设置 Kubernetes repository

1	sudo apt install -y apt-transport-https curl gnupg2 software-properties-common ca-certificates

1	curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg \| sudo apt-key add - #使用阿里云

1
2
3

sudo tee /etc/apt/sources.list.d/kubernetes.list <<EOF 
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main #阿里云源
EOF

1	sudo apt update

安装 kubeadm kubeadm kubectl

1	sudo apt-get install -y kubelet kubeadm kubectl

标记阻止自动更新

1	sudo apt-mark hold kubelet kubeadm kubectl

查看版本

1	kubectl version --client && kubeadm version

将 kubelet 设置为开机启动

1	sudo systemctl enable kubelet

拉取镜像

工作节点只拉取kube-proxy、coredns、pause

首先，用下面的命令查看需要的镜像版本

1	kubeadm config images list

系统显示

k8s.gcr.io/kube-apiserver:v1.23.0
k8s.gcr.io/kube-controller-manager:v1.23.0
k8s.gcr.io/kube-scheduler:v1.23.0
k8s.gcr.io/kube-proxy:v1.23.0
k8s.gcr.io/pause:3.6
k8s.gcr.io/etcd:3.5.1-0
k8s.gcr.io/coredns/coredns:v1.8.6

接下来，分别在 Master 节点和工作节点上新建两个名为: pull-images.sh 的文件，用来下载和对镜像进行更名

Master 节点的脚本

#!/bin/bash

echo "1.拉取镜像"

#下面的版本号要对应
KUBE_VERSION=v1.23.0
PAUSE_VERSION=3.6
CORE_DNS_VERSION=1.8.6
ETCD_VERSION=3.5.1-0

# pull kubernetes images from hub.docker.com
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$KUBE_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:$KUBE_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:$KUBE_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:$KUBE_VERSION
# pull aliyuncs mirror docker images
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$CORE_DNS_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:$ETCD_VERSION

# retag to k8s.gcr.io prefix
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$KUBE_VERSION  k8s.gcr.io/kube-proxy:$KUBE_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:$KUBE_VERSION k8s.gcr.io/kube-controller-manager:$KUBE_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:$KUBE_VERSION k8s.gcr.io/kube-apiserver:$KUBE_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:$KUBE_VERSION k8s.gcr.io/kube-scheduler:$KUBE_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION k8s.gcr.io/pause:$PAUSE_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$CORE_DNS_VERSION k8s.gcr.io/coredns/coredns:v$CORE_DNS_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:$ETCD_VERSION k8s.gcr.io/etcd:$ETCD_VERSION

# untag origin tag, the images won't be delete.
docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$KUBE_VERSION
docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:$KUBE_VERSION
docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:$KUBE_VERSION
docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:$KUBE_VERSION
docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION
docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$CORE_DNS_VERSION
docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:$ETCD_VERSION

echo "====================执行完毕======================"

echo "所需镜像:"
kubeadm config images list

echo "已安装镜像:"
sudo docker images

echo "====如果数量不匹配请多执行几次k8s_pull_images.sh====="

Node 节点的脚本

#下面的版本号要对应
KUBE_VERSION=v1.23.0
PAUSE_VERSION=3.6
CORE_DNS_VERSION=1.8.6

# pull kubernetes images from hub.docker.com
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$KUBE_VERSION

# pull aliyuncs mirror docker images
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$CORE_DNS_VERSION

# retag to k8s.gcr.io prefix
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$KUBE_VERSION  k8s.gcr.io/kube-proxy:$KUBE_VERSION

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION k8s.gcr.io/pause:$PAUSE_VERSION

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$CORE_DNS_VERSION k8s.gcr.io/coredns/coredns:v$CORE_DNS_VERSION

# untag origin tag, the images won't be delete.
docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$KUBE_VERSION

docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION

docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$CORE_DNS_VERSION

echo "====================执行完毕======================"

echo "已安装镜像:"
sudo docker images

echo "====如果数量不匹配请多执行几次k8s_pull_images.sh====="

建立好以后，执行下面的命令为脚本赋予执行权限

1	chmod a+x pull-images.sh

然后执行

1	./pull-images.sh

初始化主节点

在主节点上执行

1	sudo kubeadm init --apiserver-advertise-address=192.168.11.16 --pod-network-cidr=10.244.0.0/16

注意：用你的主节点 IP 替换 apiserver-advertise-address 的值

–apiserver-advertise-address: k8s 中的主要服务apiserver的部署地址，填自己的管理节点 ip
–pod-network-cidr: 这个是 k8s 采用的节点网络，因为我们将要使用flannel作为 k8s 的网络，作为学习环境,这里填10.244.0.0/16就行

运行完成后，你可以看到类似如下的提示

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.11.16:6443 --token pekml2.0s4kxso66inl2tww \
	--discovery-token-ca-cert-hash sha256:6f1b5b42a7a09351a8805a7f2b0bebabb07dcb4e782cc3e74461de2a16962502

按照提示执行

1	mkdir -p $HOME/.kube

1	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

1	sudo chown $(id -u):$(id -g) $HOME/.kube/config

接下来，为集群安装一个 flannel 网络

1	kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

如果该文件无法访问，也可以在当前目录下新建一个名为：kube-flannel.yml 的文件，包含以下内容：

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni-plugin
        image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.0
        command:
        - cp
        args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        volumeMounts:
        - name: cni-plugin
          mountPath: /opt/cni/bin
      - name: install-cni
        image: quay.io/coreos/flannel:v0.15.1
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.15.1
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni-plugin
        hostPath:
          path: /opt/cni/bin
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg

然后执行:

1	kubectl apply -f kube-flannel.yml

可以看到类似如下的信息

Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

验证

在主节点上执行

1	kubectl get nodes

可以看到

1 2	NAME STATUS ROLES AGE VERSION master Ready control-plane,master 21m v1.23.0

加入工作节点

在工作节点上执行

1 2	sudo kubeadm join 192.168.11.16:6443 --token pekml2.0s4kxso66inl2tww \ --discovery-token-ca-cert-hash sha256:6f1b5b42a7a09351a8805a7f2b0bebabb07dcb4e782cc3e74461de2a16962502

可以看打类似如下的信息

[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W1213 11:09:04.079403 1772383 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

当在所有工作节点上都执行了上面的命令行，可以返回到 主节点，执行

1	kubectl get nodes

如果看到类似的信息，则说明k8s集群已经安装成功

NAME      STATUS   ROLES                  AGE     VERSION
master    Ready    control-plane,master   26m     v1.23.0
node-02   Ready    <none>                 62s     v1.23.0
node-03   Ready    <none>                 2m58s   v1.23.0