k8s集群安装

虚拟机准备

我这里准备了三台虚拟机,分别部署一个master和两个node,操作系统位ubuntu 20.04。以下为特殊说明为三台机器都要做此操作

安装容器runtime

之前,我们用的容器runtime基本都是docker,但是docker并没有实现k8s的CRI,是在kubelet的有一个组件叫docker-shim做转化,在kubernetes v1.24版本以上这个组件已经废弃,这里选择containerd做容器runtime。当然,containerd是可以使用docker的镜像的。如果非要使用docker的话,被kubernetes废弃的docker-shim被docker自己维护起来了,可以试试看。但是不建议纯纯的浪费资源。

安装

1apt install -y containerd

生成默认配置

1mkdir /etc/containerd
2containerd config default > /etc/containerd/config.toml

配置systemd cgroup驱动程序

1sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

设置代理和修改pause镜像

重所周知的原因

  • 镜像加速

我这里用的网易docker源 你也可以用别的 阿里源等

限免的的 https://xxxxx.mirror.aliyuncs.com 是阿里云加速,xxxx是我屏蔽字段

https://cr.console.aliyun.com/cn-hangzhou/instances/mirrors 可以自啊这个地址申请自己的

 1sed -i 's|config_path = ""|config_path = "/etc/containerd/certs.d/"|g' /etc/containerd/config.toml
 2
 3mkdir -p /etc/containerd/certs.d/docker.io
 4mkdir -p /etc/containerd/certs.d/docker.io
 5cat >/etc/containerd/certs.d/docker.io/hosts.toml <<EOF
 6server = "https://docker.io"
 7[host."https://xxxxx.mirror.aliyuncs.com"]
 8  capabilities = ["pull","resolve"]
 9[host."https://docker.mirrors.ustc.edu.cn"]
10  capabilities = ["pull","resolve"]
11[host."https://registry-1.docker.io"]
12  capabilities = ["pull","resolve","push"]
13EOF
  • 把sandbox_image 修改成阿里云镜像版本自己看着办 不然kube-apiserver可能起不来
1vim /etc/containerd/config.toml
2sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.8"

启动

1systemctl daemon-reload
2systemctl enable containerd
3systemctl start containerd

测试

这里使用 nerdctl工具测试

nerdctl 是 containerd 房官方提供的加强版命令行工具 https://github.com/containerd/nerdctl

下载方式

1wget https://ghproxy.com/https://github.com/containerd/nerdctl/releases/download/v0.23.0/nerdctl-0.23.0-linux-amd64.tar.gz
2
3tar xzvf nerdctl-0.23.0-linux-amd64.tar.gz -C /usr/local/bin
 1nerdctl --debug pull busybox
 2
 3DEBU[0000] verification process skipped                 
 4DEBU[0000] Found hosts dir "/etc/containerd/certs.d"    
 5DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
 6DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs". 
 7DEBU[0000] fetching                                      image="docker.io/library/busybox:latest"
 8DEBU[0000] loading host directory                        dir=/etc/containerd/certs.d/docker.io
 9DEBU[0000] resolving                                     host=hub-mirror.c.163.com
10DEBU[0000] do request                                    host=hub-mirror.c.163.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.0+unknown request.method=HEAD url="http://hub-mirror.c.163.com/v2/library/busybox/manifests/latest?ns=docker.io"

看到 host=hub-mirror.c.163.com 代表配置成功

其他准备工作

防火墙

1# 查看状态
2ufw status
3# 如果打开着呢 请关闭
4ufw disable

时间同步

1apt install -y ntpdate
2ntpdate time.windows.com

关闭swap分区

1# 永久生效 需要重启
2sed -ri 's/.*swap.*/#&/' /etc/fstab
3# 临时关闭,重启后无效
4swapoff -a

将桥接的IPv4流量传递到iptables的链

  1. 在每个节点上将桥接的IPv4流量传递到iptables的链
1cat > /etc/sysctl.d/k8s.conf << EOF
2net.bridge.bridge-nf-call-ip6tables = 1
3net.bridge.bridge-nf-call-iptables = 1
4net.ipv4.ip_forward = 1
5vm.swappiness = 0
6EOF
 1# 加载br_netfilter模块
 2modprobe br_netfilter
 3# 查看是否加载
 4lsmod | grep br_netfilter
 5# 生效
 6sysctl --system
 7
 8echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
 9echo 1 > /proc/sys/net/ipv4/ip_forward
10echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

开启ipvs

在kubernetes中service有两种代理模型,一种是基于iptables,另一种是基于ipvs的。ipvs的性能要高于iptables的,但是如果要使用它,需要手动载入ipvs模块。

 1apt install -y  ipset ipvsadm
 2
 3mkdir -p /etc/sysconfig/modules
 4cat > /etc/sysconfig/modules/ipvs.modules <<EOF
 5#!/bin/bash
 6modprobe -- ip_vs
 7modprobe -- ip_vs_rr
 8modprobe -- ip_vs_wrr
 9modprobe -- ip_vs_sh
10modprobe -- nf_conntrack
11EOF

授权、运行、检查是否加载

1chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

检查是否加载

1lsmod | grep -e ipvs -e nf_conntrack
2
3sysctl --system

设置主机名

设置主机名

1hostnamectl set-hostname <hostname>

三台机器分别为

1# 192.168.56.100
2hostnamectl set-hostname k8s-master
3
4# 192.168.56.101
5hostnamectl set-hostname k8s-node1
6
7# 192.168.56.102
8hostnamectl set-hostname k8s-node2

安装kubeadm、kubelet和kubectl

安装https工具

1apt install -y apt-transport-https ca-certificates curl

下载阿里云cloud公钥

为什么下载阿里云的,不去下载 kubernetes 官方的 你懂得

1sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg

添加 Kubernetes apt 仓库

1echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

更新 apt 包索引,安装 kubelet、kubeadm 和 kubectl,并锁定其版本:

1apt update
2apt install -y kubelet kubeadm kubectl
3apt-mark hold kubelet kubeadm kubectl

查看k8s所需镜像

1kubeadm config images list
2
3egistry.k8s.io/kube-apiserver:v1.25.2
4registry.k8s.io/kube-controller-manager:v1.25.2
5registry.k8s.io/kube-scheduler:v1.25.2
6registry.k8s.io/kube-proxy:v1.25.2
7registry.k8s.io/pause:3.8
8registry.k8s.io/etcd:3.5.4-0
9registry.k8s.io/coredns/coredns:v1.9.3

初始化(只有master执行)

如果带上debug日志可以在后面加 –v=9

1kubeadm init \
2  --apiserver-advertise-address=192.168.56.100 \
3  --image-repository registry.aliyuncs.com/google_containers \
4  --kubernetes-version v1.25.2 \
5  --service-cidr=10.96.0.0/12 \
6  --pod-network-cidr=10.244.0.0/16

出现这个代表 init 成功

 1Your Kubernetes control-plane has initialized successfully!
 2
 3To start using your cluster, you need to run the following as a regular user:
 4
 5  mkdir -p $HOME/.kube
 6  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 7  sudo chown $(id -u):$(id -g) $HOME/.kube/config
 8
 9Alternatively, if you are the root user, you can run:
10
11  export KUBECONFIG=/etc/kubernetes/admin.conf
12
13You should now deploy a pod network to the cluster.
14Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
15  https://kubernetes.io/docs/concepts/cluster-administration/addons/
16
17Then you can join any number of worker nodes by running the following on each as root:
18
19kubeadm join 192.168.56.100:6443 --token qsmewy.fd3hlnkr6b3tb570 \
20        --discovery-token-ca-cert-hash sha256:08afdf5077a0ee0f72553640e09356f19846d030552c35357d05032f95a14b89

根据提示执行

1mkdir -p $HOME/.kube
2sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
3sudo chown $(id -u):$(id -g) $HOME/.kube/config

根据提示在两台node上执行命令 加入集群(这个写你自己master弹出来的命令

1kubeadm join 192.168.56.100:6443 --token qsmewy.fd3hlnkr6b3tb570 \
2        --discovery-token-ca-cert-hash sha256:08afdf5077a0ee0f72553640e09356f19846d030552c35357d05032f95a14b89

出现这个代表节点加入集群成功

1This node has joined the cluster:
2* Certificate signing request was sent to apiserver and a response was received.
3* The Kubelet was informed of the new secure connection details.
4
5Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

部署CNI网络插件

  • kubernetes支持多种网络插件,比如flannel、calico、canal等,任选一种即可,本次选择flannel
1kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

这个是网络地址,可能是失败这里提供一个yaml下载,然后 apply, kube-flannel.yml

测试

 1kubectl get node
 2
 3NAME         STATUS   ROLES           AGE   VERSION
 4k8s-master   Ready    control-plane   31m   v1.25.2
 5k8s-node1    Ready    <none>          31m   v1.25.2
 6k8s-node2    Ready    <none>          30m   v1.25.2
 7
 8kubectl get pod -n kube-system
 9
10NAME                                 READY   STATUS    RESTARTS   AGE
11coredns-c676cc86f-chtqm              1/1     Running   0          31m
12coredns-c676cc86f-ph8wl              1/1     Running   0          31m
13etcd-k8s-master                      1/1     Running   1          32m
14kube-apiserver-k8s-master            1/1     Running   1          32m
15kube-controller-manager-k8s-master   1/1     Running   1          32m
16kube-proxy-949st                     1/1     Running   0          31m
17kube-proxy-9zjnb                     1/1     Running   0          31m
18kube-proxy-g98kp                     1/1     Running   0          31m
19kube-scheduler-k8s-master            1/1     Running   1          32m
20
21kubectl get pod -n kube-flannel
22
23NAME                    READY   STATUS    RESTARTS   AGE
24kube-flannel-ds-jk8fp   1/1     Running   0          2m2s
25kube-flannel-ds-pmmcs   1/1     Running   0          2m2s
26kube-flannel-ds-r5j7s   1/1     Running   0          2m2s

创建一个 nginx pod

1kubectl run nginx --image=nginx:1.17.1
2
3kubectl get pod -owide
4NAME    READY   STATUS    RESTARTS   AGE   IP           NODE        NOMINATED NODE   READINESS GATES
5nginx   1/1     Running   0          27s   10.244.1.2   k8s-node1   <none>           <none>

创建一个 service

 1# vim nginx-svc.yaml
 2
 3apiVersion: v1
 4kind: Service
 5metadata:
 6  name: nginx
 7spec:
 8  type: ClusterIP
 9  ports:
10    - port: 8080
11      targetPort: 80
12      protocol: TCP
13      name: http
14  selector:
15    run: nginx
1kubectl apply -f nginx-svc.yaml
2kubectl get svc
3
4NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
5kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP    43m
6nginx        ClusterIP   10.110.94.194   <none>        8080/TCP   92s

之后加入node

master执行

1kubeadm token create --ttl 0 --print-join-command

执行打印出来的命令