泰格的技术分享

菜单

tanger
发布于 2023-11-05 / 7 阅读 / 0 评论 / 0 点赞

k8s集群部署（v1.23.6）

一、环境准备

前言

1、建议最小硬件配置：2核CPU、2G内存、40G硬盘

2、2台服务器，同一网段，可以互相访问，能够访问外网

3、软件环境：

(1) 操作系统：centos

(2) docker-ce-3:20.10.24-3.el9（k8s-1.23.6版本最高支持验证）

(3) Kubernetes-1.23.6 (高版本操作docker需要安装cri-dockerd，环境不好配置)

安装步骤

  1. 集群hosts规划，集群环境访问需要

  2. 服务器时间同步

  3. 禁用selinux、firewalld、swap

  4. 添加网桥过滤转发

  5. docker安装

  6. kubeadm、kubelet、kubectl安装

  7. 容器网络安装（calico）

  8. 初始化k8s集群

  9. 测试pod

二、集群hosts规划

服务器

HOSTS

192.168.3.50

k8s-master

192.168.3.51

k8s-node0

192.168.3.52

k8s-node1

修改主机名解析

#编辑文件 /etc/hosts
# 添加主机解析
# 10.0.0.12   master
# 10.0.0.13   node1

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.3.50 k8s-master
192.168.3.51 k8s-node0
192.168.3.52 k8s-node1

命令设置主机名

# 需要分别在master和node都配置上
# k8s-master/k8s-node0/k8s-node1
hostnamectl set-hostname  k8s-master

# 重启服务
systemctl reboot
# 使用主机名互相ping，看是否网络互通
ping k8s-master
ping k8s-node0
ping k8s-node1

三、时间同步

# 执行命令前 可以 使用 rpm -qa |grep chrony 查看系统是否已安装chrony，
# 没有安装环境可使用 yum install chrony 命令安装
#启动chronyd服务
systemctl start chronyd
systemctl enable chronyd

四、禁用selinux、firewalld、swap

# 关闭firewalld
systemctl stop firewalld
systemctl disable firewalld

# 禁用selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config

#禁用swap分区
vi /etc/fstab 
#注释掉下面的设置
#/dev/mapper/centos-swap swap

五、添加网桥过滤转发

cat > /etc/sysctl.d/kubernetes.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# 生效命令
sysctl --system

# 重启服务
systemctl reboot

六、安装指定版本docker

# step 1: 安装必要的一些系统工具
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
# Step 2: 添加软件源信息
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# Step 3
sudo sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo
# Step 4: 更新软件源缓存，如果centos版本大于7，使用sudo yum makecache即可
sudo yum makecache fast
# Step 5: 查看docker版本列表，x86_64（x86架构）aarch64（arm架构）
yum list docker-ce --showduplicates | sort -r

# x86_64示例：
#   Loaded plugins: branch, fastestmirror, langpacks
#   docker-ce.x86_64            17.03.1.ce-1.el7.centos            docker-ce-stable
#   docker-ce.x86_64            17.03.1.ce-1.el7.centos            @docker-ce-stable
#   docker-ce.x86_64            17.03.0.ce-1.el7.centos            docker-ce-stable

# aarch64示例：
#	docker-ce.aarch64               3:24.0.5-1.el9                 docker-ce-stable 
#	docker-ce.aarch64               3:24.0.4-1.el9                 docker-ce-stable 
#	docker-ce.aarch64               3:24.0.3-1.el9                 docker-ce-stable 
#	docker-ce.aarch64               3:24.0.2-1.el9                 docker-ce-stable 
#	docker-ce.aarch64               3:24.0.1-1.el9                 docker-ce-stable 
#	docker-ce.aarch64               3:24.0.0-1.el9                 docker-ce-stable 
#	docker-ce.aarch64               3:23.0.6-1.el9                 docker-ce-stable 
#	docker-ce.aarch64               3:23.0.5-1.el9                 docker-ce-stable 
#	docker-ce.aarch64               3:23.0.4-1.el9                 docker-ce-stable 
#	docker-ce.aarch64               3:23.0.2-1.el9                 docker-ce-stable 
#	docker-ce.aarch64               3:23.0.1-1.el9                 docker-ce-stable 
#	docker-ce.aarch64               3:23.0.0-1.el9                 docker-ce-stable 
#	docker-ce.aarch64               3:20.10.24-3.el9               docker-ce-stable

# Step 6: 安装指定版本的Docker-CE: (VERSION例如上面的3:20.10.24-3.el9)
sudo yum -y install docker-ce-3:20.10.24-3.el9

# 启动docker
systemctl start docker

# 设置开机启动
systemctl enable docker

# 配置docker，修改cgroup、log、镜像地址
cat > /etc/docker/daemon.json << EOF
{
    "registry-mirrors": [
        "https://docker.mirrors.ustc.edu.cn/",
        "https://hub-mirror.c.163.com/"
    ],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
        "cache-compress": "true",
        "cache-disabled": "false",
        "cache-max-file": "5",
        "cache-max-size": "20m",
        "env": "os,customer",
        "max-file": "2",
        "max-size": "50m"
    }
}
EOF

# 重启docker服务
systemctl restart docker

七、kubeadm、kubelet、kubectl安装

# kubernetes镜像切换成国内源，baseurl中注意架构类型
# x86架构：baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
# arm架构：baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-aarch64/

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-aarch64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# 安装指定版本 kubeadm、kubelet、kubectl
yum install -y kubelet-1.23.6 kubeadm-1.23.6 kubectl-1.23.6

systemctl enable kubelet

PS：如果找不到kubelet-1.23.6 kubeadm-1.23.6 kubectl-1.23.6安装包，使用yum clean all 清除缓存，然后yum makecache 重建缓存

PS：以上步骤操作主节点和子节点都需要进行

初始化主节点（仅master节点上操作）

kubeadm init \
  --apiserver-advertise-address=192.168.3.50 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.23.6 \
  --service-cidr=10.96.0.0/12 \
  --pod-network-cidr=10.244.0.0/16

–apiserver-advertise-address #集群通告地址(master 机器IP)

–image-repository #由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址

–kubernetes-version #K8s版本，与上面安装的一致

–service-cidr #集群内部虚拟网络，Pod统一访问入口，默认：10.96.0.0/12

–pod-network-cidr #Pod网络，与下面部署的CNI网络组件yaml中保持一致，默认：10.244.0.0/16

# 创建成功示例：
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.3.50:6443 --token 8aypwr.b8t4md9vvh4ughxs \
        --discovery-token-ca-cert-hash sha256:9c4d69585d6bbb444d61d367029f89d9b632d76487d199904459f67433c2ea1d

# 保存后续子节点加入集群的命令
kubeadm join 192.168.3.50:6443 --token 8aypwr.b8t4md9vvh4ughxs \
        --discovery-token-ca-cert-hash sha256:9c4d69585d6bbb444d61d367029f89d9b632d76487d199904459f67433c2ea1d

# 默认token有效期为24小时，当过期之后，该token就不可用了。这时就需要重新创建token，可以直接使用命令快捷生成
kubeadm token create --print-join-command

# 按要求执行后续步骤：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 查看节点状态
kubectl get node

[root@master ~]# kubectl get nodes
NAME     STATUS     ROLES                  AGE   VERSION
master   NotReady   control-plane,master   76s   v1.23.6

八、容器网络安装fannel（仅master节点上操作）

kubectl apply -f https://raw.githubusercontent.com/coreos/fannel/master/Documentation/kube-fannel.yml

九、初始化k8s集群（子节点加入到集群中）

kubeadm join 192.168.3.50:6443 --token 8aypwr.b8t4md9vvh4ughxs \
        --discovery-token-ca-cert-hash sha256:9c4d69585d6bbb444d61d367029f89d9b632d76487d199904459f67433c2ea1d

# master上查看节点状态
[root@master kuboard]# kubectl get node
NAME     STATUS   ROLES                  AGE     VERSION
master   Ready    control-plane,master   3h24m   v1.23.6
node1    Ready    <none>                 173m    v1.23.6

# 查看系统组件安装状态
kubectl get pod -n kube-system -w

十、测试pod（master节点上操作）

# 部署nginx测试
# 创建yaml文件
vim nginx.yaml

nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx

nginx-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  type: NodePort
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
# 部署nginx服务
kubectl apply -f nginx.yaml
kubectl apply -f nginx-service.yaml

# 查看服务 （Running说明启动成功）
kubectl get pod,svc

[root@master kuboard]# kubectl get pod,svc
NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-85b98978db-9ld5s   1/1     Running   0          167m

NAME                    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
service/kubernetes      ClusterIP   10.96.0.1     <none>        443/TCP        3h30m
service/nginx-service   NodePort    10.97.9.159   <none>        80:31554/TCP   167m

# 访问测试
curl http://10.97.9.159

[root@master kuboard]# curl http://10.97.9.159
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

PS：加入节点提示网络ip重复（network: failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from），可以删除对应node节点的网卡，等待系统重建即可解决

ip link set cni0 down
ip link delete cni0
#可选
kubectl delete pod -n kube-system ${podname}

node节点退出重新加入集群

  • 首先删掉节点

  • node重新加入

首先删掉节点

注意：以下操作都是在master下操作。

kubectl drain k8snode02 --delete-local-data --force --ignore-daemonsets node/k8snode02

二：删除节点

kubectl delete node k8snode02

三：确认是否已经删除

kubectl get nodes

三：生成永久Token（node加入的时候会用到）

kubeadm token create --ttl 0 --print-join-command

`kubeadm join 192.168.233.3:6443 --token rpi151.qx3660ytx2ixq8jk     --discovery-token-ca-cert-hash sha256:5cf4e801c903257b50523af245f2af16a88e78dc00be3f2acc154491ad4f32a4`#这是生成的Token,node加入时使用，此``是起到注释作用，无其他用途。

四：查看Token确认

kubeadm token list

node重新加入

注意:以下操作在node下操作

一：停掉kubelet

systemctl stop kubelet

二：删除之前的相关文件

rm -rf /etc/kubernetes/*

三：加入集群网址

kubeadm join 192.168.233.3:6443 --token rpi151.qx3660ytx2ixq8jk     --discovery-token-ca-cert-hash sha256:5cf4e801c903257b50523af245f2af16a88e78dc00be3f2acc154491ad4f32a4

交付平台规划
交付平台

评论