ICode9

精准搜索请尝试: 精确搜索
首页 > 其他分享> 文章详细

二进制部署k8s(1.18版本)+高可用版本测试

2020-09-24 17:03:31  阅读:209  来源: 互联网

标签:kube etcd kubernetes -- 1.18 版本 k8s data


二进制部署k8s(1.18版本)

部署说明

部署说明

软件名称下载地址备注
centos7.7+ https://mirrors.aliyun.com/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-Minimal-1908.iso 宿主机操作系统
kubernetes-server https://dl.k8s.io/v1.18.4/kubernetes-server-linux-amd64.tar.gz  
etcd https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz k8s数据存储
cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64%3Cbr/%3Ehttps://pkg.cfssl.org/R1.2/cfssljson_linux-amd64%3Cbr/%3Ehttps://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 证书签发工具
docker https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz CNR运行引擎
cni https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz 网络环境

部署规划

主机ip作用部署的软件备注
centos7-node4 172.17.17.85 master kube-apiserver,kube-controller-manager,kube-scheduler,docker,etcd 后期介绍master扩容
centos7-node5 172.17.17.86 node kubelet,kube-proxy,docker,etcd 后期介绍master扩容
centos7-node6 172.17.17.87 node kubelet,kube-proxy,docker,etcd 后期介绍master扩容

系统初始化(所有节点执行)

  • 软件安装路径默认路径为/data

# 更新yum源
yum -y install wget && wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo && yum -y install epel-release
# 关闭selinux,firewalld,swap
sed -i 's/enforcing/disabled/' /etc/selinux/config
systemctl disable firewalld && systemctl stop firewalld
sed -ri 's/.*swap.*/#&/' /etc/fstab && swapoff -a
# 设置好主机名与主机名解析
cat >> /etc/hosts << EOF
172.17.17.85 k8s-master-1
172.17.17.89 k8s-master-2
172.17.17.86 k8s-node5
172.17.17.87 k8s-node6
EOF
# 将桥接的IPv4流量传递到iptables的链
modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf #配置生效
#时间同步
yum install chrony -y && systemctl enable chronyd && systemctl start chronyd
timedatectl set-timezone Asia/Shanghai && timedatectl set-ntp yes

生成etcd证书配置

准备cfssl证书管理工具,使用json文件生成证书,相比openssl更方便使用

# 软件安装
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/local/bin/cfssljson
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl*

准备ca与证书配置

mkdir -p ~/TLS/{etcd,k8s} && cd ~/TLS/etcd
# 自签CA配置文件
cat > ca-config.json << EOF
{
 "signing": {
   "default": {
     "expiry": "87600h"
  },
   "profiles": {
     "www": {
       "expiry": "87600h",        
       "usages": [
           "signing",
           "key encipherment",
           "server auth",
           "client auth"
      ]
    }
  }
}
}
EOF
# 自签csr配置文件
cat > ca-csr.json << EOF
{
 "CN": "etcd CA",
 "key": {
   "algo": "rsa",
   "size": 2048
},
 "ca": {
    "expiry": "87600h"
},
 "names": [
  {
     "C": "CN",
     "L": "BJ",
     "ST": "BeiJing"
  }
]
}
EOF
# 生成CA证书
[root@k8s-master-1 etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
[root@k8s-master-1 etcd]# ls *pem
# 签发ETCD https证书

[root@k8s-master-1 etcd]# cat > server-csr.json << EOF

{ "CN": "etcd", "hosts": [ "172.17.17.85", "172.17.17.86", "172.17.17.87",

"172.17.17.89",

], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing" } ] } EOF

注:上述文件hosts字段中IP为所有etcd节点的集群内部通信IP,一个都不能少!为了方便后期扩 容可以多写几个预留的IP。

签发证书

[root@k8s-master-1 etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
[root@k8s-master-1 etcd]# ls server*pem #签发证书文件

部署ETCD集群

单个节点配置

# 安装路径准备
mkdir /data/etcd/{bin,cfg,ssl,data} -p
# 二进制文件准备
wget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz && tar xf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/etcd* /data/etcd/bin/
# 当前节点192.168.56.14配置文件
cat > /data/etcd/cfg/etcd.conf <<EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/data/etcd/data/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.17.17.85:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.17.17.85:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.17.17.85:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.17.17.85:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://172.17.17.85:2380,etcd-2=https://172.17.17.86:2380,etcd-3=https://172.17.17.87:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
# systemctl 启动管理文件配置(所有节点配置是一致的)
cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/data/etcd/cfg/etcd.conf
ExecStart=/data/etcd/bin/etcd \
  --cert-file=/data/etcd/ssl/server.pem \
  --key-file=/data/etcd/ssl/server-key.pem \
  --peer-cert-file=/data/etcd/ssl/server.pem \
  --peer-key-file=/data/etcd/ssl/server-key.pem \
  --trusted-ca-file=/data/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/data/etcd/ssl/ca.pem \
  --logger=zap
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
# 拷贝证书文件
mv ~/TLS/etcd/*pem /data/etcd/ssl
# 启动当前节点
systemctl daemon-reload && systemctl enable etcd && systemctl start etcd

配置文件字段介绍

ETCD_NAME:节点名称,集群中唯一 ETCD_DATA_DIR:数据目录 ETCD_LISTEN_PEER_URLS:集群通信监听地址 ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址 ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址 ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址 ETCD_INITIAL_CLUSTER:集群节点地址 ETCD_INITIAL_CLUSTER_TOKEN:集群Token ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new是新集群,existing表示加入已有集 群

其余两个节点配置

  1. 数据分发(从172.17.17.85分发到86,87两个节点)

    scp -r /data/etcd/ root@172.17.17.85:/opt/

    scp /usr/lib/systemd/system/etcd.service root@172.17.17.86:/usr/lib/systemd/system/

    scp /usr/lib/systemd/system/etcd.service root@172.17.17.87:/usr/lib/systemd/system/

然后在节点2和节点3分别修改etcd.conf配置文件中的节点名称和当前服务器IP:

vi /opt/etcd/cfg/etcd.conf#[Member]ETCD_NAME="etcd-1"# 修改此处,节点2改为etcd-2,节点3改为etcd-3ETCD_DATA_DIR="/var/lib/etcd/default.etcd"ETCD_LISTEN_PEER_URLS="https://172.17.17.86:2380"# 修改此处为当前服务器IPETCD_LISTEN_CLIENT_URLS="https://172.17.17.86:2379"# 修改此处为当前服务器IP#[Clustering]ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.17.17.86:2380"# 修改此处为当前服务器IPETCD_ADVERTISE_CLIENT_URLS="https://172.17.17.86:2379"# 修改此处为当前服务器IPETCD_INITIAL_CLUSTER="etcd-1=https://172.17.17.85:2380,etcd-2=https://172.17.17.86:2380,etcd-3=https://172.17.17.87:2380"ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"ETCD_INITIAL_CLUSTER_STATE="new"

最后启动etcd并设置开机启动,同上。

systemctl daemon-reload && systemctl enable etcd && systemctl start etcd

验证Etcd部署状态是否成功

[root@k8s-master-1 default.etcd]# ETCDCTL_API=3 /data/etcd/bin/etcdctl --cacert=/data/etcd/ssl/ca.pem --cert=/data/etcd/ssl/server.pem --key=/data/etcd/ssl/server-key.pem --endpoints="https://172.17.17.85:2379,https://172.17.17.86:2379,https://172.17.17.87:2379" endpoint health

正常返回结果

https://172.17.17.87:2379 is healthy: successfully committed proposal: took = 32.593008ms https://172.17.17.85:2379 is healthy: successfully committed proposal: took = 39.390838ms https://172.17.17.86:2379 is healthy: successfully committed proposal: took = 38.243128ms

集群异常排查

1. 查看/var/log/message日志或者journalctl -xe -f -uetcd
2. 一般配置文件没问题的话就ok,最大的问题还有一点就是网络通信和防火墙,注意响应的策略放开即可
3.如果集群ID重复错误   删除这个目录/data/etcd/data/default.etcd

所有节点安装docker

# 下载和解压docker二进制文件
wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz && tar xf docker-19.03.9.tgz
# 转移可执行文件
mv docker/* /usr/bin/

# 配置systemd管理docker (其余的两个节点也需要安装)
cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF

docker配置与启动

# 配置docker阿里云镜像加速和存储路径(graph)
mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
 "graph": "/data/docker",
 "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF
# 服务启动
systemctl daemon-reload && systemctl restart docker && systemctl enable docker

k8s master节点的安装与部署

当前部署master节点 ip: 172.17.17.85

生成k8s证书配置 自签证书颁发机构(CA)

cd ~/TLS/k8s
cat > ca-config.json <<EOF
{
   "signing":{
       "default":{
           "expiry":"87600h"
      },
       "profiles":{
           "kubernetes":{
               "expiry":"87600h",
               "usages":[
                   "signing",
                   "key encipherment",
                   "server auth",
                   "client auth"
              ]
          }
      }
  }
}
EOF

cat > ca-csr.json <<EOF
{
   "CN":"kubernetes",
   "key":{
       "algo":"rsa",
       "size":2048
  },
   "names":[
      {
           "C":"CN",
           "L":"Beijing",
           "ST":"Beijing",
           "O":"k8s",
           "OU":"System"
      }
  ]
}
EOF

# 生成CA证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

使用自签CA签发kube-apiserver HTTPS证书

cd ~/TLS/k8s
cat > server-csr.json << EOF
{
   "CN":"kubernetes",
   "hosts":[
       "10.0.0.1",
       "172.0.0.1",
       "127.0.0.1",
       "172.17.17.85",
       "172.17.17.86",
       "172.17.17.87",
       "172.17.17.89",
       "172.17.17.84",
"172.17.17.83",
       "kubernetes",
       "kubernetes.default",
       "kubernetes.default.svc",
       "kubernetes.default.svc.cluster",
       "kubernetes.default.svc.cluster.local"
  ],
   "key":{
       "algo":"rsa",
       "size":2048
  },
   "names":[
      {
           "C":"CN",
           "L":"BeiJing",
           "ST":"BeiJing",
           "O":"k8s",
           "OU":"System"
      }
  ]
}
EOF

注:上述文件hosts字段中IP为所有Master/LB/VIP IP一个都不能少!为了方便后期扩容可以多 写几个预留的IP。

生成apiserver证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
ls *pem

从Github下载二进制文件

# 创建软件目录
mkdir -p /data/kubernetes/{cfg,bin,ssl,logs}
# 文件下载与拷贝
wget https://dl.k8s.io/v1.18.4/kubernetes-server-linux-amd64.tar.gz && tar xf kubernetes-server-linux-amd64.tar.gz
cp kubernetes/server/bin/kube-apiserver /data/kubernetes/bin/
cp kubernetes/server/bin/kube-controller-manager /data/kubernetes/bin/
cp kubernetes/server/bin/kube-scheduler /data/kubernetes/bin/
cp kubernetes/server/bin/kubectl /usr/bin/

部署kube-apiserver

# 创建配置文件
cat > /data/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/data/kubernetes/logs \
--etcd-servers=https://172.17.17.85:2379,https://172.17.17.86:2379,https://172.17.17.87:2379 \
--bind-address=172.17.17.85 \
--secure-port=6443 \
--advertise-address=172.17.17.85 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/data/kubernetes/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/data/kubernetes/ssl/server.pem \
--kubelet-client-key=/data/kubernetes/ssl/server-key.pem \
--tls-cert-file=/data/kubernetes/ssl/server.pem \
--tls-private-key-file=/data/kubernetes/ssl/server-key.pem \
--client-ca-file=/data/kubernetes/ssl/ca.pem \
--service-account-key-file=/data/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/data/etcd/ssl/ca.pem \
--etcd-certfile=/data/etcd/ssl/server.pem \
--etcd-keyfile=/data/etcd/ssl/server-key.pem \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/data/kubernetes/logs/k8s-audit.log"
EOF

# 拷贝证书
mv ~/TLS/k8s/*pem /data/kubernetes/ssl/

注意事项:

--logtostderr:启用日志 ---v:日志等级 --log-dir:日志目录 --etcd-servers:etcd集群地址 --bind-address:监听地址 --secure-port:https安全端口 --advertise-address:集群通告地址 --allow-privileged:启用授权 --service-cluster-ip-range:Service虚拟IP地址段

--enable-admission-plugins:准入控制模块

--authorization-mode:认证授权,启用RBAC授权和节点自管理

--enable-bootstrap-token-auth:启用TLS bootstrap机制

--token-auth-file:bootstrap token文件

--service-node-port-range:Service nodeport类型默认分配端口范围

--kubelet-client-xxx:apiserver访问kubelet客户端证书

--tls-xxx-file:apiserver https证书

--etcd-xxxfile:连接Etcd集群证书 --audit-log-xxx:审计日志

启用TLS Bootstrap机制

TLS Bootstraping:Master apiserver启用TLS认证后,Node节点kubelet和kube-proxy要与kube-apiserver进行通信,必须使用CA签发的有效证书才可以,当Node节点很多时,这种客户端证书颁发需要大量工作,同样也会增加集群扩展复杂度。为了简化流程,Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书,kubelet会以一个低权限用户自动向apiserver申请证书,kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式,目前主要用于kubelet,kube-proxy还是由我们统一颁发一个证书。TLS bootstraping 工作流程:

kubernetes(七) 二进制部署k8s(1.18.4版本)

  • 根据上述配置token文件

cat > /data/kubernetes/cfg/token.csv << EOF
2b4b65d2e33e24dc0beafddda6dd4b23,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

格式:token,用户名,UID,用户组 token也可自行生成替换:

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

使用systemctl管理apiserver

生成配置文件

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF 
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/data/kubernetes/cfg/kube-apiserver.conf
ExecStart=/data/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

启动并设置开机自启动

systemctl daemon-reload && systemctl start kube-apiserver && systemctl enable kube-apiserver
  • 授权kubelet-bootstrap用户允许请求证书

    kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

部署kube-controller-manager

创建配置文件

cat > /data/kubernetes/cfg/kube-controller-manager.conf << EOF KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/data/kubernetes/logs \ --leader-elect=true \ --master=127.0.0.1:8080 \ --bind-address=127.0.0.1 \ --allocate-node-cidrs=true \ --cluster-cidr=10.244.0.0/16 \ --service-cluster-ip-range=10.0.0.0/24 \ --cluster-signing-cert-file=/data/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/data/kubernetes/ssl/ca-key.pem \ --root-ca-file=/data/kubernetes/ssl/ca.pem \ --service-account-private-key-file=/data/kubernetes/ssl/ca-key.pem \ --experimental-cluster-signing-duration=87600h0m0s" EOF

--master:通过本地非安全本地端口8080连接apiserver。

--leader-elect:当该组件启动多个时,自动选举(HA)

--cluster-signing-cert-file/--cluster-signing-key-file:自动为kubelet颁发证书的CA,与apiserver保持一致

systemctl管理controller-manager

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/data/kubernetes/cfg/kube-controller-manager.conf ExecStart=/data/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF

启动&开机自启

systemctl daemon-reload && systemctl start kube-controller-manager && systemctl enable kube-controller-manager

部署kube-scheduler

创建配置文件

cat > /data/kubernetes/cfg/kube-scheduler.conf << EOF KUBE_SCHEDULER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/data/kubernetes/logs \ --leader-elect \ --master=127.0.0.1:8080 \ --bind-address=127.0.0.1" EOF

--master:通过本地非安全本地端口8080连接apiserver。

--leader-elect:当该组件启动多个时,自动选举(HA)

systemctl管理kube-scheduler

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/data/kubernetes/cfg/kube-scheduler.conf ExecStart=/data/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF

  • 启动&开机自启

    systemctl daemon-reload && systemctl start kube-scheduler && systemctl enable kube-scheduler

至此master部署完成,集群状态查看

kubectl get cs               

返回如下结果,证明mater部署ok

NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler           Healthy   ok                  
etcd-2               Healthy   {"health":"true"}  
etcd-0               Healthy   {"health":"true"}  
etcd-1               Healthy   {"health":"true"}  

部署WorkNode

当前操作节点是: 172.17.17.85(将这个master也作为node)

  • 需要的软件:

    • kubelet

    • kube-proxy

基础软件包准备

# 创建软件目录
mkdir -p /data/kubernetes/{cfg,bin,ssl,logs}
# 文件下载与拷贝
wget https://dl.k8s.io/v1.18.4/kubernetes-server-linux-amd64.tar.gz && tar xf kubernetes-server-linux-amd64.tar.gz
cp kubernetes/server/bin/kube-proxy /data/kubernetes/bin/
cp kubernetes/server/bin/kubelet /data/kubernetes/bin/

部署kubelet

创建kubelet配置文件

cat > /data/kubernetes/cfg/kubelet.conf << EOF

KUBELET_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/data/kubernetes/logs \ --hostname-override=k8s-master-1 \ --network-plugin=cni \ --kubeconfig=/data/kubernetes/cfg/kubelet.kubeconfig \ --bootstrap-kubeconfig=/data/kubernetes/cfg/bootstrap.kubeconfig \ --config=/data/kubernetes/cfg/kubelet-config.yml \ --cert-dir=/data/kubernetes/ssl \ --pod-infra-container-image=lizhenliang/pause-amd64:3.0" EOF

--hostname-override:显示名称,集群中唯一

--network-plugin:启用CNI

--kubeconfig:空路径,会自动生成,后面用于连接apiserver

--bootstrap-kubeconfig:首次启动向apiserver申请证书

--config:配置参数文件

--cert-dir:kubelet证书生成目录

--pod-infra-container-image:管理Pod网络容器的镜像

创建参数配置文件

cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF

cat > /data/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
anonymous:
  enabled: false
webhook:
  cacheTTL: 2m0s
  enabled: true
x509:
  clientCAFile: /data/kubernetes/ssl/ca.pem
authorization:
mode: Webhook
webhook:
  cacheAuthorizedTTL: 5m0s
  cacheUnauthorizedTTL: 30s
evictionHard:
imagefs.available: 15%
memory.available: 100M
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF

生成bootstrap.kubeconfig文件

[root@k8s-master-1 ~]# KUBE_APISERVER="https://172.17.17.85:6443" [root@k8s-master-1 ~]# TOKEN="2b4b65d2e33e24dc0beafddda6dd4b23"

[root@k8s-master-1 ~]# kubectl config set-cluster kubernetes --certificate-authority=/data/kubernetes/ssl/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=bootstrap.kubeconfig

[root@k8s-master-1 ~]# kubectl config set-credentials "kubelet-bootstrap" --token=${TOKEN} --kubeconfig=bootstrap.kubeconfig

[root@k8s-master-1 ~]# kubectl config set-context default --cluster=kubernetes --user="kubelet-bootstrap" --kubeconfig=bootstrap.kubeconfig

[root@k8s-master-1 ~]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

拷贝到配置文件路径:

[root@k8s-master-1 ~]# cp bootstrap.kubeconfig /data/kubernetes/cfg

systemctl管理kubelet

  • 创建启动文件

[root@k8s-master-1 ~]# cat > /usr/lib/systemd/system/kubelet.service << EOF [Unit] Description=Kubernetes Kubelet After=docker.service

[Service] EnvironmentFile=/data/kubernetes/cfg/kubelet.conf ExecStart=/data/kubernetes/bin/kubelet $KUBELET_OPTS Restart=on-failure LimitNOFILE=65536

[Install] WantedBy=multi-user.target EOF

  • 启动kubelet&开机自启动

    systemctl daemon-reload && systemctl start kubelet && systemctl enable kubelet

    如有异常,及时查看日志解决,大多数问题会出现在/data/kubernetes/cfg/kubelet-config.yml格式上

批准kubelet证书申请并加入集群

查看kubelet证书请求

[root@k8s-master-1 ~]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-cUACWJSaXTJHpc1XIscmgbvarl1aZ3Hk79d7qjkII_A 68s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending

批准kubelet证书申请,并加入集群

[root@k8s-master-1 ~]# kubectl certificate approve node-csr-cUACWJSaXTJHpc1XIscmgbvarl1aZ3Hk79d7qjkII_A

查看节点 [root@k8s-master-1 ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master-1 NotReady <none> 4s v1.18.4

注:由于网络插件还没有部署,节点会没有准备就绪 NotReady

部署kube-proxy

创建配置文件

[root@k8s-master-1 ~]# cat > /data/kubernetes/cfg/kube-proxy.conf << EOF KUBE_PROXY_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/data/kubernetes/logs \ --config=/data/kubernetes/cfg/kube-proxy-config.yml" EOF

配置参数文件

[root@k8s-master-1 ~]# cat > /data/kubernetes/cfg/kube-proxy-config.yml << EOF kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 metricsBindAddress: 0.0.0.0:10249 clientConnection: kubeconfig: /data/kubernetes/cfg/kube-proxy.kubeconfig hostnameOverride: k8s-master-1 clusterCIDR: 10.0.0.0/24 EOF

生成kube-proxy.kubeconfig文件

生成kube-proxy证书

cd ~/TLS/k8s/
cat > kube-proxy-csr.json << EOF
{
   "CN":"system:kube-proxy",
   "hosts":[

  ],
   "key":{
       "algo":"rsa",
       "size":2048
  },
   "names":[
      {
           "C":"CN",
           "L":"BeiJing",
           "ST":"BeiJing",
           "O":"k8s",
           "OU":"System"
      }
  ]
}
# 生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

ls kube-proxy*pem
kube-proxy-key.pem kube-proxy.pem

生成kubeconfig配置文件

KUBE_APISERVER="https://172.17.17.85:6443"

kubectl config set-cluster kubernetes --certificate-authority=/data/kubernetes/ssl/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy --client-certificate=./kube-proxy.pem --client-key=./kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

# 拷贝配置到cfg下

cp kube-proxy.kubeconfig /data/kubernetes/cfg/

systemctl管理kube-proxy

  • 创建启动文件

cat > /usr/lib/systemd/system/kube-proxy.service << EOF [Unit] Description=Kubernetes Proxy After=network.target [Service] EnvironmentFile=/data/kubernetes/cfg/kube-proxy.conf ExecStart=/data/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS Restart=on-failure LimitNOFILE=65536

[Install] WantedBy=multi-user.target EOF

  • 设置开机自启动

systemctl daemon-reload && systemctl start kube-proxy && systemctl enable kube-proxy

部署CNI网络

先准备好CNI二进制文件:下载地址:https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz

wget https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz &&  mkdir /opt/cni/bin -p
tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin/
  • 部署flannel

    wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
    sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml
    kubectl apply -f kube-flannel.yml

查看部署状态

kubectl get nodes
NAME         STATUS   ROLES   AGE   VERSION
k8s-master-1   Ready   <none>   79m   v1.18.4

部署好网络插件,Node准备就绪

授权apiserver访问kubelet

[root@k8s-master-1 k8s]# cat > apiserver-to-kubelet-rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
  rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
  kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
      - ""
  resources:
    - nodes/proxy
    - nodes/stats
    - nodes/log
    - nodes/spec
    - nodes/metrics
    - pods/log
  verbs:
    - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: kubernetesuthorization.k8s.iot
name: system:kube-apiserver
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: kubernetes
EOF
## 创建授权

kubectl apply -f apiserver-to-kubelet-rbac.yaml

新增workNode

同步文件和配置

  • 将172.17.17.85上的node相关的配置拷贝到172.17.17.86,172.17.17.87上

mkdir -p /data/kubernetes/{cfg,bin,ssl,logs} #在86 87上先创建目录

# kubelet,kube-proxy拷贝
scp /data/kubernetes/bin/kubelet 172.17.17.86:/data/kubernetes/bin/
scp /data/kubernetes/bin/kubelet 172.17.17.87:/data/kubernetes/bin/
scp /data/kubernetes/bin/kube-proxy root@172.17.17.86:/data/kubernetes/bin/
scp /data/kubernetes/bin/kube-proxy root@172.17.17.87:/data/kubernetes/bin/
# cni插件拷贝
scp -rp /opt/cni/ root@172.17.17.86:/opt
scp -rp /opt/cni/ root@172.17.17.87:/opt
# 证书拷贝
scp /data/kubernetes/ssl/ca.pem 172.17.17.86:/data/kubernetes/ssl/
scp /data/kubernetes/ssl/ca.pem 172.17.17.87:/data/kubernetes/ssl/

# 配置文件拷贝
scp /data/kubernetes/cfg/kube-proxy* 172.17.17.86:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/kube-proxy* 172.17.17.87:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/kubelet* 172.17.17.86:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/kubelet* 172.17.17.87:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/bootstrap.kubeconfig 172.17.17.86:/data/kubernetes/cfg/
scp /data/kubernetes/cfg/bootstrap.kubeconfig 172.17.17.87:/data/kubernetes/cfg/
# 启动文件拷贝
scp /usr/lib/systemd/system/kubelet.service 172.17.17.86:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/kubelet.service 172.17.17.87:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/kube-proxy.service 172.17.17.86:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/kube-proxy.service 172.17.17.87:/usr/lib/systemd/system/

删除证书和配置文件 86 87 上操作

rm -rf /data/kubernetes/cfg/kubelet.kubeconfig
rm -f /data/kubernetes/ssl/kubelet*

注:这几个文件是证书申请审批后自动生成的,每个

配置新的Node节点 86 87 上操作

  • 修改kubelet和kube-proxy配置文件

vi /data/kubernetes/cfg/kubelet.conf 
--hostname-override=k8s-node5
vim /data/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: k8s-node5

vi /opt/kubernetes/cfg/kubelet.conf
--hostname-override=k8s-node6
vi /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: k8s-node6
  • 配置kubectl和kube-proxy开机启动

  • systemctl daemon-reload && systemctl start kubelet && systemctl start kube-proxy
    systemctl enable kubelet && systemctl enable kube-proxy

在master节点上准许node加入

  • 获取准入的node信息

[root@k8s-master-1 k8s]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-BKIF1VFOrkRKlbKeJc8CbEe22JLgsMwMU6aJ4KCjeMg 2m42s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending node-csr-MAr2KDH13mbed2dLM4QEmZGuikl1ooXW5ksLlkCVtOA 2m45s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending

  • 准入node

kubectl certificate approve node-csr-BKIF1VFOrkRKlbKeJc8CbEe22JLgsMwMU6aJ4KCjeMg kubectl certificate approve node-csr-MAr2KDH13mbed2dLM4QEmZGuikl1ooXW5ksLlkCVtOA

查看状态

[root@k8s-master-1 k8s]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master-1 Ready <none> 72m v1.18.4 k8s-node5 Ready <none> 60s v1.18.4 k8s-node6 Ready <none> 50s v1.18.4

部署Dashboard和CoreDNS

部署Dashboard

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

默认Dashboard只能集群内部访问,修改Service为NodePort类型,暴露到外部

vim recommended.yaml

kind: Service
apiVersion: v1
metadata:
labels:
  k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
  - port: 443
    targetPort: 8443
    nodePort: 30001
type: NodePort
selector:
  k8s-app: kubernetes-dashboard
# 部署dashboard
kubectl apply -f recommended.yml  
# 查看状态
[root@k8s-master-1 k8s]# kubectl get pods,svc -n kubernetes-dashboard
NAME                                             READY   STATUS             RESTARTS   AGE
pod/dashboard-metrics-scraper-694557449d-82n2p   0/1     ContainerCreating   0         2m2s
pod/kubernetes-dashboard-9774cc786-2fcvk         0/1     ContainerCreating   0         2m2s

NAME                               TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
service/dashboard-metrics-scraper   ClusterIP   10.0.0.107   <none>       8000/TCP       2m2s
service/kubernetes-dashboard       NodePort   10.0.0.64   <none>       443:31614/TCP   2m3s

此时可以直接访问https://NodeIP:30001 NodeIP也就是宿主机的IP

  • 创建dashboard访问token

  • kubectl create serviceaccount dashboard-admin -n kube-system
    kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
    kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')     #获取token

部署coreDNS

用于集群内部Service名称解析

kubectl apply -f coredns.yaml
[root@k8s-master-1 k8s]# kubectl get pods -n kube-system
NAME                     READY   STATUS   RESTARTS   AGE
coredns-5675d5b4d-njxc4   1/1     Running   0         30m
kube-flannel-ds-69w4v     1/1     Running   0         84m
kube-flannel-ds-6twp8     1/1     Running   5         84m
kube-flannel-ds-t6sqv     1/1     Running   0         84m

如果创建不成功报错 failed to set bridge addr: \"cni0\" already has an IP address different from 10.244.3.1/24"
ifconfig cni0 down    
ip link delete cni0
网卡删除会自动重建

DNS解析测试:

[root@k8s-master-1 k8s]# kubectl run -it --rm dns-test --image=busybox:1.28.4 sh If you don't see a command prompt, try pressing enter. / # nslookup kubernetes Server: 10.0.0.2 Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name: kubernetes Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local

高可用架构(扩容多Master架构)

Kubernetes作为容器集群系统,通过健康检查+重启策略实现了Pod故障自我修复能力,通过调度算法实现将Pod分布式部署,并保持预期副本数,根据Node失效状态自动在其他Node拉起Pod,实现了应用层的高可用性。

针对Kubernetes集群,高可用性还应包含以下两个层面的考虑:Etcd数据库的高可用性和KubernetesMaster组件的高可用性。而Etcd我们已经采用3个节点组建集群实现高可用,本节将对Master节点高可用进行说明和实施。

Master节点扮演着总控中心的角色,通过不断与工作节点上的Kubelet和kube-proxy进行通信来维护整个集群的健康工作状态。如果Master节点故障,将无法使用kubectl工具或者API做任何集群管理。

Master节点主要有三个服务kube-apiserver、kube-controller-manager和kube-scheduler,其中kube-controller-manager和kube-scheduler组件自身通过选择机制已经实现了高可用,所以Master高可用主要针对kube-apiserver组件,而该组件是以HTTP API提供服务,因此对他高可用与Web服务器类似,增加负载均衡器对其负载均衡即可,并且可水平扩容。

多Master架构图:

kubernetes(七) 二进制部署k8s(1.18.4版本)

扩容流程

新增主机:centos7-node, 角色k8s-master2

  • 系统初始化

  • 安装docker

  • wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz
    tar xf docker-19.03.9.tgz
    mv docker/* /usr/bin/
    mkdir /data/docker
    mkdir /etc/docker

创建ETCD证书目录(mkdir /data/etcd/ssl -p)

拷贝文件,从master-1拷贝到新机器


cp -rp /opt/cni/ 172.17.17.84:/opt/
scp -r /data/kubernetes root@172.17.17.84:/data
scp -r /data/etcd/ssl root@172.17.17.84:/data/etcd
scp /usr/lib/systemd/system/kube* root@172.17.17.84:/usr/lib/systemd/system
scp /usr/bin/kubectl root@172.17.17.84:/usr/bin

删除证书文件

删除kubelet证书和kubeconfig文件

rm -f /data/kubernetes/cfg/kubelet.kubeconfig

rm-f /data/kubernetes/ssl/kubelet*

修改配置文件IP和主机名

$ vim /data/kubernetes/cfg/kube-apiserver.conf
--bind-address=172.17.17.84 \
--advertise-address=172.17.17.84 \
$ vim /data/kubernetes/cfg/kubelet.conf
--hostname-override=k8s-master-2
$ vim /data/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: k8s-master-2

启动服务

systemctl daemon-reload && systemctl start docker && systemctl start kube-apiserver && systemctl start kube-controller-manager && systemctl start kube-scheduler && systemctl start kubelet && systemctl start kube-proxy

systemctl enable kube-apiserver && systemctl enable docker && systemctl enable kube-controller-manager && systemctl enable kube-scheduler && systemctl enable kubelet && systemctl enable kube-proxy

查看集群状态

[root@k8s-master-1 data]# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-1 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"} etcd-0 Healthy {"health":"true"}

k8s-mater1 准入新master节点

[root@k8s-master-1 data]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-KUYz4cMykgJVDNgnXt0okzZsJ22YjAuJ8qWnvJrnyXo 19m kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued [root@k8s-master-1 data]# kubectl certificate approve node-csr-KUYz4cMykgJVDNgnXt0okzZsJ22YjAuJ8qWnvJrnyXo^C [root@k8s-master-1 data]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master-1 Ready <none> 3d4h v1.18.4 k8s-master-2 Ready <none> 9m14s v1.18.4 k8s-node5 Ready <none> 2d23h v1.18.4 k8s-node6 Ready <none> 2d23h v1.18.4

部署ngixn负载均衡

image-20200924152210543

Nginx是一个主流Web服务和反向代理服务器,这里用四层实现对apiserver实现负载均衡。Keepalived是一个主流高可用软件,基于VIP绑定实现服务器双机热备,在上述拓扑中,Keepalived主要根据Nginx运行状态判断是否需要故障转移(偏移VIP),例如当Nginx主节点挂掉,VIP会自动绑定在Nginx备节点,从而保证VIP一直可用,实现Nginx高可用。

安装软件包(主/备)

172.17.17.81 和172.17.17.84

yum install epel-release -y

yum install nginx keepalived -y

Nginx配置文件(主/备一样)

# nginx配置
$ vim /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;

events {
  worker_connections 1024;
}
# 四层负载均衡,为两台Master apiserver组件提供负载均衡
stream {
  log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
  access_log /var/log/nginx/k8s-access.log main;
  upstream k8s-apiserver {
      server 192.168.56.14:6443;
      server 192.168.56.74:6443;
  }

  server {
      listen 6443;
      proxy_pass k8s-apiserver;
  }
}

http {
  log_format main '$remote_addr - $remote_user [$time_local] "$request" '
      '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';
  access_log /var/log/nginx/access.log main;
  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 65;
  types_hash_max_size 2048;

  include /etc/nginx/mime.types;
  default_type application/octet-stream;

  server {
      listen 80;
      server_name _;
      location / {
      }
  }
}  

keepalived配置文件(Nginx Master)

global_defs {
  notification_email {
    acassen@firewall.loc
    failover@firewall.loc
    sysadmin@firewall.loc
 
  }
  notification_email_from Alexandre.Cassen@firewall.loc
  smtp_server 127.0.0.1
  smtp_connect_timeout 30
  router_id NGINX_MASTER
}

vrrp_script check_nginx {
  script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
  state MASTER
  interface enp9s0
  virtual_router_id 51
  priority 100
  advert_int 1
  authentication {
      auth_type PASS
      auth_pass 1111
  }
  virtual_ipaddress {
      172.17.17.79/24
  }
  track_script {
      check_nginx
  }
}

vrrp_script:指定检查nginx工作状态脚本(根据nginx状态判断是否故障转移)

virtual_ipaddress:虚拟IP(VIP)

检查nginx状态脚本

vim /etc/keepalived/check_nginx.sh

#!/bin/bash count=$(ps -ef |grep nginx |egrep -cv "grep|$$") if [ "$count" -eq 0 ];then exit 1 else exit 0 fi

chmod+x /etc/keepalived/check_nginx.sh

keepalived配置文件(Nginx Backup)

vim /etc/keepalived/keepalived.conf

global_defs {
  notification_email {
    acassen@firewall.loc
    failover@firewall.loc
    sysadmin@firewall.loc

  }
  notification_email_from Alexandre.Cassen@firewall.loc
  smtp_server 127.0.0.1
  smtp_connect_timeout 30
  router_id NGINX_BACKUP
}

vrrp_script check_nginx {
  script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
  state BACKUP
  interface enp9s0
  virtual_router_id 51
  priority 90
  advert_int 1
  authentication {
      auth_type PASS
      auth_pass 1111
  }
  virtual_ipaddress {
      172.17.17.79/24
  }
  track_script {
      check_nginx
  }
}

启动并设置开机启动

systemctl daemon-reload
systemctl start nginx
systemctl start keepalived
systemctl enable nginx
systemctl enable keepalived

查看keepalived工作状态

ip  add 
enp9s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
  link/ether 10:c3:7b:49:25:2f brd ff:ff:ff:ff:ff:ff
  inet 172.17.17.83/25 brd 172.17.17.127 scope global enp9s0
      valid_lft forever preferred_lft forever
  inet 172.17.17.79/24 scope global enp9s0
      valid_lft forever preferred_lft forever
  inet6 fe80::12c3:7bff:fe49:252f/64 scope link
      valid_lft forever preferred_lft forever

可以看到,在enp9s0网卡绑定了192.168.31.88 虚拟IP,说明工作正常。

Nginx+Keepalived高可用测试关闭主节点Nginx,

测试VIP是否漂移到备节点服务器。

在Nginx Master执行 pkill nginx在Nginx Backup,

ip addr命令查看已成功绑定VIP。

访问负载均衡器测试

修改k8s-node5 和k8s-node6配置

cd /data/kubernetes/cfg/
[root@k8s-node6 cfg]# sed -i 's/172.17.17.85/172.17.17.79/g' *
[root@k8s-node6 cfg]# systemctl restart kubelet
[root@k8s-node6 cfg]# systemctl restart kube-proxy



标签:kube,etcd,kubernetes,--,1.18,版本,k8s,data
来源: https://www.cnblogs.com/zhaobin-diray/p/13724988.html

本站声明: 1. iCode9 技术分享网(下文简称本站)提供的所有内容,仅供技术学习、探讨和分享;
2. 关于本站的所有留言、评论、转载及引用,纯属内容发起人的个人观点,与本站观点和立场无关;
3. 关于本站的所有言论和文字,纯属内容发起人的个人观点,与本站观点和立场无关;
4. 本站文章均是网友提供,不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属;如您发现该文章侵犯了您的权益,可联系我们第一时间进行删除;
5. 本站为非盈利性的个人网站,所有内容不会用来进行牟利,也不会利用任何形式的广告来间接获益,纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。

专注分享技术,共同学习,共同进步。侵权联系[81616952@qq.com]

Copyright (C)ICode9.com, All Rights Reserved.

ICode9版权所有