虚拟机上部署k8s集群

2021-07-08 11:30:32 阅读：321 来源： 互联网

标签：kube name 虚拟机 sudo etc 集群 master k8s flannel

部署k8s集群

一、环境准备：

Ubuntuserver版本

Ubuntu 18.10 (GNU/Linux 4.18.0-10-generic x86_64)

下载链接：

暂无

docker版本

Docker version 18.06.1-ce, build e68fc7a

下载链接：

暂无

k8s版本

v1.13.1

下载链接：

暂无

辅助工具Xshell、vmware

Xshell 7
VMware® Workstation 16 Pro  16.1.1 build-17801498

下载链接：

暂无

二、安装Ubuntu 18.10 server

安装过程，可以一直Done,其中关键的步骤：

设置源：（错过也没关系，可以从系统内更改）

安装过程其中一步配置： http://mirrors.aliyun.com/ubuntu

三、安装docker

1.基础准备

Docker 要求 Ubuntu 系统的内核版本高于 3.10 ，查看本页面的前提条件来验证你的 Ubuntu 版本是否支持 Docker。

uname -r 
4.18.0-21-generic(主版本必须保持一致)

安装curl

sudo apt-get update && apt-get install -y curl telnet wget man \
apt-transport-https \
ca-certificates \
software-properties-common vim

查看新版本号
- Ubuntu 18.10
```
$ lsb_release -c
Codename:	cosmic
```

查看确认国内源

	$ cp /etc/apt/sources.list /etc/apt/sources.list.bak
	修改 sudo vim /etc/apt/sources.list
	输入 deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse

	$ cat /etc/apt/sources.list

2.在线安装Docker-ce(不推荐，未实践)

(建议下面的手动安装方式，因为在线可能会出现版本不一致)

注意： 该国内源目前提供 18.09版本，与k8s不符。k8s推荐安装Docker ce 18.06

安装GPG秘钥和添加国内镜像

$ curl -fsSL https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/gpg | sudo apt-key add -

添加国内源头

$ add-apt-repository \
    "deb https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu \
    $(lsb_release -cs) \
    stable"

更新国内源路径

sudo apt update

3.安装查看版本指令

sudo apt-get install -y apt-show-versions

4.查看docker-ce版本号

sudo apt-show-versions -a docker-ce

在线安装Docker-ce

sudo apt-get update && apt-get install -y docker-ce

注意到当前安装的版本是 `docker-ce_5%3a18.09.6~3-0~ubuntu-cosmic_amd64.deb`

3. 手动安装Docker(离线安装)【推荐，我一次过】

下载docker-ce_18.06.1\~ce\~3-0\~ubuntu_amd64.deb
上传到上述文件到待安装服务器master
登录待安装服务器，切换到root账户
dpkg -i docker-ce_18.06.1\~ce\~3-0\~ubuntu_amd64.deb

如果提示错误

dpkg: error: dpkg frontend is locked by another process

说明已经有其他进程在使用dpkg安装程序

sudo rm /var/lib/dpkg/lock

即可。

如果提示错误

master@master:~/package$ sudo dpkg -i docker-ce_18.06.1~ce~3-0~ubuntu_amd64.deb
 
[sudo] password for master: 
Selecting previously unselected package docker-ce.
(Reading database ... 100647 files and directories currently installed.)
Preparing to unpack docker-ce_18.06.1~ce~3-0~ubuntu_amd64.deb ...
Unpacking docker-ce (18.06.1~ce~3-0~ubuntu) ...
dpkg: dependency problems prevent configuration of docker-ce:
 docker-ce depends on libltdl7 (>= 2.4.6); however:
  Package libltdl7 is not installed.

dpkg: error processing package docker-ce (--install):
 dependency problems - leaving unconfigured
Processing triggers for man-db (2.8.4-2) ...
Processing triggers for systemd (239-7ubuntu10) ...
Errors were encountered while processing:
 docker-ce

表示当前docker-ce 依赖系统libltd17库，安装就可以了

$ apt-get install -y libltdl7

docker version

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

确保版本号是 18.06

4.启动Docker

启动docker

sudo systemctl enable docker 
sudo systemctl start docker

登录确认docker已经运行

master@ubuntu:~$ sudo docker ps 
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

四、安装k8s

1、 k8s安装环境准备

创建配置文件

sudo touch /etc/apt/sources.list.d/kubernetes.list

添加写权限

sudo chmod 666 /etc/apt/sources.list.d/kubernetes.list

再添加，内容如下:

sudo vim /etc/apt/sources.list.d/kubernetes.list

deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main

执行sudo apt update 更新操作系统源，开始会遇见如下错误

tcast@master:~$ sudo apt update
Get:1 http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial InRelease [8,993 B]
Err:1 http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial InRelease
The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY 6A030B21BA07F4FB
Hit:2 http://mirrors.aliyun.com/ubuntu cosmic InRelease
Hit:3 http://mirrors.aliyun.com/ubuntu cosmic-updates InRelease
Hit:4 http://mirrors.aliyun.com/ubuntu cosmic-backports InRelease
Hit:5 http://mirrors.aliyun.com/ubuntu cosmic-security InRelease
Err:6 https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu cosmic InRelease
Could not wait for server fd - select (11: Resource temporarily unavailable) [IP: 202.141.176.110 443]
Reading package lists… Done
W: GPG error: http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial InRelease: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY 6A030B21BA07F4FB
E: The repository ‘http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial InRelease’ is not signed.
N: Updating from such a repository can’t be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.




其中：

```bash
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6A030B21BA07F4FB

签名认证失败，需要重新生成。记住上面的NO_PUBKEY 6A030B21BA07F4FB

添加认证key

运行如下命令，添加错误中对应的key(错误中NO_PUBKEY后面的key的后8位)
```
gpg --keyserver keyserver.ubuntu.com --recv-keys BA07F4FB
```
接着运行如下命令，确认看到OK，说明成功，之后进行安装:

gpg --export --armor BA07F4FB | sudo apt-key add -


5. 再次重新`sudo apt update`更新系统下载源数据列表

```bash
master@master:~$ sudo apt update
Hit:1 https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu cosmic InRelease                  
Hit:2 http://mirrors.aliyun.com/ubuntu cosmic InRelease                                    
Hit:3 http://mirrors.aliyun.com/ubuntu cosmic-updates InRelease                            
Hit:4 http://mirrors.aliyun.com/ubuntu cosmic-backports InRelease                          
Hit:5 http://mirrors.aliyun.com/ubuntu cosmic-security InRelease                           
Get:6 http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial InRelease [8,993 B]      
Ign:7 http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial/main amd64 Packages
Get:7 http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial/main amd64 Packages [26.6 kB]
Fetched 26.6 kB in 42s (635 B/s)    
Reading package lists... Done
Building dependency tree       
Reading state information... Done
165 packages can be upgraded. Run 'apt list --upgradable' to see them.

以上没有报和错误异常，表示成功。

2、禁止基础设施

禁止防火墙

$ sudo ufw disable
Firewall stopped and disabled on system startup

关闭swap

# 成功
$ sudo swapoff -a 
# 永久关闭swap分区
$ sudo sed -i 's/.*swap.*/#&/' /etc/fstab

禁止selinux

# 安装操控selinux的命令
$ sudo apt install -y selinux-utils
# 禁止selinux
$ setenforce 0
# 重启操作系统
$ shutdown -r now
# 查看selinux是否已经关闭
$ sudo getenforce
Disabled(表示已经关闭)

3、 k8s系统网络配置

(1) 配置内核参数，将桥接的IPv4流量传递到iptables的链

创建/etc/sysctl.d/k8s.conf文件

sudo touch /etc/sysctl.d/k8s.conf

添加内容如下:

sudo vim /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness = 0

(2) 执行命令使修改生效

# 【候选】建议执行下面的命令
$ sudo modprobe br_netfilter
$ sudo sysctl -p /etc/sysctl.d/k8s.conf

5、安装k8s

注意: 切换到root用户 $ su

安装Kubernetes 目前安装版本 v1.13.1

$ apt update && apt-get install -y kubelet=1.13.1-00 kubernetes-cni=0.6.0-00 kubeadm=1.13.1-00 kubectl=1.13.1-00

设置为开机重启

$ sudo systemctl enable kubelet && systemctl start kubelet
$ sudo shutdown -r now

6、验证k8s

使用root用户登录Master主机
执行如下个命令
```
kubectl get nodes 
```

输出如下：

$ kubectl get nodes
The connection to the server localhost:8080 was refused - did you specify the right host or port?

查看当前k8s版本（说明安装完成）
```
$ kubectl version
```

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

五、部署集群

1、集群环境准备

在VMWare中创建2份完整克隆（对已经执行过一、二、三、四部分内容的主机master克隆），分别命名为UbuntuNode1和UbuntuNode2
分别对两个完整克隆的虚拟机进行如下操作，修改主机名称和静态IP

1. 使用root用户登录
2. 打开配置文件`vim /etc/cloud/cloud.cfg`
3. 修改配置`preserve_hostname: true`

修改/etc/hostname，只有一行 node1或node2

2、 master和node基础配置

给node配置hostname

node1主机

sudo vim /etc/hostname

node1

node2主机

sudo vim /etc/hostname

node2

2.确认配置的三台机器的主机名称

$ cat /etc/hosts

配置IP地址

master

sudo vim /etc/netplan/50-cloud-init.yaml

如果需要虚拟机联网，把dhcp4设置为true

network:
    ethernets:
        ens33:
            addresses: [192.168.236.177/24]
            dhcp4: false
            gateway4: 192.168.236.2
            nameservers:
                       addresses: [192.168.236.2]
            optional: true
    version: 2

重启ip配置

netplan apply

坑：配置dhcp4: true后ping不通百度

sudo vim /etc/netplan/50-cloud-init.yaml

gateway4: 255.255.255.0

netplan apply

node1

sudo vim /etc/netplan/50-cloud-init.yaml

network:
    ethernets:
        ens33:
            addresses: [192.168.236.178/24]
            dhcp4: false
            gateway4: 192.168.236.2
            nameservers:
                       addresses: [192.168.236.2]
            optional: true
    version: 2

重启ip配置

netplan apply

坑：配置dhcp4: true后ping不通百度

sudo vim /etc/netplan/50-cloud-init.yaml

gateway4: 255.255.255.0

netplan apply

node2

sudo vim /etc/netplan/50-cloud-init.yaml

network:
    ethernets:
        ens33:
            addresses: [192.168.236.179/24]
            dhcp4: false
            gateway4: 192.168.236.2
            nameservers:
                       addresses: [192.168.236.2]
            optional: true
    version: 2

重启ip配置

netplan apply

坑：配置dhcp4: true后ping不通百度

sudo vim /etc/netplan/50-cloud-init.yaml

gateway4: 255.255.255.0

netplan apply

修改hosts文件

注意： (Master、Node1、Node2都需要配置成下面的内容)

使用root用户登录

打开hosts文件 vim /etc/hosts
```
sudo vim /etc/hosts
```

输入追加如下内容

192.168.236.177 master
192.168.236.178 node1
192.168.236.179 node2

重启机器shutdown -r now(可不重启)

3、配置Master节点

创建工作目录


$ mkdir /home/master/working
$ cd /home/master/working/

创建kubeadm.conf配置文件

创建k8s的管理工具kubeadm对应的配置文件，候选操作在home/master/working/目录下

使用kubeadm配置文件，通过在配置文件中指定docker仓库地址，便于内网快速部署。

生成配置文件

kubeadm config print init-defaults ClusterConfiguration > kubeadm.conf

修改kubeadm.conf中的如下两项:

imageRepository
kubernetesVersion
```
sudo vim/home/master/working/
```

vim kubeadm.conf
# 修改 imageRepository: k8s.gcr.io
# 改为 registry.cn-beijing.aliyuncs.com/imcto
imageRepository: registry.cn-beijing.aliyuncs.com/imcto
# 修改kubernetes版本kubernetesVersion: v1.13.0
# 改为kubernetesVersion: v1.13.1
kubernetesVersion: v1.13.1

修改kubeadm.conf中的API服务器地址，后面会频繁使用这个地址。

localAPIEndpoint:

localAPIEndpoint:
  advertiseAddress: 192.168.236.177
  bindPort: 6443

注意: 192.168.236.177是master主机的ip地址

配置子网网络

networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

这里的10.244.0.0/16 和 10.96.0.0/12分别是k8s内部pods和services的子网网络，最好使用这个地址，后续flannel网络需要用到。

拉取K8s必备的模块镜像

查看一下都需要哪些镜像文件需要拉取

$ kubeadm config images list --config kubeadm.conf
registry.cn-beijing.aliyuncs.com/imcto/kube-apiserver:v1.13.1
registry.cn-beijing.aliyuncs.com/imcto/kube-controller-manager:v1.13.1
registry.cn-beijing.aliyuncs.com/imcto/kube-scheduler:v1.13.1
registry.cn-beijing.aliyuncs.com/imcto/kube-proxy:v1.13.1
registry.cn-beijing.aliyuncs.com/imcto/pause:3.1
registry.cn-beijing.aliyuncs.com/imcto/etcd:3.2.24
registry.cn-beijing.aliyuncs.com/imcto/coredns:1.2.6

拉取镜像

#下载全部当前版本的k8s所关联的镜像
kubeadm config images pull --config ./kubeadm.conf

初始化kubernetes环境

#初始化并且启动
$ sudo kubeadm init --config ./kubeadm.conf

kubeadm join 192.168.63.2:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:0c5432b503a32ef2f08efc2e5daaa3ab3ff113adede39be4e76eea8fdb66ba4a
把此段保存下来

更多kubeadm配置文件参数详见（不需要看）

kubeadm config print-defaults

k8s启动成功输出内容较多，但是记住末尾的内容

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.236.177:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:e778d3665e52f5a680a87b00c6d54df726c2eda601c0db3bfa4bb198af2262a8

按照官方提示，执行以下操作。

执行如下命令

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

创建系统服务并启动

# 启动kubelet 设置为开机自启动
$ sudo systemctl enable kubelet
# 启动k8s服务程序
$ sudo systemctl start kubelet

验证kubernetes启动结果

验证输入，注意显示master状态是NotReady，证明初始化服务器成功（切换成root用户）

$ kubectl get nodes
NAME     STATUS     ROLES    AGE   VERSION
master   NotReady   master   12m   v1.13.1

查看当前k8s集群状态

$ kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health": "true"}

目前只有一个master，还没有node，而且是NotReady状态，那么我们需要将node加入到master管理的集群中来。在加入之前，我们需要先配置k8s集群的内部通信网络，这里采用的是flannel网络。

部署集群内部通信flannel网络，此处有坑：没办法翻墙下载kube-flannel.yml。

$cd $HOME/working
$wget https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

编辑这个文件，确保flannel网络是对的,找到net-conf.json标记的内容是否正确。

 net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }

使用手动方式配置：

1.修改/etc/hosts

wget kube-flannel.yml的时候显示连接失败
是因为网站被墙了，建议在/etc/hosts文件添加一条
199.232.68.133 raw.githubusercontent.com

2.下载，但是下载的是有问题的文件，需要修改

sudo wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

3.修改，保存


---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
    - configMap
    - secret
    - emptyDir
    - hostPath
  allowedHostPaths:
    - pathPrefix: "/etc/cni/net.d"
    - pathPrefix: "/etc/kube-flannel"
    - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups: ['extensions']
    resources: ['podsecuritypolicies']
    verbs: ['use']
    resourceNames: ['psp.flannel.unprivileged']
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - amd64
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.11.0-amd64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.11.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-arm64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - arm64
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.11.0-arm64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.11.0-arm64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
             add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-arm
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - arm
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.11.0-arm
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.11.0-arm
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
             add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-ppc64le
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - ppc64le
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.11.0-ppc64le
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.11.0-ppc64le
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
             add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-s390x
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - s390x
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.11.0-s390x
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.11.0-s390x
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
             add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
EOF

这个"10.244.0.0/16"和 ./kubeadm.conf中的podsubnet的地址要一致。**

应用当前flannel配置文件

master@master:~/working$ kubectl apply -f kube-flannel.yml

输出结果如下

root@master:~/working# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

安装flannel网络前执行kubectl get nodes输出结果如下

master@master:~/working$ kubectl get node
NAME     STATUS     ROLES    AGE   VERSION
master   NotReady   master   10m   v1.13.1

安装flannel网络后执行kubectl get nodes输出结果如下

master@master:~/working$ kubectl get node
NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   10m   v1.13.1

此时master已经是Ready状态了，表示已经配置成功了，那么我们就需要配置node来加入这个集群。

3、配置node1和node2

配置Node

1 确认外部环境

确认关闭swap

apt install -y selinux-utils
sudo swapoff -a

禁止selinux
```
sudo setenforce 0
```
确认关闭防火墙
```
sudo ufw disable
```

2 配置k8s集群的Node主机环境

启动k8s后台服务

# 启动kubelet 设置为开机自启动
$ sudo systemctl enable kubelet
# 启动k8s服务程序
$ sudo systemctl start kubelet

将master机器的/etc/kubernetes/admin.conf传到到node1和node2

#将admin.conf传递给node1
sudo scp /etc/kubernetes/admin.conf master@192.168.236.178:/home/master/

#将admin.conf传递给node2
sudo scp /etc/kubernetes/admin.conf master@192.168.236.179:/home/master/

$ mkdir -p $HOME/.kube
$ sudo cp -i $HOME/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

$ mkdir -p $HOME/.kube
$ sudo cp -i $HOME/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

node1和node2分别连接master加入master集群。这里用的是kubeadm join指令

//获取join
kubeadm token create --print-join-command
//
kubeadm join 192.168.6.1:6443 --token 9j3u9g.cecwftu8ywal0sjl --discovery-token-ca-cert-hash sha256:a2600aa5707de58b49a6a6e41e52ab1aa50a5f48a138783a8f2ef6e8d8c38315

应用两个node主机分别应用flannel网络

将master中的kube-flannel.yml分别传递给两个node节点.

#将kube-flannel.yml传递给node1
sudo scp $HOME/working/kube-flannel.yml master@192.168.236.178:/home/master/

#将kube-flannel.yml传递给node2
sudo scp $HOME/working/kube-flannel.yml master@192.168.236.179:/home/master/

分别启动flannel网络

master@node1:~$ kubectl apply -f kube-flannel.yml

master@node2:~$ kubectl apply -f kube-flannel.yml

查看node是否已经加入到k8s集群中(需要等一段时间才能ready)

master@node2:~$ kubectl get nodes
NAME     STATUS   ROLES    AGE     VERSION
master   Ready    master   35m     v1.13.1
node1    Ready    <none>   2m23s   v1.13.1
node2    Ready    <none>   40s     v1.13.1

六、可能的坑

1.开启ssh，连接Xshell

安装openssh-server

sudo apt-get install openssh-server

修改配置文件

cd /etc/ssh
sudo vim sshd_config
	PermitRootLogin yes
	StrictModes yes

启动服务

初始化：sudo /etc/init.d/ssh restart（安装后第一次启动需要）

启动服务：sudo service ssh start

检查是否启动

ps -aux | grep ssh

出现sshd服务即表示启动成功

2.虚拟机动态IP与克隆主机动态IP一致

先停止虚拟机，在虚拟机上右击，设置，网络适配器，高级，mac地址生成，重启虚拟机，如果还一致，再操作一次

3.启用root账号

sudo passwd root
输入两次密码即可

命令su root 切换root账户

4.开启集群的准备操作

sudo swapoff -a
sudo setenforce 0
sudo ufw disable

5.虚拟机访问不了外网

未完

标签：kube,name,虚拟机,sudo,etc,集群,master,k8s,flannel
来源： https://blog.csdn.net/weixin_43434729/article/details/118569662

本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享；
2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关；
3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关；
4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除；
5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。

ICode9

虚拟机上部署k8s集群

部署k8s集群

一、环境准备：

二、安装Ubuntu 18.10 server

三、安装docker

1.基础准备

2.在线安装Docker-ce(不推荐，未实践)

3. 手动安装Docker(离线安装)【推荐，我一次过】

4.启动Docker

四、安装k8s

1、 k8s安装环境准备

2、 禁止基础设施

3、 k8s系统网络配置

5、安装k8s

6、验证k8s

五、部署集群

1、集群环境准备

2、 master和node基础配置

3、配置Master节点

3、配置node1和node2

六、可能的坑

1.开启ssh，连接Xshell

2.虚拟机动态IP与克隆主机动态IP一致

3.启用root账号

4.开启集群的准备操作

5.虚拟机访问不了外网

2、禁止基础设施