二进制安装多master节点

root

233
文章

0
评论

2021年8月2日23:18:39 评论 43900字阅读146分20秒

二进制安装多master节点

集群规划

k8s环境规划：

Pod网段： 10.0.0.0/16

Service网段： 10.255.0.0/16

实验环境规划：

操作系统：centos7.6

配置： 4Gib内存/6vCPU/100G硬盘

注意：也可以用4vCPU

网络：NAT

K8S集群角色	Ip	主机名	安装的组件
控制节点	192.168.1.180	hh-master01	apiserver、controller-manager、scheduler、etcd、docker、keepalived、nginx
控制节点	192.168.1.181	hh-master02	apiserver、controller-manager、scheduler、etcd、docker、keepalived、nginx
控制节点	192.168.1.182	hh-master03	apiserver、controller-manager、scheduler、etcd、docker
工作节点	192.168.1.183	hh-node01	kubelet、kube-proxy、docker、calico、coredns
Vip	192.168.1.199

1.系统初始化

Ansible-playbook Bote

2.搭建etcd集群

2.1配置etcd工作目录

//master节点执行
mkdir -p /etc/etcd
mkdir -p /etc/etcd/ssl

2.2安装签发证书工具cfssl

下载自签证书工具

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

赋予权限

chmod +x cfssl*

重命名

for x in cfssl*; do mv $x ${x%*_linux-amd64};  done

移动文件到目录 (/usr/bin)

mv cfssl* /usr/bin

产生ca请求文件证书

注：

CN：Common Name（公用名称），kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；对于 SSL 证书，一般为网站域名；而对于代码签名证书则为申请单位名称；而对于客户端证书则为证书申请者的姓名。

O：Organization（单位名称），kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；对于 SSL 证书，一般为网站域名；而对于代码签名证书则为申请单位名称；而对于客户端单位证书则为证书申请者所在单位名称

mkdir -p /data/ssl
# 自签名根证书（20年有效期）
cat > /data/ssl/ca-csr.json << EOF
{
    "CN": "kubernetes",
    "hosts": [
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ],
    "ca": {
          "expiry": "175200h"
    }
}
EOF

生成CA

[root@master03 ssl]#  cfssl gencert -initca ca-csr.json|cfssljson -bare ca
2021/07/14 19:26:14 [INFO] generating a new CA key and certificate from CSR
2021/07/14 19:26:14 [INFO] generate received request
2021/07/14 19:26:14 [INFO] received CSR
2021/07/14 19:26:14 [INFO] generating key: rsa-2048
2021/07/14 19:26:14 [INFO] encoded CSR
2021/07/14 19:26:14 [INFO] signed certificate with serial number 283971653681763962763710735735645294034226338199

xxxx

cat > /data/ssl/ca-config.json  << EOF
{
  "signing": {
      "default": {
          "expiry": "175200h"
        },
      "profiles": {
          "kubernetes": {
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "175200h"
          }
      }
  }
}
EOF

生成etcd的证书

#配置etcd证书请求，hosts的ip变成自己etcd所在节点的ip

cat > /data/ssl/etcd-csr.json << EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.1.180",
    "192.168.1.181",
    "192.168.1.182",
    "192.168.1.199"    ##VIP
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "Beijing",   ##要和ca证书的对应
    "L": "Beijing",
    "O": "k8s",
    "OU": "system"
  }]
} 
EOF

#上述文件hosts字段中IP为所有etcd节点的集群内部通信IP，可以预留几个，做扩容用

签发etcd证书

[root@master03 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson  -bare etcd

部署etcd集群

etcd下载地址：https://github.com/etcd-io/etcd/

下载类似于etcd-v3.5.0-linux-amd64.tar.gz的包

[root@master01 data]# cp -p etcd-v3.5.0-linux-amd64/etcd* /usr/local/bin/

#创建配置文件，三台master分别创建

[root@master01 work]# vim etcd.conf 
#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.180:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.180:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.180:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.180:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.180:2380,etcd2=https://192.168.1.181:2380,etcd3=https://192.168.1.182:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#注：

ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

#创建启动服务文件，三台master分别创建

[root@master01 work]# vim etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
 
[Service]
Type=notify
EnvironmentFile=-/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[root@master01 work]# cp ca*.pem /etc/etcd/ssl/

[root@master01 work]# cp etcd*.pem /etc/etcd/ssl/

[root@master01 work]# cp etcd.conf /etc/etcd/

[root@master01 work]# cp etcd.service /usr/lib/systemd/system/

[root@master01 work]# for i in xianchaomaster2 xianchaomaster3;do rsync -vaz etcd.conf $i:/etc/etcd/;done

[root@master01 work]# for i in xianchaomaster2 xianchaomaster3;do rsync -vaz etcd*.pem ca*.pem $i:/etc/etcd/ssl/;done

[root@master01 work]# for i in xianchaomaster2 xianchaomaster3;do rsync -vaz etcd.service $i:/usr/lib/systemd/system/;done

#启动etcd集群

启动etcd的时候，先启动master01的etcd服务，会一直卡住在启动的状态，然后接着再启动master02的etcd，这样master03这个节点etcd才会正常起来

[root@master01  work]# systemctl daemon-reload
[root@master01 work]# systemctl enable etcd.service
[root@master01 work]# systemctl start etcd.service

[root@master02 work]# systemctl daemon-reload
[root@master02 work]# systemctl enable etcd.service
[root@master02 work]# systemctl start etcd.service

[root@master03 work]# systemctl daemon-reload 
[root@master03 work]# systemctl enable etcd.service 
[root@master03 work]# systemctl start etcd.service

#查看etcd集群

[root@master01 ~]# /usr/local/bin/etcdctl --write-out=table --cacert=/data/etcd/ssl/ca.pem --cert=/data/etcd/ssl/etcd.pem --key=/data/etcd/ssl/etcd-key.pem --endpoints=https://192.168.1.180:2379,https://192.168.1.181:2379,https://192.168.1.182:2379  endpoint health
+----------------------------+--------+-------------+-------+
|          ENDPOINT          | HEALTH |    TOOK     | ERROR |
+----------------------------+--------+-------------+-------+
| https://192.168.1.181:2379 |   true | 11.189771ms |       |
| https://192.168.1.182:2379 |   true | 10.805193ms |       |
| https://192.168.1.180:2379 |   true | 38.046545ms |       |
+----------------------------+--------+-------------+-------+

安装kubernetes组件

下载安装包

二进制包所在的github地址如下：

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/

master:

kubectl
kube-apiserver
kube-controller-manager
kube-scheduler

node

kubelet
kube-proxy

#把kubernetes-server-linux-amd64.tar.gz上传到master01上的/data/work目录下:

[root@master01 work]# tar zxvf kubernetes-server-linux-amd64.tar.gz

[root@master01 work]# cd kubernetes/server/bin/

[root@master01 bin]# cp kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/

[root@master01 bin]# rsync -vaz kube-apiserver kube-controller-manager kube-scheduler kubectl master02:/usr/local/bin/

[root@master01 bin]# rsync -vaz kube-apiserver kube-controller-manager kube-scheduler kubectl master03:/usr/local/bin/

[root@master01 bin]# scp kubelet kube-proxy node01:/usr/local/bin/

[root@master01 bin]# cd /data/work/

[root@master01 work]# mkdir -p /etc/kubernetes/

[root@master01 work]# mkdir -p /etc/kubernetes/ssl

[root@master01 work]# mkdir /var/log/kubernetes

部署apiserver组件

#启动TLS Bootstrapping 机制

Master apiserver启用TLS认证后，每个节点的 kubelet 组件都要使用由 apiserver 使用的 CA 签发的有效证书才能与 apiserver 通讯，当Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。

为了简化流程，Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。

Bootstrap 是很多系统中都存在的程序，比如 Linux 的bootstrap，bootstrap 一般都是作为预先配置在开启或者系统启动的时候加载，这可以用来生成一个指定环境。Kubernetes 的 kubelet 在启动时同样可以加载一个这样的配置文件，这个文件的内容类似如下形式：

#TLS bootstrapping 具体引导过程

1.TLS 作用
TLS 的作用就是对通讯加密，防止中间人窃听；同时如果证书不信任的话根本就无法与 apiserver 建立连接，更不用提有没有权限向apiserver请求指定内容。

2. RBAC 作用
当 TLS 解决了通讯问题后，那么权限问题就应由 RBAC 解决(可以使用其他权限模型，如 ABAC)；RBAC 中规定了一个用户或者用户组(subject)具有请求哪些 api 的权限；在配合 TLS 加密的时候，实际上 apiserver 读取客户端证书的 CN 字段作为用户名，读取 O字段作为用户组.

以上说明：第一，想要与 apiserver 通讯就必须采用由 apiserver CA 签发的证书，这样才能形成信任关系，建立 TLS 连接；第二，可以通过证书的 CN、O 字段来提供 RBAC 所需的用户与用户组。

#kubelet 首次启动流程
TLS bootstrapping 功能是让 kubelet 组件去 apiserver 申请证书，然后用于连接 apiserver；那么第一次启动时没有证书如何连接 apiserver ?

在apiserver 配置中指定了一个 token.csv 文件，该文件中是一个预设的用户配置；同时该用户的Token 和由apiserver 的 CA签发的用户被写入了 kubelet 所使用的 bootstrap.kubeconfig 配置文件中；这样在首次请求时，kubelet 使用 bootstrap.kubeconfig 中被 apiserver CA 签发证书时信任的用户来与 apiserver 建立 TLS 通讯，使用 bootstrap.kubeconfig 中的用户 Token 来向 apiserver 声明自己的 RBAC 授权身份.
token.csv格式:

3940fd7fbb391d1b4d861ad17a1f0613,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

首次启动时，可能与遇到 kubelet 报 401 无权访问 apiserver 的错误；这是因为在默认情况下，kubelet 通过 bootstrap.kubeconfig 中的预设用户 Token 声明了自己的身份，然后创建 CSR 请求；但是不要忘记这个用户在我们不处理的情况下他没任何权限的，包括创建 CSR 请求；所以需要创建一个 ClusterRoleBinding，将预设用户 kubelet-bootstrap 与内置的 ClusterRole system:node-bootstrapper 绑定到一起，使其能够发起 CSR 请求。稍后安装kubelet的时候演示。

#创建token.csv文件

[root@master01  work]# cat > token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

#格式：token，用户名，UID，用户组

#创建csr请求文件，替换为自己机器的IP

[root@master01 work]# vim kube-apiserver-csr.json 
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.1.180",
    "192.1681181",
    "192.168.1182",
    "192.168.1.183",
    "192.168.1.199",
    "10.255.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Hubei",
      "L": "Wuhan",
      "O": "k8s",
      "OU": "system"
    }
  ]
}

#注： 如果 hosts 字段不为空则需要指定授权使用该证书的 IP 或域名列表。由于该证书后续被 kubernetes master 集群使用，需要将master节点的IP都填上，同时还需要填写 service 网络的首个IP。(一般是 kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP，如 10.255.0.1)

#生成证书

[root@master01 work]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver

#创建api-server的配置文件，替换成自己的ip

[root@master01 work]# vim kube-apiserver.conf 
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=192.168.1.180 \
  --secure-port=6443 \
  --advertise-address=192.168.1.180 \
  --insecure-port=0 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.255.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-50000 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://192.168.1.180:2379,https://192.168.1.181:2379,https://192.168.1.182:2379 \
  --enable-swagger-ui=true \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"

[root@master01 work]# cp ca*.pem /etc/kubernetes/ssl
[root@master01 work]# cp kube-apiserver*.pem /etc/kubernetes/ssl/
[root@master01 work]# cp token.csv /etc/kubernetes/
[root@master01 work]# cp kube-apiserver.conf /etc/kubernetes/
[root@master01 work]# cp kube-apiserver.service /usr/lib/systemd/system/
[root@master01 work]# rsync -vaz token.csv master02:/etc/kubernetes/
[root@master01 work]# rsync -vaz token.csv master03:/etc/kubernetes/
[root@master01 work]# rsync -vaz kube-apiserver*.pem master02:/etc/kubernetes/ssl/
[root@master01 work]# rsync -vaz kube-apiserver*.pem master03:/etc/kubernetes/ssl/
[root@master01 work]# rsync -vaz ca*.pem master02:/etc/kubernetes/ssl/
[root@master01 work]# rsync -vaz ca*.pem master03:/etc/kubernetes/ssl/
[root@master01 work]# rsync -vaz kube-apiserver.conf master02:/etc/kubernetes/
[root@master01 work]# rsync -vaz kube-apiserver.conf master03:/etc/kubernetes/
[root@master01 work]# rsync -vaz kube-apiserver.service master03:/usr/lib/systemd/system/
[root@master01 work]# rsync -vaz kube-apiserver.service master03:/usr/lib/systemd/system/

注：master02和master03配置文件kube-apiserver.conf的IP地址修改为实际的本机IP

[root@master02 ~]# cat /etc/kubernetes/kube-apiserver.conf 
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=192.168.1181 \
  --secure-port=6443 \
  --advertise-address=192.168.1.181 \
  --insecure-port=0 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.255.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-50000 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://192.168.1.180:2379,https://192.168.1181:2379,https://192.168.1.182:2379 \
  --enable-swagger-ui=true \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"

[root@master03 ~]# cat /etc/kubernetes/kube-apiserver.conf 
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=192.168.1.182 \
  --secure-port=6443 \
  --advertise-address=192.168.1.182 \
  --insecure-port=0 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.255.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-50000 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://192.168.1.180:2379,https://192.168.1.181:2379,https://192.168.1.182:2379 \
  --enable-swagger-ui=true \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"

启动服务

[root@master01 work]# systemctl daemon-reload
[root@master02 work]# systemctl daemon-reload
[root@master03 work]# systemctl daemon-reload

[root@master01 work]# systemctl enable kube-apiserver
[root@master02 work]# systemctl enable kube-apiserver
[root@master03 work]# systemctl enable kube-apiserver

[root@master01 work]# systemctl start kube-apiserver
[root@master02 work]# systemctl start kube-apiserver
[root@master03 work]# systemctl start kube-apiserver

[root@master01 work]#  systemctl status kube-apiserver
   Active: active (running) since Wed 
[root@master02 work]#  systemctl status kube-apiserver
   Active: active (running) since Wed 
[root@master03 work]#  systemctl status kube-apiserver
   Active: active (running) since Wed

[root@master01 work]#  curl --insecure https://192.168.40.180:6443/
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

上面看到401，这个是正常的的状态，还没认证

部署kubectl组件

Kubectl是客户端工具，操作k8s资源的，如增删改查等。

Kubectl操作资源的时候，怎么知道连接到哪个集群，需要一个文件/etc/kubernetes/admin.conf，kubectl会根据这个文件的配置，去访问k8s资源。/etc/kubernetes/admin.con文件记录了访问的k8s集群，和用到的证书。

可以设置一个环境变量KUBECONFIG

[root@ xianchaomaster1 ~]# export KUBECONFIG =/etc/kubernetes/admin.conf

这样在操作kubectl，就会自动加载KUBECONFIG来操作要管理哪个集群的k8s资源了

也可以按照下面方法，这个是在kubeadm初始化k8s的时候会告诉我们要用的一个方法

[root@ xianchaomaster1 ~]# cp /etc/kubernetes/admin.conf /root/.kube/config

这样我们在执行kubectl，就会加载/root/.kube/config文件，去操作k8s资源了

如果设置了KUBECONFIG，那就会先找到KUBECONFIG去操作k8s，如果没有KUBECONFIG变量，那就会使用/root/.kube/config文件决定管理哪个k8s集群的资源

#创建csr请求文件

[root@master01 work]# vim admin-csr.json 
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Hubei",
      "L": "Wuhan",
      "O": "system:masters",             
      "OU": "system"
    }
  ]
}

#说明： 后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权； kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings，如 cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定，该 Role 授予了调用kube-apiserver 的所有 API的权限； O指定该证书的 Group 为 system:masters，kubelet 使用该证书访问 kube-apiserver 时，由于证书被 CA 签名，所以认证通过，同时由于证书用户组为经过预授权的 system:masters，所以被授予访问所有 API 的权限；

注：这个admin 证书，是将来生成管理员用的kube config 配置文件用的，现在我们一般建议使用RBAC 来对kubernetes 进行角色权限控制， kubernetes 将证书中的CN 字段作为User， O 字段作为 Group； "O": "system:masters", 必须是system:masters，否则后面kubectl create clusterrolebinding报错。

#证书O配置为system:masters 在集群内部cluster-admin的clusterrolebinding将system:masters组和cluster-admin clusterrole绑定在一起

#生成证书

[root@master01 work]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
[root@master01 work]# cp admin*.pem /etc/kubernetes/ssl/

配置安全上下文

#创建kubeconfig配置文件，比较重要

kubeconfig 为 kubectl 的配置文件，包含访问 apiserver 的所有信息，如 apiserver 地址、CA 证书和自身使用的证书（这里如果报错找不到kubeconfig路径，请手动复制到相应路径下，没有则忽略）

1.设置集群参数

[root@master01 work]# kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.180:6443 --kubeconfig=kube.config

#查看kube.config内容

vim kube.config

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR0akNDQXA2Z0F3SUJBZ0lVRUVpcFFkbVRUbWpSYWV5MTMzdUhJRFVTVEVzd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RGpBTUJnTlZCQWdUQlVoMVltVnBNUTR3REFZRFZRUUhFd1ZYZFdoaApiakVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WnplWE4wWlcweEV6QVJCZ05WQkFNVENtdDFZbVZ5CmJtVjBaWE13SGhjTk1qRXdOVEV5TVRNeE16QXdXaGNOTXpFd05URXdNVE14TXpBd1dqQmhNUXN3Q1FZRFZRUUcKRXdKRFRqRU9NQXdHQTFVRUNCTUZTSFZpWldreERqQU1CZ05WQkFjVEJWZDFhR0Z1TVF3d0NnWURWUVFLRXdOcgpPSE14RHpBTkJnTlZCQXNUQm5ONWMzUmxiVEVUTUJFR0ExVUVBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxEb0s0THNYV0dLYko0UjBJSnh2T0E3a2QvM0k5M3cKckQxMzE1RXRDd1NIRXNnem5ZLzc0c05wQTJSYzdQc2NMK2ZqZTFuZU9rZ1pPbGwyT04vSTFBMi83QXd0YUt4OAp0UnlIcllNeEZyWlZ6TE9UQWxEaTZYN1RlUk9INUNMc1AxUkdqenc4OXgyVlZSd3dpNm1qc0tRcWt3U1hpbmh5CkQxaElibVU5N1h3ZEtwc1YyUkFIZkxhVUZEMkFBcDJlRW42YzZVVzNCbU5RLzdacmhVeS9FM3J1bHRYSm96NlAKd0ZZM0hGUEhZblUwN3VzRVAvSW83ZFpzc0h5WUluNVRZRjl5NTdKQmcwa09PRnJhQncxV08waWhYU0FkM01qRQoxRUFlWEhId2pXanRXRFFGMWwwWEpWaFVvL3Y2OVRtOFR2S2txdzQvUEdYRG50dmJ5S1hrNmVjQ0F3RUFBYU5tCk1HUXdEZ1lEVlIwUEFRSC9CQVFEQWdFR01CSUdBMVVkRXdFQi93UUlNQVlCQWY4Q0FRSXdIUVlEVlIwT0JCWUUKRkt2L2NkdjFjYURhRS9VNkU1V0tZNFcwMjF1eE1COEdBMVVkSXdRWU1CYUFGS3YvY2R2MWNhRGFFL1U2RTVXSwpZNFcwMjF1eE1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQWp0KzJoTU5YSVdjeWxjK1RWL05JS1FsRHRaSEJUCklRSTZYV3Q5KzFKWUNUbEMxYm5aaHExSnU1ZnB3VEJXMmdjRkRxUVRlbk5lZ0F5T2J2ejJidGNJK2ZDNkptUjgKSFg4dUpPUGJQelM0cEo5WkNsd1E4MHFJVzJYQitXMXh3OW5MSFAxdVJwZXVsSCtkeUNMeS9Zb1kwQ3FnWnc1aApBSktGSE42ckYrTUNWT0R1Tzk4ZThjTWhBcVF6U1hsb2tiVHR3Rnk3OHdnYnJaUCtybGY3eFNZL28wYytKQ1U5ClVsREFhTVJGSytvTVR4VFlicHBKMnRvOGVCemNJM2FrYjFiL2Q0cm9ESGR0U1cvclk0UzFFTTZJSGtDb0xpV1YKQ2IrVVkzb3Fqb0lBOEFHMzhZb1BiVHlqbjVuY24vOU0vVjlkS2E4RFEya011Z3dPall6alJCTFUKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://192.168.40.180:6443
  name: kubernetes
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null

2.设置客户端认证参数

[root@master01 work]# kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=kube.config

3.设置上下文参数

[root@master01 work]# kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=kube.config

4.设置当前上下文

[root@master01 work]# kubectl config use-context kubernetes --kubeconfig=kube.config

[root@master01 work]# mkdir ~/.kube -p
[root@master01 work]# cp kube.config ~/.kube/config

5.授权kubernetes证书访问kubelet api权限

这步不操作，kubectl有些权限用不了

[root@master01 work]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

#查看集群组件状态

[root@master01 ~]# kubectl cluster-info
Kubernetes control plane is running at https://192.168.1.182:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

[root@master01 work]# kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS      MESSAGE                                                                                       ERROR
controller-manager   Unhealthy   Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused   
scheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused   
etcd-0               Healthy     {"health":"true"}                                                                             
etcd-2               Healthy     {"health":"true"}                                                                             
etcd-1               Healthy     {"health":"true"}

[root@master01 work]# kubectl get all --all-namespaces
NAMESPACE   NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.255.0.1   <none>        443/TCP

#同步kubectl文件到其他节点

[root@master01 ~]# mkdir /root/.kube/
[root@master01 ~]#  mkdir /root/.kube/
[root@master01 work]# rsync -vaz /root/.kube/config xianchaomaster2:/root/.kube/
[root@master01 work]# rsync -vaz /root/.kube/config xianchaomaster3:/root/.kube/

#配置kubectl子命令补全

[root@master01 work]# yum install -y bash-completion
[root@master01 work]# source /usr/share/bash-completion/bash_completion
[root@master01 work]# source <(kubectl completion bash)
[root@master01 work]# kubectl completion bash > ~/.kube/completion.bash.inc
[root@master01 work]# source '/root/.kube/completion.bash.inc'
[root@master01 work]# source $HOME/.bash_profile

Kubectl官方备忘单：

https://kubernetes.io/zh/docs/reference/kubectl/cheatsheet/

部署kube-controller-manager组件

#创建csr请求文件

[root@master01 work]# vim kube-controller-manager-csr.json 
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.1.180",
      "192.168.1.181",
      "192.168.1.182",
      "192.168.1.199"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "Hubei",
        "L": "Wuhan",
        "O": "system:kube-controller-manager",
        "OU": "system"
      }
    ]
}

注： hosts 列表包含所有 kube-controller-manager 节点 IP； CN 为 system:kube-controller-manager、O 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限

#生成证书

[root@master01  work]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

#创建kube-controller-manager的kubeconfig

1.设置集群参数

[root@master01 work]# kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.180:6443 --kubeconfig=kube-controller-manager.kubeconfig

2.设置客户端认证参数

[root@master01 work]# kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig

3.设置上下文参数

[root@master01 work]# kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

4.设置当前上下文

[root@master01 work]# kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

#创建配置文件kube-controller-manager.conf

[root@master01 work]# vim kube-controller-manager.conf 
KUBE_CONTROLLER_MANAGER_OPTS="--port=0 \
  --secure-port=10252 \
  --bind-address=127.0.0.1 \
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --service-cluster-ip-range=10.255.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --allocate-node-cidrs=true \
  --cluster-cidr=10.0.0.0/16 \
  --experimental-cluster-signing-duration=87600h \
  --root-ca-file=/etc/kubernetes/ssl/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --horizontal-pod-autoscaler-use-rest-clients=true \
  --horizontal-pod-autoscaler-sync-period=10s \
  --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \
  --use-service-account-credentials=true \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2"

#创建启动文件

[root@master01 work]# vim kube-controller-manager.service 
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

#启动服务

[root@master01 work]# cp kube-controller-manager*.pem /etc/kubernetes/ssl/
[root@master01 work]# cp kube-controller-manager.kubeconfig /etc/kubernetes/
[root@master01 work]# cp kube-controller-manager.conf /etc/kubernetes/
[root@master01 work]# cp kube-controller-manager.service /usr/lib/systemd/system/
[root@master01 work]# rsync -vaz kube-controller-manager*.pem xianchaomaster2:/etc/kubernetes/ssl/
[root@master01 work]# rsync -vaz kube-controller-manager*.pem xianchaomaster3:/etc/kubernetes/ssl/
[root@master01 work]# rsync -vaz kube-controller-manager.kubeconfig kube-controller-manager.conf xianchaomaster2:/etc/kubernetes/
[root@master01 work]# rsync -vaz kube-controller-manager.kubeconfig kube-controller-manager.conf xianchaomaster3:/etc/kubernetes/
[root@master01 work]# rsync -vaz kube-controller-manager.service xianchaomaster2:/usr/lib/systemd/system/
[root@master01 work]# rsync -vaz kube-controller-manager.service xianchaomaster3:/usr/lib/systemd/system/

[root@master01 work]# systemctl daemon-reload 
[root@master01 work]# systemctl enable kube-controller-manager
[root@master01 work]# systemctl start kube-controller-manager
[root@master01 work]# systemctl status kube-controller-manager

[root@master02]# systemctl daemon-reload 
[root@master02]# systemctl enable kube-controller-manager
[root@master02]# systemctl start kube-controller-manager
[root@master02]#  systemctl status kube-controller-manager

[root@master03]# systemctl daemon-reload 
[root@master03]# systemctl enable kube-controller-manager
[root@master03]# systemctl start kube-controller-manager
[root@master03]# systemctl status kube-controller-manager

部署kube-scheduler组件

#创建csr请求

[root@xianchaomaster1 work]# vim kube-scheduler-csr.json 
{
    "CN": "system:kube-scheduler",
    "hosts": [
      "127.0.0.1",
      "192.168.1.180",
      "192.168.1.181",
      "192.168.1.182",
      "192.168.1.199"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "Hubei",
        "L": "Wuhan",
        "O": "system:kube-scheduler",
        "OU": "system"
      }
    ]
}

注： hosts 列表包含所有 kube-scheduler 节点 IP； CN 为 system:kube-scheduler、O 为 system:kube-scheduler，kubernetes 内置的 ClusterRoleBindings system:kube-scheduler 将赋予 kube-scheduler 工作所需的权限

#生成证书

[root@master01 work]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

#创建kube-scheduler的kubeconfig

1.设置集群参数

[root@master01 work]# kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.180:6443 --kubeconfig=kube-scheduler.kubeconfig

2.设置客户端认证参数

[root@master01  work]# kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig

3.设置上下文参数

[root@master01 work]# kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

4.设置当前上下文

[root@master01 work]# kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

#创建配置文件kube-scheduler.conf

[root@master01 work]# vim kube-scheduler.conf 
KUBE_SCHEDULER_OPTS="--address=127.0.0.1 \
--kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
--leader-elect=true \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2"

#创建服务启动文件

[root@master01 work]# vim kube-scheduler.service 
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
 
[Service]
EnvironmentFile=-/etc/kubernetes/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5
 
[Install]
WantedBy=multi-user.target

#启动服务

[root@master01 work]# cp kube-scheduler*.pem /etc/kubernetes/ssl/
[root@master01 work]# cp kube-scheduler.kubeconfig /etc/kubernetes/
[root@master01 work]# cp kube-scheduler.conf /etc/kubernetes/
[root@master01 work]# cp kube-scheduler.service /usr/lib/systemd/system/
[root@master01 work]# rsync -vaz kube-scheduler*.pem xianchaomaster2:/etc/kubernetes/ssl/
[root@master01 work]# rsync -vaz kube-scheduler*.pem xianchaomaster3:/etc/kubernetes/ssl/
[root@master01 work]# rsync -vaz kube-scheduler.kubeconfig kube-scheduler.conf xianchaomaster2:/etc/kubernetes/
[root@master01 work]# rsync -vaz kube-scheduler.kubeconfig kube-scheduler.conf xianchaomaster3:/etc/kubernetes/
[root@master01 work]# rsync -vaz kube-scheduler.service xianchaomaster2:/usr/lib/systemd/system/
[root@master01 work]# rsync -vaz kube-scheduler.service xianchaomaster3:/usr/lib/systemd/system/

[root@master01 work]# systemctl daemon-reload
[root@master01 work]# systemctl enable kube-scheduler
[root@master01 work]# systemctl start kube-scheduler
[root@master01 work]# systemctl status kube-scheduler
● kube-scheduler.service - Kubernetes Scheduler
   Active: active (running) since Wed

[root@master02]# systemctl daemon-reload
[root@master02]# systemctl enable kube-scheduler
[root@master02]# systemctl start kube-scheduler
[root@master02]# systemctl status kube-scheduler
● kube-scheduler.service - Kubernetes Scheduler
   Active: active (running) since Wed

[root@master03]# systemctl daemon-reload
[root@master03]# systemctl enable kube-scheduler
[root@master03]# systemctl start kube-scheduler
[root@master03]# systemctl status kube-scheduler
● kube-scheduler.service - Kubernetes Scheduler
   Active: active (running) since Wed

导入离线镜像压缩包

#把pause-cordns.tar.gz上传到xianchaonode1节点，手动解压

[root@node1 ~]# docker load -i pause-cordns.tar.gz

部署kubelet组件

kubelet：每个Node节点上的kubelet定期就会调用API Server的REST接口报告自身状态，API Server接收这些信息后，将节点状态信息更新到etcd中。kubelet也通过API Server监听Pod信息，从而对Node机器上的POD进行管理，如创建、删除、更新Pod

以下操作在xianchaomaster1上操作

创建kubelet-bootstrap.kubeconfig

[root@master01work]# cd /data/work/
[root@master01 work]# BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv)
[root@master01 work]# rm -r kubelet-bootstrap.kubeconfig

[root@master01 work]#  kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.40.180:6443 --kubeconfig=kubelet-bootstrap.kubeconfig

[root@master01 work]# kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.kubeconfig

[root@master01 work]# kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig

[root@master01 work]# kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig

[root@master01 work]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

#创建配置文件kubelet.json

"cgroupDriver": "systemd"要和docker的驱动一致。

address替换为自己xianchaonode1的IP地址。

[root@master01 work]# vim kubelet.json 
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/ssl/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "192.168.40.183",
  "port": 10250,
  "readOnlyPort": 10255,
  "cgroupDriver": "systemd",
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "featureGates": {
    "RotateKubeletClientCertificate": true,
    "RotateKubeletServerCertificate": true
  },
  "clusterDomain": "cluster.local.",
  "clusterDNS": ["10.255.0.2"]
}

[root@master01 work]# vim kubelet.service 
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \
  --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
  --cert-dir=/etc/kubernetes/ssl \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --config=/etc/kubernetes/kubelet.json \
  --network-plugin=cni \
  --pod-infra-container-image=k8s.gcr.io/pause:3.2 \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on-failure
RestartSec=5
 
[Install]
WantedBy=multi-user.target

#注：

–hostname-override：显示名称，集群中唯一

–network-plugin：启用CNI

–kubeconfig：空路径，会自动生成，后面用于连接apiserver

–bootstrap-kubeconfig：首次启动向apiserver申请证书

–config：配置参数文件

–cert-dir：kubelet证书生成目录

–pod-infra-container-image：管理Pod网络容器的镜像

#注：kubelete.json配置文件address改为各个节点的ip地址，在各个work节点上启动服务

[root@node1 ~]# mkdir /etc/kubernetes/ssl -p
[root@master01 work]# scp kubelet-bootstrap.kubeconfig kubelet.json xianchaonode1:/etc/kubernetes/
[root@master01 work]# scp  ca.pem xianchaonode1:/etc/kubernetes/ssl/
[root@master01 work]# scp  kubelet.service xianchaonode1:/usr/lib/systemd/system/

#启动kubelet服务

[root@node1 ~]# mkdir /var/lib/kubelet
[root@node1  ~]# mkdir /var/log/kubernetes
[root@node1  ~]#  systemctl daemon-reload
[root@node1  ~]# systemctl enable kubelet
[root@node1  ~]# systemctl start kubelet
[root@node1  ~]#  systemctl status kubelet
   Active: active (running) since

确认kubelet服务启动成功后，接着到xianchaomaster1节点上Approve一下bootstrap请求。

执行如下命令可以看到一个worker节点发送了一个 CSR 请求：

[root@master01 work]# kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-SY6gROGEmH0qVZhMVhJKKWN3UaWkKKQzV8dopoIO9Uc   87s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

[root@master01 work]# kubectl certificate approve node-csr-SY6gROGEmH0qVZhMVhJKKWN3UaWkKKQzV8dopoIO9Uc
[root@master01 work]# kubectl get csr
NAME                                                   AGE     SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-SY6gROGEmH0qVZhMVhJKKWN3UaWkKKQzV8dopoIO9Uc   2m25s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued

[root@master01 work]# kubectl get nodes
NAME    STATUS     ROLES    AGE   VERSION
xianchaonode1   NotReady   <none>   30s   v1.20.7

#注意：STATUS是NotReady表示还没有安装网络插件

部署kube-proxy组件

#创建csr请求

[root@master01 work]# vim kube-proxy-csr.json 
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Hubei",
      "L": "Wuhan",
      "O": "k8s",
      "OU": "system"
    }
   ]
}

生成证书

[root@master01 work]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

#创建kubeconfig文件

[root@master01 work]# kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.40.180:6443 --kubeconfig=kube-proxy.kubeconfig

[root@master01 work]# kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig


 
[root@master01 work]# kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig
[root@master01 work]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

#创建kube-proxy配置文件

[root@master01 work]# vim kube-proxy.yaml 
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.40.183
clientConnection:
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 192.168.40.0/24
healthzBindAddress: 192.168.40.183:10256
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.40.183:10249
mode: "ipvs"

#创建服务启动文件

[root@master01 work]# vim kube-proxy.service 
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
 
[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
  --config=/etc/kubernetes/kube-proxy.yaml \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
 
[Install]
WantedBy=multi-user.target

[root@master01 work]# scp  kube-proxy.kubeconfig kube-proxy.yaml xianchaonode1:/etc/kubernetes/
[root@master01 work]#scp  kube-proxy.service xianchaonode1:/usr/lib/systemd/system/

#启动服务

[root@node1 ~]# mkdir -p /var/lib/kube-proxy
[root@node1 ~]# systemctl daemon-reload
[root@node1 ~]# systemctl enable kube-proxy
[root@node1 ~]# systemctl  start kube-proxy
[root@node1 ~]# systemctl status kube-proxy
   Active: active (running) since Wed

部署calico组件

#解压离线镜像压缩包

#把cni.tar.gz和node.tar.gz上传到xianchaonode1节点，手动解压

[root@node1 ~]# docker load -i cni.tar.gz
[root@node1 ~]# docker load -i node.tar.gz

#把calico.yaml文件上传到xianchaomaster1上的的/data/work目录

[root@master01 work]# kubectl apply -f calico.yaml
[root@master01 ~]# kubectl get pods -n kube-system
NAME                READY   STATUS    RESTARTS   AGE
calico-node-xk7n4   1/1     Running   0          13s

[root@master01 ~]# kubectl get nodes
NAME            STATUS   ROLES    AGE   VERSION
xianchaonode1   Ready    <none>   73m   v1.20.7

部署coredns组件

[root@master01 ~]# kubectl apply -f coredns.yaml
[root@master01 ~]# kubectl get pods -n kube-system
NAME                       READY   STATUS    RESTARTS   AGE
calico-node-xk7n4          1/1     Running   0          6m6s
coredns-7bf4bd64bd-dt8dq   1/1     Running   0          51s
[root@master01 ~]# kubectl get svc -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.255.0.2   <none>        53/UDP,53/TCP,9153/TCP   12m

查看集群状态

[root@master1 ~]# kubectl get nodes
NAME    STATUS   ROLES    AGE   VERSION
xianchaonode1   Ready    <none>   38m   v1.20.7

测试k8s集群部署tomcat服务

#把tomcat.tar.gz和busybox-1-28.tar.gz上传到node1，手动解压

[root@node1 ~]# docker load -i tomcat.tar.gz
[root@node1 ~]# docker load -i busybox-1-28.tar.gz 
[root@master1 ~]# kubectl apply -f tomcat.yaml

[root@master1 ~]# kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
demo-pod   2/2     Running   0          11m
[root@master1 ~]# kubectl apply -f tomcat-service.yaml
[root@master1 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP   10.255.0.1       <none>        443/TCP          158m
tomcat       NodePort    10.255.227.179   <none>        8080:30080/TCP   19m

在浏览器访问xianchaonode1节点的ip:30080即可请求到浏览器

验证cordns是否正常

[root@master1 ~]# kubectl run busybox --image busybox:1.28 --restart=Never --rm -it busybox -- sh
/ # ping www.baidu.com
PING www.baidu.com (39.156.66.18): 56 data bytes
64 bytes from 39.156.66.18: seq=0 ttl=127 time=39.3 ms
#通过上面可以看到能访问网络
/ # nslookup kubernetes.default.svc.cluster.local
Server:		10.255.0.2
Address:	10.255.0.2:53
Name:	kubernetes.default.svc.cluster.local
Address: 10.255.0.1

/ # nslookup tomcat.default.svc.cluster.local
Server:    10.255.0.2
Address 1: 10.255.0.2 kube-dns.kube-system.svc.cluster.local

Name:      tomcat.default.svc.cluster.local
Address 1: 10.255.227.179 tomcat.default.svc.cluster.local

#注意：

busybox要用指定的1.28版本，不能用最新版本，最新版本，nslookup会解析不到dns和ip，报错如下：

/ # nslookup kubernetes.default.svc.cluster.local
Server:		10.255.0.2
Address:	10.255.0.2:53
*** Can't find kubernetes.default.svc.cluster.local: No answer
*** Can't find kubernetes.default.svc.cluster.local: No answer

10.255.0.2 就是我们coreDNS的clusterIP，说明coreDNS配置好了。

解析内部Service的名称，是通过coreDNS去解析的。

安装keepalived+nginx实现k8s apiserver高可用

把epel.repo上传到master1的/etc/yum.repos.d目录下，这样才能安装keepalived和nginx

把epel.repo传到master2、master3、node1上

[root@master1 ~]# scp /etc/yum.repos.d/epel.repo xianchaomaster2:/etc/yum.repos.d/
[root@master1 ~]# scp /etc/yum.repos.d/epel.repo xianchaomaster3:/etc/yum.repos.d/
[root@master1 ~]# scp /etc/yum.repos.d/epel.repo xianchaonode1:/etc/yum.repos.d/

1、安装nginx主备：

在master1和master2上做nginx主备安装

[root@master1 ~]#  yum install nginx keepalived -y
[root@master2 ~]#  yum install nginx keepalived -y

修改nginx配置文件。主备一样

[root@master1 ~]# cat /etc/nginx/nginx.conf
[root@master1 ~]# cat /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

# 四层负载均衡，为两台Master apiserver组件提供负载均衡

stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
       server 192.168.1.180:6443;   # master1 APISERVER IP:PORT
       server 192.168.1.181:6443;   # master2 APISERVER IP:PORT
       server 192.1681.182:6443;   # master3 APISERVER IP:PORT

    }
    
    server {
       listen 16443; # 由于nginx与master节点复用，这个监听端口不能是6443，否则会冲突
       proxy_pass k8s-apiserver;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    server {
        listen       80 default_server;
        server_name  _;

        location / {
        }
    }
}

[root@master2 ~]# cat /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

# 四层负载均衡，为两台Master apiserver组件提供负载均衡

stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
       server 192.168.1180:6443;   # master1 APISERVER IP:PORT
       server 192.168.1.181:6443;   # master2 APISERVER IP:PORT
       server 192.168.1.182:6443;   # master3 APISERVER IP:PORT

    }
    
    server {
       listen 16443; # 由于nginx与master节点复用，这个监听端口不能是6443，否则会冲突
       proxy_pass k8s-apiserver;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    server {
        listen       80 default_server;
        server_name  _;

        location / {
        }
    }
}

keepalive配置

主keepalived

[root@master1 ~]# cat /etc/keepalived/keepalived.conf 
global_defs { 
   notification_email { 
     acassen@firewall.loc 
     failover@firewall.loc 
     sysadmin@firewall.loc 
   } 
   notification_email_from Alexandre.Cassen@firewall.loc  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER 
    interface ens33  # 修改为实际网卡名
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 100    # 优先级，备服务器设置 90 
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒 
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    # 虚拟IP
    virtual_ipaddress { 
        192.168.1.199/24
    } 
    track_script {
        check_nginx
    } 
}

#vrrp_script：指定检查nginx工作状态脚本（根据nginx状态判断是否故障转移）

#virtual_ipaddress：虚拟IP（VIP）

[root@master1 ~]# cat /etc/keepalived/check_nginx.sh 
#!/bin/bash
count=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi

[root@xianchaomaster1 ~]# chmod +x  /etc/keepalived/check_nginx.sh

备keepalive

[root@master2 ~]# cat /etc/keepalived/keepalived.conf 
global_defs { 
   notification_email { 
     acassen@firewall.loc 
     failover@firewall.loc 
     sysadmin@firewall.loc 
   } 
   notification_email_from Alexandre.Cassen@firewall.loc  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_BACKUP
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state BACKUP 
    interface ens33
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 90
    advert_int 1
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    virtual_ipaddress { 
        192.168.1.199/24
    } 
    track_script {
        check_nginx
    } 
}


[root@master2 ~]# cat /etc/keepalived/check_nginx.sh 
#!/bin/bash
count=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi
[root@master2 ~]# chmod +x /etc/keepalived/check_nginx.sh

#注：keepalived根据脚本返回状态码（0为工作正常，非0不正常）判断是否故障转移。

启动服务

[root@master1 ~]# systemctl daemon-reload
[root@master1 ~]# systemctl start nginx
[root@master1 ~]# systemctl start keepalived
[root@master1 ~]# systemctl enable nginx keepalived

[root@master2 ~]# systemctl daemon-reload
[root@master2 ~]# systemctl start nginx
[root@master2 ~]# systemctl start keepalived
[root@master2 ~]# systemctl enable nginx keepalived

测试vip是否绑定成功

[root@master1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:79:9e:36 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.180/24 brd 192.168.40.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.1.199/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::b6ef:8646:1cfc:3e0c/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

测试keepalived

停掉master1上的nginx。vip会漂移到master2

[root@master1 ~]# service nginx stop

目前所有的Worker Node组件连接都还是xianchaomaster1 Node，如果不改为连接VIP走负载均衡器，那么Master还是单点故障。

因此接下来就是要改所有Worker Node（kubectl get node命令查看到的节点）组件配置文件，由原来192.168.40.180修改为192.168.40.199（VIP）。

在所有Worker Node执行：

[root@node1 ~]# sed -i 's#192.168.40.180:6443#192.168.1199:16443#' /etc/kubernetes/kubelet-bootstrap.kubeconfig

[root@node1 ~]# sed -i 's#192.168.40.180:6443#192.168.1.199:16443#' /etc/kubernetes/kubelet.json

[root@node1 ~]# sed -i 's#192.168.40.180:6443#192.168.1.199:16443#' /etc/kubernetes/kubelet.kubeconfig

[root@node1 ~]# sed -i 's#192.168.40.180:6443#192.168.1.199:16443#' /etc/kubernetes/kube-proxy.yaml

[root@node1 ~]# sed -i 's#192.168.40.180:6443#192.168.1.199:16443#' /etc/kubernetes/kube-proxy.kubeconfig

[root@node1 ~]# systemctl restart kubelet kube-proxy

这样高可用集群就安装好了

继续阅读

我的微信

这是我的微信扫一扫

jeesns项目代码部署and自动上线

二进制安装多master节点

二进制安装多master节点

1.系统初始化

2.搭建etcd集群

生成etcd的证书

部署etcd集群

安装kubernetes组件

部署apiserver组件

部署kubectl组件

部署kube-controller-manager组件

部署kube-scheduler组件

部署kubelet组件

部署kube-proxy组件

部署calico组件

查看集群状态

测试k8s集群部署tomcat服务

验证cordns是否正常

安装keepalived+nginx实现k8s apiserver高可用

keepalive配置

测试keepalived

发表评论取消回复

微信

联系方式

二进制安装多master节点

1.系统初始化

2.搭建etcd集群

生成etcd的证书

部署etcd集群

安装kubernetes组件

部署apiserver组件

部署kubectl组件

部署kube-controller-manager组件

部署kube-scheduler组件

部署kubelet组件

部署kube-proxy组件

部署calico组件

查看集群状态

测试k8s集群部署tomcat服务

验证cordns是否正常

安装keepalived+nginx实现k8s apiserver高可用

keepalive配置

测试keepalived

您可以选择一种方式赞助本站

微信钱包扫描赞助

发表评论 取消回复

登录 注册 找回密码

微信

联系方式

发表评论取消回复

登录注册找回密码