二进制安装多master节点
集群规划
k8s环境规划:
Pod网段: 10.0.0.0/16
Service网段: 10.255.0.0/16
实验环境规划:
操作系统:centos7.6
配置: 4Gib内存/6vCPU/100G硬盘
注意:也可以用4vCPU
网络:NAT
K8S集群角色 | Ip | 主机名 | 安装的组件 |
控制节点 | 192.168.1.180 | hh-master01 | apiserver、controller-manager、scheduler、etcd、docker、keepalived、nginx
|
控制节点 | 192.168.1.181 | hh-master02 | apiserver、controller-manager、scheduler、etcd、docker、keepalived、nginx
|
控制节点 | 192.168.1.182 | hh-master03 | apiserver、controller-manager、scheduler、etcd、docker
|
工作节点 | 192.168.1.183 | hh-node01 | kubelet、kube-proxy、docker、calico、coredns |
Vip | 192.168.1.199 |
1.系统初始化
2.搭建etcd集群
2.1配置etcd工作目录
//master节点执行 mkdir -p /etc/etcd mkdir -p /etc/etcd/ssl
2.2安装签发证书工具cfssl
下载自签证书工具
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
赋予权限
chmod +x cfssl*
重命名
for x in cfssl*; do mv $x ${x%*_linux-amd64}; done
移动文件到目录 (/usr/bin)
mv cfssl* /usr/bin
产生ca请求文件证书
注:
CN:Common Name(公用名称),kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name);浏览器使用该字段验证网站是否合法;对于 SSL 证书,一般为网站域名;而对于代码签名证书则为申请单位名称;而对于客户端证书则为证书申请者的姓名。
O:Organization(单位名称),kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group);对于 SSL 证书,一般为网站域名;而对于代码签名证书则为申请单位名称;而对于客户端单位证书则为证书申请者所在单位名称
mkdir -p /data/ssl # 自签名根证书(20年有效期) cat > /data/ssl/ca-csr.json << EOF { "CN": "kubernetes", "hosts": [ ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing", "O": "k8s", "OU": "System" } ], "ca": { "expiry": "175200h" } } EOF
生成CA
[root@master03 ssl]# cfssl gencert -initca ca-csr.json|cfssljson -bare ca 2021/07/14 19:26:14 [INFO] generating a new CA key and certificate from CSR 2021/07/14 19:26:14 [INFO] generate received request 2021/07/14 19:26:14 [INFO] received CSR 2021/07/14 19:26:14 [INFO] generating key: rsa-2048 2021/07/14 19:26:14 [INFO] encoded CSR 2021/07/14 19:26:14 [INFO] signed certificate with serial number 283971653681763962763710735735645294034226338199
xxxx
cat > /data/ssl/ca-config.json << EOF { "signing": { "default": { "expiry": "175200h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "175200h" } } } } EOF
生成etcd的证书
#配置etcd证书请求,hosts的ip变成自己etcd所在节点的ip
cat > /data/ssl/etcd-csr.json << EOF { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.1.180", "192.168.1.181", "192.168.1.182", "192.168.1.199" ##VIP ], "key": { "algo": "rsa", "size": 2048 }, "names": [{ "C": "CN", "ST": "Beijing", ##要和ca证书的对应 "L": "Beijing", "O": "k8s", "OU": "system" }] } EOF
#上述文件hosts字段中IP为所有etcd节点的集群内部通信IP,可以预留几个,做扩容用
签发etcd证书
[root@master03 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
部署etcd集群
etcd下载地址:https://github.com/etcd-io/etcd/
下载类似于etcd-v3.5.0-linux-amd64.tar.gz的包
[root@master01 data]# cp -p etcd-v3.5.0-linux-amd64/etcd* /usr/local/bin/
#创建配置文件,三台master分别创建
[root@master01 work]# vim etcd.conf #[Member] ETCD_NAME="etcd1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.1.180:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.1.180:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.180:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.180:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.180:2380,etcd2=https://192.168.1.181:2380,etcd3=https://192.168.1.182:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"
#注:
- ETCD_NAME:节点名称,集群中唯一
- ETCD_DATA_DIR:数据目录
- ETCD_LISTEN_PEER_URLS:集群通信监听地址
- ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址
- ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址
- ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址
- ETCD_INITIAL_CLUSTER:集群节点地址
- ETCD_INITIAL_CLUSTER_TOKEN:集群Token
- ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new是新集群,existing表示加入已有集群
#创建启动服务文件,三台master分别创建
[root@master01 work]# vim etcd.service [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=-/etc/etcd/etcd.conf WorkingDirectory=/var/lib/etcd/ ExecStart=/usr/local/bin/etcd \ --cert-file=/etc/etcd/ssl/etcd.pem \ --key-file=/etc/etcd/ssl/etcd-key.pem \ --trusted-ca-file=/etc/etcd/ssl/ca.pem \ --peer-cert-file=/etc/etcd/ssl/etcd.pem \ --peer-key-file=/etc/etcd/ssl/etcd-key.pem \ --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \ --peer-client-cert-auth \ --client-cert-auth Restart=on-failure RestartSec=5 LimitNOFILE=65536
[root@master01 work]# cp ca*.pem /etc/etcd/ssl/
[root@master01 work]# cp etcd*.pem /etc/etcd/ssl/
[root@master01 work]# cp etcd.conf /etc/etcd/
[root@master01 work]# cp etcd.service /usr/lib/systemd/system/
[root@master01 work]# for i in xianchaomaster2 xianchaomaster3;do rsync -vaz etcd.conf $i:/etc/etcd/;done
[root@master01 work]# for i in xianchaomaster2 xianchaomaster3;do rsync -vaz etcd*.pem ca*.pem $i:/etc/etcd/ssl/;done
[root@master01 work]# for i in xianchaomaster2 xianchaomaster3;do rsync -vaz etcd.service $i:/usr/lib/systemd/system/;done
#启动etcd集群
启动etcd的时候,先启动master01的etcd服务,会一直卡住在启动的状态,然后接着再启动master02的etcd,这样master03这个节点etcd才会正常起来
[root@master01 work]# systemctl daemon-reload
[root@master01 work]# systemctl enable etcd.service
[root@master01 work]# systemctl start etcd.service
[root@master02 work]# systemctl daemon-reload
[root@master02 work]# systemctl enable etcd.service
[root@master02 work]# systemctl start etcd.service
[root@master03 work]# systemctl daemon-reload
[root@master03 work]# systemctl enable etcd.service
[root@master03 work]# systemctl start etcd.service
#查看etcd集群
[root@master01 ~]# /usr/local/bin/etcdctl --write-out=table --cacert=/data/etcd/ssl/ca.pem --cert=/data/etcd/ssl/etcd.pem --key=/data/etcd/ssl/etcd-key.pem --endpoints=https://192.168.1.180:2379,https://192.168.1.181:2379,https://192.168.1.182:2379 endpoint health +----------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +----------------------------+--------+-------------+-------+ | https://192.168.1.181:2379 | true | 11.189771ms | | | https://192.168.1.182:2379 | true | 10.805193ms | | | https://192.168.1.180:2379 | true | 38.046545ms | | +----------------------------+--------+-------------+-------+
安装kubernetes组件
下载安装包
二进制包所在的github地址如下:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/
master:
- kubectl
- kube-apiserver
- kube-controller-manager
- kube-scheduler
node
- kubelet
- kube-proxy
#把kubernetes-server-linux-amd64.tar.gz上传到master01上的/data/work目录下:
[root@master01 work]# tar zxvf kubernetes-server-linux-amd64.tar.gz
[root@master01 work]# cd kubernetes/server/bin/
[root@master01 bin]# cp kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/
[root@master01 bin]# rsync -vaz kube-apiserver kube-controller-manager kube-scheduler kubectl master02:/usr/local/bin/
[root@master01 bin]# rsync -vaz kube-apiserver kube-controller-manager kube-scheduler kubectl master03:/usr/local/bin/
[root@master01 bin]# scp kubelet kube-proxy node01:/usr/local/bin/
[root@master01 bin]# cd /data/work/
[root@master01 work]# mkdir -p /etc/kubernetes/
[root@master01 work]# mkdir -p /etc/kubernetes/ssl
[root@master01 work]# mkdir /var/log/kubernetes
部署apiserver组件
#启动TLS Bootstrapping 机制
Master apiserver启用TLS认证后,每个节点的 kubelet 组件都要使用由 apiserver 使用的 CA 签发的有效证书才能与 apiserver 通讯,当Node节点很多时,这种客户端证书颁发需要大量工作,同样也会增加集群扩展复杂度。
为了简化流程,Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书,kubelet会以一个低权限用户自动向apiserver申请证书,kubelet的证书由apiserver动态签署。
Bootstrap 是很多系统中都存在的程序,比如 Linux 的bootstrap,bootstrap 一般都是作为预先配置在开启或者系统启动的时候加载,这可以用来生成一个指定环境。Kubernetes 的 kubelet 在启动时同样可以加载一个这样的配置文件,这个文件的内容类似如下形式:
#TLS bootstrapping 具体引导过程
1.TLS 作用
TLS 的作用就是对通讯加密,防止中间人窃听;同时如果证书不信任的话根本就无法与 apiserver 建立连接,更不用提有没有权限向apiserver请求指定内容。
2. RBAC 作用
当 TLS 解决了通讯问题后,那么权限问题就应由 RBAC 解决(可以使用其他权限模型,如 ABAC);RBAC 中规定了一个用户或者用户组(subject)具有请求哪些 api 的权限;在配合 TLS 加密的时候,实际上 apiserver 读取客户端证书的 CN 字段作为用户名,读取 O字段作为用户组.
以上说明:第一,想要与 apiserver 通讯就必须采用由 apiserver CA 签发的证书,这样才能形成信任关系,建立 TLS 连接;第二,可以通过证书的 CN、O 字段来提供 RBAC 所需的用户与用户组。
#kubelet 首次启动流程
TLS bootstrapping 功能是让 kubelet 组件去 apiserver 申请证书,然后用于连接 apiserver;那么第一次启动时没有证书如何连接 apiserver ?
在apiserver 配置中指定了一个 token.csv 文件,该文件中是一个预设的用户配置;同时该用户的Token 和 由apiserver 的 CA签发的用户被写入了 kubelet 所使用的 bootstrap.kubeconfig 配置文件中;这样在首次请求时,kubelet 使用 bootstrap.kubeconfig 中被 apiserver CA 签发证书时信任的用户来与 apiserver 建立 TLS 通讯,使用 bootstrap.kubeconfig 中的用户 Token 来向 apiserver 声明自己的 RBAC 授权身份.
token.csv格式:
3940fd7fbb391d1b4d861ad17a1f0613,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
首次启动时,可能与遇到 kubelet 报 401 无权访问 apiserver 的错误;这是因为在默认情况下,kubelet 通过 bootstrap.kubeconfig 中的预设用户 Token 声明了自己的身份,然后创建 CSR 请求;但是不要忘记这个用户在我们不处理的情况下他没任何权限的,包括创建 CSR 请求;所以需要创建一个 ClusterRoleBinding,将预设用户 kubelet-bootstrap 与内置的 ClusterRole system:node-bootstrapper 绑定到一起,使其能够发起 CSR 请求。稍后安装kubelet的时候演示。
#创建token.csv文件
[root@master01 work]# cat > token.csv << EOF $(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap" EOF #格式:token,用户名,UID,用户组
#创建csr请求文件,替换为自己机器的IP
[root@master01 work]# vim kube-apiserver-csr.json { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.1.180", "192.1681181", "192.168.1182", "192.168.1.183", "192.168.1.199", "10.255.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Hubei", "L": "Wuhan", "O": "k8s", "OU": "system" } ] }
#注: 如果 hosts 字段不为空则需要指定授权使用该证书的 IP 或域名列表。 由于该证书后续被 kubernetes master 集群使用,需要将master节点的IP都填上,同时还需要填写 service 网络的首个IP。(一般是 kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP,如 10.255.0.1)
#生成证书
[root@master01 work]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
#创建api-server的配置文件,替换成自己的ip
[root@master01 work]# vim kube-apiserver.conf
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
--anonymous-auth=false \
--bind-address=192.168.1.180 \
--secure-port=6443 \
--advertise-address=192.168.1.180 \
--insecure-port=0 \
--authorization-mode=Node,RBAC \
--runtime-config=api/all=true \
--enable-bootstrap-token-auth \
--service-cluster-ip-range=10.255.0.0/16 \
--token-auth-file=/etc/kubernetes/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \
--tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
--kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
--service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--etcd-cafile=/etc/etcd/ssl/ca.pem \
--etcd-certfile=/etc/etcd/ssl/etcd.pem \
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
--etcd-servers=https://192.168.1.180:2379,https://192.168.1.181:2379,https://192.168.1.182:2379 \
--enable-swagger-ui=true \
--allow-privileged=true \
--apiserver-count=3 \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/var/log/kube-apiserver-audit.log \
--event-ttl=1h \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=4"
[root@master01 work]# cp ca*.pem /etc/kubernetes/ssl [root@master01 work]# cp kube-apiserver*.pem /etc/kubernetes/ssl/ [root@master01 work]# cp token.csv /etc/kubernetes/ [root@master01 work]# cp kube-apiserver.conf /etc/kubernetes/ [root@master01 work]# cp kube-apiserver.service /usr/lib/systemd/system/ [root@master01 work]# rsync -vaz token.csv master02:/etc/kubernetes/ [root@master01 work]# rsync -vaz token.csv master03:/etc/kubernetes/ [root@master01 work]# rsync -vaz kube-apiserver*.pem master02:/etc/kubernetes/ssl/ [root@master01 work]# rsync -vaz kube-apiserver*.pem master03:/etc/kubernetes/ssl/ [root@master01 work]# rsync -vaz ca*.pem master02:/etc/kubernetes/ssl/ [root@master01 work]# rsync -vaz ca*.pem master03:/etc/kubernetes/ssl/ [root@master01 work]# rsync -vaz kube-apiserver.conf master02:/etc/kubernetes/ [root@master01 work]# rsync -vaz kube-apiserver.conf master03:/etc/kubernetes/ [root@master01 work]# rsync -vaz kube-apiserver.service master03:/usr/lib/systemd/system/ [root@master01 work]# rsync -vaz kube-apiserver.service master03:/usr/lib/systemd/system/
注:master02和master03配置文件kube-apiserver.conf的IP地址修改为实际的本机IP
[root@master02 ~]# cat /etc/kubernetes/kube-apiserver.conf KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ --anonymous-auth=false \ --bind-address=192.168.1181 \ --secure-port=6443 \ --advertise-address=192.168.1.181 \ --insecure-port=0 \ --authorization-mode=Node,RBAC \ --runtime-config=api/all=true \ --enable-bootstrap-token-auth \ --service-cluster-ip-range=10.255.0.0/16 \ --token-auth-file=/etc/kubernetes/token.csv \ --service-node-port-range=30000-50000 \ --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \ --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-issuer=https://kubernetes.default.svc.cluster.local \ --etcd-cafile=/etc/etcd/ssl/ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --etcd-servers=https://192.168.1.180:2379,https://192.168.1181:2379,https://192.168.1.182:2379 \ --enable-swagger-ui=true \ --allow-privileged=true \ --apiserver-count=3 \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/log/kube-apiserver-audit.log \ --event-ttl=1h \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=4" [root@master03 ~]# cat /etc/kubernetes/kube-apiserver.conf KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ --anonymous-auth=false \ --bind-address=192.168.1.182 \ --secure-port=6443 \ --advertise-address=192.168.1.182 \ --insecure-port=0 \ --authorization-mode=Node,RBAC \ --runtime-config=api/all=true \ --enable-bootstrap-token-auth \ --service-cluster-ip-range=10.255.0.0/16 \ --token-auth-file=/etc/kubernetes/token.csv \ --service-node-port-range=30000-50000 \ --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \ --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-issuer=https://kubernetes.default.svc.cluster.local \ --etcd-cafile=/etc/etcd/ssl/ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --etcd-servers=https://192.168.1.180:2379,https://192.168.1.181:2379,https://192.168.1.182:2379 \ --enable-swagger-ui=true \ --allow-privileged=true \ --apiserver-count=3 \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/log/kube-apiserver-audit.log \ --event-ttl=1h \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=4"
启动服务
[root@master01 work]# systemctl daemon-reload [root@master02 work]# systemctl daemon-reload [root@master03 work]# systemctl daemon-reload [root@master01 work]# systemctl enable kube-apiserver [root@master02 work]# systemctl enable kube-apiserver [root@master03 work]# systemctl enable kube-apiserver [root@master01 work]# systemctl start kube-apiserver [root@master02 work]# systemctl start kube-apiserver [root@master03 work]# systemctl start kube-apiserver [root@master01 work]# systemctl status kube-apiserver Active: active (running) since Wed [root@master02 work]# systemctl status kube-apiserver Active: active (running) since Wed [root@master03 work]# systemctl status kube-apiserver Active: active (running) since Wed
[root@master01 work]# curl --insecure https://192.168.40.180:6443/
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}
上面看到401,这个是正常的的状态,还没认证
部署kubectl组件
Kubectl是客户端工具,操作k8s资源的,如增删改查等。
Kubectl操作资源的时候,怎么知道连接到哪个集群,需要一个文件/etc/kubernetes/admin.conf,kubectl会根据这个文件的配置,去访问k8s资源。/etc/kubernetes/admin.con文件记录了访问的k8s集群,和用到的证书。
可以设置一个环境变量KUBECONFIG
[root@ xianchaomaster1 ~]# export KUBECONFIG =/etc/kubernetes/admin.conf
这样在操作kubectl,就会自动加载KUBECONFIG来操作要管理哪个集群的k8s资源了
也可以按照下面方法,这个是在kubeadm初始化k8s的时候会告诉我们要用的一个方法
[root@ xianchaomaster1 ~]# cp /etc/kubernetes/admin.conf /root/.kube/config
这样我们在执行kubectl,就会加载/root/.kube/config文件,去操作k8s资源了
如果设置了KUBECONFIG,那就会先找到KUBECONFIG去操作k8s,如果没有KUBECONFIG变量,那就会使用/root/.kube/config文件决定管理哪个k8s集群的资源
#创建csr请求文件
[root@master01 work]# vim admin-csr.json
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Hubei",
"L": "Wuhan",
"O": "system:masters",
"OU": "system"
}
]
}
#说明: 后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权; kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings,如 cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定,该 Role 授予了调用kube-apiserver 的所有 API的权限; O指定该证书的 Group 为 system:masters,kubelet 使用该证书访问 kube-apiserver 时 ,由于证书被 CA 签名,所以认证通过,同时由于证书用户组为经过预授权的 system:masters,所以被授予访问所有 API 的权限;
注: 这个admin 证书,是将来生成管理员用的kube config 配置文件用的,现在我们一般建议使用RBAC 来对kubernetes 进行角色权限控制, kubernetes 将证书中的CN 字段 作为User, O 字段作为 Group; "O": "system:masters", 必须是system:masters,否则后面kubectl create clusterrolebinding报错。
#证书O配置为system:masters 在集群内部cluster-admin的clusterrolebinding将system:masters组和cluster-admin clusterrole绑定在一起
#生成证书
[root@master01 work]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin [root@master01 work]# cp admin*.pem /etc/kubernetes/ssl/
配置安全上下文
#创建kubeconfig配置文件,比较重要
kubeconfig 为 kubectl 的配置文件,包含访问 apiserver 的所有信息,如 apiserver 地址、CA 证书和自身使用的证书(这里如果报错找不到kubeconfig路径,请手动复制到相应路径下,没有则忽略)
1.设置集群参数
[root@master01 work]# kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.180:6443 --kubeconfig=kube.config
#查看kube.config内容
vim kube.config apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR0akNDQXA2Z0F3SUJBZ0lVRUVpcFFkbVRUbWpSYWV5MTMzdUhJRFVTVEVzd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RGpBTUJnTlZCQWdUQlVoMVltVnBNUTR3REFZRFZRUUhFd1ZYZFdoaApiakVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WnplWE4wWlcweEV6QVJCZ05WQkFNVENtdDFZbVZ5CmJtVjBaWE13SGhjTk1qRXdOVEV5TVRNeE16QXdXaGNOTXpFd05URXdNVE14TXpBd1dqQmhNUXN3Q1FZRFZRUUcKRXdKRFRqRU9NQXdHQTFVRUNCTUZTSFZpWldreERqQU1CZ05WQkFjVEJWZDFhR0Z1TVF3d0NnWURWUVFLRXdOcgpPSE14RHpBTkJnTlZCQXNUQm5ONWMzUmxiVEVUTUJFR0ExVUVBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxEb0s0THNYV0dLYko0UjBJSnh2T0E3a2QvM0k5M3cKckQxMzE1RXRDd1NIRXNnem5ZLzc0c05wQTJSYzdQc2NMK2ZqZTFuZU9rZ1pPbGwyT04vSTFBMi83QXd0YUt4OAp0UnlIcllNeEZyWlZ6TE9UQWxEaTZYN1RlUk9INUNMc1AxUkdqenc4OXgyVlZSd3dpNm1qc0tRcWt3U1hpbmh5CkQxaElibVU5N1h3ZEtwc1YyUkFIZkxhVUZEMkFBcDJlRW42YzZVVzNCbU5RLzdacmhVeS9FM3J1bHRYSm96NlAKd0ZZM0hGUEhZblUwN3VzRVAvSW83ZFpzc0h5WUluNVRZRjl5NTdKQmcwa09PRnJhQncxV08waWhYU0FkM01qRQoxRUFlWEhId2pXanRXRFFGMWwwWEpWaFVvL3Y2OVRtOFR2S2txdzQvUEdYRG50dmJ5S1hrNmVjQ0F3RUFBYU5tCk1HUXdEZ1lEVlIwUEFRSC9CQVFEQWdFR01CSUdBMVVkRXdFQi93UUlNQVlCQWY4Q0FRSXdIUVlEVlIwT0JCWUUKRkt2L2NkdjFjYURhRS9VNkU1V0tZNFcwMjF1eE1COEdBMVVkSXdRWU1CYUFGS3YvY2R2MWNhRGFFL1U2RTVXSwpZNFcwMjF1eE1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQWp0KzJoTU5YSVdjeWxjK1RWL05JS1FsRHRaSEJUCklRSTZYV3Q5KzFKWUNUbEMxYm5aaHExSnU1ZnB3VEJXMmdjRkRxUVRlbk5lZ0F5T2J2ejJidGNJK2ZDNkptUjgKSFg4dUpPUGJQelM0cEo5WkNsd1E4MHFJVzJYQitXMXh3OW5MSFAxdVJwZXVsSCtkeUNMeS9Zb1kwQ3FnWnc1aApBSktGSE42ckYrTUNWT0R1Tzk4ZThjTWhBcVF6U1hsb2tiVHR3Rnk3OHdnYnJaUCtybGY3eFNZL28wYytKQ1U5ClVsREFhTVJGSytvTVR4VFlicHBKMnRvOGVCemNJM2FrYjFiL2Q0cm9ESGR0U1cvclk0UzFFTTZJSGtDb0xpV1YKQ2IrVVkzb3Fqb0lBOEFHMzhZb1BiVHlqbjVuY24vOU0vVjlkS2E4RFEya011Z3dPall6alJCTFUKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= server: https://192.168.40.180:6443 name: kubernetes contexts: null current-context: "" kind: Config preferences: {} users: null
2.设置客户端认证参数
[root@master01 work]# kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=kube.config
3.设置上下文参数
[root@master01 work]# kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=kube.config
4.设置当前上下文
[root@master01 work]# kubectl config use-context kubernetes --kubeconfig=kube.config
[root@master01 work]# mkdir ~/.kube -p [root@master01 work]# cp kube.config ~/.kube/config
5.授权kubernetes证书访问kubelet api权限
这步不操作,kubectl有些权限用不了
[root@master01 work]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes
#查看集群组件状态
[root@master01 ~]# kubectl cluster-info Kubernetes control plane is running at https://192.168.1.182:6443 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@master01 work]# kubectl get componentstatuses Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR controller-manager Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused etcd-0 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"} etcd-1 Healthy {"health":"true"}
[root@master01 work]# kubectl get all --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.255.0.1 <none> 443/TCP
#同步kubectl文件到其他节点
[root@master01 ~]# mkdir /root/.kube/ [root@master01 ~]# mkdir /root/.kube/ [root@master01 work]# rsync -vaz /root/.kube/config xianchaomaster2:/root/.kube/ [root@master01 work]# rsync -vaz /root/.kube/config xianchaomaster3:/root/.kube/
#配置kubectl子命令补全
[root@master01 work]# yum install -y bash-completion [root@master01 work]# source /usr/share/bash-completion/bash_completion [root@master01 work]# source <(kubectl completion bash) [root@master01 work]# kubectl completion bash > ~/.kube/completion.bash.inc [root@master01 work]# source '/root/.kube/completion.bash.inc' [root@master01 work]# source $HOME/.bash_profile
Kubectl官方备忘单:
https://kubernetes.io/zh/docs/reference/kubectl/cheatsheet/
部署kube-controller-manager组件
#创建csr请求文件
[root@master01 work]# vim kube-controller-manager-csr.json { "CN": "system:kube-controller-manager", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.1.180", "192.168.1.181", "192.168.1.182", "192.168.1.199" ], "names": [ { "C": "CN", "ST": "Hubei", "L": "Wuhan", "O": "system:kube-controller-manager", "OU": "system" } ] }
注: hosts 列表包含所有 kube-controller-manager 节点 IP; CN 为 system:kube-controller-manager、O 为 system:kube-controller-manager,kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限
#生成证书
[root@master01 work]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
#创建kube-controller-manager的kubeconfig
1.设置集群参数
[root@master01 work]# kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.180:6443 --kubeconfig=kube-controller-manager.kubeconfig
2.设置客户端认证参数
[root@master01 work]# kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig
3.设置上下文参数
[root@master01 work]# kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
4.设置当前上下文
[root@master01 work]# kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
#创建配置文件kube-controller-manager.conf
[root@master01 work]# vim kube-controller-manager.conf KUBE_CONTROLLER_MANAGER_OPTS="--port=0 \ --secure-port=10252 \ --bind-address=127.0.0.1 \ --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \ --service-cluster-ip-range=10.255.0.0/16 \ --cluster-name=kubernetes \ --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --allocate-node-cidrs=true \ --cluster-cidr=10.0.0.0/16 \ --experimental-cluster-signing-duration=87600h \ --root-ca-file=/etc/kubernetes/ssl/ca.pem \ --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \ --leader-elect=true \ --feature-gates=RotateKubeletServerCertificate=true \ --controllers=*,bootstrapsigner,tokencleaner \ --horizontal-pod-autoscaler-use-rest-clients=true \ --horizontal-pod-autoscaler-sync-period=10s \ --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \ --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \ --use-service-account-credentials=true \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=2"
#创建启动文件
[root@master01 work]# vim kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/etc/kubernetes/kube-controller-manager.conf ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target
#启动服务
[root@master01 work]# cp kube-controller-manager*.pem /etc/kubernetes/ssl/ [root@master01 work]# cp kube-controller-manager.kubeconfig /etc/kubernetes/ [root@master01 work]# cp kube-controller-manager.conf /etc/kubernetes/ [root@master01 work]# cp kube-controller-manager.service /usr/lib/systemd/system/ [root@master01 work]# rsync -vaz kube-controller-manager*.pem xianchaomaster2:/etc/kubernetes/ssl/ [root@master01 work]# rsync -vaz kube-controller-manager*.pem xianchaomaster3:/etc/kubernetes/ssl/ [root@master01 work]# rsync -vaz kube-controller-manager.kubeconfig kube-controller-manager.conf xianchaomaster2:/etc/kubernetes/ [root@master01 work]# rsync -vaz kube-controller-manager.kubeconfig kube-controller-manager.conf xianchaomaster3:/etc/kubernetes/ [root@master01 work]# rsync -vaz kube-controller-manager.service xianchaomaster2:/usr/lib/systemd/system/ [root@master01 work]# rsync -vaz kube-controller-manager.service xianchaomaster3:/usr/lib/systemd/system/ [root@master01 work]# systemctl daemon-reload [root@master01 work]# systemctl enable kube-controller-manager [root@master01 work]# systemctl start kube-controller-manager [root@master01 work]# systemctl status kube-controller-manager
[root@master02]# systemctl daemon-reload [root@master02]# systemctl enable kube-controller-manager [root@master02]# systemctl start kube-controller-manager [root@master02]# systemctl status kube-controller-manager
[root@master03]# systemctl daemon-reload [root@master03]# systemctl enable kube-controller-manager [root@master03]# systemctl start kube-controller-manager [root@master03]# systemctl status kube-controller-manager
部署kube-scheduler组件
#创建csr请求
[root@xianchaomaster1 work]# vim kube-scheduler-csr.json { "CN": "system:kube-scheduler", "hosts": [ "127.0.0.1", "192.168.1.180", "192.168.1.181", "192.168.1.182", "192.168.1.199" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Hubei", "L": "Wuhan", "O": "system:kube-scheduler", "OU": "system" } ] }
注: hosts 列表包含所有 kube-scheduler 节点 IP; CN 为 system:kube-scheduler、O 为 system:kube-scheduler,kubernetes 内置的 ClusterRoleBindings system:kube-scheduler 将赋予 kube-scheduler 工作所需的权限
#生成证书
[root@master01 work]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
#创建kube-scheduler的kubeconfig
1.设置集群参数
[root@master01 work]# kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.1.180:6443 --kubeconfig=kube-scheduler.kubeconfig
2.设置客户端认证参数
[root@master01 work]# kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig
3.设置上下文参数
[root@master01 work]# kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
4.设置当前上下文
[root@master01 work]# kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
#创建配置文件kube-scheduler.conf
[root@master01 work]# vim kube-scheduler.conf KUBE_SCHEDULER_OPTS="--address=127.0.0.1 \ --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \ --leader-elect=true \ --alsologtostderr=true \ --logtostderr=false \ --log-dir=/var/log/kubernetes \ --v=2"
#创建服务启动文件
[root@master01 work]# vim kube-scheduler.service [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/etc/kubernetes/kube-scheduler.conf ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target
#启动服务
[root@master01 work]# cp kube-scheduler*.pem /etc/kubernetes/ssl/ [root@master01 work]# cp kube-scheduler.kubeconfig /etc/kubernetes/ [root@master01 work]# cp kube-scheduler.conf /etc/kubernetes/ [root@master01 work]# cp kube-scheduler.service /usr/lib/systemd/system/ [root@master01 work]# rsync -vaz kube-scheduler*.pem xianchaomaster2:/etc/kubernetes/ssl/ [root@master01 work]# rsync -vaz kube-scheduler*.pem xianchaomaster3:/etc/kubernetes/ssl/ [root@master01 work]# rsync -vaz kube-scheduler.kubeconfig kube-scheduler.conf xianchaomaster2:/etc/kubernetes/ [root@master01 work]# rsync -vaz kube-scheduler.kubeconfig kube-scheduler.conf xianchaomaster3:/etc/kubernetes/ [root@master01 work]# rsync -vaz kube-scheduler.service xianchaomaster2:/usr/lib/systemd/system/ [root@master01 work]# rsync -vaz kube-scheduler.service xianchaomaster3:/usr/lib/systemd/system/ [root@master01 work]# systemctl daemon-reload [root@master01 work]# systemctl enable kube-scheduler [root@master01 work]# systemctl start kube-scheduler [root@master01 work]# systemctl status kube-scheduler ● kube-scheduler.service - Kubernetes Scheduler Active: active (running) since Wed [root@master02]# systemctl daemon-reload [root@master02]# systemctl enable kube-scheduler [root@master02]# systemctl start kube-scheduler [root@master02]# systemctl status kube-scheduler ● kube-scheduler.service - Kubernetes Scheduler Active: active (running) since Wed [root@master03]# systemctl daemon-reload [root@master03]# systemctl enable kube-scheduler [root@master03]# systemctl start kube-scheduler [root@master03]# systemctl status kube-scheduler ● kube-scheduler.service - Kubernetes Scheduler Active: active (running) since Wed
导入离线镜像压缩包
#把pause-cordns.tar.gz上传到xianchaonode1节点,手动解压
[root@node1 ~]# docker load -i pause-cordns.tar.gz
部署kubelet组件
kubelet: 每个Node节点上的kubelet定期就会调用API Server的REST接口报告自身状态,API Server接收这些信息后,将节点状态信息更新到etcd中。kubelet也通过API Server监听Pod信息,从而对Node机器上的POD进行管理,如创建、删除、更新Pod
以下操作在xianchaomaster1上操作
创建kubelet-bootstrap.kubeconfig
[root@master01work]# cd /data/work/ [root@master01 work]# BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv) [root@master01 work]# rm -r kubelet-bootstrap.kubeconfig [root@master01 work]# kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.40.180:6443 --kubeconfig=kubelet-bootstrap.kubeconfig [root@master01 work]# kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.kubeconfig [root@master01 work]# kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig [root@master01 work]# kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig [root@master01 work]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
#创建配置文件kubelet.json
"cgroupDriver": "systemd"要和docker的驱动一致。
address替换为自己xianchaonode1的IP地址。
[root@master01 work]# vim kubelet.json
{
"kind": "KubeletConfiguration",
"apiVersion": "kubelet.config.k8s.io/v1beta1",
"authentication": {
"x509": {
"clientCAFile": "/etc/kubernetes/ssl/ca.pem"
},
"webhook": {
"enabled": true,
"cacheTTL": "2m0s"
},
"anonymous": {
"enabled": false
}
},
"authorization": {
"mode": "Webhook",
"webhook": {
"cacheAuthorizedTTL": "5m0s",
"cacheUnauthorizedTTL": "30s"
}
},
"address": "192.168.40.183",
"port": 10250,
"readOnlyPort": 10255,
"cgroupDriver": "systemd",
"hairpinMode": "promiscuous-bridge",
"serializeImagePulls": false,
"featureGates": {
"RotateKubeletClientCertificate": true,
"RotateKubeletServerCertificate": true
},
"clusterDomain": "cluster.local.",
"clusterDNS": ["10.255.0.2"]
}
[root@master01 work]# vim kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \
--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
--cert-dir=/etc/kubernetes/ssl \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
--config=/etc/kubernetes/kubelet.json \
--network-plugin=cni \
--pod-infra-container-image=k8s.gcr.io/pause:3.2 \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
#注:
–hostname-override:显示名称,集群中唯一
–network-plugin:启用CNI
–kubeconfig:空路径,会自动生成,后面用于连接apiserver
–bootstrap-kubeconfig:首次启动向apiserver申请证书
–config:配置参数文件
–cert-dir:kubelet证书生成目录
–pod-infra-container-image:管理Pod网络容器的镜像
#注:kubelete.json配置文件address改为各个节点的ip地址,在各个work节点上启动服务
[root@node1 ~]# mkdir /etc/kubernetes/ssl -p [root@master01 work]# scp kubelet-bootstrap.kubeconfig kubelet.json xianchaonode1:/etc/kubernetes/ [root@master01 work]# scp ca.pem xianchaonode1:/etc/kubernetes/ssl/ [root@master01 work]# scp kubelet.service xianchaonode1:/usr/lib/systemd/system/
#启动kubelet服务
[root@node1 ~]# mkdir /var/lib/kubelet [root@node1 ~]# mkdir /var/log/kubernetes [root@node1 ~]# systemctl daemon-reload [root@node1 ~]# systemctl enable kubelet [root@node1 ~]# systemctl start kubelet [root@node1 ~]# systemctl status kubelet Active: active (running) since
确认kubelet服务启动成功后,接着到xianchaomaster1节点上Approve一下bootstrap请求。
执行如下命令可以看到一个worker节点发送了一个 CSR 请求:
[root@master01 work]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-SY6gROGEmH0qVZhMVhJKKWN3UaWkKKQzV8dopoIO9Uc 87s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending [root@master01 work]# kubectl certificate approve node-csr-SY6gROGEmH0qVZhMVhJKKWN3UaWkKKQzV8dopoIO9Uc [root@master01 work]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-SY6gROGEmH0qVZhMVhJKKWN3UaWkKKQzV8dopoIO9Uc 2m25s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued [root@master01 work]# kubectl get nodes NAME STATUS ROLES AGE VERSION xianchaonode1 NotReady <none> 30s v1.20.7
#注意:STATUS是NotReady表示还没有安装网络插件
部署kube-proxy组件
#创建csr请求
[root@master01 work]# vim kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Hubei",
"L": "Wuhan",
"O": "k8s",
"OU": "system"
}
]
}
生成证书
[root@master01 work]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
#创建kubeconfig文件
[root@master01 work]# kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.40.180:6443 --kubeconfig=kube-proxy.kubeconfig [root@master01 work]# kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig [root@master01 work]# kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig [root@master01 work]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
#创建kube-proxy配置文件
[root@master01 work]# vim kube-proxy.yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.40.183
clientConnection:
kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 192.168.40.0/24
healthzBindAddress: 192.168.40.183:10256
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.40.183:10249
mode: "ipvs"
#创建服务启动文件
[root@master01 work]# vim kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
--config=/etc/kubernetes/kube-proxy.yaml \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
[root@master01 work]# scp kube-proxy.kubeconfig kube-proxy.yaml xianchaonode1:/etc/kubernetes/ [root@master01 work]#scp kube-proxy.service xianchaonode1:/usr/lib/systemd/system/
#启动服务
[root@node1 ~]# mkdir -p /var/lib/kube-proxy [root@node1 ~]# systemctl daemon-reload [root@node1 ~]# systemctl enable kube-proxy [root@node1 ~]# systemctl start kube-proxy [root@node1 ~]# systemctl status kube-proxy Active: active (running) since Wed
部署calico组件
#解压离线镜像压缩包
#把cni.tar.gz和node.tar.gz上传到xianchaonode1节点,手动解压
[root@node1 ~]# docker load -i cni.tar.gz [root@node1 ~]# docker load -i node.tar.gz
#把calico.yaml文件上传到xianchaomaster1上的的/data/work目录
[root@master01 work]# kubectl apply -f calico.yaml [root@master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-node-xk7n4 1/1 Running 0 13s [root@master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION xianchaonode1 Ready <none> 73m v1.20.7
部署coredns组件
[root@master01 ~]# kubectl apply -f coredns.yaml [root@master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-node-xk7n4 1/1 Running 0 6m6s coredns-7bf4bd64bd-dt8dq 1/1 Running 0 51s [root@master01 ~]# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.255.0.2 <none> 53/UDP,53/TCP,9153/TCP 12m
查看集群状态
[root@master1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION xianchaonode1 Ready <none> 38m v1.20.7
测试k8s集群部署tomcat服务
#把tomcat.tar.gz和busybox-1-28.tar.gz上传到node1,手动解压
[root@node1 ~]# docker load -i tomcat.tar.gz [root@node1 ~]# docker load -i busybox-1-28.tar.gz [root@master1 ~]# kubectl apply -f tomcat.yaml [root@master1 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE demo-pod 2/2 Running 0 11m [root@master1 ~]# kubectl apply -f tomcat-service.yaml [root@master1 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.255.0.1 <none> 443/TCP 158m tomcat NodePort 10.255.227.179 <none> 8080:30080/TCP 19m
在浏览器访问xianchaonode1节点的ip:30080即可请求到浏览器
验证cordns是否正常
[root@master1 ~]# kubectl run busybox --image busybox:1.28 --restart=Never --rm -it busybox -- sh / # ping www.baidu.com PING www.baidu.com (39.156.66.18): 56 data bytes 64 bytes from 39.156.66.18: seq=0 ttl=127 time=39.3 ms #通过上面可以看到能访问网络 / # nslookup kubernetes.default.svc.cluster.local Server: 10.255.0.2 Address: 10.255.0.2:53 Name: kubernetes.default.svc.cluster.local Address: 10.255.0.1 / # nslookup tomcat.default.svc.cluster.local Server: 10.255.0.2 Address 1: 10.255.0.2 kube-dns.kube-system.svc.cluster.local Name: tomcat.default.svc.cluster.local Address 1: 10.255.227.179 tomcat.default.svc.cluster.local
#注意:
busybox要用指定的1.28版本,不能用最新版本,最新版本,nslookup会解析不到dns和ip,报错如下:
/ # nslookup kubernetes.default.svc.cluster.local Server: 10.255.0.2 Address: 10.255.0.2:53 *** Can't find kubernetes.default.svc.cluster.local: No answer *** Can't find kubernetes.default.svc.cluster.local: No answer
10.255.0.2 就是我们coreDNS的clusterIP,说明coreDNS配置好了。
解析内部Service的名称,是通过coreDNS去解析的。
安装keepalived+nginx实现k8s apiserver高可用
把epel.repo上传到master1的/etc/yum.repos.d目录下,这样才能安装keepalived和nginx
把epel.repo传到master2、master3、node1上
[root@master1 ~]# scp /etc/yum.repos.d/epel.repo xianchaomaster2:/etc/yum.repos.d/ [root@master1 ~]# scp /etc/yum.repos.d/epel.repo xianchaomaster3:/etc/yum.repos.d/ [root@master1 ~]# scp /etc/yum.repos.d/epel.repo xianchaonode1:/etc/yum.repos.d/
1、安装nginx主备:
在master1和master2上做nginx主备安装
[root@master1 ~]# yum install nginx keepalived -y [root@master2 ~]# yum install nginx keepalived -y
修改nginx配置文件。主备一样
[root@master1 ~]# cat /etc/nginx/nginx.conf [root@master1 ~]# cat /etc/nginx/nginx.conf user nginx; worker_processes auto; error_log /var/log/nginx/error.log; pid /run/nginx.pid; include /usr/share/nginx/modules/*.conf; events { worker_connections 1024; }
# 四层负载均衡,为两台Master apiserver组件提供负载均衡
stream { log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent'; access_log /var/log/nginx/k8s-access.log main; upstream k8s-apiserver { server 192.168.1.180:6443; # master1 APISERVER IP:PORT server 192.168.1.181:6443; # master2 APISERVER IP:PORT server 192.1681.182:6443; # master3 APISERVER IP:PORT } server { listen 16443; # 由于nginx与master节点复用,这个监听端口不能是6443,否则会冲突 proxy_pass k8s-apiserver; } } http { log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream; server { listen 80 default_server; server_name _; location / { } } }
[root@master2 ~]# cat /etc/nginx/nginx.conf user nginx; worker_processes auto; error_log /var/log/nginx/error.log; pid /run/nginx.pid; include /usr/share/nginx/modules/*.conf; events { worker_connections 1024; }
# 四层负载均衡,为两台Master apiserver组件提供负载均衡
stream { log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent'; access_log /var/log/nginx/k8s-access.log main; upstream k8s-apiserver { server 192.168.1180:6443; # master1 APISERVER IP:PORT server 192.168.1.181:6443; # master2 APISERVER IP:PORT server 192.168.1.182:6443; # master3 APISERVER IP:PORT } server { listen 16443; # 由于nginx与master节点复用,这个监听端口不能是6443,否则会冲突 proxy_pass k8s-apiserver; } } http { log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream; server { listen 80 default_server; server_name _; location / { } } }
keepalive配置
主keepalived
[root@master1 ~]# cat /etc/keepalived/keepalived.conf global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id NGINX_MASTER } vrrp_script check_nginx { script "/etc/keepalived/check_nginx.sh" } vrrp_instance VI_1 { state MASTER interface ens33 # 修改为实际网卡名 virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的 priority 100 # 优先级,备服务器设置 90 advert_int 1 # 指定VRRP 心跳包通告间隔时间,默认1秒 authentication { auth_type PASS auth_pass 1111 } # 虚拟IP virtual_ipaddress { 192.168.1.199/24 } track_script { check_nginx } }
#vrrp_script:指定检查nginx工作状态脚本(根据nginx状态判断是否故障转移)
#virtual_ipaddress:虚拟IP(VIP)
[root@master1 ~]# cat /etc/keepalived/check_nginx.sh #!/bin/bash count=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$") if [ "$count" -eq 0 ];then systemctl stop keepalived fi [root@xianchaomaster1 ~]# chmod +x /etc/keepalived/check_nginx.sh
备keepalive
[root@master2 ~]# cat /etc/keepalived/keepalived.conf global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id NGINX_BACKUP } vrrp_script check_nginx { script "/etc/keepalived/check_nginx.sh" } vrrp_instance VI_1 { state BACKUP interface ens33 virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.1.199/24 } track_script { check_nginx } } [root@master2 ~]# cat /etc/keepalived/check_nginx.sh #!/bin/bash count=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$") if [ "$count" -eq 0 ];then systemctl stop keepalived fi [root@master2 ~]# chmod +x /etc/keepalived/check_nginx.sh
#注:keepalived根据脚本返回状态码(0为工作正常,非0不正常)判断是否故障转移。
启动服务
[root@master1 ~]# systemctl daemon-reload [root@master1 ~]# systemctl start nginx [root@master1 ~]# systemctl start keepalived [root@master1 ~]# systemctl enable nginx keepalived [root@master2 ~]# systemctl daemon-reload [root@master2 ~]# systemctl start nginx [root@master2 ~]# systemctl start keepalived [root@master2 ~]# systemctl enable nginx keepalived
测试vip是否绑定成功
[root@master1 ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:79:9e:36 brd ff:ff:ff:ff:ff:ff inet 192.168.1.180/24 brd 192.168.40.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.1.199/24 scope global secondary ens33 valid_lft forever preferred_lft forever inet6 fe80::b6ef:8646:1cfc:3e0c/64 scope link noprefixroute valid_lft forever preferred_lft forever
测试keepalived
停掉master1上的nginx。vip会漂移到master2
[root@master1 ~]# service nginx stop
目前所有的Worker Node组件连接都还是xianchaomaster1 Node,如果不改为连接VIP走负载均衡器,那么Master还是单点故障。
因此接下来就是要改所有Worker Node(kubectl get node命令查看到的节点)组件配置文件,由原来192.168.40.180修改为192.168.40.199(VIP)。
在所有Worker Node执行:
[root@node1 ~]# sed -i 's#192.168.40.180:6443#192.168.1199:16443#' /etc/kubernetes/kubelet-bootstrap.kubeconfig [root@node1 ~]# sed -i 's#192.168.40.180:6443#192.168.1.199:16443#' /etc/kubernetes/kubelet.json [root@node1 ~]# sed -i 's#192.168.40.180:6443#192.168.1.199:16443#' /etc/kubernetes/kubelet.kubeconfig [root@node1 ~]# sed -i 's#192.168.40.180:6443#192.168.1.199:16443#' /etc/kubernetes/kube-proxy.yaml [root@node1 ~]# sed -i 's#192.168.40.180:6443#192.168.1.199:16443#' /etc/kubernetes/kube-proxy.kubeconfig [root@node1 ~]# systemctl restart kubelet kube-proxy
这样高可用集群就安装好了

评论