在 KubeSphere 中快速安装部署 Nacos
作者:老 Z,运维架构师,云原生爱好者,目前专注于云原生运维,云原生领域技术栈涉及 Kubernetes、KubeSphere、DevOps、OpenStack、Ansible 等。
简介
Nacos 集群如何在 K8s 集群上部署?Nacos 依赖的 MySQL 数据库如何在 K8S 集群上部署?GitOps 在 K8s 集群上是一种什么体验?本文将带你全面了解上述问题。
演示服务器配置
主机名 | IP | CPU | 内存 | 系统盘 | 数据盘 | 用途 |
---|---|---|---|---|---|---|
zdeops-master | 192.168.9.9 | 2 | 4 | 40 | 200 | Ansible 运维控制节点 |
ks-k8s-master-0 | 192.168.9.91 | 4 | 16 | 40 | 200+200 | KubeSphere/k8s-master/k8s-worker/Ceph |
ks-k8s-master-1 | 192.168.9.92 | 4 | 16 | 40 | 200+200 | KubeSphere/k8s-master/k8s-worker/Ceph |
ks-k8s-master-2 | 192.168.9.93 | 4 | 16 | 40 | 200+200 | KubeSphere/k8s-master/k8s-worker/Ceph |
glusterfs-node-0 | 192.168.9.95 | 2 | 8 | 40 | 200 | GlusterFS |
glusterfs-node-1 | 192.168.9.96 | 2 | 8 | 40 | 200 | GlusterFS |
glusterfs-node-2 | 192.168.9.97 | 2 | 8 | 40 | 200 | GlusterFS |
harbor | 192.168.9.89 | 2 | 8 | 40 | 200 | Harbor |
合计 | 8 | 22 | 84 | 320 | 2200 |
演示环境涉及软件版本信息
- 操作系统:CentOS-7.9-x86_64
- Ansible:2.8.20
- Harbor:2.5.1
- Nacos:v2.1.0
- MySQL:5.7.38
前提条件
准备离线镜像
此过程为可选项,离线内网环境可用,如果是直连互联网的场景,可以直接使用互联网的镜像。
在一台能同时访问互联网和内网 Harbor 仓库的服务器上进行下面的操作。
- 下载镜像
docker pull mysql:5.7.38
docker pull nacos/nacos-peer-finder-plugin:1.1
docker pull nacos/nacos-server:v2.1.0
- 重新打 tag
docker tag mysql:5.7.38 registry.zdevops.com.cn/library/mysql:5.7.38
docker tag nacos/nacos-peer-finder-plugin:1.1 registry.zdevops.com.cn/nacos/nacos-peer-finder-plugin:1.1
docker tag nacos/nacos-server:v2.1.0 registry.zdevops.com.cn/nacos/nacos-server:v2.1.0
- 推送到私有镜像仓库
# 需要提前在镜像仓库中创建 Harbor 项目
docker push registry.zdevops.com.cn/library/mysql:5.7.38
docker push registry.zdevops.com.cn/nacos/nacos-peer-finder-plugin:1.1
docker push registry.zdevops.com.cn/nacos/nacos-server:v2.1.0
- 清理临时镜像
docker rmi mysql:5.7.38
docker rmi nacos/nacos-peer-finder-plugin:1.1
docker rmi nacos/nacos-server:v2.1.0
docker rmi registry.zdevops.com.cn/library/mysql:5.7.38
docker rmi registry.zdevops.com.cn/nacos/nacos-peer-finder-plugin:1.1
docker rmi registry.zdevops.com.cn/nacos/nacos-server:v2.1.0
准备 Nacos 部署资源
- 官方资源配置清单
# 拉取官方的配置清单,各位参考着进行修改,本文不涉及修改过程
git clone https://github.com/nacos-group/nacos-k8s.git
- 初始化数据库文件
wget https://raw.githubusercontent.com/alibaba/nacos/develop/distribution/conf/nacos-mysql.sql
部署 MySQL
Nacos 需要使用 MySQL 存储配置数据,由于使用量不大,没有考虑高可用部署,直接在 K8s 上部署。也可以采用已有的 MySQL 数据库。
资源配置清单
- mysql-cm.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: nacos-mysql-config
namespace: zdevops
data:
custom.cnf: |-
[mysqld]
#performance setttings
lock_wait_timeout = 3600
open_files_limit = 65535
back_log = 1024
max_connections = 1024
max_connect_errors = 1000000
table_open_cache = 1024
table_definition_cache = 1024
thread_stack = 512K
sort_buffer_size = 4M
join_buffer_size = 4M
read_buffer_size = 8M
read_rnd_buffer_size = 4M
bulk_insert_buffer_size = 64M
thread_cache_size = 768
interactive_timeout = 600
wait_timeout = 600
tmp_table_size = 32M
max_heap_table_size = 32M
- mysql-secret.yaml
kind: Secret
apiVersion: v1
metadata:
name: nacos-mysql-secret
namespace: zdevops
data:
MYSQL_ROOT_PASSWORD: UEA4OHcwcmQ=
MYSQL_PASSWORD: UEA4OHcwcmQ=
type: Opaque
MYSQL_ROOT_PASSWORD 和 MYSQL_PASSWORD 是 MySQL 中 root 和 nacos 用户的密码。
密码需要使用 echo -n "P@88w0rd" | base64
加密的方式。
- mysql-sts.yaml
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: nacos-mysql
namespace: zdevops
labels:
app: nacos-mysql
spec:
replicas: 1
selector:
matchLabels:
app: nacos-mysql
template:
metadata:
labels:
app: nacos-mysql
spec:
volumes:
- name: host-time
hostPath:
path: /etc/localtime
type: ''
- name: config
configMap:
name: nacos-mysql-config
items:
- key: custom.cnf
path: custom.cnf
defaultMode: 420
containers:
- name: nacos-mysql
image: 'registry.zdevops.com.cn/library/mysql:5.7.38'
ports:
- name: tcp-3306
containerPort: 3306
protocol: TCP
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: nacos-mysql-secret
key: MYSQL_ROOT_PASSWORD
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: nacos-mysql-secret
key: MYSQL_PASSWORD
- name: MYSQL_DATABASE
value: "nacos"
- name: MYSQL_USER
value: "nacos"
resources:
limits:
cpu: '2'
memory: 4000Mi
requests:
cpu: 100m
memory: 500Mi
volumeMounts:
- name: host-time
mountPath: /etc/localtime
- name: data
mountPath: /var/lib/mysql
- name: config
mountPath: /etc/mysql/conf.d/custom.cnf
subPath: custom.cnf
volumeClaimTemplates:
- metadata:
name: data
namespace: zdevops
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: glusterfs
serviceName: mysql-headless
---
kind: Service
apiVersion: v1
metadata:
name: nacos-mysql-headless
namespace: zdevops
labels:
app: nacos-mysql
spec:
ports:
- name: tcp-3306
protocol: TCP
port: 3306
targetPort: 3306
selector:
app: nacos-mysql
clusterIP: None
type: ClusterIP
MYSQL_DATABASE 和 MYSQL_USER 是 MySQL 初始化时创建的 Nacos 数据库名称和 Nacos 用户名称。
- mysql-external.yaml
kind: Service
apiVersion: v1
metadata:
name: nacos-mysql-external
namespace: zdevops
labels:
app: nacos-mysql-external
spec:
ports:
- name: tcp-mysql-external
protocol: TCP
port: 3306
targetPort: 3306
nodePort: 31006
selector:
app: nacos-mysql
type: NodePort
GitOps
在运维开发服务器上操作:
# 创建新分支
[root@zdevops-master k8s-yaml]# git checkout -b main-nacos-062401 main
Switched to a new branch 'main-nacos-062401'
# 在已有代码仓库创建 nacos 目录
[root@zdevops-master k8s-yaml]# mkdir nacos
# 编辑资源配置清单
[root@zdevops-master k8s-yaml]# vi nacos/mysql-cm.yaml
[root@zdevops-master k8s-yaml]# vi nacos/mysql-secret.yaml
[root@zdevops-master k8s-yaml]# vi nacos/mysql-sts.yaml
[root@zdevops-master k8s-yaml]# vi nacos/mysql-external.yaml
# 提交 Git
[root@zdevops-master k8s-yaml]# git add nacos
[root@zdevops-master k8s-yaml]# git commit -am '添加 nacos mysql部署资源配置清单'
# 分支合并
[root@zdevops-master k8s-yaml]# git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
[root@zdevops-master k8s-yaml]# git merge main-nacos-062401
Updating e60e3e3..2d51a5a
Fast-forward
nacos/mysql-cm.yaml | 27 +++++++++++++++++++++++
nacos/mysql-external.yaml | 17 ++++++++++++++
nacos/mysql-secret.yaml | 9 ++++++++
nacos/mysql-sts.yaml | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
nodeport.md | 2 +-
5 files changed, 152 insertions(+), 1 deletion(-)
create mode 100644 nacos/mysql-cm.yaml
create mode 100644 nacos/mysql-external.yaml
create mode 100644 nacos/mysql-secret.yaml
create mode 100644 nacos/mysql-sts.yaml
# 删除临时分支
[root@zdevops-master k8s-yaml]# git branch -d main-nacos-062401
# 推送到远程仓库
[root@zdevops-master k8s-yaml]# git push
部署资源
在运维管理服务器上操作:
- 更新镜像仓库代码
[root@zdevops-master k8s-yaml]# git pull
- 部署资源
[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/mysql-cm.yaml
[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/mysql-secret.yaml
[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/mysql-sts.yaml
[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/mysql-external.yaml
验证
- 查看资源状态
[root@zdevops-master k8s-yaml]# kubectl get cm,secrets,pvc,sts,pods -n zdevops -o wide
NAME DATA AGE
configmap/nacos-mysql-config 1 3h10m
NAME TYPE DATA AGE
secret/nacos-mysql-secret Opaque 2 3h10m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/data-nacos-mysql-0 Bound pvc-13bec5c5-33e2-46df-bcdc-dc216c17f88a 5Gi RWO glusterfs 3h10m Filesystem
NAME READY AGE CONTAINERS IMAGES
statefulset.apps/nacos-mysql 1/1 3h10m nacos-mysql registry.zdevops.com.cn/library/mysql:5.7.38
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nacos-mysql-0 1/1 Running 0 32m 10.233.116.189 ks-k8s-master-2 <none> <none>
- MySQL 登录验证
# 进入 MySQL POD
[root@zdevops-master k8s-yaml]# kubectl exec -it nacos-mysql-0 -n zdevops -- /bin/bash
# 在 MySQL POD 内部以 root 用户连接 MySQL 服务器,并列出数据库
root@nacos-mysql-0:/# mysql -u root -pP@88w0rd
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.38 MySQL Community Server (GPL)
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| nacos |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.01 sec)
mysql> quit
Bye
# 在 MySQL POD 内部以 nacos 用户连接 MySQL 服务器,并列出数据库
root@nacos-mysql-0:/# mysql -u nacos -pP@88w0rd
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.7.38 MySQL Community Server (GPL)
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| nacos |
+--------------------+
2 rows in set (0.00 sec)
mysql> quit
Bye
导入 Nacos 初始化数据
本文采用将 SQL 文件复制到容器内部执行的方式导入数据。实际使用中,可以直接在集群外部使用 MySQL 客户端连接 MySQL 的 NodePort 管理数据库并导入数据。
- 获取初始化数据库文件
# 官方
wget https://raw.githubusercontent.com/alibaba/nacos/develop/distribution/conf/nacos-mysql.sql
# Gitee备用
# wget https://gitee.com/zdevops/k8s-yaml/raw/main/nacos/nacos-mysql.sql
- 将 SQL 文件复制到 MySQL 容器内部
# 拷贝文件
[root@zdevops-master k8s-yaml]# cd nacos
[root@zdevops-master nacos]# kubectl cp nacos-mysql.sql -n zdevops nacos-mysql-0:home/
# 确认文件成功拷贝
[root@zdevops-master nacos]# kubectl exec -it nacos-mysql-0 -n zdevops -- ls /home
- 登录容器内部,导入数据
# 进入 MySQL POD
[root@zdevops-master k8s-yaml]# kubectl exec -it nacos-mysql-0 -n zdevops -- /bin/bash
# 导入数据, 需要输入 nacos 用户的密码
root@nacos-mysql-0:/# mysql -u nacos -p nacos < /home/nacos-mysql.sql
Enter password:
集群模式 Nacos 部署
资源配置清单
- nacos-cm.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
name: nacos-config
namespace: zdevops
data:
mysql.host: "nacos-mysql-external.zdevops"
mysql.db.name: "nacos"
mysql.port: "3306"
mysql.user: "nacos"
mysql.password: "P@88w0rd"
- nacos-sts.yaml
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: nacos
namespace: zdevops
spec:
serviceName: nacos-headless
replicas: 3
template:
metadata:
labels:
app: nacos
annotations:
pod.alpha.kubernetes.io/initialized: "true"
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- nacos
topologyKey: "kubernetes.io/hostname"
initContainers:
- name: peer-finder-plugin-install
image: registry.zdevops.com.cn/nacos/nacos-peer-finder-plugin:1.1
imagePullPolicy: Always
volumeMounts:
- mountPath: /home/nacos/plugins/peer-finder
name: data
subPath: peer-finder
containers:
- name: nacos
imagePullPolicy: Always
image: registry.zdevops.com.cn/nacos/nacos-server:v2.1.0
resources:
limits:
cpu: '2'
memory: 4Gi
requests:
cpu: "100m"
memory: "1Gi"
ports:
- containerPort: 8848
name: client-port
- containerPort: 9848
name: client-rpc
- containerPort: 9849
name: raft-rpc
- containerPort: 7848
name: old-raft-rpc
env:
- name: NACOS_REPLICAS
value: "3"
- name: SERVICE_NAME
value: "nacos-headless"
- name: DOMAIN_NAME
value: "cluster.local"
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: MYSQL_SERVICE_HOST
valueFrom:
configMapKeyRef:
name: nacos-config
key: mysql.host
- name: MYSQL_SERVICE_DB_NAME
valueFrom:
configMapKeyRef:
name: nacos-config
key: mysql.db.name
- name: MYSQL_SERVICE_PORT
valueFrom:
configMapKeyRef:
name: nacos-config
key: mysql.port
- name: MYSQL_SERVICE_USER
valueFrom:
configMapKeyRef:
name: nacos-config
key: mysql.user
- name: MYSQL_SERVICE_PASSWORD
valueFrom:
configMapKeyRef:
name: nacos-config
key: mysql.password
- name: NACOS_SERVER_PORT
value: "8848"
- name: NACOS_APPLICATION_PORT
value: "8848"
- name: PREFER_HOST_MODE
value: "hostname"
volumeMounts:
- name: data
mountPath: /home/nacos/plugins/peer-finder
subPath: peer-finder
- name: data
mountPath: /home/nacos/data
subPath: data
- name: data
mountPath: /home/nacos/logs
subPath: logs
volumeClaimTemplates:
- metadata:
name: data
namespace: zdevops
spec:
accessModes: [ "ReadWriteMany" ]
storageClassName: "glusterfs"
resources:
requests:
storage: 5Gi
selector:
matchLabels:
app: nacos
---
apiVersion: v1
kind: Service
metadata:
name: nacos-headless
namespace: zdevops
labels:
app: nacos
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
ports:
- port: 8848
name: server
targetPort: 8848
- port: 9848
name: client-rpc
targetPort: 9848
- port: 9849
name: raft-rpc
targetPort: 9849
## 兼容1.4.x版本的选举端口
- port: 7848
name: old-raft-rpc
targetPort: 7848
clusterIP: None
selector:
app: nacos
DOMAIN_NAME 要改成实际的 K8s 的集群域名。
- nacos-external.yaml
---
apiVersion: v1
kind: Service
metadata:
name: nacos-external
namespace: zdevops
labels:
app: external
spec:
type: NodePort
ports:
- name: tcp-nacos-external
protocol: TCP
port: 8848
targetPort: 8848
nodePort: 31848
selector:
app: nacos
GitOps
在运维开发服务器上操作:
# 创建新分支
[root@zdevops-master k8s-yaml]# git checkout -b main-nacos-062402 main
Switched to a new branch 'main-nacos-062402'
# 编辑资源配置清单
[root@zdevops-master k8s-yaml]# vi nacos/nacos-cm.yaml
[root@zdevops-master k8s-yaml]# vi nacos/nacos-sts.yaml
[root@zdevops-master k8s-yaml]# vi nacos/nacos-external.yaml
# 提交 Git
[root@zdevops-master k8s-yaml]# git add nacos
[root@zdevops-master k8s-yaml]# git commit -am '添加nacos部署资源配置清单'
# 分支合并
[root@zdevops-master k8s-yaml]# git checkout main
[root@zdevops-master k8s-yaml]# git merge main-nacos-062402
# 删除临时分支
[root@zdevops-master k8s-yaml]# git branch -d main-nacos-062402
# 推送到远程仓库
[root@zdevops-master k8s-yaml]# git push
部署资源
在运维管理服务器上操作
- 更新镜像仓库代码
[root@zdevops-master k8s-yaml]# git pull
- 部署资源
[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/nacos-cm.yaml
[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/nacos-sts.yaml
[root@zdevops-master k8s-yaml]# kubectl apply -f nacos/nacos-external.yaml
验证
- 资源验证
# sts
[root@zdevops-master k8s-yaml]# kubectl get sts -n zdevops -o wide
NAME READY AGE CONTAINERS IMAGES
nacos 3/3 3h13m nacos registry.zdevops.com.cn/nacos/nacos-server:v2.1.0
nacos-mysql 1/1 26h nacos-mysql registry.zdevops.com.cn/library/mysql:5.7.38
# pods
[root@zdevops-master k8s-yaml]# kubectl get pods -n zdevops -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nacos-0 1/1 Running 0 3h12m 10.233.116.220 ks-k8s-master-2 <none> <none>
nacos-1 1/1 Running 0 3h12m 10.233.117.101 ks-k8s-master-0 <none> <none>
nacos-2 1/1 Running 0 3h12m 10.233.87.14 ks-k8s-master-1 <none> <none>
nacos-mysql-0 1/1 Running 0 23h 10.233.116.189 ks-k8s-master-2 <none> <none>
# svc
[root@zdevops-master k8s-yaml]# kubectl get svc -n zdevops -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
glusterfs-dynamic-13bec5c5-33e2-46df-bcdc-dc216c17f88a ClusterIP 10.233.48.177 <none> 1/TCP 26h <none>
glusterfs-dynamic-39795af9-8307-408c-ba2a-d0648f658f1b ClusterIP 10.233.31.22 <none> 1/TCP 3h12m <none>
glusterfs-dynamic-cc4be907-9ef3-4a69-92d7-1875f7051e36 ClusterIP 10.233.34.82 <none> 1/TCP 3h12m <none>
glusterfs-dynamic-e73324a8-3f97-4697-a7dc-67576863f21a ClusterIP 10.233.27.2 <none> 1/TCP 3h12m <none>
nacos-external NodePort 10.233.63.221 <none> 8848:30848/TCP 3h12m app=nacos
nacos-headless ClusterIP None <none> 8848/TCP,9848/TCP,9849/TCP,7848/TCP 3h12m app=nacos
nacos-mysql-external NodePort 10.233.12.228 <none> 3306:31006/TCP 23h app=nacos-mysql
nacos-mysql-headless ClusterIP None <none> 3306/TCP 26h app=nacos-mysql
- 登录 Nacos 管理控制台
在浏览器中输入 http:// 任意节点 IP:30848,输入默认的用户名和密码,nacos。
初始化配置
- 修改默认密码
- 后续配置
总结
本文详细介绍了 Nacos 集群模式在基于 KubeSphere 部署的 K8s 集群上的安装部署方法。同时,介绍了 MySQL 在 K8s 上的安装部署方法。
本文的配置方案可直接用于开发测试环境,对于生产环境也有一定的借鉴意义。