TiDB on KubeSphere: Using Cloud-Native Distributed Database on Kubernetes Platform Tailored for Hybrid Cloud
In a world where Kubernetes has become the de facto standard to build application services that span multiple containers, running a cloud-native distributed database represents an important part of the experience of using Kubernetes. In this connection, TiDB, a cloud-native, open-source NewSQL database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads, meets those needs admirably. Its architecture is suitable for Kubernetes, and it is MySQL compatible. TiDB also features horizontal scalability, strong consistency, and high availability.
In addition to TiDB, I am also using KubeSphere Container Platform, an open-source distributed operating system that manages cloud-native applications with Kubernetes as its kernel. It provides a plug-and-play architecture for the seamless integration of third-party applications to boost its ecosystem. KubeSphere can be run anywhere as it is highly pluggable without any hacking into Kubernetes.
By combining TiDB with KubeSphere, we can have Kubernetes-powered TiDB clusters, and manage the clusters using a developer-friendly wizard web UI. In this post, I will demonstrate how to deploy TiDB on KubeSphere from scratch.
Preparing Environments
As you can imagine, the very first thing to consider is to have a Kubernetes cluster so that you can deploy TiDB. Well, in this regard, the installation of Kubernetes may have haunted a large number of neophytes, especially the preparation of working machines, either physical or virtual. Besides, you also need to configure different network rules so that traffic can move smoothly among instances. Fortunately, QingCloud, the sponsor of KubeSphere, provides users with a highly functional platform that enables them to quickly deploy Kubernetes and KubeSphere at the same time (you can choose to deploy Kubernetes only). Namely, you only need to click few buttons and the platform will do the rest.
Therefore, I select QingCloud Kubernetes Engine (QKE) to prepare the environment. In fact, you can also use instances on the platform directly and deploy a highly-available Kubernetes cluster with KubeSphere installed. Here is how I deploy the cluster and TiDB:
Log in to the web console of QingCloud. Simply select KubeSphere (QKE) from the menu and create a Kubernetes cluster with KubeSphere installed. The platform allows you to install different components of KubeSphere. Here, we need to enable OpenPitrix, which powers the app management feature in KubeSphere.
Note
KubeSphere can be installed on any infrastructure. I just use QingCloud Platform as an example. See KubeSphere Documentation for more details.The cluster will be up and running in around 10 minutes. In this example, I select 3 working nodes to make sure I have enough resources for the deployment later. You can also customize configurations based on your needs. When the cluster is ready, log in to the web console of KubeSphere with the default account and password (
admin/P@88w0rd
). Here is the cluster Overview page:Use the built-in Web Kubectl from the Toolkit in the bottom-right corner to execute the following command to install TiDB Operator CRD:
kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/v1.1.6/manifests/crd.yaml
You can see the expected output as below:
customresourcedefinition.apiextensions.k8s.io/tidbclusters.pingcap.com created customresourcedefinition.apiextensions.k8s.io/backups.pingcap.com created customresourcedefinition.apiextensions.k8s.io/restores.pingcap.com created customresourcedefinition.apiextensions.k8s.io/backupschedules.pingcap.com created customresourcedefinition.apiextensions.k8s.io/tidbmonitors.pingcap.com created customresourcedefinition.apiextensions.k8s.io/tidbinitializers.pingcap.com created customresourcedefinition.apiextensions.k8s.io/tidbclusterautoscalers.pingcap.com created
Now, let's get back to the Access Control page where all the workspaces are listed. Before I proceed, first I need to create a new workspace (e.g.
dev-workspace
).In a workspace, different users have different permissions to perform varied tasks in projects. Usually, a department-wide project requires a multi-tenant system so that everyone is responsible for their own part. For demonstration purposes, I use the user
admin
in this example. You can see the official documentation of KubeSphere to know more about how the multi-tenant system works.Go to a workspace, and you can see that KubeSphere provides two methods to add Helm charts. This article only talks about the first one, which is adding an app repository in KubeSphere. Alternatively, you can also upload your own app templates and submit them to the KubeSphere App Store, which will be talked about in my next blog. Here, follow the images below to add the Helm repository of PingCap (https://charts.pingcap.org) in App Repos.
Deploying TiDB Operator
TiDB Operator is an automatic operation system for TiDB clusters in Kubernetes. It provides a full management life-cycle for TiDB, including deployment, upgrades, scaling, backup, fail-over, and configuration changes. With TiDB Operator, TiDB can run seamlessly in Kubernetes clusters deployed on public or private clouds. You need to first deploy TiDB Operator on KubeSphere.
Like I mentioned above, we need to create a project (i.e. namespace) first to run TiDB Operator.
After the project is created, navigate to Applications and click Deploy New Application.
Select From App Templates.
Switch to the PingCAP repository that stores multiple Helm charts. This article only demonstrates how to deploy TiDB Operator and TiDB clusters. You can also deploy other tools based on your needs.
Click tidb-operator and select Chart Files. You can view the configuration from the console directly or download the default
values.yaml
file. From the drop-down menu on the right, you can also select the version you want to install.Confirm your app name, version and deployment location.
You can edit the
values.yaml
file in this step, or click Deploy directly with the default configurations.In Applications, wait for TiDB Operator to be up and running.
In Workloads, you can see two Deployments created for TiDB Operator.
Deploying a TiDB Cluster
The process of deploying a TiDB cluster is similar to deploying TiDB Operator.
Also from the PingCAP repository, select tidb-cluster.
Switch to Chart Files and download the values.yaml file.
Some TiDB components require persistent volumes. In this regard, QingCloud Platform provides users with the following storage classes.
/ # kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE csi-high-capacity-legacy csi-qingcloud Delete Immediate true 71m csi-high-perf csi-qingcloud Delete Immediate true 71m csi-ssd-enterprise csi-qingcloud Delete Immediate true 71m csi-standard (default) csi-qingcloud Delete Immediate true 71m csi-super-high-perf csi-qingcloud Delete Immediate true 71m
As I installed KubeSphere through QingCloud Kubernetes Engine (QKE), all of these storage components were deployed automatically by default. The QingCloud CSI plugin implements an interface between CSI-enabled Container Orchestrator (CO) and the storage of QingCloud. If you are interested in QingCloud CSI, have a look at their GitHub repository. Select csi-standard here by replacing the default value
local-storage
of the fieldstorageClassName
withcsi-standard
invalues.yaml
. In the downloaded file, you can replace all of them directly and copy and paste it to thevalues.yaml
file.Note
Only the fieldstorageClassName
is changed to provide external persistent storage. If you want to deploy each TiDB component, such as TiKV and Placement Driver, to individual nodes, specify the fieldnodeAffinity
.Click Deploy and you can see two apps in the list as shown below:
Viewing TiDB Cluster Status
Now that we have our apps ready, we may need to focus more on observability. KubeSphere gives users a straightforward view of how apps are doing during their whole lifecycle with different metrics available on the dashboard.
Verify that all TiDB cluster Deployments are up and running in Workloads.
TiDB, TiKV and PD are all stateful applications, which can be found in StatefulSets. Note that TiKV and TiDB will be created automatically and it may take a while before they display in the list.
Click a single StatefulSet to display its detail page. This page shows metrics in line charts over a period of time. The following is an example of TiDB metrics:
View TiKV loads:
Relevant Pods are also listed. As you can see, your TiDB cluster contains three PD Pods, two TiDB Pods and three TiKV Pods.
Go to the Storage section, and you can see TiKV and PD are using persistent storage.
Volume usage is also monitored. Here is an example of TiKV:
On the Overview page, you can see a list of resource usage in the current project.
Accessing TiDB Cluster
These services just created can be accessed easily as KubeSphere tells you how a service is being exposed on which port.
In Services, you can see detailed information of all services.
As the service type is set to
NodePort
, you can access it through the Node IP address outside the cluster.Here is a test of the connection to the database through MySQL client.
[root@k8s-master1 ~]# docker run -it --rm mysql bash [root@0d7cf9d2173e:/# mysql -h 192.168.1.102 -P 32682 -u root Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 201 Server version: 5.7.25-TiDB-v4.0.6 TiDB Server (Apache License 2.0) Community Edition, MySQL 5.7 compatible Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | INFORMATION_SCHEMA | | METRICS_SCHEMA | | PERFORMANCE_SCHEMA | | mysql | | test | +--------------------+ 5 rows in set (0.01 sec) mysql>
Besides, TiDB integrates Prometheus and Grafana to monitor performance of the database cluster. As we can see above, Grafana is being exposed through
NodePort
. After you configure necessary port forwarding rules and open its port in security groups on QingCloud Platform, you can access the Grafana UI to view metrics.
Summary
I hope you guys all have successfully deployed TiDB. Both TiDB and KubeSphere are powerful tools for cloud-native applications, so in fact, I cannot showcase every aspect of them all in this post. For example, the app deployment function has much to offer for cloud-native enthusiasts like me. I will post another article on how to deploy TiDB by uploading Helm charts to the KubeSphere App Store.
If you have any questions, don't hesitate to contact us in Slack or GitHub.
References
KubeSphere GitHub: https://github.com/kubesphere/kubesphere
TiDB GitHub: https://github.com/pingcap/TiDB
TiDB Operator Documentation: https://docs.pingcap.com/tidb-in-kubernetes/stable/tidb-operator-overview
KubeSphere Introduction: https://kubesphere.io/docs/introduction/what-is-kubesphere/
KubeSphere Documentation: https://kubesphere.io/docs/