< img height="1" width="1" style="display:none;" alt="" src="https://px.ads.linkedin.com/collect/?pid=3131724&fmt=gif" />

Install Ceph

With a Ceph server, you can choose Ceph RBD or Ceph CSI as the underlying storage plugin. Ceph RBD is an in-tree storage plugin on Kubernetes, and Ceph CSI is a Container Storage Interface (CSI) driver for RBD, CephFS.

Which plugin to select for Ceph

Ceph CSI RBD is the preferred choice if you work with 14.0.0 (Nautilus)+ Ceph cluster. Here are some reasons:

  • The in-tree plugin will be deprecated in the future.
  • Ceph RBD only works on Kubernetes with hyperkube images, and hyperkube images were deprecated since Kubernetes 1.17.
  • Ceph CSI has more features such as cloning, expanding and snapshots.


Ceph-CSI needs to be installed on v1.14.0+ Kubernetes, and work with 14.0.0 (Nautilus)+ Ceph Cluster. For details about compatibility, see Ceph CSI Support Matrix.

The following is an example of KubeKey add-on configurations for Ceph CSI RBD installed by Helm Charts. As the StorageClass is not included in the chart, a StorageClass needs to be configured in the add-on config.

Chart configurations

  - clusterID: "cluster1"
      - ""     # <--TobeReplaced-->
      - ""     # <--TobeReplaced-->
      - ""    # <--TobeReplaced-->

If you want to configure more values, see chart configuration for ceph-csi-rbd.

StorageClass (including secret)

apiVersion: v1
kind: Secret
  name: csi-rbd-secret
  namespace: kube-system
  userID: admin
  userKey: "AQDoECFfYD3DGBAAm6CPhFS8TQ0Hn0aslTlovw=="    # <--ToBeReplaced-->
apiVersion: storage.k8s.io/v1
kind: StorageClass
   name: csi-rbd-sc
     storageclass.beta.kubernetes.io/is-default-class: "true"
     storageclass.kubesphere.io/supported-access-modes: '["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]'
provisioner: rbd.csi.ceph.com
   clusterID: "cluster1"
   pool: "rbd"    # <--ToBeReplaced-->
   imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: kube-system
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: kube-system
   csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
   - discard

Add-on configurations

Save the above chart config and StorageClass locally (for example, /root/ceph-csi-rbd.yaml and /root/ceph-csi-rbd-sc.yaml). The add-on configuration can be set like:

- name: ceph-csi-rbd
  namespace: kube-system
      name: ceph-csi-rbd
      repo: https://ceph.github.io/csi-charts
      valuesFile: /root/ceph-csi-rbd.yaml
- name: ceph-csi-rbd-sc
      - /root/ceph-csi-rbd-sc.yaml

Ceph RBD

KubeKey will never use hyperkube images. Hence, in-tree Ceph RBD may not work on Kubernetes installed by KubeKey. However, if your Ceph cluster is lower than 14.0.0 which means Ceph CSI can't be used, rbd provisioner can be used as a substitute for Ceph RBD. Its format is the same with in-tree Ceph RBD. The following is an example of KubeKey add-on configurations for rbd provisioner installed by Helm Charts including a StorageClass.

Chart configurations

  mon: ""    # <--ToBeReplaced-->
  adminKey: "QVFBS1JkdGRvV0lySUJBQW5LaVpSKzBRY2tjWmd6UzRJdndmQ2c9PQ=="   # <--ToBeReplaced-->
  userKey: "QVFBS1JkdGRvV0lySUJBQW5LaVpSKzBRY2tjWmd6UzRJdndmQ2c9PQ=="    # <--ToBeReplaced-->
  isDefault: false

If you want to configure more values, see chart configuration for rbd-provisioner.

Add-on configurations

Save the above chart config locally (for example, /root/rbd-provisioner.yaml). The add-on config for rbd provisioner could be like:

- name: rbd-provisioner
  namespace: kube-system
      name: rbd-provisioner
      repo: https://charts.kubesphere.io/test
      valuesFile: /root/rbd-provisioner.yaml

Receive the latest news, articles and updates from KubeSphere

Thanks for the feedback. If you have a specific question about how to use KubeSphere, ask it on Slack. Open an issue in the GitHub repo if you want to report a problem or suggest an improvement.