Recently a colleague of mine was trying to get Rook to deploy a Ceph cluster that used dedicated public and private networks to segment the Ceph replication traffic and the client access traffic to the OSDs of the cluster. In a regular Ceph deployment this is rather trivial but when in the context of Kubernetes it becomes a little more complex given that Rook is deploying the cluster containers. The following is procedure I applied to ensure my OSDs were listening on the appropriate networks.
Before we get into the steps on how to achieve this configuration lets quick take a look at the setup I used. First I have a three node Kubernetes configuration (1 master with allowed scheduling and two workers):
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-master Ready master 2d22h v1.14.0
kube-node1 Ready worker 2d22h v1.14.0
kube-node2 Ready worker 2d22h v1.14.0
On each of the nodes I have 3 network interfaces: eth0 on 10.0.0.0/24 (Kubernetes public), eth1 on 192.168.100.0/24 (Ceph private/cluster) & eth2 on 192.168.200.0/24 (Ceph public):
# ip a|grep eth[0-2]
2: eth0:mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 10.0.0.81/24 brd 10.0.0.255 scope global noprefixroute eth0
3: eth1:mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 192.168.100.81/24 brd 192.168.100.255 scope global noprefixroute eth1
4: eth2:mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 192.168.200.81/24 brd 192.168.200.255 scope global noprefixroute eth2
Before we begin lets see the current vanilla pods and namespaces on the Kubernetes cluster:
# kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-fb8b8dccf-h6wfn 1/1 Running 0 3d 10.244.1.2 kube-node2
kube-system coredns-fb8b8dccf-mv7p5 1/1 Running 0 3d 10.244.0.7 kube-master
kube-system etcd-kube-master 1/1 Running 0 3d 10.0.0.81 kube-master
kube-system kube-apiserver-kube-master 1/1 Running 0 3d 10.0.0.81 kube-master
kube-system kube-controller-manager-kube-master 1/1 Running 1 3d 10.0.0.81 kube-master
kube-system kube-flannel-ds-amd64-szhg9 1/1 Running 0 3d 10.0.0.83 kube-node2
kube-system kube-flannel-ds-amd64-t4fxs 1/1 Running 0 3d 10.0.0.82 kube-node1
kube-system kube-flannel-ds-amd64-wbsdp 1/1 Running 0 3d 10.0.0.81 kube-master
kube-system kube-proxy-sn7j7 1/1 Running 0 3d 10.0.0.83 kube-node2
kube-system kube-proxy-wtzm5 1/1 Running 0 3d 10.0.0.81 kube-master
kube-system kube-proxy-xlwd9 1/1 Running 0 3d 10.0.0.82 kube-node1
kube-system kube-scheduler-kube-master 1/1 Running 1 3d 10.0.0.81 kube-master
# kubectl get ns
NAME STATUS AGE
default Active 3d
kube-node-lease Active 3d
kube-public Active 3d
kube-system Active 3d
Before can deploy the cluster we need to create a configmap for the rook-ceph namespace. This namespace is normally created when the cluster is deployed however we want specific configuration items to be incorporated into the cluster upon deployment and so to do this we will create the rook-ceph namespace and apply a configmap that we create to that namespace.
First create a configmap file that looks like the following and notice I am referencing my Ceph cluster networks. I will save this file with an arbitrary name like config-override.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: rook-config-override
namespace: rook-ceph
data:
config: |
[global]
public network = 192.168.200.0/24
cluster network = 192.168.100.0/24
public addr = ""
cluster addr = ""
Next I will create the rook-ceph namespace:
# kubectl create namespace rook-ceph
namespace/rook-ceph created
# kubectl get ns
NAME STATUS AGE
default Active 3d1h
kube-node-lease Active 3d1h
kube-public Active 3d1h
kube-system Active 3d1h
rook-ceph Active 5s
Now we can apply the configmap we created to the newly created namespace and validate its there:
# kubectl create -f config-override.yaml
configmap/rook-config-override created
# kubectl get configmap -n rook-ceph
NAME DATA AGE
rook-config-override 1 66s
# kubectl describe configmap -n rook-ceph
Name: rook-config-override
Namespace: rook-ceph
Labels: <none>
Annotations: <none>
Data
====
config:
----
[global]
public network = 192.168.200.0/24
cluster network = 192.168.100.0/24
public addr = ""
cluster addr = ""
Events: <none>
Before we actually start to do the deploy we need to update one more thing in our Rook cluster.yaml. Inside the cluster.yaml file we need to change hostNetwork from the default of false to true:
sed -i 's/hostNetwork: false/hostNetwork: true/g' cluster.yaml
Now we can begin the process of deploying the Rook/Ceph cluster that includes launching the operator, cluster and toolbox. I will place sleep statements in between each command to ensure the pods are up before I run the next command. Also note there will be an error when creating the cluster about the rook-ceph namespace already existing and this is normal:
# kubectl create -f operator.yaml
namespace/rook-ceph-system created
customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created
clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
role.rbac.authorization.k8s.io/rook-ceph-system created
clusterrole.rbac.authorization.k8s.io/rook-ceph-global created
clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
serviceaccount/rook-ceph-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-system created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created
deployment.apps/rook-ceph-operator created
# sleep 60
# kubectl create -f cluster.yaml
serviceaccount/rook-ceph-osd created
serviceaccount/rook-ceph-mgr created
role.rbac.authorization.k8s.io/rook-ceph-osd created
role.rbac.authorization.k8s.io/rook-ceph-mgr-system created
role.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
cephcluster.ceph.rook.io/rook-ceph created
Error from server (AlreadyExists): error when creating "cluster.yaml": namespaces "rook-ceph" already exists
# sleep 60
# kubectl create -f toolbox.yaml
pod/rook-ceph-tools created
Lets validate the Rook/Ceph operator, cluster and toolbox is up and running:
# kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-fb8b8dccf-h6wfn 1/1 Running 0 3d1h 10.244.1.2 kube-node2 <none> <none>
kube-system coredns-fb8b8dccf-mv7p5 1/1 Running 0 3d1h 10.244.0.7 kube-master <none> <none>
kube-system etcd-kube-master 1/1 Running 0 3d1h 10.0.0.81 kube-master <none> <none>
kube-system kube-apiserver-kube-master 1/1 Running 0 3d1h 10.0.0.81 kube-master <none> <none>
kube-system kube-controller-manager-kube-master 1/1 Running 1 3d1h 10.0.0.81 kube-master <none> <none>
kube-system kube-flannel-ds-amd64-szhg9 1/1 Running 0 3d1h 10.0.0.83 kube-node2 <none> <none>
kube-system kube-flannel-ds-amd64-t4fxs 1/1 Running 0 3d1h 10.0.0.82 kube-node1 <none> <none>
kube-system kube-flannel-ds-amd64-wbsdp 1/1 Running 0 3d1h 10.0.0.81 kube-master <none> <none>
kube-system kube-proxy-sn7j7 1/1 Running 0 3d1h 10.0.0.83 kube-node2 <none> <none>
kube-system kube-proxy-wtzm5 1/1 Running 0 3d1h 10.0.0.81 kube-master <none> <none>
kube-system kube-proxy-xlwd9 1/1 Running 0 3d1h 10.0.0.82 kube-node1 <none> <none>
kube-system kube-scheduler-kube-master 1/1 Running 1 3d1h 10.0.0.81 kube-master <none> <none>
rook-ceph-system rook-ceph-agent-55fqp 1/1 Running 0 17m 10.0.0.83 kube-node2 <none> <none>
rook-ceph-system rook-ceph-agent-5v9v5 1/1 Running 0 17m 10.0.0.81 kube-master <none> <none>
rook-ceph-system rook-ceph-agent-spx29 1/1 Running 0 17m 10.0.0.82 kube-node1 <none> <none>
rook-ceph-system rook-ceph-operator-57547fc866-ltp8z 1/1 Running 0 18m 10.244.2.4 kube-node1 <none> <none>
rook-ceph-system rook-discover-brxmt 1/1 Running 0 17m 10.244.2.5 kube-node1 <none> <none>
rook-ceph-system rook-discover-hl748 1/1 Running 0 17m 10.244.1.8 kube-node2 <none> <none>
rook-ceph-system rook-discover-qj5kd 1/1 Running 0 17m 10.244.0.9 kube-master <none> <none>
rook-ceph rook-ceph-mgr-a-5dbb44d7f8-vzs46 1/1 Running 0 16m 10.0.0.82 kube-node1 <none> <none>
rook-ceph rook-ceph-mon-a-5fb9568cb4-gvqln 1/1 Running 0 16m 10.0.0.81 kube-master <none> <none>
rook-ceph rook-ceph-mon-b-b65c555bf-vz7ps 1/1 Running 0 16m 10.0.0.82 kube-node1 <none> <none>
rook-ceph rook-ceph-mon-c-69cf744c4d-8g4l6 1/1 Running 0 16m 10.0.0.83 kube-node2 <none> <none>
rook-ceph rook-ceph-osd-0-77499f547-d2vjx 1/1 Running 0 15m 10.0.0.81 kube-master <none> <none>
rook-ceph rook-ceph-osd-1-698f76d786-lqn4w 1/1 Running 0 15m 10.0.0.82 kube-node1 <none> <none>
rook-ceph rook-ceph-osd-2-558c59d577-wfdlr 1/1 Running 0 15m 10.0.0.83 kube-node2 <none> <none>
rook-ceph rook-ceph-osd-prepare-kube-master-p55sw 0/2 Completed 0 15m 10.0.0.81 kube-master <none> <none>
rook-ceph rook-ceph-osd-prepare-kube-node1-q7scn 0/2 Completed 0 15m 10.0.0.82 kube-node1 <none> <none>
rook-ceph rook-ceph-osd-prepare-kube-node2-8rm4d 0/2 Completed 0 15m 10.0.0.83 kube-node2 <none> <none>
rook-ceph rook-ceph-tools 1/1 Running 0 3m24s 10.244.1.9 kube-node2 <none> <none>
# kubectl -n rook-ceph exec -it rook-ceph-tools -- /bin/bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-tools /]# ceph status
cluster:
id: b58f2a5c-2fc7-43e7-b410-2d541e78a90e
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c
mgr: a(active)
osd: 3 osds: 3 up, 3 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 57 GiB used, 49 GiB / 105 GiB avail
pgs:
[root@rook-ceph-tools /]# exit
exit
At this point we have a fully operational cluster but is it really using the networks for OSD public and private traffic? Lets explore that a bit further by first running the netstat command on any node in the cluster that has an OSD pod running. Since my cluster is small I will show all 3 nodes below:
[root@kube-master]# netstat -tulpn | grep LISTEN | grep osd
tcp 0 0 192.168.100.81:6800 0.0.0.0:* LISTEN 29719/ceph-osd
tcp 0 0 192.168.200.81:6800 0.0.0.0:* LISTEN 29719/ceph-osd
tcp 0 0 192.168.200.81:6801 0.0.0.0:* LISTEN 29719/ceph-osd
tcp 0 0 192.168.100.81:6801 0.0.0.0:* LISTEN 29719/ceph-osd
[root@kube-node1]# netstat -tulpn | grep LISTEN | grep osd
tcp 0 0 192.168.100.82:6800 0.0.0.0:* LISTEN 18770/ceph-osd
tcp 0 0 192.168.100.82:6801 0.0.0.0:* LISTEN 18770/ceph-osd
tcp 0 0 192.168.200.82:6801 0.0.0.0:* LISTEN 18770/ceph-osd
tcp 0 0 192.168.200.82:6802 0.0.0.0:* LISTEN 18770/ceph-osd
[root@kube-node2]# netstat -tulpn | grep LISTEN | grep osd
tcp 0 0 192.168.100.83:6800 0.0.0.0:* LISTEN 22659/ceph-osd
tcp 0 0 192.168.200.83:6800 0.0.0.0:* LISTEN 22659/ceph-osd
tcp 0 0 192.168.200.83:6801 0.0.0.0:* LISTEN 22659/ceph-osd
tcp 0 0 192.168.100.83:6801 0.0.0.0:* LISTEN 22659/ceph-osd
From the above we should see the OSD processes listening on the corresponding public and private networks we configured in the configmap. However lets further confirm by going back into the toolbox and doing a ceph osd dump:
# kubectl -n rook-ceph exec -it rook-ceph-tools -- /bin/bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-tools]# ceph osd dump
epoch 14
fsid 05a8b767-e3e8-42aa-b792-69f479c807f7
created 2019-04-02 13:24:24.549423
modified 2019-04-02 13:25:28.441850
flags sortbitwise,recovery_deletes,purged_snapdirs
crush_version 7
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client firefly
require_osd_release mimic
max_osd 3
osd.0 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) 192.168.200.81:6800/29719 192.168.100.81:6800/29719 192.168.100.81:6801/29719 192.168.200.81:6801/29719 exists,up 2feb0edf-6652-4148-8264-6ba52d04ff80
osd.1 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) 192.168.200.82:6801/18770 192.168.100.82:6800/18770 192.168.100.82:6801/18770 192.168.200.82:6802/18770 exists,up f8df61b4-4ac8-4705-9f97-eb09a1cc0d6c
osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) 192.168.200.83:6800/22659 192.168.100.83:6800/22659 192.168.100.83:6801/22659 192.168.200.83:6801/22659 exists,up db555c80-9d81-4662-aed9-4bce1c0d5d78
As you can see it can be fairly straight forward to configure Rook to deploy a Ceph cluster using segmented networks to ensure the replication traffic runs on dedicated network and does not interfere with public client performance. Hopefully this quick demonstrate showed that.