Introduction: What is the Cluster Etcd Operator ?
Aforementioned the etcd at my previous post is managed by Cluster Etcd Operator in the OpenShift v4. I’d like to show you what the Cluster Etcd Operator is in this post. Let’s go ahead !
What is the Cluster Etcd Operator ?
This operator started to serve as of OpenShift v4.4, and it basically provides automated maintenance tasks for reducing administrative burden. It’s usually mentioned “CEO” acronym in OpenShift technical contexts.
As you can see the above file names and structure in the list, there are many sub controllers for each functionality and automated feature. For example, in automated defragmentation feature, it’s implemented as defragcontroller.go.
How does it work ?
I mentioned this operator provides automated maintenance tasks, but unfortunately it does not mean whole process is automated currently.
Some features, such as, rotating certificates, datastore defragmentation and member management under a certain condition are automated without your intervention. But backup and recovery process is required your hand yet. Fortunately CEO generates useful wrapper scripts to help the tasks easily and simply like “cluster-backup.sh”. You can just run the script for backup and restore according the instructions in OpenShift docs.
This diagram shows you how to Cluster Etcd Operator manages the Etcd static pods through StaticPodController. For beginner of kubernetes, static pod is a special pod which is managed by kubelet agent directly without master api. It’s basically the same implementation with other control plane operators in OpenShift v4 like kube apiserver operator, kube controller manager operator and kube scheduler opeerator. Because the operators has been implemented using the same api library.
In fact, more controllers are working there, but I’d rather just to show you key points how to manage the static pod here. Actually, the StaticPodController consists of multiple sub-controllers like installer_controller and prune_controller.
In usual case, if there is any update to trigger changing revision, firstly “revision-status” configmap is generated, and then etcd-pod configmap static pod contents stored would be generated appropriately according to new revision. And one-off installer pod launched and it makes the static pod update using new revision etcd-pod configmap.
“revision-status” configmap is generated by revision controller. And etcd-pod configmap has the revsion-status configmap as ownerReferences. So if removed revision-status configmap, etcd-pod configmap would be removed as well.
When the revision count is reached the maximum limit count, revision-pruner one-off pod is also launched to delete the old static pod contents through removing “revision-status” configmap. It’s cascading deletion using ownerReferences.
What is under the hood of Cluster Etcd Operator ?
This simple demo shows you how to restore the removed-member by Cluster Etcd Operator for your more understanding, when removed a member.
The test scenario is as follows.
- Firstly remove a member using “etcdctl member remove” command, then the pod would be stopped with non-zero exit code.
- Then the etcd pod starts restarting itself, and “discovery-etcd-initial-cluster” command is running and waiting to add the removed-member again. Usually “discovery-etcd-initial-cluster” command returns nothing if there is no issue on membership in the cluster. Anyway this command is for hand over current cluster member list including new member.
- As soon as the removed-member added again by clustermembercontroller,
ETCD_INITAL_CLUSTER environment variable configured by “discovery-etcd-initial-cluster” command with all member list of the cluster including removed-member. This environment variable is required when added new member to initialize. - Concurrently, existing var lib etcd member data directory is renamed for backup.
- After that, the removed-member joins the cluster next restarting etcd pod as standard scaling up ways. Standard scaling up means that usually new member is added to existing cluster, the data is synced automatically.
A picture is worth a thousand words. Let’s look actual output and logs !
For test, remove a member like the output. In this case, removed third member in the table.
If Cluster Member Controller detects that the member is removed, the removed-member is added again. And the existing data directory is renamed.
At the same time, “discovery-etcd-initial-cluster” command provides full member list for ETCD_INITIAL_CLUSTER. In this test, I ran the command manually for identify the value, but in fact this command is executed automatically when started the etcd pod.
Above operations were conducted during the etcd pod was stopped with CrashLoopBackOff. Next restart of the etcd pod, the data is synced by existing cluster and running well.
Conclusion
So far we glanced Cluster Etcd Operator. I did not touch more details for it, but I hope it’s helpful to understand Cluster Etcd Operator.
Thank you for reading.
