Etcd Cluster Restore (Removed)

Etcd support has been removed. Use NATS instead.

Many users ran into the case only one of the 3 pods of etcd cluster can work. The other 2 pods always restart and can't back to work.

Our experiences in this case it is only one reason:

The 3 pods of the etcd cluster Killed / Recreate / Destroyed at same time.

Then the cluster was broken, Only one pod can work.

Let's help the cluster back to work.

If you only have one pod can't work, just delete the PVC of the pod and delete the pod, then it should work.

1. Start the etcd cluster with bash

Because the etcd can't start, we need to start to etcd pod with bash. like this:

Change the values.yaml of helm for OpenObserve.

Add those lines:

  command: ["/bin/bash", "-c", "while true; do sleep 1; done"]
  livenessProbe:
    enabled: false
  readinessProbe:
    enabled: false

After change the config of etcd should be like this:

etcd:
  enabled: true # if true then etcd will be deployed as part of openobserve
  externalUrl: "my_custom_host.com:2379" # if bundled is false then this is required
  replicaCount: 3 # if bundled is true then this is required. should be odd number
  auth:
    token:
      enabled: false
    rbac:
      create: false
      allowNoneAuthentication: true
      rootPassword: ""
  logLevel: "info"
  extraEnvVars:
    - name: ETCD_AUTO_COMPACTION_RETENTION
      value: "1"
  command: ["/bin/bash", "-c", "while true; do sleep 1; done"]
  livenessProbe:
    enabled: false
  readinessProbe:
    enabled: false

2. Start the pod which still can work

Normally, at least you have one pod still can work. We should start the pod first because it should have the latest data.

Use any tools or directory use kubectl -n openobserve exec etcd-N -- bash login the pod.

You can use the command env | grep ETCD_INITIAL_CLUSTER to check current config for initial cluster.

!@openobserve-etcd-2:/opt/bitnami/etcd$ env | grep ETCD_INITIAL_CLUSTER
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_INITIAL_CLUSTER=openobserve-etcd-0=http://openobserve-etcd-0.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2380,openobserve-etcd-1=http://openobserve-etcd-1.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2380,openobserve-etcd-2=http://openobserve-etcd-2.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2380

The result tells us the etcd cluster starting needs to wait 3 nodes, but we know it can't, so we change the environment only to use the current node to start.

Through the command line, we know the current pod is etcd-2, so we change it like this:

Please note the pod that can work for you may be etcd-1; you need to check it first.

ETCD_INITIAL_CLUSTER=openobserve-etcd-2=http://openobserve-etcd-2.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2380

Then start the etcd of this node:

etcd

Yes, just type etcd and enter. The etcd will start, and keep the command line here.

Notice: don't close it, it will shutdown the etcd server if you close the shell.

Waiting for seconds, you can see the etcd is working.

Then let's restore other nodes.

Before switching to another pod, we need to add a new member to this cluster.

You need to create a new shell for this pod; in our case, it is etcd-2.

Check the current member of this cluster; it should only be itself.

!@openobserve-etcd-2:/opt/bitnami/etcd$ etcdctl member list
3f59fc06477e49f8, started, openobserve-etcd-2, http://openobserve-etcd-2.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2380, http://openobserve-etcd-2.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2379,http://openobserve-etcd.ziox-dev.svc.cluster.local:2379, false

Let's add a new member to this cluster:

!@openobserve-etcd-2:/opt/bitnami/etcd$ etcdctl member add openobserve-etcd-0 --peer-urls=http://openobserve-etcd-0.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2380

We use the information from ENV. Here we add the member openobserve-etcd-0. Of course, you can choose etcd-1; it doesn't matter, but next you need to restore this node first.

3. Start other pods which can't work

Earlier we added etcd-0 to the cluster, so we need to restore it first. Let's log into the pod.

First, delete the old data of this node.

rm -fR /bitnami/etcd/data/*

Second, change the environment for the cluster:

ETCD_INITIAL_CLUSTER=openobserve-etcd-0=http://openobserve-etcd-0.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2380,openobserve-etcd-2=http://openobserve-etcd-2.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2380
ETCD_INITIAL_CLUSTER_STATE=existing

Please notice it should include 2 pods, etcd-0 and etcd-2.

Then start etcd:

etcd

Yes, just type etcd and enter. The etcd will start, and keep the command line here.

Notice: don't close it, it will shutdown the etcd server if you close the shell.

Waiting for seconds, you can see the etcd is working.

You can create a new shell and log into this pod to check it:

!@openobserve-etcd-2:/opt/bitnami/etcd$ etcdctl member list
3f59fc06477e49f8, started, openobserve-etcd-2, http://openobserve-etcd-2.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2380, http://openobserve-etcd-2.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2379,http://openobserve-etcd.ziox-dev.svc.cluster.local:2379, false
51daebb86180a114, started, openobserve-etcd-0, http://openobserve-etcd-0.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2380, http://openobserve-etcd-0.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2379,http://openobserve-etcd.ziox-dev.svc.cluster.local:2379, false

It should list 2 members.

Let's add the last member; the same steps do again:

First, add etcd-1 to the cluster:

!@openobserve-etcd-2:/opt/bitnami/etcd$ etcdctl member add openobserve-etcd-1 http://openobserve-etcd-1.openobserve-etcd-headless.ziox-dev.svc.cluster.local:2380

Second, log into the pod etcd-1, delete old data and start etcd.

Finally, it should list 3 members:

Okay, the cluster restored.

Now, we need to restore the configuration of helm.

etcd:
  enabled: true # if true then etcd will be deployed as part of openobserve
  externalUrl: "my_custom_host.com:2379" # if bundled is false then this is required
  replicaCount: 3 # if bundled is true then this is required. should be odd number
  auth:
    token:
      enabled: false
    rbac:
      create: false
      allowNoneAuthentication: true
      rootPassword: ""
  logLevel: "info"
  extraEnvVars:
    - name: ETCD_AUTO_COMPACTION_RETENTION
      value: "1"
#   command: ["/bin/bash", "-c", "while true; do sleep 1; done"]
#   livenessProbe:
#     enabled: false
#   readinessProbe:
#     enabled: false

Comment the command we added at first, and then deploy to k8s again. You will see the etcd pods recreate one by one, and every pod needs 2 minutes to become green.

After 5 minutes, everything should works.

You also need to recreate OpenObserve pods because the etcd recreated; maybe some pods can't connect to etcd for a while.