Downgrade Etcd 3.3.11 to 3.2.22 for OpenShift Compatibility

While I was working on migrating etcd to my master nodes I was bitten by an incompatible etcd v3.3.11 RPM made available via RHEL Server Extras repo. Before I got to my last master the RPM was no longer available, and the scaleup playbook failed. I became aware that 3.3.11 is not compatible and should not have been made available. Unfortunately all members of my etcd cluster were already upgraded and the fix is to take down the cluster, downgrade etcd, and restore from snapshot.

Continue reading

Etcdctl v2 and v3 Aliases for Peer Authenticated Commands

Getting all the arguments to etcdctl right can be a bit of a pain. Here are a couple of aliases which take advantage of the values in the etcd.conf file. alias etcd2='. /etc/etcd/etcd.conf && \ ETCDCTL_API=2 etcdctl \ --cert-file ${ETCD_PEER_CERT_FILE} \ --key-file ${ETCD_PEER_KEY_FILE} \ --ca-file ${ETCD_PEER_TRUSTED_CA_FILE:-$ETCD_PEER_CA_FILE} \ --endpoints "${ETCD_ADVERTISE_CLIENT_URLS}"' alias etcd3='. /etc/etcd/etcd.conf && \ ETCDCTL_API=3 etcdctl \ --cert ${ETCD_PEER_CERT_FILE} \ --key ${ETCD_PEER_KEY_FILE} \ --cacert ${ETCD_PEER_TRUSTED_CA_FILE:-$ETCD_PEER_CA_FILE} \ --endpoints "${ETCD_ADVERTISE_CLIENT_URLS}"' If you are using OpenShift, you may also find that you already have some bash functions enabled by the etcd role in /etc/profile.

Continue reading

Migration of Etcd to Masters for OpenShift 3.9 to 3.10 Upgrade

As of OpenShift Container Platform 3.10 etcd is expected to run in static pods on the master nodes in the control plane. You may have a deployed an HA cluster with dedicated etcd nodes managed with systemd. How do you migrate the this new architecture? Assumptions: You are running OCP 3.9 You have multiple Master nodes You have dedicated Etcd nodes You are running RHEL, not Atomic nodes Outline:

Continue reading