The OpenShift Machine Configuration Operator applies configuration changes to to nodes using a syntax called Ignition, but managing base64 encoded text can be challenging. What if I told you that you can use plain text and normal files?

Machine Config Operator

OpenShift manages the operating system running on cluster nodes using the Machine Config Operator (MCO).

Components of the MCO include the Machine Config Sever (MCS) which provides ignition files to clients over HTTPS, and an agent running on each node called the Machine Config Daemon (MCD).

The MCD detects drift in the local machine configuration from the desired configuration expressed by the MCS.

Machine Config Pools

There are a number of settings and artifacts created during the OpenShift cluster installation process. Settings like the kubelet configuration or artifacts like ssh keys. These are all formatted into MachineConfig resources and correlated to nodes by way of Machine Config Pools (MCPs). By default there will be a pool for the control plane and a pool for the worker nodes.

All the MachineConfigs for a given pool are rendered into one large JSON blob for publishing by the MCS. Nodes are associated to MCPs by matching labels. You may need to create unique MCPs for heterogenous hardware, or you may just need to add some configuration to an existing MCP to enable multipath I/O or some other service.

Truncated Worker MCP Example

MachineConfigs are associated with the pool using machineConfigSelectors, and nodes are assocatied with a pool by nodeSelctors.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: worker
spec:
  configuration:
    ...
  machineConfigSelector:
    matchLabels:
      machineconfiguration.openshift.io/role: worker
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker: ""

MachineConfigs

Because an Ignition file is JSON formatted, a MachineConfig must be encoded into a safe representation. Here is what an example MachineConfiguration file looks like.

At a glance, maybe you can tell it is writing a file but you can’t easily read what’s in it or make changes to it.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: dne
  name: 99-worker-message
spec:
  config:
    ignition:
      version: 3.4.0
    storage:
      files:
        - contents:
            compression: ""
            source: data:,Hello%2C%20world!%0A
          mode: 420
          overwrite: true
          path: /tmp/message.txt

This text is URL encoded, but you will find base64 encoding used to for longer strings. Let’s decode the mystery text.

$ cat `which urldecode`
#!/bin/bash
python3 -c "import sys, urllib.parse as ul; print(ul.unquote_plus(sys.argv[1]))" $*

$ urldecode 'Hello%2C%20world!%0A'
Hello, world!

OK, that example was pretty clearly writing “Hello, world!”, but can you read this one:

👹 Uglier Example

In this example, the /etc/multipath.conf file contents are gzipped and then base64 encoded.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-multipathd-config
spec:
  config:
    ignition:
      version: 3.4.0
    storage:
      files:
        - contents:
            compression: gzip
            source: data:;base64,H4sIAAAAAAAC/1SPQYorMQxE9zpFgbf5yT4H+NdoPLYci3HbjSUnE0LuPnSSaeiVoBD1XjkkKXzGiS2c5lFMFm/5GFpN5PC/dXiENi+FjVFEDS3BMiNy8qMY1k+5jO5NWsXVl8F6QB8VLJa5n8kBWzH+GTm0vgsjNLfbp2mj/sH2gLasR3ETy4isocs7OUCZVzNy2O/A7CsWf+EjfaQVDwKGcp9SF66x3KfqZ1bcWQlIUuO0lbzTJzlIDWXEdftVAitS6+CfUIZKq+ReBvRVfPh+yT/o+RsAAP//NbWKB18BAAA=
          mode: 420
          path: /etc/multipath.conf
          user:
            name: root
    systemd:
      units:
        - enabled: true
          name: multipathd.service

🔥 Butane

Fortunately it is possible to write MachineConfigs in a more legible format called Butane.

With Butane you might put the text in plainly like this:

variant: openshift
version: 4.18.0
metadata:
  name: 99-worker-message
  labels:
    machineconfiguration.openshift.io/role: dne

storage:
  files:
    - path: /tmp/message.txt
      mode: 0644
      overwrite: true
      contents:
        inline: |-
          Goodbye, moon!          

Or you could reference a stand alone file.

      contents:
        local: message.txt

This is particularly helpful if you are managing a complex configuration file like multipath.conf. Just place the multipath.conf into a folder called inc/ and generate the machineconfig using the butane command.

Generate a MachineConfig with Butane

cp multipath.conf inc/
butane -d inc < 99-worker-multipath.bu > 99-worker-multipah.yaml 

Generating MachineConfigs

Once you adopt butane you will then need a scalable way to maintain your MachineConfigs to make sure they are always up to date. The following Makefile from this repo provides a way to do that.

The MachineConfigs stored in *.yaml files are generated from the Butane butane/*.bu files. Any configuration files or scripts that are included from the inc/ directory are detected by the Makefile and noted in dependencies files in .deps/.

When you run make, these dependincies are checked. If either an included file or a Butane file changes, then the MachineConfig file is automatically flagged as stale and regenerated.

⭐ Pro Tip Because dependencies are ephemeral and generated on the fly, there isn’t need to store them in git. You can safely add them to .gitignore.

Makefile

 1BUTANES = $(wildcard butane/*.bu)
 2MACHINECONFIGS = $(BUTANES:butane/%.bu=%.yaml)
 3INCLUDES_DIR = inc
 4DEPS_DIR = .deps
 5
 6# Enumerate all files that might be included
 7INCLUDE_FILES = $(wildcard $(INCLUDES_DIR)/*)
 8
 9# Create deps directory if it doesn't exist
10$(shell mkdir -p $(DEPS_DIR))
11
12all: $(MACHINECONFIGS)
13
14# Generate dependencies for each butane file
15$(DEPS_DIR)/%.d: butane/%.bu
16	@echo "Generating dependencies for $<"
17	@printf "%s: %s %s\n" "$*.yaml" "$<" "$$(grep -o 'local: [^[:space:]]*' $< | cut -d' ' -f2 | sed 's|^|$(INCLUDES_DIR)/|' | tr '\n' ' ' | sed 's/ *$$//')" > $@
18
19# Include all dependency files
20-include $(BUTANES:butane/%.bu=$(DEPS_DIR)/%.d)
21
22# Each machineconfig ends in yaml and depends on the same named file ending in bu
23%.yaml: butane/%.bu
24	butane -d $(INCLUDES_DIR) < $< > $@
25
26# rm the machineconfigs and dependency files generated from butane files
27clean:
28	rm -f $(MACHINECONFIGS)
29	rm -rf $(DEPS_DIR)

Source

Demo

📺 ASCII Screencast

Managing MachineConfig resources with Butane and Make

Summary

Leave the base64 encoding to the robots 🤖. Make things easier on yourself. Write in plain text with Butane 🔥!

References

• Demo Github Repo
• Demo Recording
• CoreOS Ignition Specification
• OpenShift 4.18 Butane Spec
• OpenShift Machine Config Operator - Github
• OpenShift 4.18 Machine Config Docs
• Download Butane