October 6, 2025

Storing Virtual Machines as Kubernetes resources in git for automated deployment promotes consistency, resilency, and accountability, but commiting secrets to git is risky and should be avoided. Use the External Secrets Operator to securely store cloud-init and other data, and sleep soundly!

Securing Cloud-init User Data

As described in this post the cloud-init userData script for a OpenShift Virtualization virtual machine may contain privileged information like activation keys for your RHEL subscription.

rh_subscription:
  org: 00000000
  activation-key: EXAMPLE
  enable-repo:
    - 'rhel-9-for-x86_64-baseos-rpms'
    - 'rhel-9-for-x86_64-appstream-rpms'

These example userData scripts for VMs: client, ldap, and nfs have placeholders which make them safe to place in a public repository.

Most of the data isn’t sensitive, and keeping the scripts in git makes for easier testing and debugging. But how can we safely manage the actual secret to be used by cloud-init if it is not commited to git?

The External Secrets Operator

The External Secrets Operator solves this issue for us by understanding how to look into secure vaults from “providers” like AWS Secrets Manager, HashiCorp Vault, or 1Password.

Once your data is within a secure vault, you will create an ExternalSecret resource, which is safe to commit to git. When you apply these resources to the cluster, the ESO will use the information in the ExternalSecret to create a Kubernetes Secret resource, securely downloading, and inserting the sensitive data into it.

1Password Provider for ESO

Using 1Password can be handy in a homelab or development environment, particularly if you already use 1Password to manage credentials in your browser. You Don’t have to deploy any infrastructure or have an AWS account to use it. Be aware there is more than one 1Password integration and you should select the 1Password-SDK External Secrets Operator provider.

Installing the External Secrets Operator

Make sure to install a version of the External Secrets Operator (ESO) that supports the 1password-sdk provider, which was introduced in version 0.17. The older 1password-connect provider is now deprecated. If the operator has been updated to version 0.17 or later by the time you read this, you may be able to use the operator directly instead of installing via the Helm chart.

Install latest upstream ESO using Helm.

$ helm repo add external-secrets https://charts.external-secrets.io

$ oc new-project external-secrets

$ helm install external-secrets \
   external-secrets/external-secrets \
   -n external-secrets

Configuring 1Password

Use the command line to setup 1Password. Grab the 1Password CLI command op from https://developer.1password.com/ or use brew.

$ brew install 1password-cli

Create a dedicated vault in 1Password for use by the ESO, so there is no chance of your personal data sloshing around in your Kubernetes environment.

$ op vault create eso --icon gears

Create a token to authenticate the ESO to access 1Password. Note that 90 days is the max lifetime allowed by 1Password.

$ TOKEN=$(
    op service-account create external-secrets-operator \
      --expires-in 90d \
      --vault eso:read_items,write_items \
    )

Place this token in a secret called onepassword-connect-token which allows ESO to authenticate to 1Password.

$ oc create secret generic onepassword-connect-token \
 --from-literal=token="$TOKEN" \
 -n external-secrets

⭐ Pro Tip: Test the token to confirm access to the vault items by setting the OP_SERVICE_ACCOUNT_TOKEN environment variable.

 $ export OP_SERVICE_ACCOUNT_TOKEN=$(oc extract secret/onepassword-connect-token \
  -n external-secrets --keys=token --to=-)

 $ op item list --vault eso
 ID                            TITLE                            VAULT            EDITED
 yzsurcc4oxfjp7qdidonudn3ne    demo autofs ldap                 eso              22 hours ago
 wretasduq3rkip7wn37njozghi    demo autofs nfs                  eso              22 hours ago
 euaujb4izjftqineetzaer3x7i    demo autofs client               eso              5 days ago

Uploading userData to the Vault

Now, we can take the example userData files and modify them to contain the sensitive data we want. In this case, the organization ID and activation key required for our subscription.

Edit the checked out copy of the {VM}/base/scripts/userData scripts, and insert the configuration that should not be stored in git.

At this point, the copies in your working directory may now look like the following. Be certain not to commit these changes to git!

rh_subscription:
  org: 12345678
  activation-key: secret-key4me
  enable-repo:
    - 'rhel-9-for-x86_64-baseos-rpms'
    - 'rhel-9-for-x86_64-appstream-rpms'

Now create 1Password items storing the modified userData for each VM.

vault=eso
for vm in client ldap nfs; do
  op item create \
    --vault "$vault" \
    --category login \
    --title "demo autofs $vm" \
    --url "https://github.com/dlbewley/demo-autofs/tree/main/${vm}/base/scripts" \
    --tags demo=autofs \
    "userData[file]=${vm}/base/scripts/userData"
done

📓 If you later make changes to the userData script you can update the copy in 1Password like this.

# download current values from 1Password to working dir
for vm in client ldap nfs; do
    op read "op://${vault}/demo autofs ${vm}/userData" \
        --out-file=${vm}/base/scripts/userData
done

# make any needed changes
vi {client,ldap,nfs}/base/scripts/userData

# upload changes to 1Password 
for vm in client ldap nfs; do
  op item edit \
    --vault "$vault" \
    --url "https://github.com/dlbewley/demo-autofs/tree/main/${vm}/base/scripts" \
    "demo autofs $vm" \
    "userData[file]=${vm}/base/scripts/userData"
done

# remove all changes made to working dir
git checkout -- {client,ldap,nfs}/base/scripts/userData

Here is a view of the 1Password vault after uploading each of our userData scripts.

Configuring the External Secrets Operator

Now that 1Password is ready, it’s time to tell the ESO about it.

Create a ClusterSecretStore associated to the 1password-sdk provider. This will be referenced by ExternalSecret resources created later, and it will use the secret token we just created to look up data in the vault.

 1---
 2apiVersion: external-secrets.io/v1
 3kind: ClusterSecretStore
 4metadata:
 5  name: 1password-sdk
 6spec:
 7  provider:
 8    onepasswordSDK:
 9      vault: eso
10      auth:
11        serviceAccountSecretRef:
12          name: onepassword-connect-token
13          key: token
14          namespace: external-secrets

Reading Secret Data from the Vault

Create an externalsecret.yaml for each VM. Here are examples for each of the VMs client, ldap, and nfs.

Notice below that we set the refreshInterval to 0 so that the ESO is not continually checking 1Password for changes to this secret. Remember this secret is only used once, when the VM boots for the first time.

If you do not disable this refresh, then it is likely that you will become rate limited.

🔐 External Secret

 1---
 2apiVersion: external-secrets.io/v1
 3kind: ExternalSecret
 4metadata:
 5name: cloudinitdisk-client
 6spec:
 7refreshPolicy: OnChange
 8refreshInterval: "0"
 9secretStoreRef:
10    kind: ClusterSecretStore
11    name: 1password-sdk
12target:
13    name: cloudinitdisk-client # this will be the name of the created secret
14    creationPolicy: Owner
15data:
16- secretKey: "userData" # this will be a field in the secret
17    remoteRef:
18    # 1password-entry-name and property
19    key: "demo autofs client/userData"

This ExternalSecret will retrieve the data at “demo autofs client/userData” create a Secret named cloudinitdisk-client.

Updating the VM Deployment

Now we must update the kustomization.yaml file that deploys the virtual machine. It should create the ExternalSecret and patch the VM to mount the subsequently created Secret holding the sensitive version of the cloud-init script.

As a reference, we can still let Kustomize generate a sample secret from our “clean” userData in git.

🪡 Virtual Machine Patches

 1---
 2apiVersion: kustomize.config.k8s.io/v1beta1
 3kind: Kustomization
 4
 5namespace: demo-client
 6
 7labels:
 8- includeSelectors: true
 9  pairs:
10    demo: client
11    app.kubernetes.io/instance: demo-autofs-client
12
13components:
14- ../../components/argocd-vm-management
15
16generatorOptions:
17disableNameSuffixHash: true
18
19configMapGenerator:
20- name: sssd-conf
21  files:
22    - scripts/sssd.conf
23    - scripts/homedir.conf
24
25secretGenerator:
26- name: cloudinitdisk-client-sample
27  files:
28    - scripts/userData
29
30resources:
31- namespace.yaml
32- externalsecret.yaml
33- virtualmachine.yaml
34
35patches:
36- target:
37    group: kubevirt.io
38    kind: VirtualMachine
39    name: .*
40    version: v1
41  patch: |-
42    - op: replace
43      path: /spec/runStrategy
44      value: Always
45    # add volumes for secret and configmap
46    - op: replace
47      path: /spec/template/spec/volumes/1/cloudInitNoCloud
48      value: {
49        "secretRef": {
50          "name": "cloudinitdisk-client"
51        }
52      }

Booting the VM

Finally, deploy the Virtual Machine using Kustomize via the oc apply -k command.

$ oc apply -k client/overlays/localnet
namespace/demo-client created
role.rbac.authorization.k8s.io/argocd-vm-management created
rolebinding.rbac.authorization.k8s.io/argocd-vm-management created
configmap/sssd-conf created
secret/cloudinitdisk-client-sample created
externalsecret.external-secrets.io/cloudinitdisk-client created # <---
virtualmachine.kubevirt.io/client created

After a moment, the ExternalSecret status should be SecretSynced and we can see that the resulting cloudinitdisk-client secret has been created.

$ oc get externalsecrets -n demo-client
NAME                   STORETYPE            STORE           REFRESH INTERVAL   STATUS         READY
cloudinitdisk-client   ClusterSecretStore   1password-sdk   0                  SecretSynced   True

$ oc get secrets -l demo=client -n demo-client
NAME                          TYPE     DATA   AGE
cloudinitdisk-client          Opaque   1      98s   # Created by ESO
cloudinitdisk-client-sample   Opaque   1      2m20s # Generated by Kustomize

Once the VM boots we can login and examine the user data imported from the secret, and confirm the information from the vault is indeed there!

[root@client ~]# grep -A2 subscription /var/lib/cloud/instance/user-data.txt
rh_subscription:
  org: 12345678
  activation-key: secret-key4me

Summary

By leveraging the External Secrets Operator and Kustomize we safely deployed fully provisioned Virtual Machines to OpenShift using a single command. 1Password was selected for it’s ubiquity and ease of setup as a provider. This pattern can be adapted for other vault providers for robust secret management in production OpenShift Virtualization environments.

Securing Cloud-init User Data with External Secrets and OpenShift Virtualization