Debugging AWS STS Authentication for OpenShift Operators
March 10, 2022
OpenShift supports granular AWS permissions for pods running cluster operators or even user applications. This enhances security by providing only the necessary privileges and nothing more. This post explores debugging authN and authZ of pods attempting to use fine grained IAM roles in combination with AWS secure token service.
The typical method of programmatic authentication to AWS uses long lived credentials in the form of any aws_access_key_id and an associated aws_secret_access_key. If those 2 items were to be accidentally leaked, it’s game over for your AWS bill.
STS adds an ephemeral third factor in aws_session_token. This token will expire after a period of time, so it must be regenerated on a regular basis. In this case, these tokens are validated and refreshed by interacting with an OpenID Connect Identity Provider created for use by OpenShift.
OpenShift projects these 3 factors into a pod which may then use them to authenticate and assume an IAM role appropriately authorized for the relevant AWS services.
Let’s explore this more deeply by looking an example where this process may not be working properly.
Degraded Ingress Cluster Operator
Multiple operators interact with STS to assume fine grained roles, but we’ll focus on the Ingress operator.
In this example, the Ingress cluster operator status is Degraded. The last message indicates the controller could not find the wildcard DNS (*.apps) entry in Route53. Normally the Ingress operator will be the device to create this record, but it can’t even search for it for some reason.
$ oc get co ingress
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
ingress False True True 18h ...\n\n
The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DNSReady=False (NoZones: The record isn't present in any zones.)
Debugging Operator STS Authentication
The Ingress operator needs to authenticate to interact with Route53 service. To do so it seeks to assume a role that was provisioned by the cloud-credential-operator, ccoctl, or by hand.
This role is named in a secret along with the location of a secret token signed by a key trused by the OpenID Connect Provider you configured in AWS IAM.
Start by describing the cluster operator (oc describe co ingress) but more importantly, check the logs of the ingress operator pod.
2022-02-25T17:08:36.019Z ERROR operator.init.controller-runtime.manager.controller.dns_controller controller/controller.go:253 Reconciler error {“name”: “default-wildcard”, “namespace”: “openshift-ingress-operator”, “error”:
failed to create DNS provider: failed to create AWS DNS manager: failed to validate aws provider service endpoints: [failed to list route53 hosted zones: WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: Couldn’t retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements
status code: 400, request id: 6aa3127f-e9b8-44e1-87e6-a739746f0370, …
It appears the token was not verified.
Mapping the Operator to a Role
Is STS authentication and “assume role with web identity” working?
The ingress operator pod should be assuming a unique role from AWS that is more fine grained than the instance profile used by the node. In this way the pod has only the premissions it requires to interact with Route53 and ELB APIs.
📓 Try to reproduce this failure by replicating what the pod is doing.
After copying the token you can use the aws sts assume-role-with-web-identity command to test with.
First, gather some details from the pod’s context. Spefically the role to be assumed, and the token to authenticate with.
Cloud-credentials Secret
The cloud-credentials secret holds the ARN of the role which the pod should assume once authenticated, and the location of the token used to authenticate with. That token will be automatically available.
Can you reach the IDP issuer’s openid-configuration?
The configuration file MUST be publicly reachable for OIDC to function as Amazon makes the call to this resource on your behalf. This must succeed on and off VPC!
ISSUER=$(cat $TOKEN | awk -F. '{ print $2 }' | base64 -d| jq -r .iss)# try this from a pod and from your laptopcurl -s $ISSUER/.well-known/openid-configuration || echo"Failure"
Can you reach the IDP issuer’s public keys?
The keys MUST be publicly reachable for OIDC to function as Amazon makes the call to this resource on your behalf. This must succeed on and off VPC!
KEYS=$(curl -s $ISSUER/.well-known/openid-configuration | jq -r .jwks_uri)# try this from a pod and from your laptopcurl -s $KEYS || echo"Failure"
Testing Authentication with the Token
Use the token to attempt to the assume the role.
First confirm you are authenitcated to AWS normally.
If you see an error like this, your pod can not authenticate or isn’t authorized to assume the role.
An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn’t retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements
Possible Failures
Are the openid-configuration and the keys publicly reachable by https? Remember this does not HAVE to be S3. If your S3 may only be private then you could use Cloudfront to expose the bucket.
Is the key id in the token the same as configured in the OIDC IDP?
Is the openid-configuration pointing to the right URL for the keys?
Is the audience in the token the same as configured in the OIDC IDP? The identity provider should have ‘sts.amazonaws.com’ and ‘openshift’ audiences.
Was credentialsMode set to manual at OpenShift install time?
Does the token service account match the trust relationship on the role? Notice the namespace and service account names
IAM Role Trust Relationship Note service account in the condition
