Blogs

Load balancing of OpenShift HA Routers Mind the GARP

OpenShift HA Routing uses haproxy application routers to get traffic into the cluster. These application routers are made redundant by running ipfailover (keepalived) pods to maintain a set of Virtual IPs on each infrastructure node where the application routers run. These VIPs are then referenced by round robin DNS records to enable a measure of load balancing. OK, so now you are load balancing at the network layer, but what about the link layer?

Continue reading

OpenShift 3.6 Upgrade Metrics Fails Missing heapster-certs Secret

After your upgrade to OpenShift v3.6 did the deployment of cluster metrics wind up with empty graphs? Check if the heapster pod failed to start due to a missing secret called heapster-certs in the openshift-infra namespace. Problem Heapster pod is failing to start $ oc get pods NAME READY STATUS RESTARTS AGE hawkular-cassandra-1-l1f3s 1/1 Running 0 9m hawkular-metrics-rdl07 1/1 Running 0 9m heapster-cfpcj 0/1 ContainerCreating 0 3m Check what volumes it is attempting to mount

Continue reading

Installing OpenShift on OpenStack

This is a work in progress The OpenShift Container Platform (OCP) can run on many types of infrastructure; from a Docker contrainer, to a single VM, to a fleet of baremetal or VMs on an infrastructure provider such as RHV, VMware, Amazon EC2, Google Compute Engine, or OpenStack Platform (OSP). This post is to document my experimentation with setting up OCP on OSP. Doc Overview So where are the docs?

Continue reading

OpenStack Network Diagram

What does the networking for OpenStack look like? Maybe something like this. # Network VLAN IP CIDR N1 Provisioning (PXE) V:310 172.23.32.0/20 N2 Internal API V:311 172.23.21.0/24 N3 Storage Network (Front) V:312 172.23.22.0/24 N4 Storage Mgmt (Back) V:313 172.23.23.0/24 N5 External Floating IPs V:179 192.0.179.0/24 N6 Public API V:177 192.0.177.0/24 N7 Overcloud Provisioning (Tenant PXE) V:314 172.

Continue reading

How to push an image to an unexposed OpenShift Docker registry

How do I push an image to the OpenShift Docker registry if it is not exposed outside the cluster? Login to a member node Get on a machine that has docker and participates in the cluster SDN or can somehow access that network. (eg. 172.30.0.0/16) Get the IP of the registry oc get svc docker-registry -n default --template "{{ .spec.clusterIP }}" SVC_REGISTRY=$(oc get svc docker-registry -n default --template "{{ .spec.clusterIP }}") Get a token for your session

Continue reading

Automated Pruning of OpenShift Artifacts; Builds, Deploys, Images

After running openshift for a while I discovered that letting builds pile up to around to around 1,200 led to what was essentially a deadlock in the scheduling of new builds. New builds were stuck in a New, waiting state indefinitely. This was fixed as of OCP 3.4.1, but it caused me to get more pro-active in the pruning of artifacts within OpenShift. I threw together a script and a playbook to deploy it.

Continue reading

Configuring OpenShift with Multiple Sharded Routers

I needed to host a service that would be consumed by a closed client that insists on speaking HTTPS on port 50,000. To solve this, I added a 2nd router deployment and used the OpenShift router sharding feature to selectively enable routes on the 2nd router by way of selectors. To summarize: Existing HA router: HTTP 80 HTTPS 443 Haproxy Stats 1,936 Added HA router: HTTP 49,999 HTTPS 50,000 Haproxy Stats 51,936 How To Open infra node firewalls Open firewall on infra nodes where router will run to allow new http and https port iptables -A OS_FIREWALL_ALLOW -m tcp -p tcp --dport 49999 -j ACCEPT iptables -A OS_FIREWALL_ALLOW -m tcp -p tcp --dport 50000 -j ACCEPT This can also be done with Ansible and the os_firewall role in your playbook.

Continue reading

OpenShift Cluster Metrics and Cassandra Troubleshooting

OpenShift gathers cluster metrics such as CPU, memory, and network bandwidth per pod which can assist in troubleshooting and capacity planning. The metrics are also used to support horizontal pod autoscaling, which makes the metrics service not just helpful, but critical to operation. Missing Liveness Probes There are 3 major components in the metrics collection process. Heapster gathers stats from Docker and feeds them to Hawkular Metrics to tuck away for safe keeping in Cassandra.

Continue reading

How to List Tags On Redhat Registry Images

Ever gone to RedHat’s container registry to search for an image and been left wondering what versions exist? Ever been frustrated by the inconsistent tag format? Is there a v or is there not a v? Me too. Docker Hub has progressed to v2, while the RedHat registry is still v1 at the moment. As long as you use the right syntax, you can use curl to query the registry API and list the tags like this:

Continue reading

Deploy Hawkular Metrics in CDK 2.1 OpenShift 3.2

Update! I failed with CDK 2.0, but CDK 2.1 works with some fiddling. In my last post I installed Red Hat Container Developer Kit to deploy OpenShift Enterprise using Vagrant. But now I want to add Hawkular Metrics to that deployment. Deploy Metrics Refer to the docs for deploying metrics in OSE. Login to the vagrant CDK VM before continuing $ cd ~/cdk/components/rhel/rhel-ose/ $ vagrant ssh $ oc login Authentication required for https://127.

Continue reading