Blogs

OpenShift Cluster Metrics and Cassandra Troubleshooting

OpenShift gathers cluster metrics such as CPU, memory, and network bandwidth per pod which can assist in troubleshooting and capacity planning. The metrics are also used to support horizontal pod autoscaling, which makes the metrics service not just helpful, but critical to operation. Missing Liveness Probes There are 3 major components in the metrics collection process. Heapster gathers stats from Docker and feeds them to Hawkular Metrics to tuck away for safe keeping in Cassandra.

Continue reading

How to List Tags On Redhat Registry Images

Ever gone to RedHat’s container registry to search for an image and been left wondering what versions exist? Ever been frustrated by the inconsistent tag format? Is there a v or is there not a v? Me too. Docker Hub has progressed to v2, while the RedHat registry is still v1 at the moment. As long as you use the right syntax, you can use curl to query the registry API and list the tags like this:

Continue reading

Deploy Hawkular Metrics in CDK 2.1 OpenShift 3.2

Update! I failed with CDK 2.0, but CDK 2.1 works with some fiddling. In my last post I installed Red Hat Container Developer Kit to deploy OpenShift Enterprise using Vagrant. But now I want to add Hawkular Metrics to that deployment. Deploy Metrics Refer to the docs for deploying metrics in OSE. Login to the vagrant CDK VM before continuing $ cd ~/cdk/components/rhel/rhel-ose/ $ vagrant ssh $ oc login Authentication required for https://127.

Continue reading

Getting Started With RedHat Container Development Kit

The RedHat Container Developer Kit allows you to deploy OpenShift on your laptop for easier testing and development. Here is how to deploy it. Register as a RedHat Developer Obtain a RH login Place credentials in ~/.vagrant.d/Vagrantfile to enable updates for VMs by automatically registering with RedHat Subscription Manager Vagrant.configure('2') do |config| config.registration.username = '<your Red Hat username>' config.registration.password = '<your Red Hat password>' end Mac OS X Prereqs Install pre-reqs:

Continue reading

Upgrading OpenShift Enterprise from 3.1 to 3.2

Upgrading from OSE 3.1 to 3.2 using the playbook went quite well for me, but there were a few issues to sort out. The issues were related to: ip failover had to be updated manually there was about 5 minutes downtime during the upgrade updates to image streams docker error messages updated policy and role bindings build strategy Source is not allowed hawkular metrics Upgrade Process Following the directions is pretty straight forward.

Continue reading

Changing the SSL Certificate for OpenShift Console

OpenShift has an internal CA for generating certificates to authenticate intra-cluster communication, but your browser doesn’t trust this CA. Perhaps you want to fix that without mucking with the internal SSL communication? I did. Here is how. This OpenShift doc explains how to do this, but it isn’t very clear, to me at least. Overview An outline of the steps: Only make changes to the public URLs and not any internal URLs.

Continue reading

OpenShift High Availability - Routing

Highly availabile containers in OpenShift are baked into the cake thanks to replication controllers and service load balancing, but there are plenty of other single points of failure. Here is how to eliminate many of those. Single Points of Failure The components of OpenShift include: Master controller manager server and API endpoint Etcd configuration and state storage Docker Registry Router haproxy This post is mostly about adding high availability to the routing layer.

Continue reading

Ansible Playbook to Prepare for OpenShift Enterprise 3.1

This playbook is written for RHEL 7.2 and OSE v3.1. It will perform the following steps which should take place before running the openshift-ansible byo playbook. Install prerequisite RPMs like docker, python, etc. Persist the systemd journal for easier debugging Setup docker ephemeral storage on 2nd disk Turn off swap Enable use of NFS in selinux Prerequisites See my Testing OpenShift Enterprise V3 post for the prereqs. The Playbook The lastest version is available here.

Continue reading

Ansible CMDB Inventory and Facts Reporting

You just deployed a complex multi-host app using Ansible. Wouldn’t it be helpful to see a overview of the deployment including hardware details? I just found ansible-cmdb which combines info from the Ansible inventory and discovered facts to create a detailed HTML report akin to a Configuration Management Database. To use it in your playbook dir, just create a directory to hold facts discovered by the setup module then generate the report.

Continue reading

Notes on SNMP MIBs OIDs and Grey Whiskers

With as many grey whiskers as I have, you would think I could grok SNMP in my sleep by now. Unfortunately, everytime I have to deal with it I get frustrated, wonder where the hell my notes from last time are, and start cursing. From now on, here are my notes! I’m typically using Zabbix to poll SNMP OIDs and place the MIBs on the Zabbix server or the Zabbix proxy responsible for SNMP in /usr/share/snmp/mibs.

Continue reading