GUI Free Life View on GitHub

Installing OpenShift on OpenStack

kubernetes openshift openstack draft

Aug 20, 2017

This is a work in progress

The OpenShift Container Platform (OCP) can run on many types of infrastructure; from a Docker contrainer, to a single VM, to a fleet of baremetal or VMs on an infrastructure provider such as RHV, VMware, Amazon EC2, Google Compute Engine, or OpenStack Platform (OSP). This post is to document my experimentation with setting up OCP on OSP.

Doc Overview

So where are the docs?

The Reference Architecture 2017 - Deploying Red Hat OpenShift Container Platform 3.4 on Red Hat OpenStack Platform 10 derives from the Redhat OpenShift on OpenStack Github repo provides the orchestration templates to stand up a infrastructure stack to run OpenShift on. At the moment the last releases are OCP 3.6 and OSP 11. Let’s target that even if it “isn’t supported”.

Antonio Gallego has created a script to prepare OpenStack for OpenShift. Does this overlap with the openshift-on-openstack repo though? The openshift-ansible repo includes a openstack dynamic inventory module, but it is not yet clear to me how best to utilize it. For my production cluster, I tend to keep my intentory synchronized to hosts.ose.example.

The openshift-ansible-contrib project has a openstack-stack role for deploying OpenShift to OpenStack, I’m not sure of the state, but there is a reference-architecture/osp-dns to deploy a DNS infra suitable for testing.

While heat and the ansible playbook will do this for you, it is interesting to look at the CLI commands to configure OpenStack for OpenShift. This scripts are sprinkled through the ref arch document as well.

OpenShift its self needs to be configured for OpenStack to make use of storage and other services provided by OpenStack. The OpenShift Ansible playbook is used to install and configure OpenShift on any platform including OpenStack and the settings will be placed in the playbook host inventory file.

Networking Overview

OpenShift highly available routing is somewhat complex on its own.

OpenShift HA Routing

Toss in the even more complex OpenStack networking, and well, hopefully you are not starting from scratch.

OpenStack Network Diagram

The reference architecture forgoes the OpenShift SDN with ovs-subnet plugin and uses Flannel. Notes on configuring Flannel networking.

Openshift on OpenStack Reference Arch Diagram

There are some drawbacks to using Flannel when it comes to isolation. An interesting alternative to Flannel could be Project Calico which uses BGP routing amongst containers while also supporting microsegmentation of traffic. develops and supports Calico commercially.

Networking Details

There will be 3 OpenStack networks in use:

Instance Details

Host IP Description 10.19.x.y (Load Balancer) Web console and API endpoint
* 10.19.x.y (Router) OpenShift routes to services handled by haproxy.
bastion   Operator access and Ansible management point
master-01   One of three redundant OpenShift Masters
master-02   One of three redundant OpenShift Masters
master-03   One of three redundant OpenShift Masters
node-01   One of two redundant Infrastructure nodes
node-02   One of two redundant Infrastructure nodes
node-03   One of two redundant Application nodes
node-04   One of two redundant Application nodes
etcd?   Etcd is not called out in the ref arch. Does it assume all-in-one master?

Infrastructure Setup

The following steps will need to take place whether by hand, or with the benefit of the OpenStack Heat templates from the OpenShift on OpenStack repo.

$ sudo yum -y install rhel-guest-image-7 # or download newer from
$ cp -p /usr/share/rhel-guest-image-7/rhel-guest-image-7*.qcow2 /tmp/
$ virt-customize \
    -a /tmp/rhel-guest-image-7*.qcow2 \
    --root-password password:<default_root_password>
$ qemu-img convert \
    -f qcow2 \
    -O raw \
    /tmp/rhel-guest-image-7*.qcow2 \
$ openstack image create \
    rhel7 \
    --container-format bare \
    --disk-format raw \
    --file /tmp/rhel7.raw \

OpenStack Prerequisites

Many of these steps are included in openshift-ansible-contrib/reference-architecture/osp-cli

#  ansible playbook to extend keystone token expiration
- hosts: controller
  become: true
  become_user: root

  - name: who am i
      msg: ""

  - name: configure keystone token expiration
      dest: /etc/keystone/keystone.conf
      regexp: '^expiration = .*'
      line: 'expiration = 7200'
      backup: yes
    notify: restart keystone

  - name: restart keystone
      name: httpd
      state: restarted
$ openstack project create ocp3 --enable
$ openstack user create ocp3 --email --project ocp3 --enable --password <password>
# create ocp3rc with credentials
$ openstack quota show ocp3
$ openstack quota set ocp3 \
              --cores 60 \
              --gigabytes 2000 \
              --instances 20 \
              --ram $(( 450 * 1024 )) \
              --volumes 30
# save the private key for later
$ openstack keypair create ocp3 > ~/.ssh/ocp3.key
$ openstack keypair show ocp3 --public-key > ~/.ssh/

Deploy a Dyanmic DNS Server

During testing create a DNS server in the project which we can be updated using nsupdate.

git clone
# real DNS servers from environment
dns_forwarders: [10.x.x.41, 10.x.x.2]
update_key: "NOT A REAL KEY"
slave_count: 2

stack_name: dns-service
external_network: public

image: rhel7
flavor: m1.small
ssh_user: cloud-user
ssh_key_name: ocp3

# NOTE: For Red Hat Enterprise Linux:
# rhn_username: "rhnusername"
# rhn_password: "NOT A REAL PASSWORD"
# rhn_pool: "pool id string"
# Either RHN or Sat6
sat6_hostname: ""
sat6_organization: ""
sat6_activationkey: ""
ansible-playbook --private-key ocp3.key -e @dns-vars.yaml -vv\
        openshift-ansible-contrib/reference-architecture/osp-dns/deploy-dns.yaml | tee dns-deploy.log

My network deployed, but I hit an error caused by the Nova scheduler configuration while deploying the instances, so I opened an issue

17-08-27 02:21:14Z [dns-service.hosts]: CREATE_FAILED  ResourceInError: resources.hosts.resources.slaves.resources.slaves.resources[1] Went to status ERROR due to \"Message: ServerGroup policy is not supported: ServerGroupAntiAffinityFilter not configured, Code: 400\"", "2017-08-27 02:21:14Z [dns-service]: CREATE_FAILED  Resource CREATE failed: ResourceInError: resources.hosts.resources.slaves.resources.slaves.resources[1] Went to status ERROR due to \"Message: ServerGroup policy is not supported: ServerGroupAntiAffinityFilter not configured, Code: 400\""]}
[stack@director ~]$ openstack server group list
| ID                                   | Name               | Policies      |
| 0eb9c161-3abd-44a6-80c7-0dea63f2060e | slave_server_group | anti-affinity |

Heat Deployment

Deploy OpenShift on OpenStack using Heat

sudo yum -y install python-heatclient openshift-heat-templates # missing in OSP 11
# or
cd ~stack/templates
git clone