This is a work in progress
The OpenShift Container Platform (OCP) can run on many types of infrastructure; from a Docker contrainer, to a single VM, to a fleet of baremetal or VMs on an infrastructure provider such as RHV, VMware, Amazon EC2, Google Compute Engine, or OpenStack Platform (OSP). This post is to document my experimentation with setting up OCP on OSP.
So where are the docs?
What does the networking for OpenStack look like? Maybe something like this.
| # | Network | VLAN | IP CIDR |
|---|---|---|---|
| N1 | Provisioning (PXE) | V:310 | 172.23.32.0/20 |
| N2 | Internal API | V:311 | 172.23.21.0/24 |
| N3 | Storage Network (Front) | V:312 | 172.23.22.0/24 |
| N4 | Storage Mgmt (Back) | V:313 | 172.23.23.0/24 |
| N5 | External Floating IPs | V:179 | 192.0.179.0/24 |
| N6 | Public API | V:177 | 192.0.177.0/24 |
| N7 | Overcloud Provisioning (Tenant PXE) | V:314 | 172.23.48.0/20 |
| N8 | Provider Network (Tenant VM with physical router) | V:175 | 192.0.175.0/24 |
| N9 | Tenant Network (tunnels) | V:317 | 172.23.96.0/20 |
| N10 | IPMI (iDRAC) | V:315 | 172.23.64.0/20 |
| N11 | Tenant IPMI (iDRAC) | V:316 | 172.23.80.0/20 |
How do I push an image to the OpenShift Docker registry if it is not exposed outside the cluster?
Login to a member node
Get on a machine that has docker and participates in the cluster SDN or can somehow access that network. (eg. 172.30.0.0/16)
Get the IP of the registry
oc get svc docker-registry -n default --template "{{ .spec.clusterIP }}"
SVC_REGISTRY=$(oc get svc docker-registry -n default --template "{{ .spec.clusterIP }}")
Get a token for your session
After running openshift for a while I discovered that letting builds pile up to around to around 1,200 led to what was essentially a deadlock in the scheduling of new builds. New builds were stuck in a New, waiting state indefinitely.
This was fixed as of OCP 3.4.1, but it caused me to get more pro-active in the pruning of artifacts within OpenShift.
I threw together a script and a playbook to deploy it. YMMV
I needed to host a service that would be consumed by a closed client that insists on speaking HTTPS on port 50,000. To solve this, I added a 2nd router deployment and used the OpenShift router sharding feature to selectively enable routes on the 2nd router by way of selectors.
To summarize:
Added HA router:
iptables -A OS_FIREWALL_ALLOW -m tcp -p tcp --dport 49999 -j ACCEPT
iptables -A OS_FIREWALL_ALLOW -m tcp -p tcp --dport 50000 -j ACCEPT
- hosts: infra-nodes
vars:
os_firewall_use_firewalld: False
os_firewall_allow:
- service: teradici-http
port: 49999/tcp
- service: teradici-https
port: 50000/tcp
roles:
- os_firewall
oa adm router or oadm router on these ports and also make sure the stats port does not clash with existing router on port 1936[root@ose-test-master-01 ~]# oc get nodes --show-labels
NAME STATUS AGE LABELS
ose-test-master-01.example.com Ready 180d kubernetes.io/hostname=ose-test-master-01.example.com,region=master,zone=rhev
ose-test-master-02.example.com Ready 180d kubernetes.io/hostname=ose-test-master-02.example.com,region=master,zone=rhev
ose-test-node-01.example.com Ready 180d ha-router=primary,kubernetes.io/hostname=ose-test-node-01.example.com,region=infra,zone=rhev
ose-test-node-02.example.com Ready 180d ha-router=primary,kubernetes.io/hostname=ose-test-node-02.example.com,region=infra,zone=rhev
ose-test-node-03.example.com Ready 180d kubernetes.io/hostname=ose-test-node-03.example.com,region=primary,zone=rhev
ose-test-node-04.example.com Ready 180d kubernetes.io/hostname=ose-test-node-04.example.com,region=primary,zone=rhev
[root@ose-test-master-01 ~]# oadm router ha-router-teradici \
--ports='49999:49999,50000:50000' \
--stats-port=51936 \
--replicas=2 \
--selector="ha-router=primary" \
--selector="region=infra" \
--labels="ha-router=teradici" \
--default-cert=201602_router_wildcard.os.example.com.pem \
--service-account=router
GOOD: I see that the ports are set properly in the haproxy.config and the service objects
OpenShift gathers cluster metrics such as CPU, memory, and network bandwidth per pod which can assist in troubleshooting and capacity planning. The metrics are also used to support horizontal pod autoscaling, which makes the metrics service not just helpful, but critical to operation.
There are 3 major components in the metrics collection process. Heapster gathers stats from Docker and feeds them to Hawkular Metrics to tuck away for safe keeping in Cassandra.
Ever gone to RedHat’s container registry to search for an image and been left wondering what versions exist? Ever been frustrated by the inconsistent tag format? Is there a v or is there not a v? Me too.
Docker Hub has progressed to v2, while the RedHat registry is still v1 at the moment. As long as you use the right syntax, you can use curl to query the registry API and list the tags like this:
Update! I failed with CDK 2.0, but CDK 2.1 works with some fiddling.
In my last post I installed Red Hat Container Developer Kit to deploy OpenShift Enterprise using Vagrant. But now I want to add Hawkular Metrics to that deployment.
Refer to the docs for deploying metrics in OSE.
Login to the vagrant CDK VM before continuing
$ cd ~/cdk/components/rhel/rhel-ose/
$ vagrant ssh
$ oc login
Authentication required for https://127.0.0.1:8443 (openshift)
Username: admin
Password: admin
Login successful.
$ oc project openshift-infra
$ oc get sa
NAME SECRETS AGE
build-controller 2 10m
builder 2 10m
daemonset-controller 2 10m
default 2 10m
deployer 2 10m
deployment-controller 2 10m
gc-controller 2 10m
hpa-controller 2 10m
job-controller 2 10m
namespace-controller 2 10m
pv-binder-controller 2 10m
pv-provisioner-controller 2 10m
pv-recycler-controller 2 10m
replication-controller 2 10m
$ oc create -f - <<API
apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-deployer
secrets:
- name: metrics-deployer
API
$ oc secrets new metrics-deployer nothing=/dev/null
$ oadm policy add-role-to-user edit system:serviceaccount:openshift-infra:metrics-deployer
$ oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:openshift-infra:heapster
From your OSE server grab /usr/share/openshift/examples/infrastructure-templates/enterprise/metrics-deployer.yaml or from here
The RedHat Container Developer Kit allows you to deploy OpenShift on your laptop for easier testing and development. Here is how to deploy it.
Place credentials in ~/.vagrant.d/Vagrantfile to enable updates for VMs by automatically registering with RedHat Subscription Manager
Vagrant.configure('2') do |config|
config.registration.username = '<your Red Hat username>'
config.registration.password = '<your Red Hat password>'
end
Install pre-reqs:
Upgrading from OSE 3.1 to 3.2 using the playbook went quite well for me, but there were a few issues to sort out.
The issues were related to:
Following the directions is pretty straight forward.
Copyright (c) 2021, Dale Bewley; all rights reserved.
Template by Bootstrapious. Ported to Hugo by DevCows.