Bare Metal OpenShift

Perspectives on the IPI Install

7 min readJan 21, 2022

Overview

In two previous posts I’ve discussed:

Why I think OpenShift on baremetal makes sense in the enterprise
What I’m am trying to accomplish with my homelab

In this and subsequent posts I will discuss the how to implement this architecture and the lessons I’ve learned during my journey. As I progress down this journey I will try and reflect on the Why and What where it makes sense.

At a high level the diagram below depicts my end state:

This post will address the lessons I learned from getting OpenShift up and running on baremetal in my homelab.

Approach

My journey toward creating my lab environment begins with the installation of Red Hat OpenShift on my 3 hosts.

There are four ways to complete this install:

Using the Assisted Installer (Tech Preview)
Installer Provisioned Infrastructure (IPI) with Redfish
Installer Provisioned Infrastructure (IPI) with IPMI
User Provisioned Infrastructure

I initially tried the Assisted Installer that is available on the Red Hat Hybrid Console to perform the installation. While I was able to get it installed, the current limitation of a single network within the GUI for the cluster made my infrastructure unstable as I layered on other products such as OpenShift Data Foundation and OpenShift Virtualization. After running my cluster for a few days I decided to abandon that approach and use the Installer Provisioned Infrastructure (IPI) method and I’m glad I did. I have since found a workaround to this limitation, but I’ll leave that for another post.

My hardware is too old to support the Redfish protocol so an IPI Install using IPMI is the direction I chose.

I mean what is a home-lab for if you can’t try different things and learn from them.

Fortunately, the Red Hat documentation on the IPI process that starts here is really quite good. However, there are some nuances that I thought needed some clarification.

Host machines configuration
Provisioner Node
Installation process

Host Machine Configurations

Red Hat recommends that each node type (master, worker, and infrastructure) have an identical configuration. For customer environments that should absolutely hold true.

In my lab I have three hosts that I will use:

2 X HP DL360G7 with different memory configurations
1 X HP DL360G8 with an higher end processor that had more cores than the G7’s

My biggest concern with the different hardware was the embedded 4 port 1Gbit NIC’s on the G8 had different interface names. The documentation here states:

Network interface names must follow the same naming convention across all nodes. For example, the first NIC name on a node, such as eth0 or eno1, must be the same name on all of the other nodes. The same principle applies to the remaining NICs on each node.

Fortunately, In each server I had Intel X520-DA2 dual port 10Gbit SFP+ cards. These cards were nearly identical such that the interface for each of the ports were consistently named.

This allowed me to create 2 X 10Gbit networks to use on my cluster.

ens1f0 — Initially used as my provisioning network and will use it for my storage network for OpenShift Data Foundation. I set this up as non-routable, but accessible by the provisioner node.
ens1f1 — This is a public network that is routable and used for internal cluster communications and communication in and out of the cluster.

Additionally it’s important that whatever NIC you use for the provisioning network be enabled to PXE boot.

A 3rd management network is required as the installer will use IPMI to turn on the machines during the installation process.

The only other nuance on networking that requires a comment is that the provisioning network should be on its own VLAN. The provisioner node will create a DHCP/TFTP server that is used by the installer to assign IP addresses to the provisioning network and as the repository for the boot images. My first installation attempt failed as I did not “VLAN-off” off the provisioner node ports, those connected to ens1f0 and the provisioner network on the provisioner host. The result was that the PXE boot found my “public” DHCP server. Whoops!

The diagram below represents the networking configuration that I used.

Provisioner Node

The provisioner node is required if you are unable to use the Redfish protocol. While it’s not difficult to set up, there are some points that I thought important to mention.

Your provisioner node is your bootstrap machine. Size it appropriately. My initial VM was too small and resulted in a failed installation.

This is the recommended sizing from Red Hat for the bootstrap machine.

vCPU’s — 4
RAM — 16GB
Storage — 100GB

One of the first things the installation process will do is to create the bootstrap node VM. That VM is created in the VM from which you run the installation script. This will require nested virtualization be configured on your host operating system. The complete Red Hat documentation for this process is listed here.

While setting up the provisioner node is not trivial. The the step-by-step documentation provided here is quite good and complete

Once the provisioner node is setup it is then time to create the install-config file that the installer will use.

Depicted below is a copy of my install-config.yaml file. A couple of point to note:

machineCIDR — should be the network that will be used for cluster communications and should be routable
Compute (worker nodes) — since this is a compact cluster where the control plane and worker nodes will run on the same host, you must inform the installer that there are no worker nodes
provisioningNetworkCIDR — should be the network that is used for provisioning. This network should be on a separate VLAN as the provisioner node will create its own DHCP Server for the PXEBoot
bootMACAddress — should the MAC Address of the network ports that will you will be PXE booting from. These ports should be on the same VLAN as the provisioning network.

apiVersion: v1               
baseDomain: dumont-lab.lan   
bootMode: legacy             
metadata:                    
  name: ocp-cluster          
networking:                  
  machineCIDR: 192.168.77.0/24
  networkType: OVNKubernetes 
compute:                     
- name: worker               
  replicas: 0                
controlPlane:                
  name: master               
  replicas: 3                
  platform:                  
    baremetal: {}            
platform:                    
  baremetal:                 
   apiVIP: 192.168.77.40     
    ingressVIP: 192.168.77.42
    provisioningNetworkCIDR: 192.168.55.0/24
    hosts:                   
      - name: dl360g7-1-master-0
        role: master         
        bmc:                 
          address: ipmi://192.168.1.247
          username: username                                                                                                        
          password: password 
        bootMACAddress: a0:36:9f:7b:e5:b4
        rootDeviceHints:     
         deviceName: "/dev/sda"
      - name: dl360g7-2-master-1
        role: master         
        bmc:                 
          address: ipmi://192.168.1.248
          username: username 
          password: password 
        bootMACAddress: a0:36:9f:15:d7:94
        rootDeviceHints:     
         deviceName: "/dev/sda"
      - name: dl360g8-3-master-2
        role: master         
        bmc:                 
          address: ipmi://192.168.1.249
          username: username 
          password: password 
        bootMACAddress: a0:36:9f:52:a3:c8
        rootDeviceHints:     
         deviceName: "/dev/sda"
pullSecret: "from your account in comsole.redhat.com" 
sshKey: "public sshkey"

Installation Process

Once the provisioner node is properly setup and the install-config.yaml is complete you are ready to run the installer using the following two steps:

Prepare the manifests:

./openshift-baremetal-install — dir ~/clusterconfigs create manifests

Execute the installer:

./openshift-baremetal-install — dir ~/clusterconfigs — log-level debug create cluster

If everything is setup properly you should have a completed Red Hat OpenShift Cluster is an hour or so depending on the speed of your network and hardware.

If it does fail, the following steps and documented here should be executed to cleanup the provisioner node and previous cluster configs before running the installer again.

Power off the bare metal nodes

ipmitool -I lanplus -U <user> -P <password> -H <management-server-ip> power off

Cleanup the provisioner node

for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {‘print $2’}); do sudo virsh destroy $i; sudo virsh undefine $i; sudo virsh vol-delete $i — pool $i; sudo virsh vol-delete $i.ign — pool $i; sudo virsh pool-destroy $i; sudo virsh pool-undefine $i; done

Cleanup the clusteconfigs directory:

rm -rf ~/clusterconfigs/auth ~/clusterconfigs/terraform* ~/clusterconfigs/tls ~/clusterconfigs/metadata.json

And try again.

Lessons Learned

It took me several tries to get this right. A couple of lessons that I learned through this process:

Pay close attention to the bare metal host networking. Make sure you understand the port names and connections
Properly setup your networking on your switches. The provisioning node will create its own DHCP/TFTP server that it wants to use. Make sure you don’t let your primary DHCP server give those ports IP addresses
Size your provisioner node correctly, it is your bootstrap node
Follow the step-by-step procedure in the Red Hat documentation to setup the provisioner node — it is complete and works well
Make sure you know how to recover from a failed installation. You probably won’t get it right the first time. If you understand the process to cleanup it will certainly expedite your next install attempt

Please let me know in the comments if you found this useful or if you would suggest