Restricted network bare metal UPI installation on OpenShift v4.6

6 min readNov 22, 2020

OpenShift v4.6 just has been released, it’s a first version supported Extended Update Support(EUS) on OpenShift. I walk you through bare metal installation under restricted network that I reviewed. I hope this article help your practical installation plan.

Summary about the installation

Basically this installation procedures are based on official documentation. But you would need additional systems for its prerequisites, such as bastion server, registry server and so on.

Installing a cluster on bare metal in a restricted network

Overall system diagram is as follows.

DNS records

You should appropriately register all the DNS records to each name server(restricted side and other side name servers). You need to register IPs for API and ingress wild card VIP to each the name server in duplicate.

DNS record list for the restricted network, it’s required those to access to OCP cluster under the restricted network.

DNS record list for the other side network, it’s required those to access to OCP cluster under not restricted network.

Host list

Load Balancer configuration

You can configure a haproxy on bastion host server as follows.

API Load Balancer

Application Ingress Load Balancer

Refer Networking requirements for user-provisioned infrastructure for more specification details of the Load balancer.

You can install the haproxy package on the bastion server, and configure it as follows.

Mirror registry server configuration

First, install the oc CLI with the same version of the OpenShift v4.6.

Refer Installing the CLI on Linux for more details.

You should work those tasks on the image-down host.

user1@image-down ~$ dig +short quay.io
54.210.241.179
52.0.92.170
52.201.153.168
:
user1@image-down ~$ oc version
Client Version: 4.6.1

Store the pull secret downloaded from cloud.redhat.com to a file. Then create a source file for arguments of oc CLI and a directory for mirror images.

user1@image-down ~$ cat <<EOF > pull-secret-for-RH-registries.json
{"auths":{"cloud.openshift.com":{"auth":...}}}
EOF

user1@image-down ~$ mkdir ./copied_images

user1@image-down ~$ cat <<EOF > source_file_for_copied_images
export OCP_RELEASE=4.6.1
export LOCAL_REGISTRY=mirror.reg.priv.local:5000
export LOCAL_REPOSITORY=ocp4/openshift4
export PRODUCT_REPO=openshift-release-dev
export LOCAL_SECRET_JSON=/home/user1/pull-secret-for-RH-registries.json
export RELEASE_NAME=ocp-release
export ARCHITECTURE=x86_64
export REMOVABLE_MEDIA_PATH=/home/user1/copied_images
EOF

user1@image-down ~$ source source_file_for_copied_images

Take a note imageContentSources details from output of “ — dry-run” option for creating install-config.yaml.

Download all the images to the directory you created previously.

Transfer the images to the bastion server using “rsync -a” or “tar”.

Upload all the images to the mirror registry from the bastion server

You should also install oc CLI as the same version with target OCP on the bastion server either. Those tasks are conducted on the bastion server.

user1@bastion ~$ dig +short mirror.reg.priv.local
192.168.9.50

user1@bastion ~$ oc version
Client Version: 4.6.1

Create the pull secret JSON file for the mirror registry, and source file for oc CLI arguments.

Upload all the mirror images to the mirror registry, if you can see the following message, Go lang specification changes about certifications and mirror registry certification created without SAN cause it. You can run the command with “GODEBUG=x509ignoreCN=0” as a workaround of this issue.

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

Refer Need to document TLS CommonName deprecation for more details of this issue.

Originally we should extract openshift-install installer binary from the mirror image, but currently it’s not available due to the known issue. So you should install using the same version installer you can download cloud.redhat.com instead of extracting it from the image.

$ GODEBUG=x509ignoreCN=0 oc adm -a ${LOCAL_SECRET_JSON} release extract \
  --command=openshift-install \
  "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}"
error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:...: unauthorized: access to the requested resource is not authorized

Refer here for the known issue details.

Installation process

Those tasks should be conducted on the bastion server, and you should generate SSH key pair before the following tasks.

Generating an SSH private key and adding it to the agent

And create install-config.yaml with the mirror registry CA, the generated SSH key pair and imageContentSources.

Modify the <your installation directory>/manifests/cluster-scheduler-02-config.yml Kubernetes manifest file to prevent pods from being scheduled on the control plane machines for stable running control plain.

Configure DocumentRoot for providing Ignition files through your web server(httpd).

/var/www/html/ocp46rt/
├── ign
    ├── bootstrap.ign (owner: apache, group: apache, 0644)
    ├── master.ign    (owner: apache, group: apache, 0644)
    └── worker.ign    (owner: apache, group: apache, 0644)

Generate the Ignition files, and copy the files under DocumentRoot.

user2@bastion ~$ openshift-install create ignition-configs --dir install_dir
INFO Consuming Worker Machines from target directory 
INFO Consuming Master Machines from target directory 
INFO Consuming Common Manifests from target directory 
INFO Consuming Openshift Manifests from target directory 
INFO Consuming OpenShift Install (Manifests) from target directory 
INFO Ignition-Configs created in: install_dir and install_dir/auth

user2@bastion ~$ cp install_dir/*.ign /var/www/html/ocp46rt/ign/
user2@bastion ~$ sudo chown apache:apache -R /var/www/html/ocp46rt/*

Ok, we’re ready to run the installer now. You can run Bootstrap node first with rhcos-4.6.1-x86_64-live.x86_64.iso, then Master node 1~3 can run. And you also install RHCOS of each node using coreos-installer. It’s new method and suggested way to initialize RHCOS as of OCPv4.6. But you can also install with the same way like before versions till.

Refer Advanced RHCOS installation reference for more details.

In my environment, DHCP was already active, and I should enforce to configure the static IP addresses using “ — append-karg” for suppressing the DHCP interference at that time. But usually it may configure custom network using nmcli, and the configurations are copied nmconnection files of the NetworkManager by “ — copy-network” option. RHCOS will initialize with the network configuration.

For instance of the master node RHCOS,

Wait for completing the bootstrap process after run Bootstrap and Master nodes.

user2@bastion ~$ openshift-install wait-for bootstrap-complete \
                 --dir install_dir --log-level debug
DEBUG OpenShift Installer 4.6.1                    
DEBUG Built from commit ebdbda57fc18d3b73e69f0f2cc499ddfca7e6593 
INFO Waiting up to 20m0s for the Kubernetes API at https://api.ocp46rt.example.com:6443... 
:
INFO API v1.19.0+d59ce34 up                       
INFO Waiting up to 30m0s for bootstrapping to complete... 
:
INFO It is now safe to remove the bootstrap resources 
DEBUG Time elapsed per stage:                      
DEBUG Bootstrap Complete: 7m58s                    
DEBUG                API: 2m52s                    
INFO Time elapsed: 7m58s

After the bootstrap process, you can log in the OCP cluster and add worker nodes.

user2@bastion ~$ export KUBECONFIG=/home/user2/install_dir/auth/kubeconfig

user2@bastion ~$ oc get node
NAME                         STATUS   ROLES    AGE    VERSION
master1.ocp46rt.priv.local   Ready    master   122m   v1.19.0+d59ce34
master2.ocp46rt.priv.local   Ready    master   64m    v1.19.0+d59ce34
master3.ocp46rt.priv.local   Ready    master   17m    v1.19.0+d59ce34

You can add worker nodes through the same procedures with Master nodes, if worker nodes are added, CSRs are issued and approve them to register worker nodes to the cluster.

After that, you should configure Image Registry appropriately. Refer Image registry storage configuration for more details.

You can complete the installation process after verifying all cluster operator is AVAILABLE: True.

Complete installing

Wow, we can complete to install OCPv4.6 bare metal. You may consider post-installation node tasks after that as follows.

Thank you for reading.