Table of Contents
Introduction
Installing an OKD cluster using the User-Provisioned Infrastructure (UPI) method involves setting up a Kubernetes-based OpenShift platform on either virtual machines or bare metal servers. This approach gives you full control over the underlying infrastructure, allowing for tailored configurations that suit your specific needs. The process typically includes preparing the hardware or virtual environment, configuring network settings, and installing required dependencies. You’ll then deploy the OKD cluster using installation scripts and configuration files provided by OKD, ensuring that each node is correctly integrated into the cluster. This method is ideal for users seeking customization and control over their OKD deployment environment.
Minimum System Requirements
In OKD cluster, except bastion node all other nodes will be installed with Fedora Core OS
| IP Address | Server Name | CPU | Memory | Disk Size |
|---|---|---|---|---|
| 192.168.11.107 | bootstrap.okdcls.linuxsysadmins.lan | 4 | 16 | 100 GB |
| 192.168.11.11 | bastion.okd.linuxsysadmins.lan | 2 | 4 GB | 50 GB |
| 192.168.11.201 | control-plane1.okdcls.linuxsysadmins.lan | 8 | 16 GB | 200 GB |
| 192.168.11.202 | control-plane2.okdcls.linuxsysadmins.lan | 8 | 16 GB | 200 GB |
| 192.168.11.203 | control-plane3.okdcls.linuxsysadmins.lan | 8 | 16 GB | 200 GB |
| 192.168.11.204 | compute1.okdcls.linuxsysadmins.lan | 8 | 16 GB | 200 GB |
| 192.168.11.205 | compute2.okdcls.linuxsysadmins.lan | 8 | 16 GB | 200 GB |
| 192.168.11.206 | compute3.okdcls.linuxsysadmins.lan | 8 | 16 GB | 200 GB |
DNS Requirement
If you are looking to setup the DNS server for 3 node OKD Cluster refer to below URL.
| Component | DNS Record |
|---|---|
| Kubernetes API | api.okdcls.linuxsysadmins.lan api-int.okdcls.linuxsysadmins.lan |
| Routes | *.apps.okdcls.linuxsysadmins.lan |
| Bootstrap machine | bootstrap.okdcls.linuxsysadmins.lan |
| Control plane machines | control-plane1.okdcls.linuxsysadmins.lan control-plane2.okdcls.linuxsysadmins.lan control-plane3.okdcls.linuxsysadmins.lan |
| Compute machines | compute1.okdcls.linuxsysadmins.lan compute2.okdcls.linuxsysadmins.lan compute3.okdcls.linuxsysadmins.lan |
DHCP Requirement
To assign the IP for bootstrap, master and worker nodes during the bootstrapping we need to have a DHCP server.
We have an existing article on setting up DHCP server for a single node cluster.
The updated DHCP article for 3 node OKD cluster can be found below
Load Balancing Requirement
We need to have a working LB to distribute the API (6443/TCP, 22623/TCP) and application Ingress traffics (443/TCP, 80/TCP), an existing guide on setting up HAProxy available from here.
Before proceeding with further steps make sure to verify all DNS are resolvable.
$ dig +noall +answer 192.168.11.100 api.okdcls.linuxsysadmins.lan
$ dig +noall +answer 192.168.11.100 api-int.okdcls.linuxsysadmins.lan
$ dig +noall +answer 192.168.11.100 console-openshift-console.apps.okdcls.linuxsysadmins.la
$ dig +noall +answer 192.168.11.100 someapp.apps.okdcls.linuxsysadmins.lan
$ dig +noall +answer 192.168.11.107 bootstrap.okdcls.linuxsysadmins.lan
$ dig +noall +answer 192.168.11.100 -x 192.168.11.11
$ dig +noall +answer 192.168.11.100 -x 192.168.11.201
$ dig +noall +answer 192.168.11.100 -x 192.168.11.202
$ dig +noall +answer 192.168.11.100 -x 192.168.11.203
$ dig +noall +answer 192.168.11.100 -x 192.168.11.204Installation URL Exclusions
OKD Cluster firewall requirement is a huge list, it can be found here.
SSH Authentication
After installing the cluster, if troubleshooting is required, we need to access the control plane or worker nodes via SSH. To log in to the cluster nodes, we must copy the SSH keys from the bastion nodes to the cluster nodes for the coreos user. While this process is handled by the installer, it is necessary to generate an SSH key and append it to the prepared config file.
Obtain a Pull secret
To access content from the container catalog registry, such as images and operators, a valid pull secret is required. This secret can be obtained through the Red Hat OpenShift Cluster Manager cloud console. Please note that the pull secret is only valid for 12 to 24 hours. For instance, if you initiate cluster preparation today and plan to resume the installation tomorrow at the same time, a new pull secret will be necessary; otherwise, the installation may fail.
https://console.redhat.com/openshift/install/pull-secretCreating Installation configuration file
Prepare the install-config.yaml appropriate to your environment requirement.
- Here the my base domain is linuxsysadmins.lan
- The replica for worker node should be 0, we need to manually install the worker node after the control-planes.
- In my case okdcls is the cluster name.
- My all okdcls cluster nodes will reside under 192.168.11.0/24 subnet
$ cat > install-config.yamlMake the necessary changes.
apiVersion: v1
baseDomain: linuxsysadmins.lan
compute:
- hyperthreading: Enabled
name: worker
replicas: 0
controlPlane:
hyperthreading: Enabled
name: master
replicas: 3
metadata:
name: okdcls
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
pullSecret: '{"auth.com"}}}'
sshKey: 'ssh-rsa AAAAIGkrSprpi7Dc= ansible@bastion.okd.linuxsysadmins.lan'
Append the prepared SSH key and pull secret with in single quote ”.
Preparing OKD Cluster Installer and Client (OC)
Download the OKD Cluster installer and oc client packages from below URL.
Pick your right version from official GitHub Repo.
While preparing this post the version used is 4.15.0-0.okd-2024-03-10-010116
https://github.com/okd-project/okd/releasesSet the version and arch as a variable
$ OKD_VERSION=4.15.0-0.okd-2024-03-10-010116
$ ARCH=x86_64Download the client and installer tar files, or manually obtain them by selecting matching versions from the official GitHub URL mentioned above.
$ cd ~
$ curl -L https://github.com/okd-project/okd/releases/download/$OKD_VERSION/openshift-client-linux-$OKD_VERSION.tar.gz -o oc.tar.gz
$ curl -L https://github.com/okd-project/okd/releases/download/$OKD_VERSION/openshift-install-linux-$OKD_VERSION.tar.gz -o openshift-install-linux.tar.gzExtract the client packages and move /usr/local/bin
$ tar xvf openshift-client-linux-4.15.0-0.okd-2024-03-10-010116.tar.gz
$ mv oc kubectl /usr/local/bin/
$ restorecon -RFv /usr/local/bin/*Extract the OKD Cluster Installer
$ mkdir 3nodecls
$ tar xvf openshift-install-linux-4.15.0-0.okd-2024-03-10-010116.tar.gzGenerating k8s manifest and Ignition config files
Once we are done with all above mandatory requirements, prepare the ignition files
$ cp install-config.yaml 3nodecls/Generate the k8s manifests and ignition config files.
$ ./openshift-install create manifests --dir 3nodecls[ansible@bastion 3nodecls]$ tree .
.
├── manifests
│  ├── cluster-config.yaml
│  ├── cluster-dns-02-config.yml
│  ├── cluster-infrastructure-02-config.yml
│  ├── cluster-ingress-02-config.yml
│  ├── cluster-network-02-config.yml
│  ├── cluster-proxy-01-config.yaml
│  ├── cluster-scheduler-02-config.yml
│  ├── cvo-overrides.yaml
│  ├── kube-cloud-config.yaml
│  ├── kube-system-configmap-root-ca.yaml
│  ├── machine-config-server-tls-secret.yaml
│  └── openshift-config-secret-pull-secret.yaml
└── openshift
├── 99_feature-gate.yaml
├── 99_kubeadmin-password-secret.yaml
├── 99_openshift-cluster-api_master-user-data-secret.yaml
├── 99_openshift-cluster-api_worker-user-data-secret.yaml
├── 99_openshift-machineconfig_99-master-ssh.yaml
├── 99_openshift-machineconfig_99-worker-ssh.yaml
└── openshift-install-manifests.yaml
2 directories, 19 files
[ansible@bastion 3nodecls]$
After executing the ignition command, only the ignition and authentication-related files will be present in the installation directory.
$ ./openshift-install create ignition-configs --dir 3nodecls[ansible@bastion 3nodecls]$ tree . . ├── auth │  ├── kubeadmin-password │  └── kubeconfig ├── bootstrap.ign ├── master.ign ├── metadata.json └── worker.ign 1 directory, 6 files [ansible@bastion 3nodecls]$
The final output will have similar to above.
Ignition file distribution
To perform the bootstrap the ignition file should be available to all the nodes, to distribute the ignition file we need to have a web server.
$ dnf install httpd -y
$ systemctl enable httpd --now
$ firewalld-cmd --add-service=http --permanentCopy all the ignition files to anyone of web server.
$ scp *.ign root@192.168.0.105:/var/www/html/okd/Restore the SELinux labels by running $ sudo restorecon -RFv /var/www/html/okd/*
Make sure to access the ignition file from anyone of node in the same subnet.
$ curl http://192.168.0.105/okd/bootstrap.ign
$ curl http://192.168.0.105/okd/master.ign
$ curl http://192.168.0.105/okd/worker.ignObtain Fedora CoreOS ISO
Installing Fedora CoreOS on all cluster nodes will be performed using an ISO image, download the Fedora CoreOS using below commands
[ansible@bastion ~]$ ./openshift-install coreos print-stream-json | grep '\.iso[^.]'
"location": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20231101.3.0/aarch64/fedora-coreos-39.20231101.3.0-live.aarch64.iso",
"location": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20231101.3.0/ppc64le/fedora-coreos-39.20231101.3.0-live.ppc64le.iso",
"location": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20231101.3.0/s390x/fedora-coreos-39.20231101.3.0-live.s390x.iso",
"location": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20231101.3.0/x86_64/fedora-coreos-39.20231101.3.0-live.x86_64.iso",
[ansible@bastion ~]$
Download the x86_64 ISO using wget and copy to the ISO file to proxmox ISO datastore.
Bootstrapping OKD Cluster using FCOS ISO
First, transfer the ISO image from the Bastion node and upload it to the Proxmox ISO data store. Then, attach the ISO to all cluster nodes, adjust the boot order as needed, and start the virtual machine.
Initially, start the bootstrap process on both the bootstrap and master nodes. Once booting is complete, you should have access to a command line interface under the coreos user.
Verify that our API is resolvable from current shell. If any of the DNS entries cannot be resolved, the installation will fail. Ensure that DNS resolution issues are resolved before proceeding.
$ curl -v https://api.okdcls.linuxsysadmins.lan:6443By pointing the ignition file and available local disk install the Fedora CoreOS.
$ sudo coreos-installer install --ignition-url=http://192.168.0.105/okd/bootstrap.ign /dev/sda --insecure-ignition
$ sudo coreos-installer install --ignition-url=http://192.168.0.105/okd/master.ign /dev/sda --insecure-ignition
$ sudo coreos-installer install --ignition-url=http://192.168.0.105/okd/worker.ign /dev/sda --insecure-ignitionEnsure that you use the correct ignition files for each node sequentially. While the master nodes are being configured, begin bootstrapping the remaining worker nodes.
The installation process typically takes 2 to 3 minutes to complete. Once the installation is finished, shut down the nodes, unmount the ISO from the virtual machine, and adjust the boot order to prioritize booting from the disk.
Monitoring the OKD Cluster bootstrap progress
By running below command from the bastion node we can monitor the installation progress.
$ ./openshift-install --dir 3nodecls/ wait-for bootstrap-complete --log-level=infoIt takes more than 30 minutes and depends on your assigned resources.
[ansible@bastion ~]$ ./openshift-install --dir 3nodecls/ wait-for bootstrap-complete --log-level=info
INFO Waiting up to 20m0s (until 5:01PM +04) for the Kubernetes API at https://api.okdcls.linuxsysadmins.lan:6443...
INFO API v1.28.2-3598+6e2789bbd58938-dirty up
INFO Waiting up to 30m0s (until 5:14PM +04) for bootstrapping to complete...
INFO It is now safe to remove the bootstrap resources
INFO Time elapsed: 28m34s
[ansible@bastion ~]$Once we get to see the above bootstrap status message its safe to shutdown or remove the bootstrap server from the cluster. Make sure to disable the DNS entries, DHCP entries and LB entry from the HAProxy.
Verifying the Installation
To verify the installation, log in to the cluster using the kubeconfig file that was generated during the manifest generation stage.
[ansible@bastion ~]$ export KUBECONFIG=3nodecls/auth/kubeconfigPrint the user who is authenticated and print the available nodes after the bootstrapping.
[ansible@bastion ~]$ oc get nodes
NAME STATUS ROLES AGE VERSION
control-plane1.okdcls.linuxsysadmins.lan Ready control-plane,master,worker 28m v1.28.7+6e2789b
control-plane2.okdcls.linuxsysadmins.lan Ready control-plane,master,worker 28m v1.28.7+6e2789b
control-plane3.okdcls.linuxsysadmins.lan Ready control-plane,master,worker 28m v1.28.7+6e2789b
[ansible@bastion ~]$
Here we could see the compute nodes as not listed. To bring the compute nodes we need to approve them.
Approve the certificate request from worker nodes.
[ansible@bastion ~]$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve
certificatesigningrequest.certificates.k8s.io/csr-4j486 approved
certificatesigningrequest.certificates.k8s.io/csr-9tv4s approved
certificatesigningrequest.certificates.k8s.io/csr-s4csx approved
[ansible@bastion ~]$ Now we are good with listing all the nodes.
[ansible@bastion ~]$ oc get nodes
NAME STATUS ROLES AGE VERSION
compute1.okdcls.linuxsysadmins.lan Ready worker 5m49s v1.28.7+6e2789b
compute2.okdcls.linuxsysadmins.lan Ready worker 5m34s v1.28.7+6e2789b
compute3.okdcls.linuxsysadmins.lan Ready worker 5m57s v1.28.7+6e2789b
control-plane1.okdcls.linuxsysadmins.lan Ready control-plane,master 36m v1.28.7+6e2789b
control-plane2.okdcls.linuxsysadmins.lan Ready control-plane,master 36m v1.28.7+6e2789b
control-plane3.okdcls.linuxsysadmins.lan Ready control-plane,master 36m v1.28.7+6e2789b
[ansible@bastion ~]$ List all the nodes using -o wide option.
[ansible@bastion ~]$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
compute1.okdcls.linuxsysadmins.lan Ready worker 145m v1.28.7+6e2789b 192.168.11.204 <none> Fedora CoreOS 39.20240210.3.0 6.7.4-200.fc39.x86_64 cri-o://1.28.2
compute2.okdcls.linuxsysadmins.lan Ready worker 145m v1.28.7+6e2789b 192.168.11.205 <none> Fedora CoreOS 39.20240210.3.0 6.7.4-200.fc39.x86_64 cri-o://1.28.2
compute3.okdcls.linuxsysadmins.lan Ready worker 145m v1.28.7+6e2789b 192.168.11.206 <none> Fedora CoreOS 39.20240210.3.0 6.7.4-200.fc39.x86_64 cri-o://1.28.2
control-plane1.okdcls.linuxsysadmins.lan Ready control-plane,master 176m v1.28.7+6e2789b 192.168.11.201 <none> Fedora CoreOS 39.20240210.3.0 6.7.4-200.fc39.x86_64 cri-o://1.28.2
control-plane2.okdcls.linuxsysadmins.lan Ready control-plane,master 176m v1.28.7+6e2789b 192.168.11.202 <none> Fedora CoreOS 39.20240210.3.0 6.7.4-200.fc39.x86_64 cri-o://1.28.2
control-plane3.okdcls.linuxsysadmins.lan Ready control-plane,master 176m v1.28.7+6e2789b 192.168.11.203 <none> Fedora CoreOS 39.20240210.3.0 6.7.4-200.fc39.x86_64 cri-o://1.28.2
[ansible@bastion ~]$List all the pods across all the namespaces.
[ansible@bastion ~]$ oc get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
openshift-apiserver-operator openshift-apiserver-operator-66d44b7798-xqz9n 1/1 Running 1 (27h ago) 27h
openshift-apiserver apiserver-86987b956-ctjlt 2/2 Running 0 27h
openshift-apiserver apiserver-86987b956-nbm9k 2/2 Running 0 27h
openshift-apiserver apiserver-86987b956-qh4nk 2/2 Running 1 (27h ago) 27h
openshift-authentication-operator authentication-operator-766d9779d4-fzd7k 1/1 Running 1 (27h ago) 27h
openshift-authentication oauth-openshift-7bc695c5b7-qv85w 1/1 Running 1 (27h ago) 27h
openshift-authentication oauth-openshift-7bc695c5b7-t9mxb 1/1 Running 0 27h
openshift-authentication oauth-openshift-7bc695c5b7-w2qwf 1/1 Running 0 27h
openshift-cloud-controller-manager-operator cluster-cloud-controller-manager-operator-66b4887675-pbm6j 3/3 Running 6 (27h ago) 27h
openshift-cloud-credential-operator cloud-credential-operator-5db944c458-gx9ww 2/2 Running 0 27h
openshift-cluster-machine-approver machine-approver-77b9fcf5c7-f5rcj 2/2 Running 1 (27h ago) 27h
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-64c9946fd-jqbd2 1/1 Running 0 27h
openshift-cluster-node-tuning-operator tuned-6m9st 1/1 Running 0 27h
openshift-cluster-node-tuning-operator tuned-bq9bh 1/1 Running 0 27h
openshift-cluster-node-tuning-operator tuned-bshh8 1/1 Running 0 27h
openshift-cluster-node-tuning-operator tuned-nlqhz 1/1 Running 0 27h
openshift-cluster-node-tuning-operator tuned-v7sz9 1/1 Running 0 27h
openshift-cluster-node-tuning-operator tuned-wf2w9 1/1 Running 0 27h
openshift-cluster-samples-operator cluster-samples-operator-54985c54c9-mpt6w 2/2 Running 0 27h
openshift-cluster-storage-operator cluster-storage-operator-7665d759d6-dnh42 1/1 Running 1 (27h ago) 27h
openshift-cluster-storage-operator csi-snapshot-controller-585dd5b969-9rzdl 1/1 Running 0 27h
openshift-cluster-storage-operator csi-snapshot-controller-585dd5b969-h7zz4 1/1 Running 0 27h
openshift-cluster-storage-operator csi-snapshot-controller-operator-7ccf586476-hkk9l 1/1 Running 0 27h
openshift-cluster-storage-operator csi-snapshot-webhook-56c558db8-7p9xq 1/1 Running 0 27h
openshift-cluster-storage-operator csi-snapshot-webhook-56c558db8-tgjz2 1/1 Running 0 27h
openshift-cluster-version cluster-version-operator-7b85459cfd-xrtsq 1/1 Running 0 27h
openshift-config-operator openshift-config-operator-c645b84c6-z5pc5 1/1 Running 1 (27h ago) 27h
openshift-console-operator console-operator-7b9c78dccc-vxw6h 2/2 Running 0 27h
openshift-console console-7cfd466999-bpwph 1/1 Running 10 (25h ago) 25h
openshift-console console-7cfd466999-s8l79 1/1 Running 10 (25h ago) 25h
openshift-console downloads-6d5dc6fc54-8qzxb 1/1 Running 1 (27h ago) 27h
openshift-console downloads-6d5dc6fc54-lds4d 1/1 Running 0 27h
openshift-controller-manager-operator openshift-controller-manager-operator-9f84455b9-pnfps 1/1 Running 1 (27h ago) 27h
openshift-controller-manager controller-manager-64d688f85c-ltkhv 1/1 Running 0 20h
openshift-controller-manager controller-manager-64d688f85c-xld8t 1/1 Running 0 20h
openshift-controller-manager controller-manager-64d688f85c-z6zgg 1/1 Running 0 20h
openshift-dns-operator dns-operator-668f779d47-bx8t4 2/2 Running 0 27h
openshift-dns dns-default-2vxj6 2/2 Running 0 27h
openshift-dns dns-default-5kv8w 2/2 Running 0 27h
openshift-dns dns-default-8sjj6 2/2 Running 0 27h
openshift-dns dns-default-l9xkw 2/2 Running 0 27h
openshift-dns dns-default-tm64d 2/2 Running 0 27h
openshift-dns dns-default-vwz7c 2/2 Running 0 27h
openshift-dns node-resolver-4jkbk 1/1 Running 0 27h
openshift-dns node-resolver-7w4f8 1/1 Running 0 27h
openshift-dns node-resolver-882jj 1/1 Running 0 27h
openshift-dns node-resolver-kvd84 1/1 Running 0 27h
openshift-dns node-resolver-slxn7 1/1 Running 0 27h
openshift-dns node-resolver-wv446 1/1 Running 0 27h
openshift-etcd-operator etcd-operator-64f9b89996-84fmt 1/1 Running 1 (27h ago) 27h
openshift-etcd etcd-control-plane1.okdcls.linuxsysadmins.lan 4/4 Running 0 27h
openshift-etcd etcd-control-plane2.okdcls.linuxsysadmins.lan 4/4 Running 0 27h
openshift-etcd etcd-control-plane3.okdcls.linuxsysadmins.lan 4/4 Running 0 27h
openshift-etcd etcd-guard-control-plane1.okdcls.linuxsysadmins.lan 1/1 Running 0 27h
openshift-etcd etcd-guard-control-plane2.okdcls.linuxsysadmins.lan 1/1 Running 0 27h
openshift-etcd etcd-guard-control-plane3.okdcls.linuxsysadmins.lan 1/1 Running 0 27h
openshift-etcd installer-4-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd installer-6-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd installer-6-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd installer-8-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd installer-8-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd installer-8-retry-1-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd installer-9-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd installer-9-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd installer-9-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd revision-pruner-8-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd revision-pruner-8-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd revision-pruner-8-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd revision-pruner-9-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd revision-pruner-9-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-etcd revision-pruner-9-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-image-registry cluster-image-registry-operator-58ff44549-gj52d 1/1 Running 0 27h
openshift-image-registry image-pruner-28715040-vhxt7 0/1 Completed 0 16h
openshift-image-registry node-ca-27b4x 1/1 Running 0 27h
openshift-image-registry node-ca-dvr4w 1/1 Running 0 27h
openshift-image-registry node-ca-lm8mc 1/1 Running 0 27h
openshift-image-registry node-ca-mskfg 1/1 Running 0 27h
openshift-image-registry node-ca-qqg7j 1/1 Running 0 27h
openshift-image-registry node-ca-s59h4 1/1 Running 0 27h
openshift-ingress-canary ingress-canary-5rdst 1/1 Running 0 27h
openshift-ingress-canary ingress-canary-6vpj4 1/1 Running 0 27h
openshift-ingress-canary ingress-canary-8ftfp 1/1 Running 0 27h
openshift-ingress-canary ingress-canary-9sm8q 1/1 Running 0 27h
openshift-ingress-canary ingress-canary-rfkqm 1/1 Running 0 27h
openshift-ingress-canary ingress-canary-rl8tr 1/1 Running 0 27h
openshift-ingress-operator ingress-operator-6d9c9b9d7c-4bxrw 2/2 Running 2 (27h ago) 27h
openshift-ingress router-default-68d4d5649d-g7ljq 1/1 Running 0 25h
openshift-ingress router-default-68d4d5649d-pvmbf 1/1 Running 0 25h
openshift-insights insights-operator-79458b87cc-kjmr4 1/1 Running 1 (27h ago) 27h
openshift-kube-apiserver-operator kube-apiserver-operator-5c5b686ff9-d8jtq 1/1 Running 1 (27h ago) 27h
openshift-kube-apiserver installer-10-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 25h
openshift-kube-apiserver installer-10-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 25h
openshift-kube-apiserver installer-10-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 25h
openshift-kube-apiserver installer-11-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 22h
openshift-kube-apiserver installer-11-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 22h
openshift-kube-apiserver installer-11-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 22h
openshift-kube-apiserver installer-12-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 22h
openshift-kube-apiserver installer-12-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 22h
openshift-kube-apiserver installer-12-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 22h
openshift-kube-apiserver installer-13-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 21h
openshift-kube-apiserver installer-13-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 21h
openshift-kube-apiserver installer-13-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 21h
openshift-kube-apiserver installer-14-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 20h
openshift-kube-apiserver installer-14-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 20h
openshift-kube-apiserver installer-14-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 20h
openshift-kube-apiserver kube-apiserver-control-plane1.okdcls.linuxsysadmins.lan 5/5 Running 0 20h
openshift-kube-apiserver kube-apiserver-control-plane2.okdcls.linuxsysadmins.lan 5/5 Running 0 20h
openshift-kube-apiserver kube-apiserver-control-plane3.okdcls.linuxsysadmins.lan 5/5 Running 0 20h
openshift-kube-apiserver kube-apiserver-guard-control-plane1.okdcls.linuxsysadmins.lan 1/1 Running 0 27h
openshift-kube-apiserver kube-apiserver-guard-control-plane2.okdcls.linuxsysadmins.lan 1/1 Running 0 27h
openshift-kube-apiserver kube-apiserver-guard-control-plane3.okdcls.linuxsysadmins.lan 1/1 Running 0 27h
openshift-kube-apiserver revision-pruner-10-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 25h
openshift-kube-apiserver revision-pruner-10-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 25h
openshift-kube-apiserver revision-pruner-10-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 25h
openshift-kube-apiserver revision-pruner-11-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 22h
openshift-kube-apiserver revision-pruner-11-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 22h
openshift-kube-apiserver revision-pruner-11-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 22h
openshift-kube-apiserver revision-pruner-12-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 22h
openshift-kube-apiserver revision-pruner-12-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 22h
openshift-kube-apiserver revision-pruner-12-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 22h
openshift-kube-apiserver revision-pruner-13-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 21h
openshift-kube-apiserver revision-pruner-13-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 21h
openshift-kube-apiserver revision-pruner-13-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 21h
openshift-kube-apiserver revision-pruner-14-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 20h
openshift-kube-apiserver revision-pruner-14-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 20h
openshift-kube-apiserver revision-pruner-14-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 20h
openshift-kube-controller-manager-operator kube-controller-manager-operator-58f696cbbf-cb2f8 1/1 Running 1 (27h ago) 27h
openshift-kube-controller-manager installer-4-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-controller-manager installer-4-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-controller-manager installer-5-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-controller-manager installer-5-retry-1-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-controller-manager installer-6-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-controller-manager installer-6-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-controller-manager installer-6-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-controller-manager kube-controller-manager-control-plane1.okdcls.linuxsysadmins.lan 4/4 Running 1 (27h ago) 27h
openshift-kube-controller-manager kube-controller-manager-control-plane2.okdcls.linuxsysadmins.lan 4/4 Running 0 27h
openshift-kube-controller-manager kube-controller-manager-control-plane3.okdcls.linuxsysadmins.lan 4/4 Running 1 (27h ago) 27h
openshift-kube-controller-manager kube-controller-manager-guard-control-plane1.okdcls.linuxsysadmins.lan 1/1 Running 0 27h
openshift-kube-controller-manager kube-controller-manager-guard-control-plane2.okdcls.linuxsysadmins.lan 1/1 Running 0 27h
openshift-kube-controller-manager kube-controller-manager-guard-control-plane3.okdcls.linuxsysadmins.lan 1/1 Running 0 27h
openshift-kube-scheduler-operator openshift-kube-scheduler-operator-dc89c89fc-m7w2w 1/1 Running 1 (27h ago) 27h
openshift-kube-scheduler installer-6-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-scheduler installer-6-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-scheduler installer-7-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-scheduler installer-7-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-scheduler installer-7-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-scheduler openshift-kube-scheduler-control-plane1.okdcls.linuxsysadmins.lan 3/3 Running 0 27h
openshift-kube-scheduler openshift-kube-scheduler-control-plane2.okdcls.linuxsysadmins.lan 3/3 Running 0 27h
openshift-kube-scheduler openshift-kube-scheduler-control-plane3.okdcls.linuxsysadmins.lan 3/3 Running 0 27h
openshift-kube-scheduler openshift-kube-scheduler-guard-control-plane1.okdcls.linuxsysadmins.lan 1/1 Running 0 27h
openshift-kube-scheduler openshift-kube-scheduler-guard-control-plane2.okdcls.linuxsysadmins.lan 1/1 Running 0 27h
openshift-kube-scheduler openshift-kube-scheduler-guard-control-plane3.okdcls.linuxsysadmins.lan 1/1 Running 0 27h
openshift-kube-scheduler revision-pruner-7-control-plane1.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-scheduler revision-pruner-7-control-plane2.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-scheduler revision-pruner-7-control-plane3.okdcls.linuxsysadmins.lan 0/1 Completed 0 27h
openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-7df68c455f-zn6fz 1/1 Running 1 (27h ago) 27h
openshift-kube-storage-version-migrator migrator-7dc4b56b9c-jgq26 1/1 Running 0 27h
openshift-machine-api cluster-autoscaler-operator-64759f8bc6-5vb8x 2/2 Running 0 27h
openshift-machine-api cluster-baremetal-operator-5785c4dd4d-b9prf 2/2 Running 0 27h
openshift-machine-api control-plane-machine-set-operator-7fbb586545-mvc28 1/1 Running 0 27h
openshift-machine-api machine-api-operator-6599bd4cbd-ps57l 2/2 Running 0 27h
openshift-machine-config-operator machine-config-controller-84c8d5686b-qxbfs 2/2 Running 0 27h
openshift-machine-config-operator machine-config-daemon-4czg7 2/2 Running 0 27h
openshift-machine-config-operator machine-config-daemon-5zxh8 2/2 Running 0 27h
openshift-machine-config-operator machine-config-daemon-bb6qm 2/2 Running 0 27h
openshift-machine-config-operator machine-config-daemon-nchk6 2/2 Running 0 27h
openshift-machine-config-operator machine-config-daemon-trb44 2/2 Running 0 27h
openshift-machine-config-operator machine-config-daemon-x2t7m 2/2 Running 0 27h
openshift-machine-config-operator machine-config-operator-5974f7756-pw455 2/2 Running 0 27h
openshift-machine-config-operator machine-config-server-27vsk 1/1 Running 0 27h
openshift-machine-config-operator machine-config-server-nhmv8 1/1 Running 0 27h
openshift-machine-config-operator machine-config-server-xzfsq 1/1 Running 0 27h
openshift-marketplace community-operators-82fpz 1/1 Running 0 27h
openshift-marketplace marketplace-operator-6cccff5dfc-t6nhc 1/1 Running 0 27h
openshift-monitoring alertmanager-main-0 6/6 Running 0 27h
openshift-monitoring alertmanager-main-1 6/6 Running 0 27h
openshift-monitoring cluster-monitoring-operator-76dd7766d8-wcwm7 1/1 Running 0 27h
openshift-monitoring kube-state-metrics-64d7486765-5v5mw 3/3 Running 0 27h
openshift-monitoring monitoring-plugin-59464d9c55-pk5m2 1/1 Running 0 27h
openshift-monitoring monitoring-plugin-59464d9c55-pz4qm 1/1 Running 0 27h
openshift-monitoring node-exporter-5bn4p 2/2 Running 0 27h
openshift-monitoring node-exporter-8l487 2/2 Running 0 27h
openshift-monitoring node-exporter-cqp7x 2/2 Running 0 27h
openshift-monitoring node-exporter-h2pg4 2/2 Running 0 27h
openshift-monitoring node-exporter-tzcxt 2/2 Running 0 27h
openshift-monitoring node-exporter-w2r97 2/2 Running 0 27h
openshift-monitoring openshift-state-metrics-b7f48f7cf-8hrpl 3/3 Running 0 27h
openshift-monitoring prometheus-adapter-8bb7ddf8d-m8gcq 1/1 Running 0 20h
openshift-monitoring prometheus-adapter-8bb7ddf8d-ncvc6 1/1 Running 0 20h
openshift-monitoring prometheus-k8s-0 6/6 Running 0 27h
openshift-monitoring prometheus-k8s-1 6/6 Running 0 27h
openshift-monitoring prometheus-operator-7c5d9bbff5-txm79 2/2 Running 0 27h
openshift-monitoring prometheus-operator-admission-webhook-7dbd4f5c44-2qxzw 1/1 Running 0 27h
openshift-monitoring prometheus-operator-admission-webhook-7dbd4f5c44-sqgxc 1/1 Running 0 27h
openshift-monitoring telemeter-client-5d69d47bb5-hr54l 3/3 Running 0 27h
openshift-monitoring thanos-querier-5896947ffd-nl5xd 6/6 Running 0 27h
openshift-monitoring thanos-querier-5896947ffd-xjdxg 6/6 Running 0 27h
openshift-multus multus-2wnrl 1/1 Running 1 (27h ago) 27h
openshift-multus multus-9xmx4 1/1 Running 0 27h
openshift-multus multus-9zm2z 1/1 Running 1 (27h ago) 27h
openshift-multus multus-additional-cni-plugins-cgwrg 1/1 Running 0 27h
openshift-multus multus-additional-cni-plugins-hq74h 1/1 Running 0 27h
openshift-multus multus-additional-cni-plugins-lgvkk 1/1 Running 0 27h
openshift-multus multus-additional-cni-plugins-spz27 1/1 Running 0 27h
openshift-multus multus-additional-cni-plugins-thtgf 1/1 Running 0 27h
openshift-multus multus-additional-cni-plugins-x68s4 1/1 Running 0 27h
openshift-multus multus-admission-controller-88ff5469d-g2x92 2/2 Running 0 27h
openshift-multus multus-admission-controller-88ff5469d-qlzw8 2/2 Running 0 27h
openshift-multus multus-f9n6l 1/1 Running 2 (27h ago) 27h
openshift-multus multus-zntl2 1/1 Running 1 (27h ago) 27h
openshift-multus multus-zvg44 1/1 Running 0 27h
openshift-multus network-metrics-daemon-5q59d 2/2 Running 0 27h
openshift-multus network-metrics-daemon-czszz 2/2 Running 0 27h
openshift-multus network-metrics-daemon-hw9dk 2/2 Running 0 27h
openshift-multus network-metrics-daemon-jpxxb 2/2 Running 0 27h
openshift-multus network-metrics-daemon-lfv6r 2/2 Running 0 27h
openshift-multus network-metrics-daemon-vjprt 2/2 Running 0 27h
openshift-network-diagnostics network-check-source-5f9b68d7f8-xz8z6 1/1 Running 0 27h
openshift-network-diagnostics network-check-target-4n65j 1/1 Running 0 27h
openshift-network-diagnostics network-check-target-5fnqx 1/1 Running 0 27h
openshift-network-diagnostics network-check-target-g66w2 1/1 Running 0 27h
openshift-network-diagnostics network-check-target-gk82x 1/1 Running 0 27h
openshift-network-diagnostics network-check-target-qtg9x 1/1 Running 0 27h
openshift-network-diagnostics network-check-target-rqb6c 1/1 Running 0 27h
openshift-network-node-identity network-node-identity-h89ll 2/2 Running 0 27h
openshift-network-node-identity network-node-identity-mzrwl 2/2 Running 0 27h
openshift-network-node-identity network-node-identity-xdf7m 2/2 Running 0 27h
openshift-network-operator network-operator-656d4d696f-4s2cv 1/1 Running 1 (27h ago) 27h
openshift-oauth-apiserver apiserver-7df57cbfff-wz2dz 1/1 Running 0 27h
openshift-oauth-apiserver apiserver-7df57cbfff-z7t29 1/1 Running 0 27h
openshift-oauth-apiserver apiserver-7df57cbfff-zghjw 1/1 Running 0 27h
openshift-operator-lifecycle-manager catalog-operator-5db8cc987c-kjqv7 1/1 Running 0 27h
openshift-operator-lifecycle-manager collect-profiles-28716000-wffdw 0/1 Completed 0 43m
openshift-operator-lifecycle-manager collect-profiles-28716015-mvmn8 0/1 Completed 0 28m
openshift-operator-lifecycle-manager collect-profiles-28716030-mpz5w 0/1 Completed 0 13m
openshift-operator-lifecycle-manager olm-operator-65b47cf747-mzk52 1/1 Running 0 27h
openshift-operator-lifecycle-manager package-server-manager-7d888667b7-p5cq4 2/2 Running 1 (27h ago) 27h
openshift-operator-lifecycle-manager packageserver-5569d94df4-jdgvq 1/1 Running 0 27h
openshift-operator-lifecycle-manager packageserver-5569d94df4-qfnt9 1/1 Running 0 27h
openshift-ovn-kubernetes ovnkube-control-plane-68798d4445-8pkbb 2/2 Running 0 27h
openshift-ovn-kubernetes ovnkube-control-plane-68798d4445-wwc5q 2/2 Running 0 27h
openshift-ovn-kubernetes ovnkube-node-455dg 8/8 Running 0 27h
openshift-ovn-kubernetes ovnkube-node-5jtbs 8/8 Running 0 27h
openshift-ovn-kubernetes ovnkube-node-7zh45 8/8 Running 0 27h
openshift-ovn-kubernetes ovnkube-node-d9dkq 8/8 Running 0 27h
openshift-ovn-kubernetes ovnkube-node-ld2ll 8/8 Running 0 27h
openshift-ovn-kubernetes ovnkube-node-s4n6v 8/8 Running 0 27h
openshift-route-controller-manager route-controller-manager-9f595b59d-7tbbc 1/1 Running 0 20h
openshift-route-controller-manager route-controller-manager-9f595b59d-csxw9 1/1 Running 0 20h
openshift-route-controller-manager route-controller-manager-9f595b59d-vn2wr 1/1 Running 0 20h
openshift-service-ca-operator service-ca-operator-5c5d464f8-4g2wj 1/1 Running 1 (27h ago) 27h
openshift-service-ca service-ca-6ff7684868-7qqr8 1/1 Running 0 27h
[ansible@bastion ~]$ Listing operators
List all the installed operators, since we have used pull secret all available operators should be installed.
[ansible@bastion ~]$ oc get clusteroperators
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.15.0-0.okd-2024-03-10-010116 True False False 25h
baremetal 4.15.0-0.okd-2024-03-10-010116 True False False 27h
cloud-controller-manager 4.15.0-0.okd-2024-03-10-010116 True False False 27h
cloud-credential 4.15.0-0.okd-2024-03-10-010116 True False False 27h
cluster-autoscaler 4.15.0-0.okd-2024-03-10-010116 True False False 27h
config-operator 4.15.0-0.okd-2024-03-10-010116 True False False 27h
console 4.15.0-0.okd-2024-03-10-010116 True False False 25h
control-plane-machine-set 4.15.0-0.okd-2024-03-10-010116 True False False 27h
csi-snapshot-controller 4.15.0-0.okd-2024-03-10-010116 True False False 27h
dns 4.15.0-0.okd-2024-03-10-010116 True False False 27h
etcd 4.15.0-0.okd-2024-03-10-010116 True False False 27h
image-registry 4.15.0-0.okd-2024-03-10-010116 True False False 27h
ingress 4.15.0-0.okd-2024-03-10-010116 True False False 27h
insights 4.15.0-0.okd-2024-03-10-010116 True False False 27h
kube-apiserver 4.15.0-0.okd-2024-03-10-010116 True False False 27h
kube-controller-manager 4.15.0-0.okd-2024-03-10-010116 True False False 27h
kube-scheduler 4.15.0-0.okd-2024-03-10-010116 True False False 27h
kube-storage-version-migrator 4.15.0-0.okd-2024-03-10-010116 True False False 27h
machine-api 4.15.0-0.okd-2024-03-10-010116 True False False 27h
machine-approver 4.15.0-0.okd-2024-03-10-010116 True False False 27h
machine-config 4.15.0-0.okd-2024-03-10-010116 True False False 27h
marketplace 4.15.0-0.okd-2024-03-10-010116 True False False 27h
monitoring 4.15.0-0.okd-2024-03-10-010116 True False False 27h
network 4.15.0-0.okd-2024-03-10-010116 True False False 27h
node-tuning 4.15.0-0.okd-2024-03-10-010116 True False False 27h
openshift-apiserver 4.15.0-0.okd-2024-03-10-010116 True False False 27h
openshift-controller-manager 4.15.0-0.okd-2024-03-10-010116 True False False 27h
openshift-samples 4.15.0-0.okd-2024-03-10-010116 True False False 27h
operator-lifecycle-manager 4.15.0-0.okd-2024-03-10-010116 True False False 27h
operator-lifecycle-manager-catalog 4.15.0-0.okd-2024-03-10-010116 True False False 27h
operator-lifecycle-manager-packageserver 4.15.0-0.okd-2024-03-10-010116 True False False 27h
service-ca 4.15.0-0.okd-2024-03-10-010116 True False False 27h
storage 4.15.0-0.okd-2024-03-10-010116 True False False 27h
[ansible@bastion ~]$ Accessing OpenShift / OKD Cluster Console
To know the user who authenticated to the cluster from CLI
[ansible@bastion ~]$ oc whoami
system:admin
[ansible@bastion ~]$If we need to know the URL of OKD console.
[ansible@bastion ~]$ oc whoami --show-console
https://console-openshift-console.apps.okdcls.linuxsysadmins.lan
[ansible@bastion ~]$ The credentials to login the console, it can be find under the ignition generated location.
Username: kubeadmin
Password: S5J5x-QPmKu-J2fAK-zEvjt
That’s it, we are done with installing OKD HA cluster on top of a virtual machine.