You can upgrade the CDF in the following ways:
This section provides information about upgrading CDF manually.
Ensure that you have downloaded the ESM for Fusion package on all the CDF nodes. You need the following file in the package for upgrading the CDF:
cdf-xxxx.xx.xxxx
Ensure that you have minimum 50 GB free space in master node and 30 GB free space in worker node.
Create a backup directory with minimum 30 GB of space on every node of your cluster:
mkdir /tmp/upgrade-backup
If you do not create a backup directory, the backup files will be stores in the default location (\tmp).
Install socat and container-selinux packages on all nodes in the cluster by using the command:
yum install <package_name>
Ensure that you have appropriate permission to restart nodes. You may need to restart nodes if there is an issue during upgrade.
Ensure that all nodes are currently running:
kubectl get nodes
Ensure that all pods are currently running:
<K8S_HOME>/bin/kube-status.sh
Run the following commands on each node:
cd <fusion_download_directory>/installers/cdf-xxxx.xx.xxxx
./upgrade.sh -t <path_to_backup_directory> -i
Example:
cd /opt/esm-cmd-center-installer-for-fusion-x.x.x.x/installers/cdf-2020.05.x.x.x.x
./upgrade.sh -t /tmp/upgrade-backup -i
NOTE:If you do not specify -t <path>, the backup files will be stored in the default location (\tmp).
Run the following commands on one of the master nodes:
cd <fusion_download_directory>/installers/cdf-xxxx.xx.xxxx
./upgrade.sh -u
Example:
cd /opt/esm-cmd-center-installer-for-fusion-x.x.x.x/installers/cdf-2020.05.xxxx
./upgrade.sh -u
(Optional) Clean unused docker daemon images by executing the following command on all the worker and master nodes:
cd <fusion_download_directory>/installers/cdf-xxxx.xx.x.x.x.x
./upgrade.sh -c
Example:
cd /opt/esm-cmd-center-installer-for-fusion-x.x.x.x/installers/cdf-2020.05.x.x.x.x
./upgrade.sh -c
Ensure that upgrade is successful by verifying the following on all the nodes:
Check the CDF version by executing the command:
cat <K8S_HOME>/version.txt
For example:
cat /opt/arcsight/kubernetes/100.txt
Check the current status of CDF pods by executing the command:
<K8S_HOME>/bin/kube-status.sh
For example:
/opt/arcsight/kubernetes/bin/kube-status.sh
NOTE:If the pods are not in running state, execute the following command to recreate main cluster services:
<K8S_HOME>/bin/kube-restart.sh
This section provides workarounds for the following problems that might occur during the CDF upgrade:
If any of the upgrade process fails, complete the following steps:
To ensure that kubelet is running, execute the following command:
kubectl get pod -all-namespaces
Rerun the upgrade command.
If upgrade process timeouts, complete the following steps:
Restart the node.
To ensure that all the pods are in running state, execute the following command:
<K8S_HOME>/bin/kube-status.sh
For example:
/opt/arcsight/kubernetes/bin/kube-status.sh
Rerun the upgrade command.
You can run the CDF automated upgrade with a single command, requiring no interaction until completion of each phase. Typically, the upgrade process takes around 1 hour for a 3x3 cluster.
The automated upgrade allows you to upgrade the CDF from any host (known as the upgrade manager). The upgrade manager can be one of the following:
One of the cluster nodes
A host outside the cluster in a secure network location
The automated upgrade process generates the following directories:
/tmp/autoUpgrade directory on the upgrade manager. This directory stores the upgrade process steps and logs.
/tmp/CDF_xxxx_upgrade, a backup directory, on every node. This directory is approximately 1.7 GB.
A working directory on the upgrade manager and every node at the location provided by the -d parameter. The upgrade package will be copied to this directory. This directory is approximately 9 GB. The automated upgrade process deletes this directory after the upgrade.
NOTE:You can create the working directory manually on upgrade manager and every node, and then pass as -dparameter to the auto-upgrade script. If you are a non-root user on the nodes inside the cluster, make sure you have permission to this directory.
Ensure that you have downloaded the ESM for Fusion installer package to the download directory (<download_directory>) on the upgrade manager.
Verify that you have CDF xxxx.xx upgrade packages in the following location:
<fusion_download_directory>/installers/cdf-xxxx.xx.x.x.x.x/autoUpgrade.sh
For example:
cd /opt/esm-cmd-center-installer-for-fusion-x.x.x.x/installers/cdf-2020.05.x.x.x.x/autoUpgrade.sh
Install socat and container-selinux packages on all nodes in the cluster by using the command:
yum install <package_name>
To configure passwordless SSH communication between the upgrade manager and all the nodes in the cluster, complete the following steps:
To generate key pair, run the following command on the upgrade manager:
ssh-keygen -t rsa
To copy the generated public key to every node of your cluster, run the following command on the upgrade manager:
ssh-copy-id -i ~/.ssh/id_rsa.pub root@<node_fqdn_or_ip>
Change to the directory where you have downloaded the latest CDF package:
cd /<download-directory>/installers/cdf-xxxx.xx.xxxx/
Run the following command for automatic upgrade:
./autoUpgrade.sh -d /path/to/workinig_directory -n {any_cluster_node_adress_or_ip}
For example:
./autoUpgrade.sh -d /tmp/upgrade -n pueas-ansi-node1.swinfra.net
Ensure that upgrade is successful by verifying the following on all the nodes:
Check the CDF version by executing the command:
cat <K8S_HOME>/version.txt
For example:
cat /opt/arcsight/kubernetes/100.txt
Check the current status of CDF pods by executing the command:
<K8S_HOME>/bin/kube-status.sh
For example:
opt/arcsight/kubernetes/bin/kube-status.sh
NOTE:If the pods are not in running state, execute the following command to recreate main cluster services:
<K8S_HOME>/bin/kube-restart.sh
Delete the auto-upgrade temporary directory and backup directory from the upgrade manager.
The auto-upgrade temporary directory contains the upgrade steps and logs. If you want to upgrade another cluster from the same upgrade manager, remove that directory.
rm -rf /tmp/autoUpgrade
rm -rf /tmp/CDF_xxxx_upgrade
If the automated upgrade fails, run autoUpgrade.sh again as outlined above. The process may take several attempts to succeed.
In some cases, the automatic upgrade may return an error message about the upgrade process still running and the existence of a *.lock file which prevents autoupgrade.sh to continue. This file is automatically deleted in a few minutes. Alternatively, you can manually delete this file. Once the file is deleted either automatically or manually, run autoUpgrade.sh again.
If the automated upgrade process is still unsuccessful, continue the process on the failed node using the procedure outlined in Manually Upgrading the CDF.