For a client I automated the patching process of Linux operating systems and middleware. I create a python action for that in StackStorm. The script retrieved all servers from the CMDB and patched per data center.

 

Non cluster hosts parallel

All non cluster hosts where patched in parallel with the Python ParallelSSHClient.

Cluster hosts rolling upgrade

For hosts in a cluster it’s a bit harder. The minimum amount of nodes in a cluster is most of the time 3 and only one node can be down without downtime of the cluster. So a rolling upgrade is required, patching one node at a time. Some may find this a bit scary to automate, but I think it’s actually quite safe to do. Only start patching if the cluster status is healthy. After patching a node, ONLY continue with the next node if the cluster is completely healthy again.

I created standalone upgrade scripts written in Python for:


Elasticsearch rolling upgrade

 


Couchbase rolling upgrade

 


CouchDB rolling upgrade