proxmox-upgrade-cluster

proxmox-upgrade-cluster.sh is a Bash script that performs a rolling upgrade of a Proxmox cluster. It automates the upgrade and optional reboot of one or more nodes.

Features

• Automatically determine cluster members (-c HOSTNAME).
• Upgrade all or specific nodes (-n HOSTNAME).
• When discovering nodes via -c, upgrade them in ascending order of running guest count so the lightest node migrates first (override with --preserve-discovery-order).
• Migrate running guests off each node via Proxmox HA maintenance mode before upgrading (override with --no-maintenance-mode).
• Pre-flight and during-upgrade safety checks (cluster reachable, no offline members, no running tasks or guests on the target).
• Reboot nodes only when needed, or can optionally force a reboot (--force-reboot).
• Optionally force package reinstallation after upgrade (--pkg-reinstall PACKAGE).
• No-op mode (--dry-run).
• Reuse a single SSH connection per host via connection multiplexing (ControlMaster/ControlPath/ControlPersist), so the many per-node commands avoid a fresh TCP and auth handshake each time (disable with --no-ssh-multiplexing).
• Multiple levels of verbosity (-v, -vv, -vvv).

Requirements

• Bash 4.4 or newer
• SSH access to all Proxmox nodes
• jq installed on your system and on PATH.

Assumptions, omissions, and possible future features

• Cluster HA is enabled with all guests managed.
• SSH is available to all nodes (outside of reboots) to all nodes while the running.
• That running this script is not stopped mid-run. Currently, it doesn't to any cleanup or checking to make sure the cluster is left in a working state. Manual intervention may be needed if exiting or errors occur.
• SSH Key Auth is set up and working for the root user. Users other than root have not been tested.
• No rebalancing or guest migration outside the cluster HA. Recommended rebalancing after either manually or with other tools.

Usage

NAME
    proxmox-upgrade-cluster.sh - Perform a rolling upgrade for a Proxmox cluster

SYNOPSIS
    proxmox-upgrade-cluster.sh [OPTIONS] --cluster-node HOSTNAME

    proxmox-upgrade-cluster.sh [OPTIONS] --node HOSTNAME [--node HOSTNAME]...

    proxmox-upgrade-cluster.sh --help

OPTIONS

    -c HOSTNAME, --cluster-node HOSTNAME
        A node in a cluster to pull all nodes from.

    -n HOSTNAME, --node HOSTNAME
        Node(s) to upgrade. Can be passed multiple times.

    -u USER, --ssh-user USER
        SSH user to authenticate with. Defaults to "root".

    -o SSH_OPT, --ssh-opt SSH_OPT
        Options to pass to ssh. Can be passed multiple times.

    --ssh-allow-password-auth
        Allow ssh password auth. Default is to force SSH key auth with
        'PasswordAuthentication=no'.

    --no-ssh-multiplexing
        Disable SSH connection multiplexing. By default a single master
        connection per host is reused (ControlMaster/ControlPath/ControlPersist)
        so the many per-node commands avoid a fresh TCP+auth handshake each time.

    --cluster-node-use-ip
        When using '--cluster-node', use the IP address instead of the node name.

    --dry-run
        Enable dry-run mode; no destructive actions are taken.

    --pkg-reinstall PACKAGE
        Package(s) on the hosts to reinstall with 'apt-get reinstall' post
        upgrade. Can be passed multiple times. Defaults to none.

    --ignore-task-type TYPE
        Task type(s) to ignore when checking for running cluster tasks.
        Can be passed multiple times. Default of 'vncproxy termproxy spiceproxy'.

    --force-upgrade
        Force all nodes to upgrade, and not only those with available upgrades.

    --reboot-only
        Skip apt-update and apt-get dist-upgrade entirely. Filter
        cluster_nodes via node_needs_reboot and reboot just the ones
        running a kernel older than the one staged for boot. Mutually
        exclusive with --force-upgrade, --force-reboot, and
        --skip-reboot.

    --no-maintenance-mode
        Don't set node to maintenance mode when upgrading. This will disable
        HA migrations. Default: maintenance mode is enabled.

    --force-reboot
        Force all nodes to be rebooted during upgrade, and not only those that
        aren't booted with the same kernel as the currently installed one.
        Mutually exclusive with --skip-reboot.

    --skip-reboot
        Skip the reboot step entirely, even when a new kernel is staged for
        boot. Mutually exclusive with --force-reboot.

    --reboot-timeout SECONDS
        Maximum number of seconds to wait for a node to reboot and come back
        up before aborting the upgrade. Defaults to 900 (15 minutes).

    --allow-running-guests
        Disable check for running guests on the node prior to upgrade.

    --allow-running-tasks
        Disable check for running tasks on the cluster prior to upgrade.

    --preserve-discovery-order
        When using --cluster-node, do not reorder the upgrade sequence
        ascending by running guest count. Default is to sort so the node
        with the fewest guests upgrades first.

    -v, --verbose
        Log actions and details. Each repetition unlocks an additional log
        level (all lower levels stay enabled):

            (none)    INFO — status, warnings, errors
            -v        + VERBOSE — high-level script-flow notes
            -vv       + DEBUG — per-node ssh commands as they run
            -vvv      + DEBUG2 — full pvesh JSON responses; timestamps
                       also gain millisecond precision
            -vvvv     + DEBUG3 — internal background-job lifecycle

        Levels beyond -vvvv are for script development:

            -vvvvv    also pass -v to ssh (remote login verbose)
            -vvvvvv   also enable bash 'set -x' (command trace)
            -vvvvvvv  also pass another -v to ssh (ssh -vv)

    -h, --help
        Show this message.

EXAMPLE

    Upgrade all nodes in a cluster, retrieving the cluster nodes from 'pve1':

        proxmox-upgrade-cluster.sh -c pve1


    Upgrade only nodes pve2, pve3:

        proxmox-upgrade-cluster.sh -n pve2 -n pve3


    Reach the cluster through a bastion via an ssh ProxyCommand. Pass each
    ssh option as its own --ssh-opt value:

        proxmox-upgrade-cluster.sh -c pve1 \
            --ssh-opt "-o ProxyCommand=ssh -W %h:%p bastion.example.com"

Example very-verbose dry run

  ./proxmox-upgrade-cluster.sh -c pve1 -vv --dry-run
[2026-05-27 17:02:36][INFO   ] Running in dry run mode.
[2026-05-27 17:02:36][INFO   ] Getting all cluster nodes from node 'pve1'...
[2026-05-27 17:02:36][DEBUG  ][pve1] Running command 'whoami'
[2026-05-27 17:02:36][DEBUG  ][pve1] Node is up.
[2026-05-27 17:02:36][DEBUG  ][pve1] Running command 'hash pvesh'
[2026-05-27 17:02:36][DEBUG  ][pve1] Running command 'pvesh get cluster/status  --output-form=json'
[2026-05-27 17:02:38][INFO   ] Using 'pve1 pve3 pve2' as all nodes to check.
[2026-05-27 17:02:38][INFO   ] Checking if any nodes are currently down...
[2026-05-27 17:02:38][DEBUG  ][pve1] Running command 'whoami'
[2026-05-27 17:02:38][DEBUG  ][pve3] Running command 'whoami'
[2026-05-27 17:02:38][DEBUG  ][pve2] Running command 'whoami'
[2026-05-27 17:02:38][DEBUG  ][pve1] Node is up.
[2026-05-27 17:02:38][DEBUG  ][pve2] Node is up.
[2026-05-27 17:02:38][DEBUG  ][pve3] Node is up.
[2026-05-27 17:02:38][INFO   ] All nodes are up.
[2026-05-27 17:02:38][INFO   ] Checking if any nodes are not proxmox...
[2026-05-27 17:02:38][DEBUG  ][pve1] Running command 'hash pvesh'
[2026-05-27 17:02:38][DEBUG  ][pve3] Running command 'hash pvesh'
[2026-05-27 17:02:38][DEBUG  ][pve2] Running command 'hash pvesh'
[2026-05-27 17:02:38][INFO   ] All nodes are proxmox.
[2026-05-27 17:02:38][INFO   ] Checking if any nodes are currently not online...
[2026-05-27 17:02:38][DEBUG  ][pve1] Running command 'pvesh get cluster/ha/status/manager_status  --output-form=json'
[2026-05-27 17:02:40][INFO   ] All nodes are online.
[2026-05-27 17:02:40][INFO   ] Checking if any nodes currently have tasks running...
[2026-05-27 17:02:40][DEBUG  ][pve1] Running command 'pvesh get nodes/$(hostname)/tasks --source=active --output-form=json'
[2026-05-27 17:02:40][DEBUG  ][pve3] Running command 'pvesh get nodes/$(hostname)/tasks --source=active --output-form=json'
[2026-05-27 17:02:40][DEBUG  ][pve2] Running command 'pvesh get nodes/$(hostname)/tasks --source=active --output-form=json'
[2026-05-27 17:02:41][DEBUG  ][node_not_running_task][pve2] Task Count: 0
[2026-05-27 17:02:41][DEBUG  ][node_not_running_task][pve3] Task Count: 0
[2026-05-27 17:02:41][DEBUG  ][node_not_running_task][pve1] Task Count: 0
[2026-05-27 17:02:41][INFO   ] No tasks are running.
[2026-05-27 17:02:41][INFO   ] Checking for updates on all nodes...
[2026-05-27 17:02:41][DEBUG  ][pve1] Running command 'DEBIAN_FRONTEND=noninteractive apt-get update'
[2026-05-27 17:02:41][DEBUG  ][pve3] Running command 'DEBIAN_FRONTEND=noninteractive apt-get update'
[2026-05-27 17:02:41][DEBUG  ][pve2] Running command 'DEBIAN_FRONTEND=noninteractive apt-get update'
[2026-05-27 17:02:41][VERBOSE][pve2][apt] Hit:1 https://deb.debian.org/debian trixie InRelease
[2026-05-27 17:02:41][VERBOSE][pve3][apt] Hit:1 https://deb.debian.org/debian trixie InRelease
[2026-05-27 17:02:41][VERBOSE][pve1][apt] Hit:1 https://deb.debian.org/debian trixie InRelease
[2026-05-27 17:02:41][VERBOSE][pve2][apt] Hit:2 https://deb.debian.org/debian trixie-updates InRelease
[2026-05-27 17:02:41][VERBOSE][pve3][apt] Hit:2 https://deb.debian.org/debian trixie-updates InRelease
[2026-05-27 17:02:41][VERBOSE][pve1][apt] Hit:2 https://deb.debian.org/debian trixie-updates InRelease
[2026-05-27 17:02:41][VERBOSE][pve3][apt] Hit:3 https://deb.debian.org/debian-security trixie-security InRelease
[2026-05-27 17:02:41][VERBOSE][pve1][apt] Hit:3 https://deb.debian.org/debian-security trixie-security InRelease
[2026-05-27 17:02:41][VERBOSE][pve2][apt] Hit:3 https://deb.debian.org/debian-security trixie-security InRelease
[2026-05-27 17:02:41][VERBOSE][pve3][apt] Hit:4 https://boomshankerx.github.io/proxmox-truenas-apt trixie InRelease
[2026-05-27 17:02:41][VERBOSE][pve1][apt] Hit:4 https://boomshankerx.github.io/proxmox-truenas-apt trixie InRelease
[2026-05-27 17:02:41][VERBOSE][pve2][apt] Hit:4 https://boomshankerx.github.io/proxmox-truenas-apt trixie InRelease
[2026-05-27 17:02:43][VERBOSE][pve2][apt] Hit:5 http://download.proxmox.com/debian/ceph-squid trixie InRelease
[2026-05-27 17:02:43][VERBOSE][pve3][apt] Hit:5 http://download.proxmox.com/debian/ceph-squid trixie InRelease
[2026-05-27 17:02:43][VERBOSE][pve1][apt] Hit:5 http://download.proxmox.com/debian/ceph-squid trixie InRelease
[2026-05-27 17:02:43][VERBOSE][pve2][apt] Hit:6 http://download.proxmox.com/debian/pve trixie InRelease
[2026-05-27 17:02:43][VERBOSE][pve3][apt] Hit:6 http://download.proxmox.com/debian/pve trixie InRelease
[2026-05-27 17:02:43][VERBOSE][pve1][apt] Hit:6 http://download.proxmox.com/debian/pve trixie InRelease
[2026-05-27 17:02:43][VERBOSE][pve3][apt] Reading package lists...
[2026-05-27 17:02:43][VERBOSE][pve2][apt] Reading package lists...
[2026-05-27 17:02:43][VERBOSE][pve1][apt] Reading package lists...
[2026-05-27 17:02:43][DEBUG  ][pve1] Running command 'DEBIAN_FRONTEND=noninteractive apt-get -qq -s upgrade'
[2026-05-27 17:02:44][INFO   ][pve1] No updates available.
[2026-05-27 17:02:44][VERBOSE][pve1] Removed from upgrade sequence.
[2026-05-27 17:02:44][DEBUG  ][pve3] Running command 'DEBIAN_FRONTEND=noninteractive apt-get -qq -s upgrade'
[2026-05-27 17:02:44][INFO   ][pve3] No updates available.
[2026-05-27 17:02:44][VERBOSE][pve3] Removed from upgrade sequence.
[2026-05-27 17:02:45][DEBUG  ][pve2] Running command 'DEBIAN_FRONTEND=noninteractive apt-get -qq -s upgrade'
[2026-05-27 17:02:45][INFO   ][pve2] No updates available.
[2026-05-27 17:02:45][VERBOSE][pve2] Removed from upgrade sequence.
[2026-05-27 17:02:45][INFO   ] No nodes need updates. Exiting.

Testing

Tests are written using ShellSpec and run via Nix development shell:

nix develop -c shellspec spec/

Run a specific test file or tag:

nix develop -c shellspec spec/logging_spec.sh
nix develop -c shellspec spec/node_functions_spec.sh:@1-1

Tests mock all SSH and external commands. No Proxmox cluster is required to run the suite.

License

This script is provided as-is under the GNU v3 License. Use at your own risk.