fix: resize OneKE disks via Ruby API #PROESP-2752 - #196
Open
raulluzon wants to merge 33 commits into
Open
Conversation
raulluzon
changed the title
The amazon.aws.s3_object list mode fails with a hard error when the bucket does not exist (auto-creation was removed in amazon.aws 6.0.0). Add an idempotent amazon.aws.s3_bucket task before the list/download tasks so the bucket is created on first deployment and silently reused on all subsequent deployments. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…failure The open5gs_vm image (id: 260) advertises kex-strict-s-v00@openssh.com but fails to complete the handshake when strict KEX mode is activated, causing Stage 3 (wait_for_connection) to time out with "Connection closed by UNKNOWN port 65535". Adding -o KexAlgorithms=curve25519-sha256 prevents the client from engaging strict mode, which allows the key exchange to complete. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
raulluzon
force-pushed
the
fix/uma-test-raul-oneke-onevm-path
branch
from
June 26, 2026 08:06
24d9931 to
36be98e
Compare
Version 2.7.6 has been removed from the open5gs/latest PPA. The only available version is now 2.8.0~jammy5. Added update_cache: yes to ensure the APT index is refreshed after adding the PPA. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Follows the installation fix in commit 6a7128e; the README and result template still referenced the old pinned version. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… in public.yaml Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
unattended-upgrades holds /var/lib/dpkg/lock-frontend on VM boot, causing apt installs to fail with rc 100 when Ansible runs immediately after the VM becomes reachable. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…meout Passive wait could timeout (>5 min) if unattended-upgrades was running. Stop the service first so the lock is released immediately. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Stopping unattended-upgrades via systemd sends SIGTERM but the in-flight dpkg operation keeps holding lock-frontend until it completes. Use fuser -k to immediately kill any process holding the dpkg/apt locks, then repair any interrupted dpkg state before proceeding with apt installs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… user profile All VM Terraform templates used \$USER[SSH_PUBLIC_KEY], which expands to the SSH key stored in the OpenNebula account of the deploying user. When that user is a human operator rather than Jenkins, the provisioned VM gets the wrong key in authorized_keys and Jenkins can never authenticate. Read the key at pipeline time from /var/lib/jenkins/.ssh/id_ed25519.pub via an Ansible lookup (always runs on the controller) and inject it as jenkins_ssh_public_key. Replace \$USER[SSH_PUBLIC_KEY] with this variable in all 23 VM component templates. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ents ProxyJump (-J) does not forward -o KexAlgorithms to the bastion hop; it uses system defaults for the proxy connection. Bastions running the Open5GS appliance image advertise kex-strict-s-v00@openssh.com, which causes the proxy hop to fail with "kex_exchange_identification: Connection closed by remote host" even when KexAlgorithms=curve25519-sha256 is set for the target connection. Replace -J with an explicit ProxyCommand that runs ssh+nc and carries KexAlgorithms=curve25519-sha256 for the bastion hop, and also sets it for the final hop. Fixes SSH connectivity for all 21 VM components that use a bastion proxy. Multi-hop variants (oneKE, open5gs_k8s, ocf) are left unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nection timeout Ubuntu VMs deployed via OpenNebula one-appliance trigger unattended-upgrades at first boot, which runs in the background for ~16 minutes and restarts openssh-server mid-session. This causes Ansible wait_for_connection to fail with "Connection to <ip> closed by remote host" before provisioning can start. Two-part fix applied to all VM components: 1. Add START_SCRIPT_BASE64 to the OpenNebula VM context: stops and masks unattended-upgrades, apt-daily-upgrade, and apt-daily services at first boot, before the apt-daily-upgrade.service timer fires at ~30 seconds post-boot. 2. Increase wait_for_connection timeout from 200s to 600s (10 min per phase, ~20 min total) as a safety net for any remaining boot-time delays. For ks8500_runner and loadcore_agent the existing START_SCRIPT is extended with the unattended-upgrades stop/mask commands prepended. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… open5gs_vm The previous START_SCRIPT_BASE64 masked apt service units but not their timers (apt-daily.timer and apt-daily-upgrade.timer), allowing unattended-upgrades to run at first boot via the timer path and restart sshd during Ansible provisioning. Also enables ansible_pipelining on open5gs_vm to avoid the strict-KEX bug (CVE-2023-48795 mitigation in OpenSSH 8.9) which causes SFTP to hang when the default KEX proposal includes kex-strict-c-v00@openssh.com. With pipelining on, Ansible sends module code via SSH exec stdin instead of SFTP, bypassing the hang entirely. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… transfers With pipelining enabled, the template module serialises the full hostvars dict and sends it through the nc ProxyCommand chain. This is large enough to stall the pipe. The shell module was unaffected because its payload is tiny. Removing pipelining restores SFTP-based file transfer, which works correctly with the existing KexAlgorithms=curve25519-sha256 fix. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…-proxied connections The Jenkins global ~/.ansible.cfg sets ControlMaster=auto with ControlPersist=60s. When Ansible connects to the open5gs_vm via nc ProxyCommand, the ControlMaster is created but the nc tunnel drops after the idle period between wait_for_connection and the first real task. This leaves a stale socket file that causes all subsequent SSH connections to hang indefinitely. -S none in ansible_ssh_common_args uses the OpenSSH -S flag which directly overwrites options.control_path, bypassing the first-wins rule that applies to -o options. This definitively disables connection sharing regardless of ansible.cfg. ansible_pipelining: false explicitly overrides the global transfer_method = piped setting in ansible.cfg so SFTP is used for file transfers instead of piped stdin. Also: clean up any stale socket at the start of Stage 3 as a belt-and-suspenders measure, and increase wait_for_connection timeout to 1800s to handle slow VM boots where cloud-init apt operations can take over 16 minutes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…p task The delegate_to: localhost task in Stage 3 was inheriting ansible_user=jenkins from the open5gs_vm inventory entry, causing Ansible to SSH to localhost rather than running the file deletion locally. Added vars: ansible_connection: local to force local execution. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ommand ansible_pipelining: false forced sftp-based file upload for become: true tasks, which opens a separate SSH sub-connection through the nc ProxyCommand. That connection is closed by the remote host. The uma-test site succeeded with pipelining enabled — the stale ControlMaster was the real root cause, now fixed with -S none. Pipelining through nc ProxyCommand works for all task sizes here. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…etplan resilience ControlMaster=no caused every task to open a fresh SSH tunnel. When netplan apply briefly disrupts the VXLAN interface (192.168.199.x), the next connection attempt catches the window and fails with "Connection closed by remote host." Using ControlMaster=auto with ControlPersist=3600 keeps a single tunnel alive through the disruption. The original stale-socket bug (60s expiry) is prevented by the longer persist window plus the cleanup task that removes old sockets at Stage 3 start. Socket path is explicit (-S /tmp/cm-<ip>-22-jenkins) so cleanup is exact. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…n apply ControlMaster=auto caused silent socket failures (empty error, earlier than previous failures). Reverting to -S none/-o ControlMaster=no which was stable through task 16 in the previous run. The remaining blocker was netplan apply briefly disrupting the VXLAN interface (192.168.199.x), causing the next apt_repository task to get "Connection closed by remote host". Adding wait_for_connection after netplan apply (same when condition) ensures the SSH connection is re-established before apt tasks run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On slow-booting VMs (>15 min), unattended-upgrades runs during the cloud-init window and may restart sshd after wait_for_connection succeeds. The five variable-loading tasks that follow run on the Ansible controller (no SSH), so they appear OK. The first real remote task (Set site ssh key) then hits the restarting sshd and gets "Connection closed by remote host." A second wait_for_connection (delay 5s, timeout 300s) after variable loading but before the first remote task absorbs any sshd restart window. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On slow-booting VMs unattended-upgrades runs during cloud-init and periodically restarts sshd, which kills active Ansible connections. Adding a shell task right after the initial wait_for_connection to stop unattended-upgrades and release dpkg locks. ignore_unreachable/ignore_errors tolerates the case where sshd dies mid-task; the follow-up wait_for_connection absorbs the restart window before any real configuration work begins. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…sshd disruption Replace systemctl stop with mask --now for unattended-upgrades and apt-daily timers in both the pre-install stage and the installation block. Also write a needrestart override (restart=l) so that needrestart does not auto-restart sshd after mongodb-org or open5gs packages are installed, which was causing UNREACHABLE failures mid-installation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… tasks cloud-init restarts sshd multiple times during early VM boot. The previous wait_for_connection only confirms reachability at that instant; a second restart window during cloud-init can hit the Create-new-user task. Adding cloud-init status --wait (with ignore_unreachable and a recovery wait_for_connection) ensures cloud-init is fully done before any state-changing remote tasks run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nsible wait_for_connection The ProxyCommand tunnels (Jenkins → bastion → VM via nc) were being killed by a ~946-second TCP idle timeout on the client network because no data flowed while Ansible waited for the ping module response. Each attempt consumed the full 946 s before failing, causing the 1800-second wait_for_connection timeout to be exceeded after only two retries. Adds -o ServerAliveInterval=30 -o ServerAliveCountMax=10 to both the outer ansible_ssh_common_args and the inner ProxyCommand ssh invocation across all 21 VM component playbooks, keeping both the Jenkins→bastion and the end-to-end SSH connections alive during idle periods. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dpkg --configure -a runs the openssh-server post-install script which calls invoke-rc.d ssh restart, dropping the Ansible SSH session and causing UNREACHABLE. The task had no ignore_unreachable guard, so the play failed. Add a policy-rc.d stub that returns 101 (action refused) before running dpkg --configure -a so invoke-rc.d refuses all service restarts during package configuration. Remove it immediately after. Also add ignore_unreachable: true as a safety net and an unconditional wait_for_connection after the task to reconnect if sshd does restart. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…shd restart dpkg --configure -a is called with policy-rc.d (exit 101) during the early apt-masking step, before any non-ignored Ansible tasks. Any pending openssh-server post-install script is resolved here without restarting sshd, eliminating the deferred restart that caused UNREACHABLE on Set TN ssh key as authorized in new user (Bug 18). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… remote task wait_for_connection timed out at elapsed=321s against timeout=300s on raul12 (Bug 19): SSH dropped again after the load_variables include, narrowly missing the deadline. Extend the first check to 600s and guard it with ignore_unreachable/ignore_errors, then confirm stability with a second bounded wait, mirroring the existing cloud-init check/confirm pattern below it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…figure wait_for_connection timed out at elapsed=322s against timeout=120s on raul13 (Bug 20): recovery after the guarded dpkg --configure -a UNREACHABLE genuinely takes longer than 120s. Extend to 600s and guard with ignore_unreachable/ignore_errors, then confirm stability with a second bounded wait, matching the pattern already applied to the pre-install stabilize checks in component_playbook.yaml (Bug 19). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…oy apply The OpenNebula terraform provider hard-errors with "AR not found" when an opennebula_virtual_network_address_range's AR was already destroyed as a side effect of the first (ignored) apply, but the resource is still tracked in state. Parse that failure for the affected VNet IDs and remove just the matching stale AR entries from state before the second, must-succeed apply. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…RNAL error
The first observed failure mode ("AR (ID:Y) not found") is not the only one
OpenNebula's provider produces for an already-gone AR; a second run hit
"AR (ID:Y): OpenNebula error [INTERNAL]: [one.vn.free_ar] Address Range does
not exist" instead, which the previous regex didn't match. Match on the
common "Virtual network (ID:X): AR (ID:Y)" prefix instead of the trailing
error text, since that phrase is specific to this failure regardless of
wording.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…urces
The previous selectattr('values.virtual_network_id', 'in', stale_ar_vnet_ids)
pipeline silently matched nothing whenever virtual_network_id was stored as
an int in state (stale_ar_vnet_ids from regex_findall is always strings),
so the cleanup step never actually ran despite correctly identifying the
affected VNet IDs. Replace the selectattr/map pipeline with a direct loop
over state resources with an explicit `| string` cast in the per-item when
condition, which can't silently type-mismatch the way selectattr('in') did.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…sion `.values` and (per-resource) `.values` again are dict keys in terraform show -json's output, but Jinja2 dot-notation resolves them to Python's built-in dict.values() method instead of the "values" key, since attribute lookup is tried before key lookup. Combined with `| default([])`, this silently produced an empty list rather than erroring, so the loop always had zero items and "skipping" was the *expected* (but wrong) outcome for every run so far. Switch both accesses to bracket notation, which resolves against dict keys directly. Verified in isolation with a hand-crafted terraform show -json payload before landing this. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deployment currently fails in the 6G-Library OneKE component because oneKE/code/component_playbook.yaml calls the hardcoded /usr/bin/onevm disk-resize, but Jenkins does not have a working OpenNebula CLI at that path.
Replace the hardcoded CLI invocation with the validated Ruby OpenNebula API path using OpenNebula::VirtualMachine#disk_resize(1, 30 * 1024).
Validation evidence from the UMA debugging session: