Hardware verification: bootmode apply during rebuild (3 Dell firmware questions)

[sadsfae]

Summary

Hardware verification needed for the fix in fix: apply host bootmode during rebuild (Gerrit review.gerrithub.io/c/quadsproject/quads/+/1252105, closes #734).

The change is correct as far as the repo can prove: Badfish.set_bios_attribute no longer reboots internally, so the existing reboot_server(graceful=False) at the end of reboot_for_rebuild (src/quads/plugins/builtin/release/standard.py:400-406) is the single apply point for the scheduled BIOS job. A 7-agent expert review and refutation pass confirmed the internal reboot was a latent double-reboot bug from birth (added in ce5dcfc next to a caller that rebooted again immediately after; return value ignored, no job wait), and that reverting would be worse: the first of the two cycles applies and consumes the one-time PXE boot, leaving the second without an override.

Three Dell-firmware behavior questions remain unverifiable from the repo (no iDRAC in CI, no external doc access during the review). None are introduced by this change; they existed in 2.2 and in PS1. One verification run on a mismatched host would settle all three.

Questions to verify on hardware

1. iDRAC9: second ApplyTime PATCH while a BIOS config job is pending: merge or reject?

   • Flow on a mismatched host: boot_to_type runs first (since c9f60c3 it is unconditional) and PATCHes OneTimeBootMode/OneTimeDev to /redfish/v1/Systems/<id>/Bios/Settings with @Redfish.SettingsApplyTime: OnReset (badfish.py:1077-1086, 1121-1129, 1131-1136), then _apply_bootmode PATCHes only BootMode on the same resource (badfish.py:306-375, insist=False).
   • In-repo evidence says the PATCH schedules the config job on iDRAC9 and the explicit create-job POST is then refused 400 "Pending configuration values are already committed, unable to perform another set operation" (commit 7b26bcb, issue Dell iDRAC: boot_to_type fails with 400 'Pending configuration values are already committed' (double-scheduled BIOS job) #726; tolerated only in create_job, badfish.py:950-968). That exact message appears nowhere else in repo history; there is zero in-repo evidence of a PATCH-time rejection (no IDRAC.SYS011 anywhere). External claims of SYS011 on the PATCH path (bmclib#467, libredfish#105, Dell idrac_bios) could not be verified and are not settled by repo evidence.
   • If the second PATCH is rejected, the code chain is proven to abort the rebuild: patch_bios 400 with insist=False -> error_handler -> BadfishException -> plugin set_bios_attribute returns False (plugins/builtin/hardware/badfish.py:127-137) -> _apply_bootmode returns False -> reboot_for_rebuild returns False (standard.py:401-403).
   • 2.2 ran the same two-PATCH sequence in production but only for non-default boot orders (edit ce5dcfc:src/quads/tools/move_and_rebuild.py), so the sample is small; absence of reports is weak evidence.
   • Given code: does a second PATCH while pending succeed (merging pending values) or return 400? If it succeeds, nothing to do. If 400, the fix is a single merged PATCH carrying OneTime attrs + BootMode (or clear-pending-then-set), applied at the release layer.

2. iDRAC10 (Dell 17G, e.g. R670): does the OnReset PATCH alone schedule the BIOS config job, or is a DellJobService POST required?

   • set_bios_attribute never calls clear_job_queue and never calls create_bios_config_job; boot_to always does after its PATCH (badfish.py:1077-1086), and create_job accepts the 201 Dell returns on 17G (commit 35d1951, tests/unit/test_badfish_idrac10.py TestCreateJobAccepts201).
   • The only "PATCH auto-schedules" statement in the code is explicitly iDRAC9-scoped (comment badfish.py:960-963, commit 7b26bcb). No statement exists either way for iDRAC10.
   • Worst case: mismatch + _is_one_time_boot_set True (standard.py:395-398) means boot_to_type is skipped, no job is ever created, and the BootMode change silently never applies on 17G, reintroducing Host bootmode (UEFI/BIOS) not applied during rebuild (regression from 2.2) #734.
   • Given code: on a 17G host, PATCH BootMode with OnReset, reboot once, then read back Bios.Attributes.BootMode and the iDRAC job queue. Does the job exist and the attribute commit? If not, set_bios_attribute (or its release-layer caller) must create the BIOS config job after the PATCH.

3. One-time boot after a same-reset BootMode switch: does the pre-switch one-time boot still fire?

   • send_one_time_boot derives the sequence from the CURRENT committed BootMode (get_boot_seq/get_bios_boot_mode, badfish.py:221-247): OneTimeUefiBootSeq + OneTimeUefiBootSeqDev when UEFI, OneTimeBootSeq + OneTimeBootSeqDev when BIOS. check_device validates the yaml device against the current-mode sequence (badfish.py:1181-1191, 463-490).
   • Device names are mode-specific in repo data: conf/idrac_interfaces.yml uses NIC.PxeDevice./RAID.SL./BOSS.SL.* for r660/r6625/r760 and NIC.Integrated./NIC.Slot. for the rest; README documents the names as reported by badfish --check-boot.
   • With _apply_bootmode now staging BootMode in the same pending set and a single reboot applying both, the host POSTs in the new mode with a one-time target named for the old mode. If the firmware ignores the old-mode one-time, no PXE is attempted on the only reboot. This failure is not silent end to end: validation later flags hosts that were not kickstarted (c9f60c3).
   • Also check the retry path: _is_one_time_boot_set picks the sequence from the current mode, so after an aborted attempt the 'already set' decision can disagree with the new mode.
   • Given code: on a mismatched host (BIOS -> UEFI and UEFI -> BIOS) with the KVM attached, watch whether the single reset PXE boots. If not, the one-time boot must be derived from the desired BootMode (the yaml only carries one name list per model, so a per-model/per-mode map or two-phase apply and reboot is needed).

Verification procedure (one host, one move/rebuild per direction)

1. Host with bootmode set to the opposite of its current BootMode (e.g. current BIOS, desired Uefi).
2. Run a rebuild (or call reboot_for_rebuild directly through the move flow) with badfish debug logs and KVM attached.
3. Observe: PXE boot on the first reset; after completion read Bios.Attributes.BootMode (committed value) and the iDRAC job queue (a BIOS config job should have existed and completed); check the rebuild reaches Foreman validation.
4. Repeat once with a second rebuild to exercise the _is_one_time_boot_set retry path.

Already verified (no action needed)

• The double reboot was real in PS1 (set_bios_attribute internal reboot + reboot_server(graceful=False)), confirmed by review.
• Removing it: no other caller of set_bios_attribute exists in the repo (only the hardware plugin path); external badfish.py is a library with no CLI entry point.
• Single reboot is the documented apply point: patch_bios always sends SettingsApplyTime: OnReset; on iDRAC9 that schedules the BIOS config job that runs at the next reset.
• Regression test: tests/unit/test_badfish_idrac10.py TestSetBiosAttribute asserts the setter patches without rebooting and no-ops when the value already matches.

Skipped options (with reasons, from the review): revert to PS1 (double reboot, consumes the one-time boot); create the job inside set_bios_attribute (clear_job_queue would delete the one-time-boot job created by boot_to in the same run; a second create-job POST on iDRAC10 risks an untolerated 400); merge into a single PATCH (does not fix question 3: the yaml is a single mode-agnostic list per model and check_device validates against the current-mode sequence only, so a desired-mode device name may not exist in the yaml at all).

[sadsfae]

Extending question 3: the persistent boot order leg has the same pre-switch-mode dependency. When a custom boot_order is used, prepare_host_hardware -> change_boot writes BootSeq/UefiBootSeq to BootSources/Settings keyed off the current committed mode (patch_boot_seq, badfish.py:810, seq cached by get_boot_devices), with its own config job and reboot (badfish.py:769-779). That apply happens while the host is still in the old mode, so it is correct at apply time; it only becomes stale because the later BootMode switch makes the other mode's sequence the active one. It is second-order relative to question 3: if the old-mode one-time boot does not fire on the same-reset switch, the rebuild is broken regardless of the persistent order.

Add to the verification run: on the mismatched host with a custom boot order, after the switch read both BootSources/Settings sequences and confirm the active mode's sequence contains the PXE NIC, and confirm the one-time boot fires on the single reset.

If the one-time leg is shown to fail, the mode-agnostic option already in tree is set_next_boot_pxe (System.BootSourceOverrideTarget: Pxe/Once; interfaces/hardware.py:87, dispatchers/hardware.py:229): one PATCH, no BootMode dependence, no yaml change.