[nelljerram]

Why

Three interlocking improvements to the OpenStack mech driver's Neutron-DB access:

• Speed up Neutron→etcd resync at scale by replacing the O(N) per-port / per-SG DB query pattern with bulk prefetch. Expected ~42x speedup on the WorkloadEndpoint resync at scale=1000 (mock-DB measurement; Lens will publish the real DevStack number once this lands).
• Fix CORE-11889 -- the "AssertionError: do not call blocking functions from the mainloop" traces in neutron-server.log on Noble, caused by SQLAlchemy connection-fairy finalisation happening on the eventlet hub greenlet.
• Fix intermittent QueuePool timeouts observed at scale=1000+ in the resync scale test, where the deprecated subtransactions=True path failed to return connections to the pool reliably.

What

1. Modernize transaction management

Replace session.begin(subtransactions=True) (deprecated in SQLAlchemy 1.4, removed in 2.0) with neutron_lib's db_api.CONTEXT_WRITER.using() at every Neutron-DB access point in the postcommit-hook paths in the mech driver and the read paths in the resync runner.

We use CONTEXT_WRITER (not CONTEXT_READER) on those paths, even on apparent "read" paths, because some Neutron helpers transitively write to the DB -- most notably _ensure_default_security_group(), which lazily creates the per-project default SG inside what looks like a pure read. oslo.db's enginefacade strictly forbids upgrading a READER to a WRITER mid-transaction, so a READER scope would break those paths. No realistic cost: postcommit hooks run AFTER Neutron's main writer transaction has committed; the resync runs in a dedicated worker process.

2. Bulk-fetch per-item DB data during resync

Replace the O(N) DB round-trip pattern in both WorkloadEndpointSyncer and PolicySyncer with bulk queries at the start of each syncer's get_from_neutron(), cached for the duration of the pass.

• PolicySyncer: a single get_security_group_rules(filters={"security_group_id": [all_sg_ids]}) call replaces N per-SG calls. Rules are grouped in self._rules_by_sg and consumed by neutron_to_etcd_write_data. Performed inside get_from_neutron, after the base class's etcd read, so the cache reflects a Neutron snapshot strictly later than the etcd snapshot -- preserving the etcd-first ordering's CAS-based race protection against concurrent dynamic SG-rule updates.
• WorkloadEndpointSyncer: _prefetch_bulk_port_data() populates self._bulk with IPAllocation, FloatingIP, Network, SecurityGroup-port-binding, QoS bandwidth + packet-rate rule, and Subnet rows, indexed appropriately. get_extra_port_information() consults self._bulk in O(1) when populated; otherwise (e.g. postcommit-hook single-port writes) it falls through to the original per-port path unchanged. add_port_qos() reads from the same bulk cache when available.
• A subnet missing from the bulk cache falls back to a per-subnet get_subnet(), mirroring add_port_gateways().

Both syncers restrict the bulk-fetch to the scope's port / SG IDs to avoid over-fetching narrow resyncs.

3. Close admin-context sessions in the port-status loop

_loop_writing_port_statuses creates its own admin context and previously never explicitly closed the resulting session. Under newer eventlet + SQLAlchemy versions, the connection-fairy's finalisation (rollback + return-to-pool) can fire on the eventlet hub greenlet at unpredictable moments, tripping the "do not call blocking functions from the mainloop" assertion (CORE-11889).

Add a _close_session_safely() helper and call it after each port-status update (plus once on loop exit) so the connection is returned to the pool deterministically inside the worker greenlet, not via opaque finalisation.

A mid-PR refactor attempted to replace this helper with db_api.CONTEXT_WRITER.using(admin_context) per iteration on the theory that an enginefacade-managed scope was cleaner. That regressed CI -- Ml2Plugin.update_port_statuses is decorated with @retry_db_errors, which refuses to run inside an existing transaction (it manages its own writer scope so it can retry on conflicts). Reverted; the _close_session_safely form is correct because the work inside the loop already manages its own transaction and only needs deterministic session cleanup afterwards.

4. Cleanup enabled by (1)

• Move the MySQL-driver sanity check (PyMySQL-only, per oslo.db bug 1350149) to start of day -- read cfg.CONF.database.connection directly and raise on the first call to CalicoMechanismDriver.initialize() or networking_calico.resync.cli.main(), instead of lazily on first DB access.
• Explicitly register the oslo.db [database] option group before that read, since ML2's mechanism_manager.initialize() runs before anything else has imported oslo.db's enginefacade (which is what would otherwise side-effect-register it).
• Eliminate the _txn_from_context() wrappers entirely. After (1), they were one-line wrappers around db_api.CONTEXT_WRITER.using plus a lazy mysql check that has now moved to start of day. Drop the txn_from_context parameter plumbed through every ResourceSyncer constructor. Call sites in mech_calico.py, syncer.py, and scope.py use db_api.CONTEXT_WRITER.using directly.

Release Note

For OpenStack, resync between the Neutron DB and Calico's datastore is now dramatically faster than in earlier releases (on the order of tens of times faster for a deployment with thousands of ports).  The driver also uses modern Neutron/SQLAlchemy database-access patterns, which avoids some connection-pool and event-loop problems that could previously occur during resync at scale.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[github-actions]

CI triage — Calico

Recommendation: Could not retrieve failure details for this run.

Workflow header below:

Workflow 07dc7a7f-a0d2-463f-a603-16fa52e71e85

verdict: failed
pipeline: Calico
branch: pull-request-12881
commit: networking-calico: modernize DB access and improve resync performance
reason: test

Failed: Check Python / Check Python files

[nelljerram]

This PR delivers about a 44x speed-up in resync performance, when the Neutron DB has 3000 standing ports, mostly delivered by bulk upfront reading of all the Neutron tables that will be required for those ports - instead of reading data as required on a port-by-port basis. The speed-up can be seen in the following two visualizations of the performance data that we are now tracking in our Lens Elastic DB. The first one shows total time (ms) for the resync:

In CI for the master branch ("(null)") and PRs 12770 and 12889 - which are all prior to this current PR - total time is about 127s. In CI for this PR (12881), total time is 2.7s.

The second one shows the breakdown across the components of a resync op: subnets, policy, endpoints and Felix config:

It shows that most of the improvement is for endpoints, with a smaller contribution from policy (which also now involves some pre-fetching), with subnets staying as they were and Felix config insignificant.

[nelljerram]

For completeness here's the breakdown one again with a linear Y axis:

[nelljerram]

While reviewing the latest CI run's neutron-server.log I spotted one new WARNING that's worth calling out (the other things in the log -- "Felix on host semaphore-vm failed to check in" and "Key changed or deleted; restart election" -- are pre-existing scale-test cross-talk, see below).

WARNING neutron.plugins.ml2.ovo_rpc [...] This handler is supposed to handle AFTER events, as in 'AFTER it's committed', not BEFORE. Offending resource event: security_group, after_create.

The traceback:

update_port_postcommit
  → endpoint_syncer.write_endpoint
    → policy_syncer.write_sgs_to_etcd
      → self.db.get_security_group_rules(...)
        → _ensure_default_security_group(context, project_id)
          → create_security_group(...)
            → registry.publish(SECURITY_GROUP, AFTER_CREATE, ...)
              → ml2/ovo_rpc._is_session_semantic_violated() → WARNS

This is direct fallout of wrapping update_port_postcommit in db_api.CONTEXT_WRITER.using(plugin_context). Inside that wrap, get_security_group_rules lazily creates the per-project default SG (on the very first SG read in a fresh project), and the resulting AFTER_CREATE notification fires while our outer writer transaction is still open. Neutron's ovo_rpc handler warns because its contract assumes "AFTER" notifications fire post-commit.

The functional concern would be: if our outer transaction rolled back after the notification fired, an OVO subscriber would be told about an SG that doesn't actually exist. In our paths the outer transaction always commits (postcommit hooks don't raise to roll the outer transaction back), so by the time anything acts on the OVO notification, the SG is committed.

Frequency: 1 occurrence per project, on the first SG access. The whole 24h CI window had a single occurrence (admin project's default SG creation).

I'm accepting this as a known quirk of the nested-write pattern we needed for _ensure_default_security_group. It's the trade-off for picking CONTEXT_WRITER over CONTEXT_READER (the latter would forbid the lazy SG creation entirely). If it becomes more frequent or causes real bugs, the cleanest fix would be to pre-create the default SG at start of day rather than on first port operation.

---

For completeness on the other log noise:

• "Felix on host semaphore-vm failed to check in" (3×) + "Key changed or deleted; restart election" (3×) — same root cause, etcd lease expiry under scale-test CPU pressure. Each pair fires at the same instant on the same mod_revision. The single-VM CI is running real Felix, neutron-server (4 forks), MariaDB, etcd, and the test driver all competing for cores; Felix gets starved off-CPU long enough that its etcd lease lapses, the watcher sees the resulting DELETE events, and the same starvation costs the neutron-election fork its election key. Both recover within a second (Felix re-publishes, another fork takes the election then hands it back). Not introduced by this PR; an artefact of the scale test sharing a host with Felix.

[nelljerram]

Docs at tigera/docs#2800