Skip to content

Commit ae63aa1

Browse files
nelljerramclaude
nelljerram
and
claude
committed
Document the OpenStack Neutron-etcd resync process
Add a new networking/openstack/resync.mdx page covering the resync behaviour as it now exists after the post-v3.32 changes: one-shot startup resync (gated by [calico] startup_resync), the on-demand calico-resync command, per-item INFO logging, and performance and concurrency characteristics. Also note the removal of the old periodic resync and deprecation of resync_interval_secs and resync_max_interval_secs. Cross-reference the new page from configuration.mdx and add it to the calico sidebar. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 49d1108 commit ae63aa1

3 files changed

Lines changed: 189 additions & 0 deletions

File tree

‎calico/networking/openstack/configuration.mdx‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,3 +59,20 @@ When specified, the value of `openstack_region` must be a string of lower case a
5959
characters or '-', starting and ending with an alphanumeric character, and must match the value of
6060
[`OpenStackRegion`](../../reference/felix/configuration.mdx#openstack-specific-configuration)
6161
configured for the Felixes in the same region.
62+
63+
The `[calico] startup_resync` option controls whether the $[prodname] driver
64+
reconciles the Neutron database into etcd when Neutron server starts. See
65+
[Resync between Neutron and etcd](resync.mdx) for details.
66+
67+
| Setting | Default Value | Meaning |
68+
| -------------- | ------------- | ---------------------------------------------------------------------------------------- |
69+
| startup_resync | always | Whether to run a full Neutron DB &rarr; etcd resync when Neutron server starts. Set to `never` to suppress it. |
70+
71+
:::note
72+
73+
The `resync_interval_secs` and `resync_max_interval_secs` options that
74+
configured the old periodic resync are deprecated and have no effect. See
75+
[Resync between Neutron and
76+
etcd](resync.mdx#upgrade-notes-for-existing-deployments).
77+
78+
:::

‎calico/networking/openstack/resync.mdx‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
---
2+
description: How Calico reconciles the Neutron database with its etcd datastore, including startup resync and the on-demand calico-resync command.
3+
---
4+
5+
# Resync between Neutron and etcd
6+
7+
## Big picture
8+
9+
In a $[prodname] OpenStack deployment, the Neutron database is the source of
10+
truth for networking state, and the $[prodname] driver mirrors the parts that
11+
$[prodname] needs &mdash; ports, subnets, and security groups &mdash; into the
12+
$[prodname] etcd datastore as `WorkloadEndpoint`, `Subnet`, and `NetworkPolicy`
13+
resources (plus `LiveMigration` entries while an instance is migrating).
14+
15+
In normal operation the driver keeps etcd up to date incrementally, writing to
16+
etcd from the Neutron post-commit hooks as ports and security groups change. A
17+
_resync_ is a reconciliation pass that reads the current Neutron state and the
18+
current etcd state and corrects any difference between them &mdash; creating
19+
anything that is missing from etcd, updating anything that has drifted, and
20+
deleting anything in etcd that no longer corresponds to Neutron. Its purpose
21+
is to recover from the rare cases where the incremental path can leave etcd out
22+
of date, for example if the Neutron server crashed part way through handling a
23+
request, or if etcd was briefly unreachable when an update happened.
24+
25+
## When resync runs
26+
27+
There are two ways a resync happens:
28+
29+
- **Startup resync** &mdash; a one-shot resync that runs automatically when the
30+
Neutron server starts or is restarted. This is the normal, hands-off
31+
mechanism, and is enabled by default.
32+
33+
- **On-demand resync** &mdash; run explicitly by an operator using the
34+
[`calico-resync`](#the-calico-resync-command) command, optionally scoped to
35+
particular networks, subnets, ports, or security groups.
36+
37+
:::note
38+
39+
In releases up to and including v3.32, the driver also ran a _periodic_ resync,
40+
repeating roughly every 30 seconds, governed by the `resync_interval_secs` and
41+
`resync_max_interval_secs` options. That periodic resync has been removed
42+
because it was never needed for correct mainline operation, and it was
43+
misleading to retain it as though it was. The related config options are now
44+
deprecated no-ops, retained only so that an existing `neutron.conf` does not
45+
break. See [Upgrade notes](#upgrade-notes-for-existing-deployments) below.
46+
47+
:::
48+
49+
## Startup resync
50+
51+
When the Neutron server starts or restarts, the $[prodname] driver performs a
52+
single all-resource resync. Running once on startup is sufficient because the
53+
incremental post-commit path keeps etcd current thereafter.
54+
55+
The startup resync is controlled by the `startup_resync` option in the
56+
`[calico]` section of `/etc/neutron/neutron.conf`:
57+
58+
| Setting | Default | Meaning |
59+
| -------------- | ------- | ---------------------------------------------------------------- |
60+
| startup_resync | always | Whether to run a full Neutron&nbsp;DB&nbsp;&rarr;&nbsp;etcd resync when the Neutron server starts. Set to `never` to suppress the startup resync worker entirely. |
61+
62+
In almost all deployments you should leave `startup_resync = always`. Setting
63+
it to `never` is intended for cases where you want full control over when
64+
resyncs run; for example:
65+
66+
- a deployment where you prefer to trigger reconciliation manually with
67+
`calico-resync` at a chosen time
68+
69+
- a deployment with multiple Neutron servers, where you want to dictate that
70+
only one of those performs startup resyncs; in this case you would configure
71+
`always` on the chosen control node and `never` on the others.
72+
73+
:::note
74+
75+
Unlike the driver's other background jobs (etcd compaction, agent-status
76+
watching, and endpoint-status watching), which elect a single active worker
77+
dynamically through an etcd key so that the work fails over cleanly, the
78+
startup resync is governed by this static `startup_resync` config switch. It
79+
runs once rather than continuously, so operator-level control over whether and
80+
where it runs is more useful than automatic failover.
81+
82+
:::
83+
84+
## The calico-resync command
85+
86+
`calico-resync` is a console script, installed with the $[prodname] networking
87+
driver, that drives exactly the same resync logic as the startup resync but
88+
out-of-band, on demand. Run it on a Neutron control node that has access to
89+
`/etc/neutron/neutron.conf` and to the Neutron database.
90+
91+
With no scope flags, it performs a full resync of all resources:
92+
93+
```bash
94+
calico-resync
95+
```
96+
97+
By default it reads `/etc/neutron/neutron.conf`; use `--config-file`
98+
(repeatable, for layered config) to point at a different file.
99+
100+
### Scoping a resync
101+
102+
You can restrict a resync to specific resources, which is much faster than a
103+
full resync and useful when you know which objects need attention:
104+
105+
| Flag | Effect |
106+
| -------------------------- | -------------------------------------------------------------------- |
107+
| `--network ID` | Resync this network, and its subnets and ports. Repeatable. |
108+
| `--subnet ID` | Resync this subnet, and the ports on it. Repeatable. |
109+
| `--port ID` | Resync this port. Repeatable. |
110+
| `--security-group ID` | Resync this security group's `NetworkPolicy`. Repeatable. |
111+
| `--include-sgs-for-ports` | When resyncing ports, also resync the security groups they belong to. Off by default, because the port&rarr;security-group binding is expressed through labels, so a port-only resync is usually sufficient. |
112+
113+
For example, to reconcile a single port and the security groups it references:
114+
115+
```bash
116+
calico-resync --port <port ID> --include-sgs-for-ports
117+
```
118+
119+
### Output
120+
121+
`calico-resync` prints a JSON `ResyncResult` to stdout, reporting an overall
122+
`ok` flag, per-phase timings, and per-kind counters of how many resources were
123+
found already correct, created, updated, and deleted (the endpoint phase
124+
additionally reports `LiveMigration` activity). Use `-o PATH` / `--output
125+
PATH` to write the JSON to a file instead; this is convenient for tooling,
126+
since depending on your logging configuration oslo.log may also emit lines to
127+
stdout and make the JSON harder to parse.
128+
129+
## Observability
130+
131+
During a resync &mdash; whether startup or on-demand &mdash; the driver logs an
132+
`INFO`-level line for each resource that it has to reconcile, so you can see
133+
exactly what diverged and why resync intervened:
134+
135+
```
136+
Resync creating <kind> <name> in etcd
137+
Resync updating <kind> <name> in etcd: old=<existing>, new=<replacement>
138+
Resync deleting <kind> <name> from etcd
139+
```
140+
141+
Resources whose etcd data already matches Neutron are not enumerated at `INFO`
142+
(they are logged only at `DEBUG`); they are still counted in the per-kind
143+
summary totals. Failures to write to or delete from etcd remain at `WARNING`,
144+
since those can indicate a genuine race rather than routine reconciliation.
145+
146+
## Performance and concurrency
147+
148+
The resync code reads Neutron state in bulk rather than issuing per-port and
149+
per-security-group queries, which makes a large-scale resync dramatically
150+
faster than in earlier releases (on the order of tens of times faster for a
151+
deployment with thousands of ports). The driver also uses modern
152+
Neutron/SQLAlchemy database-access patterns, which avoids some connection-pool
153+
and event-loop problems that could previously occur during resync at scale.
154+
155+
A resync runs in its own process, separate from the Neutron server processes
156+
that handle API requests and post-commit updates. A long-running resync does
157+
not meaningfully block dynamic operations such as creating ports: you can run
158+
`calico-resync` on a busy deployment without stalling normal Neutron activity.
159+
160+
## Upgrade notes for existing deployments
161+
162+
If you are upgrading from v3.32 or earlier:
163+
164+
- No configuration change is required. The driver now runs a resync on Neutron
165+
server startup by default, and no longer runs a periodic resync.
166+
167+
- `resync_interval_secs` and `resync_max_interval_secs` are deprecated and have
168+
no effect. You can leave them in `neutron.conf` (they will not cause an
169+
error), but you can also remove them. If you previously relied on the
170+
periodic resync to recover from drift, use the startup resync (the default)
171+
and run `calico-resync` on demand when you need an immediate reconciliation.

‎sidebars-calico.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -328,6 +328,7 @@ module.exports = {
328328
'networking/openstack/host-routes',
329329
'networking/openstack/multiple-regions',
330330
'networking/openstack/live-migration',
331+
'networking/openstack/resync',
331332
'networking/openstack/kuryr',
332333
'networking/openstack/neutron-api',
333334
],

0 commit comments

Comments
 (0)