Skip to content

masks: no brush release crashes or ends a creation session, a click's source is drawn and grabbed, and the overlay composites all the masks draw and no more - #1632

Merged
Jiyone merged 13 commits into
masterfrom
bug-1621
Nov 11, 2026
Merged

Jiyone merged 13 commits into
masterfrom
bug-1621

Conversation

@Jiyone

@Jiyone Jiyone commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Fixes #1621
Fixes #1609

What crashed

In a brush creation session, the release that ends a stroke goes through dt_masks_events_button_released(). That dispatcher read form_visible before calling the shape's handler, then used it afterwards for the group selection change and for _set_hinter_message().

When the release's press had gathered no point, the brush handler ended the creation session through dt_masks_set_edit_mode() → dt_masks_change_form_gui(). That frees the unfinished shape that dt_masks_gui_form_save_creation() had just made visible. The dispatcher then read freed memory, and _set_hinter_message() called a function pointer out of it: SIGSEGV at the end of the second stroke.

A press gathers no point when it carries Shift and the release does not, when it never reaches the masks, or when the device sends the release twice. A tablet makes the last two likely.

Measured under ASan, the three steps are:

  • allocated in dt_masks_gui_form_save_creation();
  • freed in _brush_events_button_released() → dt_masks_set_edit_mode() → dt_masks_change_form_gui();
  • read in dt_masks_events_button_released().

The change

  1. masks: a shape's event handler may replace the visible form, so the dispatcher reads it again after. _set_hinter_message() now reads the visible form itself instead of taking one from its caller. The release and scroll dispatchers read the visible form again after the handler. This also covers the dt_masks_cow_touch() clone that the earlier re-read was there for. The press dispatcher computes shape_was_selected before the handler runs. This removes the whole class of defect from the dispatchers, not just the brush path.
  2. brush: a release whose press gathered no point leaves the creation session armed. Such a release now draws nothing and leaves the session as it was. A creation session ends only through the user's own ways out:
    • Escape;
    • the context menu entry;
    • the armed toolbar button clicked again;
    • BackSpace on a polygon with no node placed;
    • entering edit mode;
    • giving another module the focus;
    • the first edit made off the canvas, as master's d9783a1296 has it: a module's control, a style, a jump in the history;
    • the darkroom tearing down.
  3. A click without a drag still makes a brush, and that brush can now be seen and used. A click reaches the release with one point and is padded with a second node 0.01 px away: a one-point brush, whose mask is the full disc of its border. An earlier commit on this branch dropped it instead; that was reverted (6e833b21b5), since the brush is wanted. What was wrong was the drawing:
    • brush: a brush shorter than a pixel is drawn by its border, and so is its source. Its centreline has no length, so unselected it showed only a speck, and its clone source was not drawn at all: the link is anchored at the centreline's midpoint, and a line with no length has none. The link falls back to the first nodes, and the source is the border carried onto the source node, the shift the hit test already uses.
    • brush: a dot draws its dashed border only while selected, like any brush. The commit above drew it in every state; that is taken back. An unselected dot shows its centreline, and its source is still drawn.
    • brush: a dot's source is hit anywhere inside its drawn border. It could only be grabbed within the cursor's reach of its centre, as for a circle's source now anywhere inside.
  4. masks: the overlay composites a clone source wherever it lies. The overlay composites only the rectangle it bounds, and the bound held no clone source: in a creation session the source of a shape just made was cut to the box of the shapes already made. An outline's box now holds the shape's box carried onto its source, and both bounds use it. The composited rectangle is also cut to the view: one reaching out of it is never covered by an expose, and the overlay repainted continuously while a selected shape's bound crossed the edge.
  5. masks: a brush trace drawn in a creation session is shown outside the shapes already made. The brush trace, its cursor and a clone's source cross are stroked by cairo, which the rasteriser's touched rectangle never saw: from the second stroke of a session on, the trace was drawn but shown only inside the first strokes' box. The points were recorded; only the drawing was cut. dt_stroke_raster_touch_path() now lets what cairo strokes declare what it covers. This is Bug: Brush shape's cursor remains hidden by the bounding box of the previous brush shape #1609: the cursor and the path hidden behind the previous brush shape's box, only from a session's second stroke on, which is why it looked random.
  6. masks: a dot's source and the overlay bound are drawn without single-use helpers. No change of behaviour. It follows a review against AGENTS.md and SonarCloud's findings on this pull request. The callback that only forced the border flag for the dot's source ring, the box-bound helper and the frame-cut helper are folded into their callers. dt_stroke_raster_touch_path() loses a flag that was TRUE at every call, and a NULL test on a context every caller holds.
  7. masks: a point that is not finite is no sample of an outline's box. A gradient's border joins its two curves with an (INFINITY, INFINITY) separator, and the box counted it, so the box was infinite. The bound of a selected shape that (4) builds from that box carried the infinity into _rect_include(), whose conversion to int is undefined. On x86 it gives INT_MIN, which happened to cover everything; nothing was measured on ARM. The session clip, on master too, was a rectangle of infinite width that let nothing through, so in a gradient creation session the gradient already made was not drawn while the next one was placed. The box now leaves out any point that is not finite.
  8. masks: a brush's and a polygon's border header repeats the outline's. The border buffer is index-aligned with the points, and its header, three entries per node, was zeros: the image's origin. Nothing reads them, since the pipeline's area starts after the header. But every box over the whole buffer counted them, so the box of every brush and polygon reached the top-left corner of the image, and so did the rectangle composited and cleared each frame and the creation session's bound. The border's header now repeats the outline's. A polygon still writes each node's border handle into the first of the three entries.

Documented in doc/masks-gui.md, doc/masks-geometry.md and doc/overlay-raster.md.

Measured

A shim loaded with LD_PRELOAD plays brush strokes on "diffuse or sharpen". It calls dt_control_button_pressed() / dt_control_mouse_moved() / dt_control_button_released() on the GUI thread, with tablet pointer input set. Build: Debug + ASan, CPU only.

second stroke before after
release without a press heap-use-after-free clean
Shift press, plain release heap-use-after-free clean
second release heap-use-after-free clean
two ordinary strokes clean clean

After the change, a point-less release leaves the unfinished shape and the dev->forms count unchanged, creation stays set, and a third stroke is saved.

For the drawing, shims on a RelWithDebInfo build in Xephyr (1280×900, a 2400×1600 JPEG at fit zoom) drive retouch's shape tools and log what the overlay strokes and the rectangle it composites (MASKS_DUMP_OVERLAY):

case before after
a click, unselected a speck, no source, no arrow the speck, a 32×32 px source ring and the arrow; no ring at the shape
the click's source, hovered 8–15 px from its centre (ring radius 16 px) nothing selected source selected
session: circle of radius 0.3, source down to row 485 composited down to row 323 down to row 511
session: a click, source at row 389 composited down to row 230 down to row 424
a dot selected near the view's left edge 57 redraws in one second, nothing moving 1
session: second stroke, cursor at (400, 518) composited up to (199, 187) up to (418, 536)

For the cleanup (6), the dot shim (drawn, idle, then hovered inside its source's border at 60 % of its radius) and the two-stroke session shim were replayed on the builds before and after it. The 5 overlay images and the 7 text dumps are identical, and the dot's source is still hit.

For (7), a shim makes a gradient in exposure's drawn mask, dumps the frame with the session still open, then selects the gradient the way Escape does. The box was x −1684 to inf, y −642 to inf; it is x −1684 to 4082, y −642 to 2242. The selected gradient draws the same pixels as before. The session frame draws 3230 more, all within 2 px of where the gradient is drawn once selected; master drew none of them either. The dot and two-stroke shims dump identical overlays. Also checked by hand: the gradient session, Escape, then a retouch dot and two brush strokes.

For (8), at fit zoom in a 500×648 view with the shapes at the bottom right of the image, on the builds before and after it:

shape composited rectangle before after
brush 500×421, 0.31 ms per frame 204×202, 0.10 ms (400 frames each)
dot 247×133 88×87
polygon, a node selected 500×457 213×263
two-stroke creation session 418×473 387×473

Every overlay draws the same pixels, the selected polygon node's border handle included, and nothing painted is left outside the rectangle. A mouse move costs the same, about 180 µs, inside the old box or outside both. Also checked by hand: a retouch dot, a polygon node's border handle once the node is clicked, and two brush strokes.

Gates run, all pass:

  • check_layering.sh
  • check_module_boundaries.sh
  • check_return_types.py
  • check_list_order.py
  • pragma_once_to_guards.py --verify
  • include_graph.py --summary (cycles 0)
  • check_statelessness.sh
  • check_conditional_includes.sh origin/master
  • check_unused_includes.sh --changed origin/master

Both files compile without warnings under clang-19.

Rebased on master e19adbd0b9. The commits apply unchanged, except one line of doc/masks-gui.md and the message of 03a1d170af, which now count the first edit made off the canvas among the ways a session ends, and the revert's message, which names the rebased commit. On the rebased head, on a clang-19 RelWithDebInfo build, the gates above pass except check_statelessness.sh, which cannot read that build's LTO objects: stroke_raster.c, the one file of this branch it covers, was compiled without LTO under -Werror instead, and holds no static data. check_layering.sh reports cycles 0 and 182 layering violations, its baseline; check_unused_includes.sh introduces none in the 4 files it checks. ctest passes 32 of 32, check_it_runs.sh exports and exits cleanly, and the touched files compile without warnings. Before the rebase, check_statelessness.sh passed on a clang-19 Debug build without LTO, the touched files compiled without warnings in Debug too, and the pre-commit image_test.sh smoke test (8 raws) passed on every commit touching src/.

Not tested:

  • after the rebase, the shims were not replayed; master's change to how a session ends was checked against this branch by reading the code only;
  • a real tablet;
  • a two-point stroke (a press and a single motion): the code keeps it, but only the click alone was played;
  • the reporter's GCC 13 LTO build;
  • the full image_test.sh bank. The changes are in GUI event handling and overlay drawing, which the pixel pipeline never runs;
  • the cursor alone, no button pressed, in a session: in the shim it was composited over the whole view before the change as after, so only the drag is measured;
  • a selected shape's source cut by the old header bound outside a session: the mechanism is in the code, but the shapes played here fit within its margin;
  • a source outline under a non-affine distortion after the module (lens correction, perspective), where carrying the border is an approximation, as the hit test already makes.

Found on the way, filed separately

🤖 Generated with Claude Code

t and others added 13 commits November 11, 2026 15:56
…ispatcher reads it again after

dt_masks_events_button_released() read form_visible before calling the shape's handler and used it afterwards for the group selection change and the hint. A brush release that gathered no points ends the creation session through dt_masks_set_edit_mode() and dt_masks_change_form_gui(), which frees the unfinished shape: the dispatcher then read freed memory, and _set_hinter_message() called a function pointer out of it (SIGSEGV at the end of a second brush stroke).

_set_hinter_message() now reads the visible form itself instead of taking one from its caller. The release and scroll dispatchers read the visible form again after the handler, which also covers the dt_masks_cow_touch() clone the earlier re-read was there for, and the press dispatcher computes shape_was_selected before the handler runs.

Reproduced under ASan with a shim playing two brush strokes on "diffuse or sharpen" through dt_control_button_pressed/mouse_moved/button_released: a second stroke without a press, with a Shift press, or followed by a second release read freed memory 3 times out of 3; with this change the same runs, and two ordinary strokes, are clean.

Fixes #1621

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssion armed

A brush release in a creation session with no point gathered -- the press carried Shift and the release did not, the press did not reach the masks, or the device sent the release twice -- ended the session through dt_masks_set_edit_mode() and dt_masks_change_form_gui(), disarming the tool after an ordinary stroke for no gesture of the user's. It now draws nothing and leaves the session as it was. A creation session ends only through the user's own ways out: Escape, the context menu entry, the armed toolbar button clicked again, BackSpace on a polygon with no node placed, entering edit mode, giving another module the focus, the first edit made off the canvas, the darkroom tearing down.

Measured under ASan with a shim playing three brush strokes on "diffuse or sharpen": when the second stroke has no press, a Shift press, or a second release, that release leaves the unfinished shape and dev->forms unchanged with creation still set, and the third stroke is saved; no run reads freed memory.

Refs #1621

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… border gap fillers take their points as const

A release whose press gathered no point no longer ends the creation session, so `_brush_events_button_released()` stops using `module`; the parameter stays, since the shape function table fixes the signature.

`_brush_points_recurs_border_gaps()` and `_brush_points_recurs_border_small_gaps()` only read the corner and the two border points they are given.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A release that gathered a single point, a click without a drag, was padded with a second node 0.01 away and saved as a brush. It is now dropped like a release with no point: nothing is saved and the creation session stays armed for the next stroke.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ays armed"

This reverts commit a5e813f. A click makes a one-point brush again, padded with a helper node: the brush is wanted, what was wrong is that it could hardly be seen, and that is a matter for the overlay, not a reason to drop the shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its source

A click without a drag makes a brush whose centreline has no length. Unselected, a brush draws its centreline only, which for such a dot is a speck the width of the line, and its clone source was not drawn at all: the link is anchored at the centreline's midpoint and a line with no length has none.

A brush whose centreline is shorter than a pixel now draws its dashed border in every state, not only while selected. The link's ends fall back to the first nodes, and the source is the border shifted onto the source node, as the hit test already shifts it, drawn solid like every source outline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The overlay composites only the rectangle it bounds, and the bound held no clone source. A creation session's box was the union of its shapes' spines and borders, so a source outside them was drawn into the canvas and never composited: in retouch, the source of a shape just made was cut to the shape's box. Outside a session, the box was the node headers with a fixed margin, which a source drawn as large as a border can exceed.

An outline's box now holds the shape's box carried onto its source, by the shift between the first points of the two outlines, and both bounds take it: the session's is the union of its outlines' boxes, the selected shape's includes its own.

The rectangle is also cut to the frame. One reaching out of the view is never covered by an expose, so the damage record asked a redraw of it after every frame, and the overlay repainted continuously while a selected shape's bound crossed the edge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A brush's source is hit within the cursor's reach of its centreline carried onto the source. A dot's centreline is a speck, so its source, drawn as its border carried there, could only be grabbed at its very centre: measured, up to 4 px from it, and nothing at all between 8 and 15 px inside a 16 px ring, not even the shape.

A dot's source is now hit anywhere inside its border carried onto the source, as a circle's source is inside its outline. The centreline end and the dot test move above the hit test that now uses them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… shapes already made

In a creation session the overlay composites the box of the shapes already made plus what the rasteriser touched. A brush's trace and cursor, and a clone's source cross, are stroked by cairo, which the rasteriser never sees: from the second stroke of a session on, the trace was drawn into the canvas and shown only inside the first strokes' box. The points were recorded; only the drawing was cut. The first stroke hid it, the bound being the whole canvas while the session has no box.

dt_stroke_raster_touch_path() adds what the current path covers, its extents grown by half the line width, to the surface's touched rectangle, and the brush trace, its cursor and dt_draw_cross() declare their strokes with it. Measured: the rectangle composited at a second stroke's last motion ended at (199, 187) with the cursor at (400, 518); it ends at (418, 536).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…use helpers

A dot's source ring is its border path built by _brush_draw_shape() and stroked by dt_draw_source_shape(), which strokes the current path when given no callback: the wrapper that only forced the border flag, and its static pointer, are gone. The outline's box joins the cairo bound through _bound_include_points(), with its four corners. _overlay_dirty() unions the rasteriser's rectangle with the cairo bound and cuts the result to the frame itself, with gdk_rectangle_union() and gdk_rectangle_intersect().

dt_stroke_raster_touch_path() takes the context alone: its stroke flag was TRUE at every call, and its NULL test covered a context every caller holds. It declares one variable per statement, _brush_is_dot() leaves its loop by a return rather than a condition on the length, and the brush's button release marks widget_x and widget_y unused, as SonarCloud asked on this pull request.

Measured with the same shims on the builds before and after: a dot drawn, idle, then hovered inside its source's border at 60 % of its radius, and two strokes of one creation session. The 5 overlay images and the 7 text dumps are identical, and the source is still hit. lizard: brush.c 674 -> 672, masks_gui.c 1286 -> 1279, stroke_raster.c 135 -> 134.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A gradient's border joins its two curves with an (INFINITY, INFINITY) separator, and the outline's box counted it: the box was infinite. No hit test could rule a gradient out. The session clip, a rectangle of infinite width, let nothing through, so in a gradient creation session the gradient already made was not drawn while the next one was placed. And the overlay's bounds carried the infinity into _rect_include(), whose conversion to int is undefined: INT_MIN on x86, which here happened to make a rectangle covering everything.

The box now leaves out any point that is not finite.

Measured with a shim that makes a gradient in exposure's drawn mask, dumps the frame with the session still open, then selects the gradient the way Escape does. The box was x -1684 to inf, y -642 to inf; it is x -1684 to 4082, y -642 to 2242. The selected gradient draws the same pixels; the session frame draws 3230 more, all within 2 px of where the gradient is drawn once selected. The dot and two-stroke shims dump identical overlays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The border buffer is index-aligned with the points, and its header, three entries per node, was zeros: the image's origin. Nothing reads them, the pipeline's area starting after the header, but every box taken over the whole buffer counted them. The box of every brush and polygon reached the top-left corner of the image, and with it the rectangle the overlay composites and clears each frame and the creation session's bound.

The border's header now repeats the outline's. A polygon still writes each node's border handle into the first of the three entries.

Measured at fit zoom in a 500x648 view, shapes at the bottom right of the image. A brush's box went from x 0..2407, y 0..1529 to x 1717..2407, y 1097..1529, its composited rectangle from 500x421 to 204x202, and the composite from 0.31 to 0.10 ms per frame over 400 frames. A dot's rectangle went from 247x133 to 88x87, a polygon's from 500x457 to 213x263, a two-stroke creation session's from 418x473 to 387x473. Every overlay drew the same pixels, and nothing painted was left outside the rectangle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A brush shorter than a pixel drew its dashed border in every state. It goes back to being drawn only on the selected brush, as for every other brush; an unselected dot shows its centreline, and its clone source is still drawn as its border carried onto the source, with the link anchored at its first nodes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
14 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@Jiyone Jiyone changed the title masks: a brush release that gathered no point no longer frees the shape the dispatcher still reads, and keeps creation armed masks: no brush release crashes or ends a creation session, a click's source is drawn and grabbed, and the overlay composites all the masks draw and no more Nov 11, 2026
@Jiyone
Jiyone merged commit f76012b into master Nov 11, 2026
84 of 98 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

Bug: Ansel crashed whilst drawing a mask Bug: Brush shape's cursor remains hidden by the bounding box of the previous brush shape

1 participant