Skip to content

Limit IDs passed to location assocations query - #189

Open
Rovack wants to merge 10 commits into
developfrom
feature/avoid-large-associations-query
Open

Limit IDs passed to location assocations query#189
Rovack wants to merge 10 commits into
developfrom
feature/avoid-large-associations-query

Conversation

@Rovack

@Rovack Rovack commented Mar 13, 2026

Copy link
Copy Markdown
Member
  • When locationFieldsOnly is true, get all datafields in the 1st query, so the 2nd one doesn't need a giant IN with countless IDs.
  • When locationFieldsOnly is false, disallow fetching >200 locations.

- When locationFieldsOnly is true, get all datafields in the 1st query,
  so the 2nd one doesn't need a giant IN with countless IDs.
- When locationFieldsOnly is false, disallow fetching >200 locations.
@Rovack
Rovack temporarily deployed to CI_CD_PIPELINE March 13, 2026 21:59 — with GitHub Actions Inactive

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR introduces a blocking regression in the locationFieldsOnly code path and changes response-limiting behavior without safeguards/tests.

Blocking issues

  • Broken API behavior: Location.search() now returns raw locationIds when locationFieldsOnly is truthy (src/models/location.js:590-592), but the controller still assumes Sequelize model instances and calls location.get(...) on each result (src/controllers/locations.js:259-260). This will throw at runtime (get is not a function) for locationFieldsOnly requests.
  • Potential regression with no validation/tests: the new hard cap (Math.min(requestedLimit, 200)) for non-locationFieldsOnly responses (src/controllers/locations.js:236-240) silently changes API behavior for clients requesting >200 results, including paginated flows, and no accompanying tests/documentation are present to prove this is safe.

@Rovack Rovack changed the title Limit IDs passed to location assocations query WIP: Limit IDs passed to location assocations query Mar 13, 2026
@Rovack Rovack changed the title WIP: Limit IDs passed to location assocations query [WIP] Limit IDs passed to location assocations query Mar 13, 2026
@Rovack
Rovack marked this pull request as draft March 13, 2026 22:01

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 346a456f41

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/models/location.js
Comment thread src/controllers/locations.js Outdated
@Rovack
Rovack marked this pull request as ready for review March 13, 2026 22:09
@Rovack
Rovack temporarily deployed to CI_CD_PIPELINE March 13, 2026 22:09 — with GitHub Actions Inactive
@Rovack
Rovack marked this pull request as draft March 13, 2026 22:09

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR introduces a blocking API regression in the locationFieldsOnly path and is not safe to approve.

Blocking issues

  • Broken API behavior: Location.search now returns raw plain objects for locationFieldsOnly (locationsWithAssociations = locationStubs), but the controller still calls location.get({ plain: true }) on every result. This will throw TypeError: location.get is not a function at runtime for GET /locations with locationFieldsOnly=true (src/models/location.js, src/controllers/locations.js).
  • Behavioral regression: the locationFieldsOnly response is now built from findUniqueLocationStubs, whose query selects only DISTINCT Location.id (+ optional sort attribute) with raw: true. Even if the runtime error were fixed, this path no longer reliably returns expected location fields/association structure used to compute closed, changing API response semantics (src/models/location.js).

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bb912756e1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/controllers/locations.js Outdated
@Rovack
Rovack marked this pull request as ready for review March 13, 2026 22:24
@Rovack
Rovack temporarily deployed to CI_CD_PIPELINE March 13, 2026 22:24 — with GitHub Actions Inactive
@Rovack
Rovack marked this pull request as draft March 13, 2026 22:24

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not safe to approve. The refactor introduces API behavior regressions in locationFieldsOnly responses and pagination semantics, with no accompanying test coverage for the new risky paths.

Blocking issues

  • Broken API behavior for locationFieldsOnly: Location.search now returns locationStubs directly (src/models/location.js:592-594), but those stubs are produced by a raw: true query selecting only DISTINCT Location.id (plus optional sort attr) (src/models/location.js:374-379). This means callers can receive incomplete/flattened records instead of full location fields, and closed computation in the controller can run with missing EventRelatedInfos/Services data (src/controllers/locations.js:263-271).
  • Pagination regression: offset is now computed from capped limit (src/controllers/locations.js:242-243) instead of requested pageSize. For non-locationFieldsOnly requests where pageSize > 200, pages will overlap/skip (e.g., page 1 offset 0, page 2 offset 200 instead of 300), causing duplicated/missing results and inconsistent paging behavior.
  • Risky logic change lacks targeted tests: no integration tests cover locationFieldsOnly response shape/closed correctness or pagination behavior with capped detailed results, so these regressions would not be caught pre-merge.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e39f2f156d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/models/location.js
Comment thread src/controllers/locations.js Outdated
- Remove attribute filtering from the query for location stubs.
- Dedupe in memory, as the query doesn't do DISTINCT anymore.
@Rovack
Rovack marked this pull request as ready for review March 13, 2026 22:45
@Rovack
Rovack temporarily deployed to CI_CD_PIPELINE March 13, 2026 22:45 — with GitHub Actions Inactive
@Rovack
Rovack marked this pull request as draft March 13, 2026 22:45

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found blocking API-behavior regressions in pagination and locationFieldsOnly result shaping, so this PR is not safe to approve yet.

Blocking issues

  • src/controllers/locations.js + src/models/location.js: pagination semantics are now inconsistent when locationFieldsOnly is false and pageSize > 200. The code caps limit to 200 but still computes paginationCount using the original pageSize, which can cause clients to stop paging early and miss results.
  • src/models/location.js (findUniqueLocationStubs + search locationFieldsOnly path): locationFieldsOnly now returns rows from a raw: true include query, then deduplicates by id. Raw include rows do not provide the nested Services/EventRelatedInfos structure expected by src/controllers/locations.js (isLocationClosed(...)), so closed computation and response shape can be incorrect.
  • src/models/location.js: deduplicating raw joined rows before returning stubs drops multi-row association data (keeps first row per location). In the locationFieldsOnly fast-path this can silently remove service/schedule context, creating incorrect API output.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bd996a66b4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/models/location.js Outdated
- Revert to using pageSize as before, but capping both it _and_
  maxResults (not limit directly) if asking for location details.
@Rovack
Rovack marked this pull request as ready for review March 13, 2026 22:53
@Rovack
Rovack temporarily deployed to CI_CD_PIPELINE March 13, 2026 22:53 — with GitHub Actions Inactive
@Rovack
Rovack marked this pull request as draft March 13, 2026 22:53

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not safe to approve. The PR changes introduce likely API regressions in locationFieldsOnly responses and closed computation, with no accompanying tests proving compatibility.

Blocking issues

  • locationFieldsOnly now returns raw stub rows (src/models/location.js:589) produced by a raw: true multi-join query (src/models/location.js:376-419) instead of hydrated location models. This changes response shape and can expose flattened join columns (for example association-prefixed fields) rather than the prior contract.
  • Controller logic still expects association arrays (EventRelatedInfos, Services) to compute closed (src/controllers/locations.js:266-273), but raw stubs do not provide that structure. This can silently produce incorrect closed values and behavior regressions for occasion-based queries.
  • Risky behavioral change lacks targeted tests for locationFieldsOnly output schema and closed calculation after the query-path rewrite. Given the response-path change, missing regression tests is blocking.

Rovack added 2 commits March 14, 2026 00:58
- Re-add selectedAttributeForOrderBy to findAll, so when it's a
  computed column it's still selected and therefore exists for WHERE.
@Rovack
Rovack marked this pull request as ready for review March 13, 2026 23:19
@Rovack
Rovack temporarily deployed to CI_CD_PIPELINE March 13, 2026 23:19 — with GitHub Actions Inactive
@Rovack
Rovack marked this pull request as draft March 13, 2026 23:19

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found blocking regressions in the new locationFieldsOnly flow and query shape changes that can alter API behavior.

Blocking issues

  • Broken API behavior for locationFieldsOnly=true: Location.search now returns locationStubs directly when locationFieldsOnly is set, but those stubs come from findUniqueLocationStubs with raw: true and joined includes. In src/controllers/locations.js, the formatter expects nested EventRelatedInfos/Services and strips them; with raw rows these are not present as expected, which can break closed calculation and response shape.
  • findUniqueLocationStubs removed SQL-level DISTINCT id selection and now fetches broader joined rows (including always-joined EventRelatedInfo and Service.HolidaySchedule) before deduping in memory. This changes result ordering/selection behavior under joins and creates regression risk for pagination/sort correctness in Location.search (especially when limit/offset are applied after dedupe).
  • Risky logic changes are not accompanied by targeted tests for: (a) locationFieldsOnly response structure and closed derivation, and (b) pagination/sort stability after the DISTINCT removal and in-memory dedupe path.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6ffc84ee25

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/models/location.js
- Even when using the `raw: true` query, without associations, use a
  literal to add a column with the event list from EventRelatedInfos.
- Standardize the response structure by returning it as objects
  resembling the (relevant parts of) EventRelatedInfos.
@Rovack
Rovack marked this pull request as ready for review March 14, 2026 00:31
@Rovack
Rovack temporarily deployed to CI_CD_PIPELINE March 14, 2026 00:31 — with GitHub Actions Inactive
@Rovack
Rovack marked this pull request as draft March 14, 2026 00:31

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The refactor introduces a likely API regression in locationFieldsOnly mode and lacks coverage for the new query/response behavior.

Blocking issues

  • Broken API behavior risk in src/models/location.js + src/controllers/locations.js: Location.search now returns locationStubs (raw rows) directly when locationFieldsOnly=true. Those stubs are produced with raw: true and joined includes, then only EventRelatedInfos is reconstructed. Services is not reconstructed as an array/object, so downstream logic in locations controller (isLocationClosed(occasion, EventRelatedInfos, Services)) can receive incorrect/empty service data and compute wrong closed values.
  • Payload shape regression risk for locationFieldsOnly responses: because stubs come from raw: true joined queries, response objects may contain flattened association columns instead of the prior model/plain shape. This is a behavior change to a public API path and is not guarded by tests in this PR.
  • Missing tests for risky logic changes: no tests were added for (a) locationFieldsOnly result shape/closed computation after switching to stubs, and (b) the new detailed-results cap behavior (capDetailedLocations) affecting limit/minResults. Given the scope of query and response-path changes, this is a blocking gap.

@Rovack Rovack changed the title [WIP] Limit IDs passed to location assocations query Limit IDs passed to location assocations query Mar 14, 2026
- Remove services from the signature and calls to isLocationClosed.
- To still be able to omit it from the returned location objects,
  configure varsIgnorePattern for eslint.

(Doesn't really has to be part of this PR, but both changes are good to
have regardless, and should make Codex less confused.)
@Rovack
Rovack marked this pull request as ready for review March 14, 2026 00:43
@Rovack
Rovack temporarily deployed to CI_CD_PIPELINE March 14, 2026 00:43 — with GitHub Actions Inactive

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not safe to approve. The PR introduces a likely API behavior regression in the locationFieldsOnly path and lacks coverage for high-risk query/response-shape changes.

Blocking issues

  • locationFieldsOnly can now return unintended fields/shape changes. In src/models/location.js, findUniqueLocationStubs still joins broad associations (Organization, PhysicalAddress, Phone, Service, etc.), and Location.search now returns those stubs directly when locationFieldsOnly is true. In src/controllers/locations.js, only EventRelatedInfos and Services are stripped, so other joined fields can leak into the response, changing API behavior.
  • Risky query-path refactor has no accompanying tests in this PR. The move from findUniqueLocationIds to raw findUniqueLocationStubs + reconstructed EventRelatedInfos + capped detailed pagination changes response construction and pagination behavior; without tests for locationFieldsOnly response contract and pagination/order semantics, regression risk is too high.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6bc827a61e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/models/location.js
Comment on lines +386 to +390
attributes: {
include: [
EVENTS_WITH_INFO_COLUMN_SUBQUERY,
...(selectedAttributeForOrderBy ? [selectedAttributeForOrderBy] : []),
],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restore distinct ID projection in stub lookup query

Location.search still uses this helper for pagination/count paths ((await ...).length), but this change replaced the old DISTINCT Location.id projection with a full-row select (plus extra subqueries/joins) and then deduplicates in JavaScript. On broad searches this pulls and materializes many joined rows per location just to count IDs, which can significantly increase query latency and memory use and make /locations requests time out under production-sized datasets.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant