Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 48 additions & 3 deletions docs/rollups/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,29 @@ for it.
An entry takes whatever `RollupRequest` carries, minus the range and the dataset. A field added to
the request arrives here on its own.

`RollupSummaries` takes the same shape on its own `requests` prop. A site narrowing a question for
both writes the narrowing once and passes the constant to each:

```typescript
const searches = { paths: ["/liju/search/", "/cidian/search/"], param: "term" };

new RollupQueries(this, "RainlyticsRollups", {
table,
workgroup,
requests: { searches },
});
new RollupSummaries(this, "RainlyticsSummaries", {
table,
workgroup,
requests: { searches },
});
```

A saved query and a stored summary that drift apart answer two questions under one name. The
constant keeps them in step, and
[reading a precomputed answer](#a-summary-answers-the-question-it-was-computed-with) is where the
command line picks the same narrowing up.

A fact that belongs to every question, such as the host of one site on a distribution serving
several, is a variable spread into each entry. Every key has to name a rollup being saved, and a
mistyped one fails at synthesis. The alternative is a deployed query still counting whatever it
Expand Down Expand Up @@ -550,7 +573,18 @@ count every combination of them. `RollupSummaries` computes the unfiltered form
and [`requests`](../summary-schedule/) is where a deployment adds a narrowed one under a name of its
own.

A run whose filters no stored summary matches is told what was stored:
A run that names none of those options takes the ones the summaries were computed with. The
deployment declared its narrowing once and the command reads that copy back. A shell alias never has
to carry a second one. Standard error says which filters the run took:

```text
Took --path /liju/search/ /cidian/search/ from the summaries. Those options
were left off this command line, and the answer covers the narrowing the
deployment computes.
```

An option somebody typed stays theirs. A run whose filters no stored summary matches is told what
was stored:

```text
The stored pageviews summaries answer a different question.
Expand All @@ -560,9 +594,20 @@ on RollupSummaries is where a narrowed one is added. --query answers this run
from Athena at the cost a query reports.
```

A change to `requests` leaves both narrowings in the bucket. Over a span holding some of each, the
command names the option that would settle it and stops:

```text
The stored pageviews summaries over that span were not all computed the same
way, and this run named nothing to settle it with.
--path: some windows computed with /guides/, others with the whole distribution
```

Typing the one you want settles it. So does a span on one side of the change.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

`--limit` is the one option where the two only have to overlap. A summary computed with the top
hundred paths holds the top twenty inside it. The reverse loses rows nobody counted, and it is
refused.
hundred paths holds the top twenty inside it. The reverse loses rows nobody counted. A run that
typed the row count is refused, and a run that typed none takes the stored one.

### Several windows add up, and the ranking is approximate

Expand Down
32 changes: 32 additions & 0 deletions src/cli/rollup-options.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ import {
summaryBucketFrom,
} from "./option-values.js";
import { defaultLimit, defaultParam } from "./rollup-help.js";
import type { NarrowingOption } from "./summary-question.js";
import { narrowingOptions } from "./summary-question.js";

/** What one rollup was asked for, read off the command line. */
export interface RollupAsked {
Expand All @@ -34,6 +36,20 @@ export interface RollupAsked {
*/
readonly range: TimeRange;

/**
* The narrowing options this command line actually named.
*
* Read before the defaults go in, and this is the only place the two can
* still be told apart. `rollupRequest` fills in a value for every field of
* every question. A `RollupRequest` carrying `limit: 20` says nothing about
* whether anybody typed `--limit 20`.
*
* `summary-adoption.ts` is what needs the difference. A filter nobody named
* is one the stored summaries can supply, and a filter somebody named is one
* they have to match.
*/
readonly named: ReadonlySet<NarrowingOption>;

/** The bucket holding the precomputed answers, where one is known. */
readonly summaries: string | undefined;

Expand All @@ -50,6 +66,21 @@ export interface RollupAsked {
readonly region: string | undefined;
}

/**
* The narrowing options that arrived on one command line.
*
* `narrowingOptions` spells each one as a reader types it, and the parser keys
* on the long name without the dashes. An option nobody gave is absent from
* the values, whatever default the request will carry for it.
*/
function namedOn(context: CommandContext): ReadonlySet<NarrowingOption> {
return new Set(
narrowingOptions.filter(
(option) => context.options[option.slice(2)] !== undefined,
),
);
}

/**
* One command line, read as a rollup request.
*
Expand All @@ -67,6 +98,7 @@ export function requestFrom(
return {
database,
range,
named: namedOn(context),
workgroup: chosen(context.options["workgroup"]) ?? defaultWorkgroupName,
region: chosen(context.options["region"]),
summaries: summaryBucketFrom(context.options["summaries"]),
Expand Down
180 changes: 180 additions & 0 deletions src/cli/summary-adoption.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
// Settling the question a run is answered under, and stopping the runs the
// bucket cannot answer.
//
// A site that narrows a question used to say so three times. Once on
// `RollupQueries`, once on `RollupSummaries`, and again on every command line
// reading the answers back. The third copy is the one with nowhere to read
// from, and a deployment that changes its narrowing leaves every shell alias
// behind. A stored summary records the question it was computed with. The
// command reads that copy back, and the third one goes away.
//
// An option nobody named is the only kind taken. `rollupRequest` fills in a
// default for every field of every question, and by the time a
// `RollupRequest` exists a default and a value somebody chose are the same
// thing. `RollupAsked.named` carries the difference, read off the command line
// before the defaults go in.
//
// A named option that disagrees with the stored summaries is refused here too.
// Adoption fills gaps and never overrides one, and `refuseAnotherQuestion` at
// the foot of this file is the check that makes that safe to say.

import type { RollupSummary, SummaryQuestion } from "../rollup-summaries.js";
import type { Rollup } from "../rollups.js";
import type {
NarrowingOption,
StoredDisagreement,
} from "./summary-question.js";
import { narrowedBy, narrowingText } from "./summary-narrowings.js";
import { narrowingOptions, questionDifferences } from "./summary-question.js";
import {
answersSomethingElse,
computedMoreThanOneWay,
} from "./summary-refusals.js";

/** What a run settled on, once the stored narrowing had been read. */
export interface AdoptedQuestion {
/** The question the stored summaries are then checked against. */
readonly question: SummaryQuestion;

/**
* The filters taken from them, as the command line that would have asked.
*
* In the order help prints the options, and empty on a run that narrowed
* everything the summaries narrow.
*/
readonly adopted: readonly string[];
}

/**
* The question asked, with the filters nobody named taken from the summaries.
*
* A command line that narrows nothing is answered under the narrowing its
* deployment declared, and {@link AdoptedQuestion.adopted} is what standard
* error then says it took. A command line that narrows something keeps what it
* was given, and `refuseAnotherQuestion` decides whether the summaries hold
* that answer.
*
* @throws {Error} where the summaries answer a filter nobody named two ways.
* Taking one of the two would answer part of the span under a narrowing the
* rest was never computed with, and the question has to be settled before
* anything can be checked against it.
*/
export function adoptedQuestion(
rollup: Rollup,
asked: SummaryQuestion,
named: ReadonlySet<NarrowingOption>,
summaries: readonly RollupSummary[],
): AdoptedQuestion {
const [first] = summaries;

if (first === undefined) {
return { question: asked, adopted: [] };
}

const elsewhere = storedElsewhere(rollup, asked, summaries);
const wanted = narrowingOptions.filter(
(option) => !named.has(option) && elsewhere.has(option),
);
const disagreements = disagreedOn(rollup, wanted, first, summaries);

if (disagreements.length > 0) {
throw computedMoreThanOneWay(rollup, disagreements);
}

let question = asked;

for (const option of wanted) {
question = narrowedBy(option, question, first.question);
}

return {
question,
adopted: wanted.map((option) => narrowingText(option, first.question)),
};
}

/** The options any stored summary was computed with differently. */
function storedElsewhere(
rollup: Rollup,
asked: SummaryQuestion,
summaries: readonly RollupSummary[],
): ReadonlySet<NarrowingOption> {
return new Set(
summaries
.flatMap((summary) =>
questionDifferences(rollup, asked, summary.question),
)
.map((difference) => difference.option),
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
);
}

/**
* The options among `wanted` that the summaries answer two ways.
*
* Every summary is compared against `first`, both ways round, since `--limit`
* reports a difference in one direction alone. A window computed with the top
* hundred rows covers a run asking for twenty, and a window computed with the
* top one does not. Comparing `first` against itself finds nothing, which is
* why it stays in the list it is compared against.
*/
function disagreedOn(
rollup: Rollup,
wanted: readonly NarrowingOption[],
first: RollupSummary,
summaries: readonly RollupSummary[],
): readonly StoredDisagreement[] {
const found = new Map<NarrowingOption, Set<string>>();

for (const other of summaries) {
const between = [
...questionDifferences(rollup, first.question, other.question),
...questionDifferences(rollup, other.question, first.question),
];

for (const difference of between) {
const values = found.get(difference.option) ?? new Set<string>();

found.set(
difference.option,
values.add(difference.asked).add(difference.computed),
);
}
}

return wanted.flatMap((option) => {
const values = found.get(option);

return values === undefined ? [] : [{ option, computed: [...values] }];
});
}

/**
* Stops a run whose filters no stored summary was computed with.
*
* Every window is checked rather than the first alone. A deployment whose
* question changed halfway through the span has summaries of both, and the
* older half answers something the command line never asked.
*
* The question checked is the settled one. A filter the command line named is
* still its own, and a filter taken from the summaries matches them by
* construction.
*
* @throws {Error} naming every option a reader would have to change.
*/
export function refuseAnotherQuestion(
rollup: Rollup,
question: SummaryQuestion,
summaries: readonly RollupSummary[],
): void {
const differences = new Map(
summaries
.flatMap((summary) =>
questionDifferences(rollup, question, summary.question),
)
.map((difference) => [difference.option, difference]),
);

if (differences.size > 0) {
throw answersSomethingElse(rollup, [...differences.values()]);
}
}
Loading