feat: the picture by week and month, and All time from months kept for good (ADRs 0063, 0064)

README.md

@@ -19,7 +19,9 @@ Two rules shape everything here:
   recordings, no cookie banners to click through. Raw addresses are dropped after 72 hours, and
   the key that tells one visitor's activity from another's is worked out with a secret that
   changes every day, so once an address is gone there is no key that joins a reader's activity
-  from one day to the next — and no screen or figure ever tries to.
+  from one day to the next — and no screen or figure ever tries to. Every report of a visit is
+  deleted after twelve months; what is kept after that is each month's totals, until the website
+  is removed.
 
 ## Where this is today
 
@@ -37,8 +39,15 @@ and the [master plan](#master-plan) below it says what does not yet.
   website. Removing a website deletes everything ever measured for it and nothing brings it back,
   so it asks you to type the website's address before it will go ahead. Anyone who can change a
   website's settings can rename it; removing one is the owner's alone.
-- A dashboard showing page views, daily visitors, a daily traffic graph whose days can be pressed
-  to look at one on its own, and the pages a period's traffic went to, for a website you own. A
+- A dashboard showing page views, daily visitors, a traffic graph drawn by the day, the week or the
+  month whose bars can be pressed to look at the days they cover, and the pages a period's traffic
+  went to, for a website you own. "All time" shows the figures and the graph, by the month or the
+  year, back to the first month the website was measured — or, on an installation that first runs
+  this release more than about eleven months after that, the first month it could still count —
+  long after the reports of those months are gone; the lists beneath it cover the last twelve
+  months, which is as far back as anything about a single visit is kept. Nothing is ever compared
+  with days that are no longer kept, so a period whose earlier stretch reaches back past twelve
+  months shows no rise or fall at all. A
   site running both the tracker and its own server's reports is counted once per page delivered
   rather than once per report, so running the product properly does not double what it tells you.
   Traffic arriving through a pool of rented addresses is counted as the one operator it is, rather
@@ -115,8 +124,9 @@ and the [master plan](#master-plan) below it says what does not yet.
   ends a visit.
 - Every figure on the overview and the whole User journey list can be kept to the visits judged to
   be people, with one switch beside the period. It is the same arithmetic over fewer visits, so the
-  cards, the graph and every list add up to each other, and the choice travels in the address with
-  the period so a link opens on exactly what was being looked at.
+  cards, the graph and every list add up to each other — under "All time" the cards and the graph
+  do, and the lists beneath them cover the last twelve months — and the choice travels in the
+  address with the period so a link opens on exactly what was being looked at.
 
 - Two ways of establishing that a crawler really is whose crawler it says it is, which between
   them are the only basis on which a visit is ever reported as confirmed. Most companies publish a
@@ -135,7 +145,7 @@ and the [master plan](#master-plan) below it says what does not yet.
 ## Master plan
 
 Every capability the product is meant to have, phase by phase, in the order the work is picked
-up. Each line is marked with one of four states. Last reviewed 2026-09-21.
+up. Each line is marked with one of four states. Last reviewed 2026-10-30.
 
 - **Done** — shipped and visible on the dashboard, or collected on every capture surface.
 - **Partly done** — a real piece ships; the line says what is still missing.
@@ -144,7 +154,8 @@ up. Each line is marked with one of four states. Last reviewed 2026-09-21.
 
 ### Phase 1 — Public statements
 
-- **Done** · README: retention is 12 months, not unlimited.
+- **Done** · README: every report of a visit is kept for 12 months; each website's monthly totals
+  until the website is removed.
 - **Done** · README: a page counts as read only when its arrival reached us.
 - **Done** · README: multi-customer analysis is planned, not present.
 - **Done** · README: the daily key does not recognise a returning reader.
@@ -388,8 +399,11 @@ up. Each line is marked with one of four states. Last reviewed 2026-09-21.
 
 **5.6 Trends**
 
-- **Done** · Totals and people only over time; change against the period before; the earlier
-  period as a dashed line; days that can be pressed.
+- **Done** · Totals and people only over time, by days, weeks (Monday to Sunday) or months; change
+  against the period before; the earlier period as a dashed line, lined up week for week or month
+  for month; bars that can be pressed to look at the days they cover.
+- **Done** · All time: the headline figures and the graph by months or years, back to the first
+  month kept.
 - **Partly done** · The "Who came" chart stacks four tones over time; AI and search are folded
   into Machinery.
 - **Not started** · AI crawlers and search crawlers as lines of their own.
@@ -807,8 +821,9 @@ somebody to embed.
 
 Self-hosting is not a crippled tier. The full detection engine, every screen that shows or
 judges your traffic, and unlimited websites and traffic are free, and every edition keeps twelve
-months of traffic and verdicts. What the hosted service adds today is the running of it: an
-account anyone can create, plans and billing. Analysis that only exists because it sees many
+months of traffic and verdicts, and each website's monthly totals until the website is removed.
+What the hosted service adds today is the running of it: an account anyone can create, plans and
+billing. Analysis that only exists because it sees many
 customers' traffic at once — which no single installation could produce for itself — is planned
 for that edition and not yet built. The commercial part lives in its own repository and is not
 required to build or run anything here.

backend/migrations/clickhouse/0014_months_kept_after_their_reports.sql

@@ -0,0 +1,86 @@
+-- ---------------------------------------------------------------------------
+-- 0014_months_kept_after_their_reports — each website's totals for each
+-- calendar month, kept until the website is removed.
+--
+-- Every report, verdict and name answer is deleted twelve months after it
+-- arrived (0001, 0003, 0010), and nothing here changes that. What outlives
+-- them is one row per website per finished month: the month's totals, counted
+-- by the dashboard's own arithmetic while every report of the month was still
+-- held. This widens what the product keeps — these rows have no TTL and go
+-- only when the website does — and it is the whole of the widening: a month
+-- holds how much was read, by how many daily visitors, on which day, weekday
+-- and hour, how many visits of each kind there were and the pages they read,
+-- and, on the month it falls in, the website's first recorded day. Nothing
+-- about any one visit: no address, no page address and no user agent.
+--
+-- The same figures are kept a second time for the visits judged to be people,
+-- on the same row, so the two can never describe different counts.
+--
+-- A row says how it was counted: in which zone, by which rules, how far
+-- judging had reached and which version of the arithmetic ran. That is what
+-- lets the engine count a month again while its reports remain — after a
+-- change of zone, a newer ruleset or a correction to the arithmetic — and
+-- leave it alone once they are gone.
+--
+-- ReplacingMergeTree on counted_at, the instant the count was taken rather
+-- than the instant it was written, so the newest count wins whichever order
+-- two engines' writes arrive in; readers take the newest themselves rather
+-- than wait for a merge. Partitioned by year, which every row of a website and
+-- month falls inside once, so replacing a row never spans two partitions.
+--
+-- Touches no existing row, and is safe to run again. There is no backfill
+-- here: counting needs each website's zone, which only the control plane
+-- holds, so the engine counts every month still held itself, from its first
+-- run on, the oldest first and a few months per website per run.
+-- ---------------------------------------------------------------------------
+
+CREATE TABLE IF NOT EXISTS consolidated_months
+(
+  site_id                       UUID,
+
+  -- The first of the month on the website's calendar, and the zone that
+  -- calendar was read in when the month was counted.
+  month                         Date,
+  time_zone                     LowCardinality(String),
+
+  -- The stretch counted: from the month's first instant to its last.
+  begins_at                     DateTime64(3, 'UTC'),
+  ends_at                       DateTime64(3, 'UTC'),
+
+  -- The website's first recorded day, on the row of the month it falls in.
+  first_day                     Nullable(Date),
+  days_measured                 UInt8,
+
+  page_views                    UInt64,
+  visitors                      UInt64,
+  best_day                      Nullable(Date),
+  best_day_page_views           UInt64,
+  page_views_by_weekday         Array(UInt64),
+  page_views_by_hour            Array(UInt64),
+
+  people_page_views             UInt64,
+  people_visitors               UInt64,
+  people_best_day               Nullable(Date),
+  people_best_day_page_views    UInt64,
+  people_page_views_by_weekday  Array(UInt64),
+  people_page_views_by_hour     Array(UInt64),
+
+  -- How many visits of each kind began in the month, by the newest verdict
+  -- each had when it was counted, spelled as 0003 spells a category.
+  verdict_categories            Array(LowCardinality(String)),
+  verdict_visits                Array(UInt64),
+  verdict_page_views            Array(UInt64),
+
+  -- How it was counted.
+  ruleset_major                 UInt16,
+  ruleset_minor                 UInt16,
+  ruleset_judged_to             DateTime64(3, 'UTC'),
+  counting_rules                UInt16,
+  counted_to                    DateTime64(3, 'UTC'),
+  judged_to                     DateTime64(3, 'UTC'),
+  counted_at                    DateTime64(3, 'UTC')
+)
+ENGINE = ReplacingMergeTree(counted_at)
+PARTITION BY toYear(month)
+ORDER BY (site_id, month)
+SETTINGS index_granularity = 8192;

backend/src/Dewiride.Analytics.Api/Analytics/QueryVocabulary.cs

@@ -0,0 +1,180 @@
+using System.Collections.Frozen;
+using Dewiride.Analytics.Application.Analytics;
+using Dewiride.Analytics.Application.Sessions;
+using Dewiride.Analytics.Application.Tenancy;
+using Dewiride.Analytics.Classification;
+using Microsoft.AspNetCore.Http.HttpResults;
+
+namespace Dewiride.Analytics.Api.Analytics;
+
+/// <summary>
+/// The words the questions about a website's numbers are asked in, and how a question that cannot
+/// be answered as asked is refused.
+/// </summary>
+/// <remarks>
+/// Shared by every endpoint that answers such a question, so a word means one thing on all of them
+/// and a refusal reads the same wherever it is met. Each table is written out rather than derived
+/// from an enumeration, so renaming a member in C# cannot change an address the dashboard already
+/// asks for, and anything not in a table is refused before the compiler is reached — a second answer
+/// to the question the compiler already settles by taking every identifier from a fixed table of its
+/// own.
+/// </remarks>
+internal static class QueryVocabulary
+{
+    /// <summary>What is said when a question is asked about a population this product cannot separate.</summary>
+    public const string UnknownPopulation = "Ask about everybody, or only about people.";
+
+    /// <summary>What is said when a series is asked for in buckets of a size this product does not cut.</summary>
+    public const string UnknownGranularity = "Ask for buckets of an hour, a day, a week or a month.";
+
+    /// <summary>What is said when a website's whole record is asked for in buckets it is not cut into.</summary>
+    public const string UnknownKeptGranularity = "Ask for buckets of a month or a year.";
+
+    /// <summary>What is said when a series is asked for a measure this product does not count.</summary>
+    public const string UnknownMetric = "Ask for either page views or visitors.";
+
+    /// <summary>Which measure a series counts, by the word used on the wire.</summary>
+    public static readonly FrozenDictionary<string, TimeSeriesMetric> Metrics =
+        new Dictionary<string, TimeSeriesMetric>(StringComparer.OrdinalIgnoreCase)
+        {
+            ["pageviews"] = TimeSeriesMetric.PageViews,
+            ["visitors"] = TimeSeriesMetric.Visitors,
+        }.ToFrozenDictionary(StringComparer.OrdinalIgnoreCase);
+
+    /// <summary>
+    /// The spelling each measure is reported back under.
+    /// </summary>
+    /// <remarks>
+    /// The answer names what it counted rather than repeating the caller's own spelling, so a
+    /// request for <c>PageViews</c> and one for <c>pageviews</c> produce identical documents.
+    /// </remarks>
+    public static readonly FrozenDictionary<TimeSeriesMetric, string> MetricNames =
+        Metrics.ToFrozenDictionary(entry => entry.Value, entry => entry.Key);
+
+    /// <summary>
+    /// Who an answer may be about, by the word used on the wire.
+    /// </summary>
+    /// <remarks>
+    /// One word: everybody is what a question means when nobody is named, and is never spelled.
+    /// </remarks>
+    public static readonly FrozenDictionary<string, Population> Populations =
+        new Dictionary<string, Population>(StringComparer.OrdinalIgnoreCase)
+        {
+            ["people"] = Population.People,
+        }.ToFrozenDictionary(StringComparer.OrdinalIgnoreCase);
+
+    /// <summary>
+    /// How wide each bucket of a series may be, by the word used on the wire.
+    /// </summary>
+    /// <remarks>
+    /// A week runs Monday to Sunday and a month is the calendar month, both where the website is.
+    /// </remarks>
+    public static readonly FrozenDictionary<string, TimeGranularity> Granularities =
+        new Dictionary<string, TimeGranularity>(StringComparer.OrdinalIgnoreCase)
+        {
+            ["hour"] = TimeGranularity.Hour,
+            ["day"] = TimeGranularity.Day,
+            ["week"] = TimeGranularity.Week,
+            ["month"] = TimeGranularity.Month,
+        }.ToFrozenDictionary(StringComparer.OrdinalIgnoreCase);
+
+    /// <summary>The spelling each bucket width is reported back under.</summary>
+    public static readonly FrozenDictionary<TimeGranularity, string> GranularityNames =
+        Granularities.ToFrozenDictionary(entry => entry.Value, entry => entry.Key);
+
+    /// <summary>
+    /// How wide each bucket of a website's whole record may be, by the word used on the wire.
+    /// </summary>
+    /// <remarks>
+    /// A month or a year, each on the website's calendar: the record reaches back beyond the reports a
+    /// finer bucket would be counted from.
+    /// </remarks>
+    public static readonly FrozenDictionary<string, KeptGranularity> KeptGranularities =
+        new Dictionary<string, KeptGranularity>(StringComparer.OrdinalIgnoreCase)
+        {
+            ["month"] = KeptGranularity.Month,
+            ["year"] = KeptGranularity.Year,
+        }.ToFrozenDictionary(StringComparer.OrdinalIgnoreCase);
+
+    /// <summary>The spelling each bucket width of a whole record is reported back under.</summary>
+    public static readonly FrozenDictionary<KeptGranularity, string> KeptGranularityNames =
+        KeptGranularities.ToFrozenDictionary(entry => entry.Value, entry => entry.Key);
+
+    /// <summary>
+    /// Reads who a question is asked about.
+    /// </summary>
+    /// <remarks>
+    /// Nothing named is everybody, which is what every question means until somebody is named.
+    /// Called before the site is resolved on the same terms as every other check, so a word naming
+    /// no population is refused identically whether or not the site exists.
+    /// </remarks>
+    /// <param name="asked">What the caller supplied, or nothing.</param>
+    /// <param name="population">Who it names, where it names anyone this product can separate.</param>
+    /// <returns><see langword="true"/> when the request can be answered as asked.</returns>
+    public static bool TryReadPopulation(string? asked, out Population population)
+    {
+        if (string.IsNullOrEmpty(asked))
+        {
+            population = Population.Everybody;
+
+            return true;
+        }
+
+        return Populations.TryGetValue(asked, out population);
+    }
+
+    /// <summary>
+    /// Refuses a question that cannot be answered as asked.
+    /// </summary>
+    /// <param name="detail">Why, in a sentence the caller can act on.</param>
+    /// <returns>The refusal.</returns>
+    public static ProblemHttpResult Unusable(string? detail) =>
+        TypedResults.Problem(
+            title: "That request could not be answered as asked.",
+            detail: detail,
+            statusCode: StatusCodes.Status400BadRequest);
+
+    /// <summary>
+    /// Where an answer counted from verdicts stops being complete.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// The earlier of two instants, as <see cref="JudgingReach.Settled"/> explains. A site judging
+    /// has not started on is complete nowhere in the window being read, which is said by answering
+    /// with the window's own start.
+    /// </para>
+    /// <para>
+    /// Read before the answer it qualifies, never after. The engine stores a pass's verdicts before
+    /// it moves its bookmark past them, so a bookmark read first can only be behind what an answer
+    /// read afterwards holds — which washes a bucket that had in fact settled — whereas read second
+    /// it can be ahead of it, and call a stretch complete whose verdicts the answer never saw.
+    /// </para>
+    /// </remarks>
+    /// <param name="scope">The site being read.</param>
+    /// <param name="range">The window being read, whose start is the answer for a site judging has not started on.</param>
+    /// <param name="progress">Where judging has reached.</param>
+    /// <param name="settings">The engine's own idle timeout.</param>
+    /// <param name="clock">The present moment.</param>
+    /// <param name="cancellationToken">Cancellation token.</param>
+    /// <returns>The instant.</returns>
+    public static async Task<DateTimeOffset> JudgedUpToAsync(
+        TenantScope scope,
+        TimeRange range,
+        IClassificationProgressStore progress,
+        ClassificationOptions settings,
+        TimeProvider clock,
+        CancellationToken cancellationToken)
+    {
+        ArgumentNullException.ThrowIfNull(scope);
+        ArgumentNullException.ThrowIfNull(progress);
+        ArgumentNullException.ThrowIfNull(settings);
+        ArgumentNullException.ThrowIfNull(clock);
+
+        var caughtUp = await progress
+                .CaughtUpToAsync(scope.SiteId, RulesetVersion.Current, cancellationToken)
+                .ConfigureAwait(false)
+            ?? range.From;
+
+        return JudgingReach.Settled(caughtUp, clock.GetUtcNow(), settings.IdleTimeout);
+    }
+}

backend/src/Dewiride.Analytics.Api/Analytics/RequestedWindow.cs

@@ -37,6 +37,19 @@ internal static class RequestedWindow
     /// </remarks>
     public static readonly TimeSpan LongestByHour = TimeSpan.FromDays(31);
 
+    /// <summary>
+    /// Longest window a series may be cut into buckets of one size over.
+    /// </summary>
+    /// <remarks>
+    /// Only hours are held to a month. A year of hours is thousands of points, while a year of
+    /// days, weeks or months is few enough for a chart to show every one of them, so those may be
+    /// asked for across the longest window any question may cover.
+    /// </remarks>
+    /// <param name="granularity">How wide each bucket is.</param>
+    /// <returns>The longest window.</returns>
+    public static TimeSpan LongestFor(TimeGranularity granularity) =>
+        granularity == TimeGranularity.Hour ? LongestByHour : Longest;
+
     /// <summary>
     /// Resolves the window, filling in whichever end was left out.
     /// </summary>