feat: a website's posts on Insights, their views kept a month at a time (ADR 0069)

README.md

@@ -20,8 +20,8 @@ Two rules shape everything here:
   the key that tells one visitor's activity from another's is worked out with a secret that
   changes every day, so once an address is gone there is no key that joins a reader's activity
   from one day to the next — and no screen or figure ever tries to. Every report of a visit is
-  deleted after twelve months; what is kept after that is each month's totals, until the website
-  is removed.
+  deleted after twelve months; what is kept after that is each month's totals, and how many times
+  each of the website's posts was read that month, until the website is removed.
 
 ## Where this is today
 
@@ -67,8 +67,10 @@ and the [master plan](#master-plan) below it says what does not yet.
 - A website's posts, known from its sitemap. Give its settings the sitemap's address and the engine
   reads it, and once each page it lists, for that page's title and publish date, keeping to one
   request a second and to the website's own addresses. Settings says how many posts it found, how
-  many say when they were published, and in plain words anything that got in the way. How it behaves,
-  and how to switch it off, is under [Reading a website's sitemap](#reading-a-websites-sitemap).
+  many say when they were published, and in plain words anything that got in the way. Insights shows
+  them: how many there are, the newest ten with how often each was read, the most read ever, and a
+  calendar of the last twelve months of posting. How it behaves, and how to switch it off, is under
+  [Reading a website's sitemap](#reading-a-websites-sitemap).
 - Where your readers are, by country, by town, and by the network they arrived over, ranked by how
   many people were in each place rather than by how much browsing they did. Towns are named as an
   estimate, because that is what they are; the network view is there because a hundred rented
@@ -169,8 +171,8 @@ up. Each line is marked with one of four states. Last reviewed 2026-11-07.
 
 ### Phase 1 — Public statements
 
-- **Done** · README: every report of a visit is kept for 12 months; each website's monthly totals
-  until the website is removed.
+- **Done** · README: every report of a visit is kept for 12 months; each website's monthly totals,
+  and each post's views month by month, until the website is removed.
 - **Done** · README: a page counts as read only when its arrival reached us.
 - **Done** · README: multi-customer analysis is planned, not present.
 - **Done** · README: the daily key does not recognise a returning reader.
@@ -455,7 +457,8 @@ up. Each line is marked with one of four states. Last reviewed 2026-11-07.
 
 - **Done** · All-time page views and daily visitors, the best day ever, the busiest day of the week
   and hour, every month and year by total views or by average per day, and the same for people only.
-- **Not started** · Posts: how many there are, the latest, the most read, and when they were published.
+- **Done** · Posts from the website's own sitemap: how many, the newest ten with their views, the most
+  read ever, and a calendar of the last 12 months of posting.
 
 ### Phase 6 — AI analytics
 
@@ -822,6 +825,8 @@ fifty thousand posts take most of a day, the newest first. It reads at most 500
 
 What it keeps is each page's address, title and publish date, with when the sitemap says the page
 last changed. Removing the sitemap in Settings deletes all of it, and so does removing the website.
+How many times each post was read is kept month by month with the website's monthly totals, and
+goes only when the website is removed.
 
 **Letting it in, or keeping it out**
 

backend/migrations/clickhouse/0015_posts_read_month_by_month.sql

@@ -0,0 +1,67 @@
+-- ---------------------------------------------------------------------------
+-- 0015_posts_read_month_by_month — how many times each of a website's posts
+-- was read in each calendar month, kept until the website is removed.
+--
+-- Every report is deleted twelve months after it arrived (0001), and nothing
+-- here changes that. What outlives them is one row per website per finished
+-- month: the paths of the website's posts, as its sitemap lists them, each
+-- with the pages delivered there that month, counted by the arithmetic of the
+-- dashboard's list of pages while every report of the month was still held.
+-- This widens what the product keeps — these rows have no TTL and go only
+-- when the website does — and it is the whole of the widening: the month,
+-- the website's own post paths, and two counts beside each, for everybody
+-- and for the visits judged to be people. Nothing about any one visit: no
+-- address, no user agent, no visitor key, and no path but the website's own
+-- posts, and of those only the ones somebody read that month.
+--
+-- A row says how it was counted, as a month's totals say it (0014), and
+-- which list of posts it was counted for, as a fingerprint of their paths.
+-- That is what lets the engine count a month's posts again while its reports
+-- remain — after a change of zone, a newer ruleset, a correction to the
+-- arithmetic or a post found later — and leave it alone once they are gone.
+--
+-- ReplacingMergeTree on counted_at, partitioned by year and sorted by website
+-- and month, for the reasons 0014 gives; readers take the newest row of each
+-- month themselves rather than wait for a merge.
+--
+-- Touches no existing row, and is safe to run again. There is no backfill
+-- here: the engine counts the posts of every month still held itself, from
+-- its first run on, the oldest first and a few months per website per run.
+-- ---------------------------------------------------------------------------
+
+CREATE TABLE IF NOT EXISTS consolidated_post_months
+(
+  site_id            UUID,
+
+  -- The first of the month on the website's calendar, and the zone that
+  -- calendar was read in when the month was counted.
+  month              Date,
+  time_zone          LowCardinality(String),
+
+  -- The stretch counted: from the month's first instant to its last.
+  begins_at          DateTime64(3, 'UTC'),
+  ends_at            DateTime64(3, 'UTC'),
+
+  -- The paths a page was delivered at, in ordinal order, each with the
+  -- pages delivered there to everybody and to the visits judged to be
+  -- people, nought where none went to a person.
+  paths              Array(String),
+  page_views         Array(UInt64),
+  people_page_views  Array(UInt64),
+
+  -- Which list of posts was counted.
+  posts_fingerprint  String,
+
+  -- How it was counted.
+  ruleset_major      UInt16,
+  ruleset_minor      UInt16,
+  ruleset_judged_to  DateTime64(3, 'UTC'),
+  counting_rules     UInt16,
+  counted_to         DateTime64(3, 'UTC'),
+  judged_to          DateTime64(3, 'UTC'),
+  counted_at         DateTime64(3, 'UTC')
+)
+ENGINE = ReplacingMergeTree(counted_at)
+PARTITION BY toYear(month)
+ORDER BY (site_id, month)
+SETTINGS index_granularity = 8192;

backend/src/Dewiride.Analytics.Api/Contracts/AnalyticsContracts.cs

@@ -1039,6 +1039,61 @@ public sealed record InsightsBestDay(string Day, long PageViews);
 /// </param>
 public readonly record struct InsightsMonth(string Month, long? PageViews, int DaysMeasured);
 
+/// <summary>
+/// What a website's posts have done, all from one reading.
+/// </summary>
+/// <param name="Posts">Posts its sitemap lists now: the figure Settings gives.</param>
+/// <param name="Dated">Those of them whose page said when it was published, as Settings counts them.</param>
+/// <param name="ViewsFrom">
+/// The month, as <c>yyyy-MM</c> on the website's calendar, the posts' views are counted from: the first
+/// month whose posts were kept, or the first answered from its reports where that is earlier. Nothing
+/// where nothing of the website is held, when no post has a view.
+/// </param>
+/// <param name="Latest">Up to ten of the posts with a publishing date, the newest first.</param>
+/// <param name="MostRead">The post read the most, or nothing where none of them has been read.</param>
+/// <param name="CalendarFrom">
+/// The calendar's first day, as <c>yyyy-MM-dd</c>: the first of the month eleven months before this one,
+/// on the website's calendar.
+/// </param>
+/// <param name="CalendarTo">The calendar's last day, as <c>yyyy-MM-dd</c>: today, on the website's calendar.</param>
+/// <param name="Calendar">Each day of the calendar a post was published on, the oldest first.</param>
+public sealed record InsightsPostsResponse(
+    int Posts,
+    int Dated,
+    string? ViewsFrom,
+    IReadOnlyList<InsightsPost> Latest,
+    InsightsPost? MostRead,
+    string CalendarFrom,
+    string CalendarTo,
+    IReadOnlyList<PostingDayResponse> Calendar);
+
+/// <summary>
+/// One of a website's posts, with how much it was read.
+/// </summary>
+/// <remarks>
+/// The address and the title were written by the website, through its sitemap and its page. They are
+/// text to show, and an address is to be linked to only where it is a web address.
+/// </remarks>
+/// <param name="Path">The path the post is read at.</param>
+/// <param name="Address">Its address, as the website's sitemap gave it.</param>
+/// <param name="Title">What the post says it is called, or nothing where it said nothing.</param>
+/// <param name="Published">
+/// The day it was published, as <c>yyyy-MM-dd</c> on the website's calendar, or nothing where it did not
+/// say.
+/// </param>
+/// <param name="PageViews">
+/// Pages delivered at it since <see cref="InsightsPostsResponse.ViewsFrom"/>, at its own path and where it
+/// landed, to whoever the reading is about.
+/// </param>
+public sealed record InsightsPost(string Path, string Address, string? Title, string? Published, long PageViews);
+
+/// <summary>
+/// One day posts were published on.
+/// </summary>
+/// <param name="Day">The day, as <c>yyyy-MM-dd</c> on the website's calendar.</param>
+/// <param name="Posts">How many posts were published on it.</param>
+public readonly record struct PostingDayResponse(string Day, int Posts);
+
 /// <summary>
 /// One measure across a website's whole record, month by month or year by year.
 /// </summary>

backend/src/Dewiride.Analytics.Api/Contracts/ApiJsonContext.cs

@@ -48,6 +48,7 @@ namespace Dewiride.Analytics.Api.Contracts;
 [JsonSerializable(typeof(AllTimeSeriesResponse))]
 [JsonSerializable(typeof(AllTimeTrafficSeriesResponse))]
 [JsonSerializable(typeof(InsightsResponse))]
+[JsonSerializable(typeof(InsightsPostsResponse))]
 [JsonSerializable(typeof(AddSiteRequest))]
 [JsonSerializable(typeof(SiteSummary))]
 [JsonSerializable(typeof(ActionsResponse))]

backend/src/Dewiride.Analytics.Api/Endpoints/InsightsPostsEndpoints.cs

@@ -0,0 +1,205 @@
+using System.Collections.Immutable;
+using System.Globalization;
+using Dewiride.Analytics.Api.Contracts;
+using Dewiride.Analytics.Application.Analytics;
+using Dewiride.Analytics.Application.Posts;
+using Dewiride.Analytics.Application.Tenancy;
+using Microsoft.AspNetCore.Http.HttpResults;
+using static Dewiride.Analytics.Api.Analytics.QueryVocabulary;
+
+namespace Dewiride.Analytics.Api.Endpoints;
+
+/// <summary>
+/// Reading what a website's posts have done: how many its sitemap lists, the newest and the most read
+/// with their views across its whole record, and when the last twelve months' were published.
+/// </summary>
+/// <remarks>
+/// <para>
+/// Beside the readings of the whole record, and on its terms: a post's views are read from what was kept
+/// of each month and, from the first month nothing was kept of the posts of, from the reports, by the
+/// arithmetic of the dashboard's list of pages (<see cref="PostsInsights"/>). So a post's views are never
+/// behind the record's own figures.
+/// </para>
+/// <para>
+/// On the same terms as every other read of a website's numbers: everything the caller supplied is
+/// checked before the website is resolved, and a website that does not exist and one the caller has
+/// no role on are answered identically. A title and an address are the website's own text: they travel
+/// in the answer as values and are never logged.
+/// </para>
+/// </remarks>
+internal static class InsightsPostsEndpoints
+{
+    /// <summary>How a day is written in an answer.</summary>
+    private const string DayFormat = "yyyy-MM-dd";
+
+    /// <summary>How a month is written in an answer.</summary>
+    private const string MonthFormat = "yyyy-MM";
+
+    /// <summary>
+    /// Maps the question about a website's posts.
+    /// </summary>
+    /// <param name="routes">The route builder.</param>
+    public static void MapInsightsPosts(this IEndpointRouteBuilder routes)
+    {
+        ArgumentNullException.ThrowIfNull(routes);
+
+        routes.MapGet("/api/sites/{siteId:guid}/insights/posts", PostsAsync)
+            .WithName("SiteInsightsPosts")
+            .WithSummary("Returns what a website's posts have done: how many its sitemap lists, the newest ten and the most read with their views across its whole record, and the days of the last twelve months a post was published on.");
+    }
+
+    /// <summary>
+    /// Answers what a website's posts have done.
+    /// </summary>
+    /// <remarks>
+    /// A website that lists no posts is answered at once, from the control plane alone. Otherwise only the
+    /// posts shown are named.
+    /// </remarks>
+    private static async Task<Results<Ok<InsightsPostsResponse>, NotFound, ProblemHttpResult>> PostsAsync(
+        [AsParameters] WholeRecordParameters parameters,
+        ITenantScopeProvider scopes,
+        ISitePosts posts,
+        ITelemetryQueries telemetry,
+        TimeProvider clock,
+        CancellationToken cancellationToken)
+    {
+        if (!TryReadPopulation(parameters.Only, out var population))
+        {
+            return Unusable(UnknownPopulation);
+        }
+
+        var scope = await scopes.ResolveAsync(parameters.SiteId, cancellationToken).ConfigureAwait(false);
+
+        if (scope is null)
+        {
+            return TypedResults.NotFound();
+        }
+
+        var asOf = clock.GetUtcNow();
+        var listed = await posts.ListAsync(scope, cancellationToken).ConfigureAwait(false);
+        var viewing = listed.IsEmpty
+            ? PostViewing.Nothing
+            : await PostViewsAsync(scope, listed, population, telemetry, asOf, cancellationToken).ConfigureAwait(false);
+        var reading = PostsInsights.Read(
+            listed,
+            viewing.Views,
+            TimeZoneInfo.FindSystemTimeZoneById(scope.TimeZoneId),
+            asOf);
+        var named = await NameAsync(scope, reading, posts, cancellationToken).ConfigureAwait(false);
+
+        return TypedResults.Ok(
+            new InsightsPostsResponse(
+                reading.Posts,
+                reading.Dated,
+                viewing.From?.ToString(MonthFormat, CultureInfo.InvariantCulture),
+                [.. reading.Latest.Select(post => Described(post, named)).OfType<InsightsPost>()],
+                reading.MostRead is { } most ? Described(most, named) : null,
+                reading.CalendarFrom.ToString(DayFormat, CultureInfo.InvariantCulture),
+                reading.CalendarTo.ToString(DayFormat, CultureInfo.InvariantCulture),
+                [
+                    .. reading.Calendar.Select(day => new PostingDayResponse(
+                        day.Day.ToString(DayFormat, CultureInfo.InvariantCulture),
+                        day.Posts)),
+                ]));
+    }
+
+    /// <summary>
+    /// Counts the pages delivered at a website's posts across its whole record: what was kept of each
+    /// month before the first answered from its reports, and the reports from there to the present.
+    /// </summary>
+    /// <remarks>
+    /// Every month instant comes from the store, through the record the other readings of it read, so the
+    /// stretch counted from the reports begins where a month of the record begins.
+    /// </remarks>
+    /// <returns>The views, and the month they are counted from; none where nothing of the website is held.</returns>
+    private static async Task<PostViewing> PostViewsAsync(
+        TenantScope scope,
+        ImmutableArray<ListedPost> listed,
+        Population population,
+        ITelemetryQueries telemetry,
+        DateTimeOffset asOf,
+        CancellationToken cancellationToken)
+    {
+        var record = await telemetry
+            .GetKeptMonthsAsync(scope, new KeptMonthsQuery(asOf), cancellationToken)
+            .ConfigureAwait(false);
+        var postMonths = await telemetry
+            .GetKeptPostMonthsAsync(scope, new KeptPostMonthsQuery(asOf), cancellationToken)
+            .ConfigureAwait(false);
+
+        if (PostsInsights.LiveFrom(record.Months, postMonths, asOf) is not { } liveFrom)
+        {
+            return PostViewing.Nothing;
+        }
+
+        var kept = postMonths.Any(month => month < liveFrom.Month)
+            ? await telemetry
+                .GetKeptPostViewsAsync(
+                    scope,
+                    new KeptPostViewsQuery(asOf, liveFrom.Month) { Population = population },
+                    cancellationToken)
+                .ConfigureAwait(false)
+            : [];
+        var live = liveFrom.Begins < asOf
+            ? await telemetry
+                .GetPostViewsAsync(
+                    scope,
+                    new PostViewsQuery(new TimeRange(liveFrom.Begins, asOf), PostsInsights.CountedPaths(listed))
+                    {
+                        Population = population,
+                    },
+                    cancellationToken)
+                .ConfigureAwait(false)
+            : [];
+
+        return new PostViewing([.. kept, .. live], PostsInsights.ViewsFrom(postMonths, liveFrom));
+    }
+
+    /// <summary>
+    /// Names the posts an answer shows — the newest and the most read — and no others.
+    /// </summary>
+    /// <returns>Each of them still listed, by its path.</returns>
+    private static async Task<Dictionary<string, NamedPost>> NameAsync(
+        TenantScope scope,
+        PostsReading reading,
+        ISitePosts posts,
+        CancellationToken cancellationToken)
+    {
+        string[] shown =
+        [
+            .. reading.Latest
+                .Select(post => post.Path)
+                .Append(reading.MostRead?.Path)
+                .OfType<string>()
+                .Distinct(StringComparer.Ordinal),
+        ];
+
+        var named = shown.Length == 0
+            ? []
+            : await posts.NamedAsync(scope, shown, cancellationToken).ConfigureAwait(false);
+
+        return named.ToDictionary(post => post.Path, StringComparer.Ordinal);
+    }
+
+    /// <summary>
+    /// A post as the answer carries it, or nothing where it stopped being listed while it was read.
+    /// </summary>
+    private static InsightsPost? Described(ShownPost post, Dictionary<string, NamedPost> named) =>
+        named.TryGetValue(post.Path, out var name)
+            ? new InsightsPost(
+                post.Path,
+                name.Address,
+                name.Title,
+                post.Published?.ToString(DayFormat, CultureInfo.InvariantCulture),
+                post.PageViews)
+            : null;
+
+    /// <summary>The pages delivered at a website's posts, and the month they are counted from.</summary>
+    /// <param name="Views">Each path counted, from what was kept and from the reports.</param>
+    /// <param name="From">The month the views are counted from, or nothing where nothing of the website is held.</param>
+    private sealed record PostViewing(ImmutableArray<PathViews> Views, DateOnly? From)
+    {
+        /// <summary>No views, counted from nowhere.</summary>
+        public static PostViewing Nothing { get; } = new([], null);
+    }
+}