Add structurally anonymous data collection for privacy-preserving demographic surveys

[mjumbewu]

Problem

Shareabouts currently supports two tiers of data on places and submissions:

1. Public data — returned by default in API responses
2. Private data — stored on the place/submission, hidden unless the requester has elevated access (include_private)

There is no mechanism for collecting data that is structurally decoupled from the place or submission that prompted it. This matters for demographic surveys or any other case where sensitive data is involved. For example, if a city government collects an attribute like is_citizen on a place submission, that data is stored directly in the place's data blob and is trivially joinable to the submitter's identity. Even the private_ prefix only provides access-control separation, not structural separation.

We need a third tier:

3. Anonymous data — values that are detached from the originating place or submission at the storage layer, associated only with the dataset and logical submission set, and accessible only in aggregate.

Proposed Design

Data Model

Add a new AnonymousValues model:

AnonymousValues
┌────────────────────────────────────────┐
│ id          UUID (PK, auto-generated)  │
│ dataset     FK → DataSet (CASCADE)     │
│ set_name    TEXT (indexed)             │
│ data        JSONB                      │
└────────────────────────────────────────┘

Key properties:

• No foreign key to Place or Submission. This is the core privacy guarantee. There is no join path from an anonymous values row back to the individual who submitted it.
• No timestamp. Including created_datetime would allow correlation with submission ordering, undermining anonymity.
• UUID primary key. Avoids sequential ID correlation.
• One row per submission event that includes anonymous attributes. All anonymous attributes from a single submission are stored together in one data JSONB object, preserving the ability to cross-tabulate (e.g., "age distribution among respondents who identified as Asian").
• set_name identifies the logical submission set the data is associated with. Uses the same values as Submission.set_name (e.g., "comments", "support"). For anonymous data originating from place creation, use "places".
• Cascade delete on DataSet. When a dataset is deleted, all associated AnonymousValues rows are deleted.

API: Writing Anonymous Data

Anonymous data is submitted through existing place and submission creation endpoints using an anonymous_ prefix, analogous to how private_ works today.

Example request:

POST /api/v2/owner/datasets/my-ds/places/
Content-Type: application/json

{
  "type": "Feature",
  "geometry": { "type": "Point", "coordinates": [-75.16, 39.95] },
  "properties": {
    "location_type": "suggestion",
    "description": "Add a bike lane here",
    "private_email": "jane@example.com",
    "anonymous_age": "25-34",
    "anonymous_race": "Asian"
  }
}

What the server does:

1. Strips anonymous_-prefixed attributes from the submission data.
2. Creates the Place with only public and private data in its data blob:

   { "location_type": "suggestion", "description": "Add a bike lane here", "private_email": "jane@example.com" }

3. Creates a new AnonymousValues record (with no link to the place):

   { "id": "<uuid>", "dataset_id": 1, "set_name": "places", "data": {"age": "25-34", "race": "Asian"} }

The same mechanism applies to submission creation (e.g., POST .../places/42/comments), using the submission's set_name as the AnonymousValues.set_name.

API: Reading Anonymous Data

Anonymous data is never included in individual place or submission representations. Places and submissions have no knowledge of anonymous data at all — no attribute hints, no flags, nothing.

Anonymous data is accessible only at the dataset or submission set level via the include_anonymous query parameter:

Dataset-level request:

GET /api/v2/owner/datasets/my-ds/?include_anonymous

Response (dataset representation with anonymous data):

{
  "url": "...",
  "owner": { "username": "owner" },
  "slug": "my-ds",
  "display_name": "My Dataset",
  "anonymous_data": {
    "places": [
      { "age": "25-34", "race": "Asian" },
      { "age": "35-44", "race": "Black" }
    ],
    "comments": [
      { "age": "18-24" }
    ]
  }
}

Submission set-level request:

GET /api/v2/owner/datasets/my-ds/comments?include_anonymous

Response (submission set representation with anonymous data):

{
  "metadata": { "...": "..." },
  "results": [ "..." ],
  "anonymous_data": [
    { "age": "18-24" },
    { "age": "25-34" }
  ]
}

Place list-level request:

GET /api/v2/owner/datasets/my-ds/places?include_anonymous

Response (place list representation with anonymous data):

{
  "metadata": { "...": "..." },
  "features": [ "..." ],
  "anonymous_data": [
    { "age": "25-34", "race": "Asian" },
    { "age": "35-44", "race": "Black" }
  ]
}

Permissions

Access to anonymous data is controlled via DataPermission, not hardcoded to the dataset owner. This allows fine-grained configuration of who can retrieve aggregate demographic data.

When a request includes include_anonymous but the requester does not have the appropriate DataPermission, the parameter is silently ignored (consistent with how include_private behaves).

Deduplication

Deduplication of anonymous data from repeat submitters is client-managed. The API does not attempt to deduplicate.

A recommended client-side pattern: generate a stable per-user identifier and submit it as an anonymous attribute (e.g., anonymous_dedup_uuid). This UUID is stored in the AnonymousValues.data blob alongside other demographic attributes, allowing deduplication during analysis without compromising anonymity — the dedup UUID cannot be joined back to any place or submission.

Privacy Considerations

• Structural guarantee: There is no foreign key, timestamp, or sequential ID that links an AnonymousValues row to a specific place, submission, or user. This is by design and is the primary defense against compelled disclosure.
• Cross-tabulation tradeoff: Storing multiple anonymous attributes together in one row enables demographic cross-tabulation but increases re-identification risk in small datasets (e.g., if only one respondent is age 65+ and Asian). This is an accepted tradeoff — the alternative (one row per attribute) would make demographic analysis significantly less useful.
• Immutability in practice: Because there is no link from a place/submission to its anonymous values, updating or deleting a specific person's anonymous data after the fact is not possible. Deleting a place does not delete the associated anonymous values. This is an accepted tradeoff for structural separation.

Alternatives Considered

Keep anonymous data in the place/submission data blob

Filter anonymous_-prefixed attributes at the serializer level, similar to how private_ works. Zero new models required, but anonymous values remain in the data blob and are joinable to the submitter via direct database access. Rejected because the privacy guarantee must be structural, not just access-control.

AnonymousValue model (one row per attribute)

A model with name, id, and value fields — one row per anonymous attribute per submission. Avoids cross-tabulation risk but makes demographic analysis (e.g., "age distribution among Asian respondents") impossible. Rejected in favor of cross-tabulation support.

JSONB field on DataSet

An anonymous_data JSONB field on DataSet that accumulates all anonymous values. Simple but creates a write bottleneck: concurrent submissions would need to read-modify-write the same JSONB field, leading to race conditions. Rejected for concurrency concerns.

Separate submission set

Use a dedicated set_name (e.g., "demographics") and store anonymous data as regular submissions. Zero new models, but submissions have sequential IDs and timestamps that could be correlated with place creation order, undermining anonymity from dataset owners. Rejected because anonymity must extend to dataset administrators.