feat(data): serve contiguous data by content digest

docs/glossary.md

@@ -223,6 +223,18 @@ implementations including filesystem and S3.
 **Contiguous Data Store** - Storage backend for complete transaction data.
 Manages both data files and verification metadata.
 
+<a id="content-digest"></a> **Content Digest (ar-io-digest)** - The SHA-256
+hash of a piece of contiguous data, base64url-encoded. It is emitted on data
+responses as the `X-AR-IO-Digest` header and is the key under which the
+[contiguous data store](#contiguous-data-store) addresses bytes on disk
+(`data/<h0:2>/<h2:4>/<hash>`). Because the same value identifies content
+across the cache, the index, and the response header, it doubles as a stable
+content address. The `GET /ar-io/digest/{digest}` endpoint serves bytes
+directly by this value; such responses are inherently self-verifying (the
+bytes provably hash to the requested digest) and immutable, but local-cache
+only — there is no on-demand fetch by content hash, since Arweave addresses
+data by [item ID](#item-id), not by content hash.
+
 ## Data Verification
 
 **Data Verification** - The process of cryptographically verifying data

docs/openapi.yaml

@@ -1745,6 +1745,115 @@ paths:
         '416':
           description: Range not satisfiable for HEAD request
 
+  '/ar-io/digest/{digest}':
+    get:
+      tags: [Data]
+      summary: Get data by its content digest (SHA-256)
+      description: |
+        Retrieve contiguous data addressed by its SHA-256 content digest — the
+        same base64url value emitted in the `X-AR-IO-Digest` response header on
+        every data response (and used as the gateway's on-disk cache key).
+
+        This is a content-addressed endpoint: the bytes returned provably hash
+        to the requested digest, so the response is always self-verifying
+        (`X-AR-IO-Verified: true`) and immutable (`Cache-Control: …, immutable`).
+
+        Local-cache only: the gateway can only serve a digest it has already
+        materialized (via prior retrieval or bundle unbundling). There is no
+        on-demand fetch by content hash, because Arweave and peers address data
+        by transaction/data-item id, not by content hash — so a digest the node
+        has never stored returns 404.
+
+        For header parity with `/raw/{txId}`, a representative id that resolves
+        to this digest is used to populate the full id-scoped header set
+        (`X-AR-IO-Data-Id`, tags, owner, signature, root offsets), which are
+        then covered by the HTTPSIG signature when signing is enabled.
+      parameters:
+        - name: digest
+          in: path
+          required: true
+          schema:
+            $ref: '#/components/schemas/Base64Url43'
+          description: base64url-encoded SHA-256 content digest (X-AR-IO-Digest)
+        - name: Range
+          in: header
+          required: false
+          schema:
+            $ref: '#/components/schemas/ByteRange'
+          description: Byte range(s) to retrieve
+      responses:
+        '200':
+          description: |
+            Successful response. Emits the same header set as `/raw/{txId}`
+            (see that endpoint for the full list), with `X-AR-IO-Verified`
+            always `true` and an immutable `Cache-Control`.
+          headers:
+            Content-Type:
+              schema:
+                type: string
+                example: application/octet-stream
+            Content-Length:
+              schema:
+                type: string
+                example: 1024
+            Cache-Control:
+              schema:
+                type: string
+                example: public, max-age=2592000, immutable
+            X-AR-IO-Digest:
+              schema:
+                type: string
+                example: '4ROTs2lTPAKbr8Y41WrjHu-2q-7S-m-yTuO7fAUzZI4'
+            ETag:
+              schema:
+                type: string
+                example: '4ROTs2lTPAKbr8Y41WrjHu-2q-7S-m-yTuO7fAUzZI4'
+            Content-Digest:
+              schema:
+                type: string
+                description: RFC 9530 compliant digest header with SHA-256
+                example: 'sha-256=:4ROTs2lTPAKbr8Y41WrjHu+2q+7S+m+yTuO7fAUzZI4=:'
+            X-AR-IO-Verified:
+              schema:
+                $ref: '#/components/schemas/VerificationStatus'
+                example: true
+            X-AR-IO-Data-Id:
+              schema:
+                type: string
+              description: A representative id that resolves to this digest
+        '206':
+          description: Partial content for range requests
+        '400':
+          description: Malformed digest (not a canonical 43-char base64url SHA-256)
+        '404':
+          description: No content for this digest in the local cache
+        '416':
+          description: Range not satisfiable
+        '451':
+          description: Content blocked by this node's content policy
+    head:
+      tags: [Data]
+      summary: Get headers for data by its content digest
+      description: |
+        Existence check / header retrieval for content addressed by its SHA-256
+        digest. Returns the same headers as the GET response with no body.
+      parameters:
+        - name: digest
+          in: path
+          required: true
+          schema:
+            $ref: '#/components/schemas/Base64Url43'
+          description: base64url-encoded SHA-256 content digest (X-AR-IO-Digest)
+      responses:
+        '200':
+          description: Successful response (headers only)
+        '400':
+          description: Malformed digest
+        '404':
+          description: No content for this digest in the local cache
+        '451':
+          description: Content blocked by this node's content policy
+
   # Network and Node Status
   '/info':
     get:

src/constants.ts

@@ -88,6 +88,11 @@ export const verificationPriorities = {
 export const DATA_PATH_REGEX =
   /^\/?([a-zA-Z0-9-_]{43})\/?$|^\/?([a-zA-Z0-9-_]{43})\/(.*)$/i;
 export const RAW_DATA_PATH_REGEX = /^\/raw\/([a-zA-Z0-9-_]{43})\/?$/i;
+// Content-addressed data: base64url SHA-256 digest (43 chars), the value
+// emitted as X-AR-IO-Digest. Distinct prefix from /raw/:txid because a
+// digest is indistinguishable from a 43-char txid by shape alone.
+export const DIGEST_DATA_PATH_REGEX =
+  /^\/ar-io\/digest\/([a-zA-Z0-9-_]{43})\/?$/i;
 export const FARCASTER_FRAME_DATA_PATH_REGEX =
   /^\/local\/farcaster\/frame\/([a-zA-Z0-9-_]{43})\/?$/i;
 

src/data/read-through-data-cache.test.ts

@@ -110,6 +110,22 @@ describe('ReadThroughDataCache', function () {
         return undefined;
       },
 
+      getDataAttributesByHash: async (hash: string) => {
+        if (hash === 'knownHash') {
+          return {
+            hash: 'knownHash',
+            size: 100,
+            contentType: 'text/plain',
+            id: 'knownId',
+          };
+        }
+        // Indexed in contiguous_data but the blob is missing from the store.
+        if (hash === 'indexedButNoBlob') {
+          return { hash: 'indexedButNoBlob', size: 50 };
+        }
+        return undefined;
+      },
+
       // eslint-disable-next-line no-empty-pattern
       saveDataContentAttributes: async ({}: {
         id: string;
@@ -169,6 +185,52 @@ describe('ReadThroughDataCache', function () {
     mock.restoreAll();
   });
 
+  describe('getDataByHash', () => {
+    it('streams indexed content addressed by hash, marked self-verifying', async () => {
+      const result = await readThroughDataCache.getDataByHash('knownHash');
+
+      assert.equal(result.hash, 'knownHash');
+      assert.equal(result.size, 100);
+      assert.equal(result.totalSize, 100);
+      assert.equal(result.sourceContentType, 'text/plain');
+      // Content-addressed reads are self-verifying and always local-cache.
+      assert.equal(result.verified, true);
+      assert.equal(result.trusted, true);
+      assert.equal(result.cached, true);
+      // The single internal lookup also surfaces the representative id.
+      assert.equal(result.representativeId, 'knownId');
+
+      const chunks: Buffer[] = [];
+      for await (const chunk of result.stream) {
+        chunks.push(Buffer.from(chunk));
+      }
+      assert.equal(Buffer.concat(chunks).toString(), 'simulated data');
+    });
+
+    it('honors a byte region', async () => {
+      const result = await readThroughDataCache.getDataByHash('knownHash', {
+        offset: 0,
+        size: 4,
+      });
+      assert.equal(result.size, 4);
+      assert.equal(result.totalSize, 100);
+    });
+
+    it('rejects when the hash is not indexed', async () => {
+      await assert.rejects(
+        readThroughDataCache.getDataByHash('unknownHash'),
+        /No content indexed/,
+      );
+    });
+
+    it('rejects when indexed but the blob is missing from the store', async () => {
+      await assert.rejects(
+        readThroughDataCache.getDataByHash('indexedButNoBlob'),
+        /No cached data/,
+      );
+    });
+  });
+
   describe('getCachedData', () => {
     it('should return data from cache when available', async () => {
       let calledWithArgument: string;

src/data/read-through-data-cache.ts

@@ -20,6 +20,7 @@ import * as metrics from '../metrics.js';
 import { KvJsonStore } from '../store/kv-attributes-store.js';
 import { startChildSpan } from '../tracing.js';
 import {
+  ByHashData,
   ContiguousData,
   ContiguousDataAttributesStore,
   ContiguousDataIndex,
@@ -435,6 +436,64 @@ export class ReadThroughDataCache implements ContiguousDataSource {
     return undefined;
   }
 
+  /**
+   * Serve contiguous data addressed directly by its content hash (the
+   * value emitted as X-AR-IO-Digest and used as the on-disk cache key).
+   *
+   * Unlike {@link getData}, there is no id, no manifest/ArNS resolution, and
+   * no upstream fall-through: Arweave and peers address by transaction id,
+   * not by content hash, so a hash we have never materialized cannot be
+   * fetched on demand. The endpoint therefore serves only content already
+   * present in the local content store. Because the store is keyed by the
+   * SHA-256 of the bytes, a successful read is self-verifying — the bytes
+   * provably hash to the requested digest — so the result is reported as
+   * verified, trusted, and cached.
+   *
+   * @throws if no content is indexed for the hash, or the indexed blob is
+   *   missing from the store (evicted/pruned between index and read).
+   */
+  async getDataByHash(
+    hash: string,
+    region?: {
+      offset: number;
+      size: number;
+    },
+  ): Promise<ByHashData> {
+    const attributes =
+      await this.contiguousDataIndex.getDataAttributesByHash(hash);
+    if (attributes === undefined) {
+      throw new Error(`No content indexed for hash: ${hash}`);
+    }
+
+    const cacheStream = await this.dataStore.get(hash, region);
+    if (cacheStream === undefined) {
+      throw new Error(`No cached data found for hash: ${hash}`);
+    }
+
+    const requestType = region !== undefined ? 'range' : 'full';
+    metrics.getDataStreamSuccessesTotal.inc({
+      class: this.constructor.name,
+      source: 'cache',
+      request_type: requestType,
+    });
+
+    const totalSize = attributes.size;
+    return {
+      hash,
+      stream: cacheStream,
+      size: region?.size ?? totalSize,
+      totalSize,
+      sourceContentType: attributes.contentType,
+      // Content-addressed: the bytes provably hash to the requested digest.
+      verified: true,
+      trusted: true,
+      cached: true,
+      // A representative id resolving to this hash (the same single lookup
+      // above), so callers need not re-query to emit id-scoped headers.
+      representativeId: attributes.id,
+    };
+  }
+
   async getData({
     id,
     requestAttributes,

src/database/sql/data/content-attributes.sql

@@ -155,6 +155,30 @@ FROM (
 )
 LIMIT 1
 
+-- selectDataAttributesByHash
+-- Reverse lookup: resolve content metadata directly from the content hash
+-- (the value emitted as X-AR-IO-Digest and used as the on-disk cache key).
+-- contiguous_data is keyed by hash (primary-key point lookup); the LEFT JOIN
+-- additionally surfaces one representative id that resolves to this hash
+-- (via the contiguous_data_hash index) so the content-addressed endpoint can
+-- emit id-scoped response headers. Many ids may share a hash (byte-identical
+-- content under different signed envelopes); the ORDER BY makes the choice
+-- deterministic and prefers the strongest provenance — a verified id over a
+-- trusted one over an arbitrary one — so the representative is stable across
+-- requests rather than dependent on index iteration order.
+SELECT
+  cd.hash,
+  cd.data_size,
+  cd.original_source_content_type,
+  cdi.id AS id
+FROM contiguous_data cd
+LEFT JOIN contiguous_data_ids cdi ON cdi.contiguous_data_hash = cd.hash
+WHERE cd.hash = :hash
+ORDER BY cdi.verified DESC NULLS LAST,
+  cdi.trusted DESC NULLS LAST,
+  cdi.id ASC
+LIMIT 1
+
 -- selectDataParent
 SELECT
   cdip.parent_id,