Repository navigation
fix[next]: compile against content-addressed host views of connectivity tables - #2953
Conversation
…ay library Persistent translation-cache keys fingerprint the connectivity tables in the offset provider. A process-pool compile worker only receives a host (numpy) copy of each table, while serial and thread compilation key the original, so on GPU backends the two keys differed and the cache written in one mode missed in the other. Gather connectivities now get their own deconstructor: domain, codomain, skip value and a content fingerprint of the host table, without the connectivity class or array library. The table fingerprint is memoized per connectivity, so a device table is copied to the host once per process.
…ory order A host copy of a connectivity table can come back in a different memory order than the original (C vs. F), which split the persistent cache key again for F-ordered tables. Translation does not depend on table strides, so the table content is fingerprinted in C order.
Compilation receives a host snapshot of every gather connectivity: a NumPy connectivity over a C-contiguous copy of the table, memoized per source connectivity. The parent process, compile threads and process-pool workers (which rehydrate the shipped table directly from the memory map) then see the same objects, independent of the device and memory layout of the runtime tables, and compile-time domain inference no longer reads device memory. Runtime calls still receive the original offset provider.
Compilation now runs against a CompileTimeConnectivity for every gather connectivity: a read-only, C-contiguous host table memory-mapped from a file in the session cache directory named by the digest of its content. It is fingerprinted by the digest instead of the table and pickled as a reference to its file, so - the table is copied from the device and hashed once per runtime connectivity, not on every fingerprint or task, - the host copy lives in the page cache instead of process memory and is shared with the process-pool workers, which no longer re-hash it, - shipping no longer needs separate file references, and rehydration keeps the memory map instead of copying it into a newly allocated buffer. This supersedes the content fingerprint of runtime connectivities from the previous commits, which is reverted. ADR 0024 gets an amendment.
|
🤖 Written by an agent on behalf of @havogt |
…ce orchestration - make_compilation_task converts the offset provider to its compile-time view, and CompilationTask carries the compilable itself: the with_refs variant is gone, so every runner and every caller compiles the same input. - The table digest is xxh64 over dtype, shape and the C-contiguous buffer, without copies and independent of numpy's pickling internals. - CompileTimeConnectivity derives its digest from the table when it is not given, so copies made through its constructor stay valid, and writes its file on pickling if needed. A failed table write leaves no file behind. - The dace orchestration path compiles against the same view. - Stale comments and the ADR amendment are updated.
- A worker unpickles the compilable inside the task, so an unreadable connectivity table file fails that task instead of breaking the pool. - The digest of a compile-time connectivity is always derived from its table (xxh3_128); only unpickling trusts the content-addressed file name, and checks shape and dtype of the mapped table. The class is private. - A failed rename of a table file no longer leaves the temporary file. - Tests with independent witnesses: identical gtfn translation against the runtime tables, a persistent cache filled in process mode hitting in the other modes, and the table content seen by every runner. - ADR 0024 and ADR 0019 notes updated.
egparedes
left a comment
egparedes
left a comment
There was a problem hiding this comment.
Looks good, just a few minor comments and questions
| ) | ||
| def _table_digest(table: np.ndarray) -> str: | ||
| """Digest of the content of a C-contiguous `table`, hashed in place.""" | ||
| hasher = xxhash.xxh3_128(b"table\0") |
There was a problem hiding this comment.
Question: can you explain why you chose this specific xxh3_128 digest algorithm?
There was a problem hiding this comment.
The digest is the only identity of a table. It names the content-addressed file, it is all of the table that goes into the translation-cache key, and unpickling trusts it. So a collision would silently compile against another table. 128 bits make that negligible even across many tables and long-lived persistent caches. xxh3 is the fastest variant in xxhash, which we already depend on, and it hashes the buffer in place without copies (~20 GB/s here). It doesn't need to be cryptographic, since the inputs aren't adversarial.
🤖 Written by an agent on behalf of @havogt
| _connectivity_from_file, | ||
| ( | ||
| _dump_connectivity(self.connectivity), | ||
| self.connectivity.domain, | ||
| self.connectivity.codomain, | ||
| self.connectivity.skip_value, | ||
| str(path), |
There was a problem hiding this comment.
Just a question: passing _CompileTimeConnectivity.__dict__["_from_file"] as callable and then adding _CompileTimeConnectivity as first argument would be equivalent to the (slightly redundant) _connectivity_from_file function, right?
There was a problem hiding this comment.
Right, the helper was redundant. __dict__["_from_file"] is the raw classmethod object, which is neither callable nor picklable (TypeError: cannot pickle 'classmethod' object). The bound classmethod pickles by reference, though, so __reduce__ now returns type(self)._from_file and the helper is gone.
🤖 Written by an agent on behalf of @havogt
|
|
||
| #: Compile-time views of runtime connectivities, keyed by the `id` of the | ||
| #: runtime connectivity (validated against a weakref, since ids can be reused). | ||
| _compile_time_connectivities: dict[int, tuple[weakref.ref, _CompileTimeConnectivity]] = {} |
There was a problem hiding this comment.
Question: wouldn't it be better to implement this as a WeakValueDictionary?
There was a problem hiding this comment.
That would be weak in the wrong direction. An entry has to live as long as the source connectivity, because its id is the key and can be reused once the source is gone. The view outlives its source, since the frontend's in-memory caches keep compilables alive. With a WeakValueDictionary[id, view], a new connectivity that gets a freed id would be served the old view. What we need is a WeakKeyDictionary keyed by the connectivity, but connectivities aren't hashable (the dataclass hash covers the ndarray). The (weakref(source), view) entry is that WeakKeyDictionary, keyed by identity.
🤖 Written by an agent on behalf of @havogt
| Return the compile-time view of `offset_provider`. | ||
|
|
||
| Every gather connectivity is replaced by a read-only, C-contiguous host | ||
| copy of it, memory-mapped from a file named by the digest of its content, | ||
| so that compilation depends neither on the device nor on the memory layout | ||
| of the tables, and does not read device memory after the conversion. Other | ||
| entries and the order of the entries are kept. |
There was a problem hiding this comment.
This docstring duplicates most of the _CompileTimeConnectivity docstring, I guess one of them should refer to the other to avoid duplication
There was a problem hiding this comment.
Agreed. The function docstring now just refers to _CompileTimeConnectivity.
🤖 Written by an agent on behalf of @havogt
…smethod Drop the redundant module-level helper and the docstring duplication in compile_time_offset_provider.
…ectivity # Conflicts: # src/gt4py/next/program_processors/runners/dace/program.py
Description
Compilation received the runtime offset provider, so it depended on where and how connectivity tables are stored:
GT4PY_BUILD_JOBS_MODE=process, workers get a host copy of each table, while serial and thread compilation use the device original. On GPU backends the compiled input and its persistent translation-cache key therefore differed between modes. In an icon4py warm start, thread mode on a cache filled by process mode re-translated about 90 of 110 programs._max_domain_range_sizes,_unstructured_translate_range_statically) reduced device arrays, which means implicit syncs and cupy 0-d results (see fix[next]: format domain warnings for array scalars #2924).Compilation now runs against a compile-time view of the offset provider.
make_compilation_taskreplaces each gather connectivity, once per runtime connectivity, by a compile-time connectivity: a read-only, C-contiguous host table, memory-mapped from a file in the session cache directory named by a digest of its content (dtype, shape and data, hashed in place). It is fingerprinted by the digest instead of the table and pickled as a reference to its file, which replaces the previous file references and thewith_refsvariant ofCompilationTask. Workers unpickle the compilable inside the task, so an unreadable table file fails only that task instead of breaking the pool. All runners therefore compile the same input with the same key, and the host tables live in the page cache, shared with the workers. Compilation that bypassesmake_compilation_task(e.g. a directBackend.compile) still sees the runtime tables. Execution always receives the runtime offset provider.ADR 0024 gets an amendment and ADR 0019 a note. The amendment also states the new contract: tables must not be modified in place after they have been used for compilation.
Translation-cache keys of programs with connectivities change on all backends, so they are re-translated once.
Supersedes #2951. #2952 is not addressed here.
Requirements