Repository navigation
Conversation
Select NumPy 2 on Python 3.13, pin the docdata wheel that the index intermittently fails to serve, check installed requirements, run mypy for each matrix version, and let the matrix finish after a failure.
Answer path, intersection, union and negated queries over KGEs and the ULTRA, TRIX and Flock foundation models with CQD beam search, optionally calibrated by score adapters fitted on source graphs with frozen backbones. Native UltraQuery, GNN-QE, QTO, CQD, CQD-Hybrid, ConE and CLMPT load the released checkpoints and are tested against fixtures exported from the pinned upstream code. Evaluation reads the released UltraQuery and +H files, filters easy and hard answers, scores ties by sort order and by their exact expectation, and resumes from verified checkpoints. Deterministic message passing splits hub nodes into order-preserving partial sums, ordered reductions prefetch edges, and device-side checks replace per-query host synchronization; predictions are bitwise unchanged.
`python -m benchmarks.cqa {plus_h,ultraquery} COMMAND` selects any methods,
datasets and query types from compact public recipes, freezes inputs, query
batches and source in a study, verifies validation predictions against oracles
exported from the pinned upstream code, and runs and reports the test split.
With --image, commands run in a pinned, network-free Docker runtime; the host
needs only Python and Docker. Paired controls measure learned adapters, the
corrected +H answer filters and inference graphs.
The evaluation protocol and every known deviation from the publications are
documented in benchmarks/cqa/PROTOCOL.md.
…ministic backbones
prepare_adapter_data(hardness='balanced') gives each source query a missing-link level in rotation and trains it on that level's hard answers; queries can carry explicit positives. The default path is unchanged.
KGFM recipes accept operators such as 'product:chains', so one recipe can score some query types with a second adapter of the same t-norm.
Bit-exact comparison with fixture scores recorded on another CPU failed on CI runners. The joint DNF projection batch is now rebuilt on the test machine; the fixture comparison keeps its 3e-5 tolerance.
… the state token once
AtomicBatchCache gains a row granularity: rows are cached and evicted one by
one, so a row computed in one atomic batch is reused by any later batch
instead of only by an identical batch. KGFM evaluation enables it with the
raw_cache_granularity option ('batch' stays the default). AtomicScorer
memoizes the model-state token for the duration of an evaluation context
rather than hashing all parameters for every atomic batch.
Scores are unchanged up to float rounding from the batch shapes. On the S1
validation screens with atomic batches of 16, TRIX runs 5.5x faster and ULTRA
1.6x, and all 26 datasets match their previous results.
…t the row cache The docs describe `sort` ties as PyTorch's default descending argsort, which is not guaranteed stable, and `expected` as the exact expectation under random order within ties. +H is described as balancing hard answers across the number of links that must be predicted, and the examples pass the required --split. Also documents the state-token assumption of the row cache, types its new methods, and tests the row-granular cache through KGFM evaluation.
`evaluate` defaulted to the test split, so an exploratory run could score test queries by accident. The flag is now required rather than defaulted to valid: the host forwards only non-default options into the pinned container, whose older code would otherwise fall back to its own test default.
…d ranks `report` now writes suite-effects.json/.md: for every adapter recipe, the suite-level MRR difference to its no-adapter control, to UltraQuery, and of the control to UltraQuery, overall, for EPFO and negation types and per UltraQuery family. Query types within a dataset and datasets within a group weigh equally. Seed replicates are averaged per query before differencing; the control is shared across seeds. Intervals come from a query bootstrap stratified by dataset and query type, conditional on the frozen models, adapters and seeds; per-seed point estimates are listed too. Only complete test runs with the suite's target answer filter are compared.
… profile
- +H now ships the 2i/3i-trained adapters, the recipe selected for both
suites.
- The h100 profile is the configuration measured on full validation
splits: batches of 16, a 24 GiB per-row raw-score cache in host
memory and a 512 MiB calibrated cache.
…hecks verify now reuses a passing integration check of an unchanged ULTRA/TRIX entry (same entry, inputs and source), so an interrupted verification resumes and the checks can run as separate jobs. An incomplete check still requires a fresh directory.
…points setup now pins and downloads the ULTRA 4g and 50g checkpoints from the pinned ULTRA tree.
code_identity resolved the checkout root but not the harness directory, so study commands failed when the code was imported through a symlinked path (for example a home directory mounted under another prefix).
report --workers N summarizes the rank traces and computes the paired, graph, filter and suite effects in N processes. Suite effects read each dataset's traces once for the seed mean and every single seed, instead of once per seed, and all datasets of all comparisons run together. Every bootstrap seeds its own generator, so the outputs are byte-identical for any number of workers.
…idence verify now also reuses passing parity evidence of an unchanged baseline entry checked against the same oracles (and comparison oracle), so parity checks can run as separate guarded jobs and an interrupted verification resumes without repeating them.
…red types The pinned +H CQD has no negation in its beam search, so oracles of CQD and CQD-Hybrid with CQD-A signed-atom negation cannot cover the five negated types. The exporter now declares exactly those types as exempt; verify accepts no other exemption, still evaluates the exempt queries, and records them as checked without an upstream reference (fixtures compare them with CQD-A outputs).
… of per-query rows Chained QTO projections recompute almost a whole relation matrix for every query behind a 512 MiB row cache, which puts full +H and UltraQuery test splits at days per dataset. With matrix_bytes (and host_matrix_bytes), each relation's thresholded calibrated matrix is built once from the same canonical 100-head blocks and kept in CSR form on the GPU, spilling to host memory; projections from at most eight heads keep computing rows unless the matrix is already on the device. Scores are identical: tests compare both paths bit for bit on a three-block random graph across all 16 shapes, small caches, invalidation and host spill, and against the upstream fixture. The six QTO recipes enable it (6 GiB on the device, 32 GiB in host memory).
…dapters membership_threshold zeroes memberships at or below a fixed value before observed facts override them, the hand-set calibration UltraQuery LP applies. It lets that calibration run inside the beam-search executor as a lower reference point for the learned adapter. The executor already treats zero memberships as ordinary values, also under negation. Adapters without a threshold keep their configuration, so existing artifacts and cache keys are unchanged; the threshold is saved, loaded and part of the cache key.
The independent KGFM reference now zeroes memberships at or below an adapter's membership_threshold before observed facts override them, as the production adapter does, so frozen studies can verify entries that run UltraQuery LP's fixed calibration in the beam-search executor. The identity control is still checked without a threshold, matching the paired control the evaluator builds. Tests compare executor and reference on every query shape with ties at zero membership inside the beams.
…nference Pure PyTorch reimplementation of KG-ICL (Cui et al., NeurIPS 2024) that loads the official nju-websoft/KG-ICL model_best.tar checkpoints strictly. Prompt examples and extra prompt entities are drawn from generators seeded by the prompt seed and the public relation ID; prompt encodings are cached per relation. Inference runs either upstream's hop-by-hop expansion in PyTorch, one query at a time, or batched Triton kernels whose rows are independent of the batch. Fixtures come from the pinned official code with the documented bug-fix patch applied.
The Triton path keeps states entity-major, [entities, queries, dim]: one program aggregates an entity's incoming edges for a block of four queries whose source states are contiguous, and the attention's source term Ws_attn h is projected once per entity and layer instead of once per edge. GRU updates use 3xTF32 tensor-core dots where TF32 exists and IEEE FMA dots otherwise. Rows keep a fixed operation order without atomics, so they stay bitwise independent of the query batch; entities with more than 256 incoming edges are summed in ordered 64-edge partial sums. On an RTX 5070 Laptop GPU the FB15k-237 kernels take 0.93 ms per query in batches of 8, down from 4.5 ms. During inference the prompt encoder projects entity and relation tables before the per-edge gather, which bounds the memory of hub-sized prompt graphs, and prompt distances use boolean-mask BFS. prompt_identity() describes prompt sampling for persistent score caches. Tests cover hub rows, padded query blocks and the distance mask on CUDA, and rows that are bitwise independent of batching in the query evaluator. Gradient parity uses atol=1e-4, as official gradients reach about 450.
…m runners kgfm_zero_shot.py evaluates KG-ICL with a prompt seed (--prompt-seed) and, optionally, the official answer-distance mask (--distance-mask), and labels FB15k-237 and the NELL-995 variants as related to its FB V1 and NELL V1 pretraining graphs. kgfm_upstream.py runs the pinned official KG-ICL in its own interpreter (--upstream-python) with the bug-fix patch applied to a temporary copy (--upstream-variant unpatched times the released code). Both implementations read the same graph, DICE's seeded prompt graphs written in the official case format and the same distance mask. --allow-unchecked-gpu records an unavailable nvidia-smi process check instead of aborting, and the float64 diagnostic promotes the official loader's incidence matrices.
…al recipes kgicl-adapter evaluates KG-ICL like the ULTRA and TRIX adapters. The catalog pins the official repository and commit; score adapters bind to the backbone through the state fingerprint of its checkpoint, which keeps the weights under 'state_dict'; cached adapter training rows record the prompt sampling identity. Prompt graphs come from each benchmark's inference graph with seed 0, or a recipe's prompt_seed. The +H and UltraQuery recipes are marked finalized=false and list their blockers: the product adapter is not fitted yet. They cache the per-layer relation tables of every query relation (projection_cache_mb 512), which makes rows on FB15k237 about 2.5 times faster than the 64 MiB default. Validation-only runs no longer require the +H test reduction labels, and the benchmark CLI accepts --model KGICL. The paired KGFM pipeline tests also run with KG-ICL.
…rison docs/kgicl.md covers checkpoints, prompt graphs, the official evaluation protocol, training, determinism, the five corrected upstream defects with their measured effects on five of the paper's zero-shot datasets, fixture and real-graph verification against the pinned official code, and the provisional CQA recipes. The README adds a KG-ICL section and its zero-shot rows for all twelve datasets, with Partial pretraining exposure for FB15k-237 and the NELL-995 variants; docs/kgfm_benchmarks.md documents that exposure with exact overlap counts. docs/kgfm_inference.md compares warm inference with the official code on FB15k-237, WN18RR and YAGO3-10 on an RTX 5070 Laptop GPU and reports query-evaluator row throughput on CQA graphs.
KG-ICL supports the same scoring techniques as the other graph-based foundation models (static_preprocess_funcs.py rejects NegSampleMargin for it), so the sweep excludes it like ULTRA, TRIX and Flock.
The KG-ICL product adapter is fitted with the shared ULTRA/TRIX source recipe (training seed 2026090851); the file holds the bytes the frozen test studies pinned. Both kgfm_kgicl.json recipes are now final, with the reference entry and its identity control; the docs, the CQA README and the changelog no longer call them provisional.
… as strings pyarrow 26 requires NumPy 2, which the NumPy 1.26.4 pin excludes before Python 3.13, so importing dicee failed on 3.11 and 3.12. polars 2.0 removed read_csv's dtypes argument, and its replacement needs one dtype per column of the file; infer_schema_length=0 reads every column as String, as the pandas reader does, on every polars version.
… and negation from known facts - QueryScoreAdapter.fixed_calibration: CQD-Hybrid's min-max and QTO's softmax times the observed degree, plus the plain softmax and a tie-free cap for analysis; optional masking of known-tail logits; recipes may break ties among known facts by the model. - UltraQuery executor: opt-in exact traversal of the observed facts at every projection and softmax-degree projections in place of the sigmoid. - ULTRA relation_conditioning='query' conditions relation reasoning on the queried relation, so UltraQuery's weights run as a frozen ULTRA backbone. - Engine option negation='observed': a negated branch removes its exact known answers. - Recipe checks for the new options and Flock's walk_num and test_samples.
…new calibrations
- verify runs its checks in parallel with --gpus; combined reports keep
candidate and context identity.
- The independent KGFM reference covers the training-free calibrations, the
observed traversal and negation from known facts.
- PROTOCOL.md documents the two opt-in options that are not upstream
methods: UltraQuery's observed traversal and UltraQuery's weights as a
frozen ULTRA backbone. Every adapter file under benchmarks/adapters is
tracked.
fixed_calibration='softmax-mass' scales the softmax over tails so that the observed tails together hold their count (FIT, Yin et al., 2024, Section 5.2), capped at QTO's 0.9999 below the membership of an observed fact. A row without observed tails keeps the plain softmax, and known-tail masking is refused because the factor divides by the softmax mass on those tails. The KGFM reference oracle covers the new calibration.
difficulty-report --direct reads the results of direct evaluate runs, which carry no bundle hash; the bundle then only pins the label-generation inputs and the filter corrections. Without the flag such results are refused as before, and the report records its result_mode.
setup.py allows torch>=2.5.1, which ships Triton 3.1. Two Triton 3.1
limits broke tests there:
- The first KG-ICL GRU layer multiplied the constant zero state in
three tf32x3 dots. Triton 3.1 aborts the whole process on such a dot
("DotOperandEncodingAttr non-NvidiaMmaEncodingAttr parent not
supported yet"). The state is zero, so the first layer now adds only
the hidden biases. On Triton 3.7 the outputs are bitwise equal to the
old kernel, and the first layer runs 4 dots instead of 7.
- The oracle kernels in tests/test_inference_kernels.py imported tl
inside a helper function. Triton 3.1 looks up kernel names only in
the module globals, so the imports move to module level.
(cherry picked from commit eb030f0315854122f39c79aca870f47274029f38)
test_flock_fused_fixed_walks_match_official_checkpoint failed on every CUDA setup (torch 2.5.1 on an RTX 4070 Ti SUPER, torch 2.9.1 on an H100 NVL) by about 1e-3. The reference scores are IEEE float32 runs of the official Flock code on the CPU. Flock's GRU runs in cuDNN, which uses TF32 by default, and torch.set_float32_matmul_precision does not change that. The autouse fixture now also turns off TF32 in cuDNN (convolutions and RNNs), as the KGFM benchmark runners do, and restores the settings after each test. On PyTorch with per-backend precision, it reads the cuDNN precision objects from the class, because an imported torch.backends.cudnn.rnn submodule hides the module attribute. With IEEE cuDNN the largest differences are 6e-6 (entity) and 2e-6 (relation). The module passes (32 tests) with torch 2.5.1 and 2.9.1. (cherry picked from commit 983275aef0b6cd63ecb136f59ee253a07dfed938)
ReadFromDisk chose the split of each file in dataset_dir with a substring test on the full path. A parent folder whose name contains "train", "valid" or "test" misrouted the files. Under a pytest temporary folder (/tmp/pytest-of-<user>/...), valid.txt became the test set and raw_valid_set stayed unset. The reader now tests the file name only, as the research checkout on kiowl does. The new test fails with the old rule and passes with the new one. UMLS loads the same rows as before.
Resolve the conflicts with develop's PyArrow and Polars fixes, keeping both sides: - read_with_polars: name the columns after reading, as develop does, since files may contain trailing fields; both sides read every column as a string. - requirements.txt, setup.py: keep the split by Python version (PyArrow below 26 with the NumPy 1.26 pin before Python 3.13, NumPy 2 from 3.13), with develop's comment.
Since pytest 9.2, monkeypatch.setattr records the old value of an attribute without a data descriptor and restores it with delattr. The precision objects of PyTorch (for example _FP32Precision) serve fp32_precision through __getattr__ and __setattr__. So the undo of the autouse fixture in tests/test_inference_kernels.py raised AttributeError at teardown and stopped CI. The fixture now saves each setting, sets IEEE precision and restores the settings in a finally block, as 98b3ae7 does for the KGFM cache test. Checked with pytest 9.2.0 and torch 2.14.1 (6 passed, 51 skipped on the CPU, settings restored) and with pytest 8.3.4 and torch 2.5.1.
Resolve the conflict in KGE.answer_multi_hop_query by keeping this branch's query engine, which already combines sigmoid memberships by default, and adding develop's warning for raw logits with the product T-norm or a negation query, with its explanation in the docstring. Develop's new tests in test_query_generator.py stubbed the atomic logits through KGE.predict, which the engine no longer calls. Their stub now provides the same logits through a table model, and their assertions are unchanged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP
Complex Query Answering using Knowledge Graph Foundation Models