Skip to content

KGFM CQA - #464

Draft
LckyLke wants to merge 41 commits into
developfrom
kgfm_cqd
Draft

LckyLke wants to merge 41 commits into
developfrom
kgfm_cqd

Conversation

@LckyLke

@LckyLke LckyLke commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

WIP

Complex Query Answering using Knowledge Graph Foundation Models

@LckyLke LckyLke changed the title Kgfm cqd KGFM CQA Sep 22, 2026
Select NumPy 2 on Python 3.13, pin the docdata wheel that the index
intermittently fails to serve, check installed requirements, run mypy for each
matrix version, and let the matrix finish after a failure.
Answer path, intersection, union and negated queries over KGEs and the ULTRA,
TRIX and Flock foundation models with CQD beam search, optionally calibrated by
score adapters fitted on source graphs with frozen backbones. Native
UltraQuery, GNN-QE, QTO, CQD, CQD-Hybrid, ConE and CLMPT load the released
checkpoints and are tested against fixtures exported from the pinned upstream
code. Evaluation reads the released UltraQuery and +H files, filters easy and
hard answers, scores ties by sort order and by their exact expectation, and
resumes from verified checkpoints.

Deterministic message passing splits hub nodes into order-preserving partial
sums, ordered reductions prefetch edges, and device-side checks replace
per-query host synchronization; predictions are bitwise unchanged.
  
  `python -m benchmarks.cqa {plus_h,ultraquery} COMMAND` selects any methods,
  datasets and query types from compact public recipes, freezes inputs, query
  batches and source in a study, verifies validation predictions against oracles
  exported from the pinned upstream code, and runs and reports the test split.
  With --image, commands run in a pinned, network-free Docker runtime; the host
  needs only Python and Docker. Paired controls measure learned adapters, the
  corrected +H answer filters and inference graphs.
  
  The evaluation protocol and every known deviation from the publications are
  documented in benchmarks/cqa/PROTOCOL.md.
LckyLke added 24 commits October 2, 2026 09:16
prepare_adapter_data(hardness='balanced') gives each source query a missing-link level in rotation and trains it on that level's hard answers; queries can carry explicit positives. The default path is unchanged.
KGFM recipes accept operators such as 'product:chains', so one recipe can score some query types with a second adapter of the same t-norm.
Bit-exact comparison with fixture scores recorded on another CPU failed on CI runners. The joint DNF projection batch is now rebuilt on the test machine; the fixture comparison keeps its 3e-5 tolerance.
… the state token once

AtomicBatchCache gains a row granularity: rows are cached and evicted one by
one, so a row computed in one atomic batch is reused by any later batch
instead of only by an identical batch. KGFM evaluation enables it with the
raw_cache_granularity option ('batch' stays the default). AtomicScorer
memoizes the model-state token for the duration of an evaluation context
rather than hashing all parameters for every atomic batch.

Scores are unchanged up to float rounding from the batch shapes. On the S1
validation screens with atomic batches of 16, TRIX runs 5.5x faster and ULTRA
1.6x, and all 26 datasets match their previous results.
…t the row cache

  
  The docs describe `sort` ties as PyTorch's default descending argsort, which is
  not guaranteed stable, and `expected` as the exact expectation under random
  order within ties. +H is described as balancing hard answers across the number
  of links that must be predicted, and the examples pass the required --split.
  
  Also documents the state-token assumption of the row cache, types its new
  methods, and tests the row-granular cache through KGFM evaluation.
`evaluate` defaulted to the test split, so an exploratory run could score
test queries by accident. The flag is now required rather than defaulted to
valid: the host forwards only non-default options into the pinned container,
whose older code would otherwise fall back to its own test default.
…d ranks

  
  `report` now writes suite-effects.json/.md: for every adapter recipe, the
  suite-level MRR difference to its no-adapter control, to UltraQuery, and of
  the control to UltraQuery, overall, for EPFO and negation types and per
  UltraQuery family. Query types within a dataset and datasets within a group
  weigh equally. Seed replicates are averaged per query before differencing;
  the control is shared across seeds. Intervals come from a query bootstrap
  stratified by dataset and query type, conditional on the frozen models,
  adapters and seeds; per-seed point estimates are listed too. Only complete
  test runs with the suite's target answer filter are compared.
… profile

  
  - +H now ships the 2i/3i-trained adapters, the recipe selected for both
    suites.
  - The h100 profile is the configuration measured on full validation
    splits: batches of 16, a 24 GiB per-row raw-score cache in host
    memory and a 512 MiB calibrated cache.
…hecks

verify now reuses a passing integration check of an unchanged ULTRA/TRIX
entry (same entry, inputs and source), so an interrupted verification
resumes and the checks can run as separate jobs. An incomplete check
still requires a fresh directory.
…points

  
  setup now pins and downloads the ULTRA 4g and 50g checkpoints from the
  pinned ULTRA tree.
code_identity resolved the checkout root but not the harness directory,
so study commands failed when the code was imported through a symlinked
path (for example a home directory mounted under another prefix).
report --workers N summarizes the rank traces and computes the paired,
graph, filter and suite effects in N processes. Suite effects read each
dataset's traces once for the seed mean and every single seed, instead
of once per seed, and all datasets of all comparisons run together.
Every bootstrap seeds its own generator, so the outputs are
byte-identical for any number of workers.
…idence

verify now also reuses passing parity evidence of an unchanged baseline
entry checked against the same oracles (and comparison oracle), so parity
checks can run as separate guarded jobs and an interrupted verification
resumes without repeating them.
…red types

The pinned +H CQD has no negation in its beam search, so oracles of CQD and
CQD-Hybrid with CQD-A signed-atom negation cannot cover the five negated
types. The exporter now declares exactly those types as exempt; verify
accepts no other exemption, still evaluates the exempt queries, and records
them as checked without an upstream reference (fixtures compare them with
CQD-A outputs).
… of per-query rows

Chained QTO projections recompute almost a whole relation matrix for every
query behind a 512 MiB row cache, which puts full +H and UltraQuery test
splits at days per dataset. With matrix_bytes (and host_matrix_bytes), each
relation's thresholded calibrated matrix is built once from the same
canonical 100-head blocks and kept in CSR form on the GPU, spilling to host
memory; projections from at most eight heads keep computing rows unless the
matrix is already on the device. Scores are identical: tests compare both
paths bit for bit on a three-block random graph across all 16 shapes, small
caches, invalidation and host spill, and against the upstream fixture. The
six QTO recipes enable it (6 GiB on the device, 32 GiB in host memory).
…dapters

membership_threshold zeroes memberships at or below a fixed value before
observed facts override them, the hand-set calibration UltraQuery LP applies.
It lets that calibration run inside the beam-search executor as a lower
reference point for the learned adapter. The executor already treats zero
memberships as ordinary values, also under negation. Adapters without a
threshold keep their configuration, so existing artifacts and cache keys are
unchanged; the threshold is saved, loaded and part of the cache key.
The independent KGFM reference now zeroes memberships at or below an
adapter's membership_threshold before observed facts override them, as the
production adapter does, so frozen studies can verify entries that run
UltraQuery LP's fixed calibration in the beam-search executor. The identity
control is still checked without a threshold, matching the paired control
the evaluator builds. Tests compare executor and reference on every query
shape with ties at zero membership inside the beams.
…nference

Pure PyTorch reimplementation of KG-ICL (Cui et al., NeurIPS 2024) that loads
the official nju-websoft/KG-ICL model_best.tar checkpoints strictly. Prompt
examples and extra prompt entities are drawn from generators seeded by the
prompt seed and the public relation ID; prompt encodings are cached per
relation. Inference runs either upstream's hop-by-hop expansion in PyTorch,
one query at a time, or batched Triton kernels whose rows are independent of
the batch. Fixtures come from the pinned official code with the documented
bug-fix patch applied.
The Triton path keeps states entity-major, [entities, queries, dim]: one
program aggregates an entity's incoming edges for a block of four queries
whose source states are contiguous, and the attention's source term Ws_attn h
is projected once per entity and layer instead of once per edge. GRU updates
use 3xTF32 tensor-core dots where TF32 exists and IEEE FMA dots otherwise.
Rows keep a fixed operation order without atomics, so they stay bitwise
independent of the query batch; entities with more than 256 incoming edges
are summed in ordered 64-edge partial sums. On an RTX 5070 Laptop GPU the
FB15k-237 kernels take 0.93 ms per query in batches of 8, down from 4.5 ms.

During inference the prompt encoder projects entity and relation tables
before the per-edge gather, which bounds the memory of hub-sized prompt
graphs, and prompt distances use boolean-mask BFS. prompt_identity()
describes prompt sampling for persistent score caches. Tests cover hub
rows, padded query blocks and the distance mask on CUDA, and rows that are
bitwise independent of batching in the query evaluator. Gradient parity
uses atol=1e-4, as official gradients reach about 450.
…m runners

kgfm_zero_shot.py evaluates KG-ICL with a prompt seed (--prompt-seed) and,
optionally, the official answer-distance mask (--distance-mask), and labels
FB15k-237 and the NELL-995 variants as related to its FB V1 and NELL V1
pretraining graphs. kgfm_upstream.py runs the pinned official KG-ICL in its
own interpreter (--upstream-python) with the bug-fix patch applied to a
temporary copy (--upstream-variant unpatched times the released code). Both
implementations read the same graph, DICE's seeded prompt graphs written in
the official case format and the same distance mask. --allow-unchecked-gpu
records an unavailable nvidia-smi process check instead of aborting, and the
float64 diagnostic promotes the official loader's incidence matrices.
…al recipes

kgicl-adapter evaluates KG-ICL like the ULTRA and TRIX adapters. The
catalog pins the official repository and commit; score adapters bind to
the backbone through the state fingerprint of its checkpoint, which keeps
the weights under 'state_dict'; cached adapter training rows record the
prompt sampling identity. Prompt graphs come from each benchmark's
inference graph with seed 0, or a recipe's prompt_seed.

The +H and UltraQuery recipes are marked finalized=false and list their
blockers: the product adapter is not fitted yet. They cache the per-layer
relation tables of every query relation (projection_cache_mb 512), which
makes rows on FB15k237 about 2.5 times faster than the 64 MiB default.
Validation-only runs no longer require the +H test reduction labels, and
the benchmark CLI accepts --model KGICL. The paired KGFM pipeline tests
also run with KG-ICL.
…rison

docs/kgicl.md covers checkpoints, prompt graphs, the official evaluation
protocol, training, determinism, the five corrected upstream defects with
their measured effects on five of the paper's zero-shot datasets, fixture and
real-graph verification against the pinned official code, and the
provisional CQA recipes. The README adds a KG-ICL section and its zero-shot
rows for all twelve datasets, with Partial pretraining exposure for FB15k-237
and the NELL-995 variants; docs/kgfm_benchmarks.md documents that exposure
with exact overlap counts. docs/kgfm_inference.md compares warm inference
with the official code on FB15k-237, WN18RR and YAGO3-10 on an RTX 5070
Laptop GPU and reports query-evaluator row throughput on CQA graphs.
KG-ICL supports the same scoring techniques as the other graph-based
foundation models (static_preprocess_funcs.py rejects NegSampleMargin for
it), so the sweep excludes it like ULTRA, TRIX and Flock.
  
  The KG-ICL product adapter is fitted with the shared ULTRA/TRIX source
  recipe (training seed 2026090851); the file holds the bytes the frozen test
  studies pinned. Both kgfm_kgicl.json recipes are now final, with the
  reference entry and its identity control; the docs, the CQA README and the
  changelog no longer call them provisional.
… as strings

pyarrow 26 requires NumPy 2, which the NumPy 1.26.4 pin excludes before Python 3.13,
so importing dicee failed on 3.11 and 3.12. polars 2.0 removed read_csv's dtypes
argument, and its replacement needs one dtype per column of the file;
infer_schema_length=0 reads every column as String, as the pandas reader does, on
every polars version.
… and negation from known facts

- QueryScoreAdapter.fixed_calibration: CQD-Hybrid's min-max and QTO's softmax times the
  observed degree, plus the plain softmax and a tie-free cap for analysis; optional
  masking of known-tail logits; recipes may break ties among known facts by the model.
- UltraQuery executor: opt-in exact traversal of the observed facts at every projection
  and softmax-degree projections in place of the sigmoid.
- ULTRA relation_conditioning='query' conditions relation reasoning on the queried
  relation, so UltraQuery's weights run as a frozen ULTRA backbone.
- Engine option negation='observed': a negated branch removes its exact known answers.
- Recipe checks for the new options and Flock's walk_num and test_samples.
…new calibrations

  
  - verify runs its checks in parallel with --gpus; combined reports keep
    candidate and context identity.
  - The independent KGFM reference covers the training-free calibrations, the
    observed traversal and negation from known facts.
  - PROTOCOL.md documents the two opt-in options that are not upstream
    methods: UltraQuery's observed traversal and UltraQuery's weights as a
    frozen ULTRA backbone. Every adapter file under benchmarks/adapters is
    tracked.
fixed_calibration='softmax-mass' scales the softmax over tails so that the
observed tails together hold their count (FIT, Yin et al., 2024, Section 5.2),
capped at QTO's 0.9999 below the membership of an observed fact. A row without
observed tails keeps the plain softmax, and known-tail masking is refused
because the factor divides by the softmax mass on those tails. The KGFM
reference oracle covers the new calibration.
difficulty-report --direct reads the results of direct evaluate runs, which
carry no bundle hash; the bundle then only pins the label-generation inputs and
the filter corrections. Without the flag such results are refused as before,
and the report records its result_mode.
setup.py allows torch>=2.5.1, which ships Triton 3.1. Two Triton 3.1
limits broke tests there:

- The first KG-ICL GRU layer multiplied the constant zero state in
  three tf32x3 dots. Triton 3.1 aborts the whole process on such a dot
  ("DotOperandEncodingAttr non-NvidiaMmaEncodingAttr parent not
  supported yet"). The state is zero, so the first layer now adds only
  the hidden biases. On Triton 3.7 the outputs are bitwise equal to the
  old kernel, and the first layer runs 4 dots instead of 7.
- The oracle kernels in tests/test_inference_kernels.py imported tl
  inside a helper function. Triton 3.1 looks up kernel names only in
  the module globals, so the imports move to module level.

(cherry picked from commit eb030f0315854122f39c79aca870f47274029f38)
test_flock_fused_fixed_walks_match_official_checkpoint failed on every
CUDA setup (torch 2.5.1 on an RTX 4070 Ti SUPER, torch 2.9.1 on an
H100 NVL) by about 1e-3. The reference scores are IEEE float32 runs of
the official Flock code on the CPU. Flock's GRU runs in cuDNN, which
uses TF32 by default, and torch.set_float32_matmul_precision does not
change that.

The autouse fixture now also turns off TF32 in cuDNN (convolutions and
RNNs), as the KGFM benchmark runners do, and restores the settings
after each test. On PyTorch with per-backend precision, it reads the
cuDNN precision objects from the class, because an imported
torch.backends.cudnn.rnn submodule hides the module attribute. With
IEEE cuDNN the largest differences are 6e-6 (entity) and 2e-6
(relation). The module passes (32 tests) with torch 2.5.1 and 2.9.1.

(cherry picked from commit 983275aef0b6cd63ecb136f59ee253a07dfed938)
ReadFromDisk chose the split of each file in dataset_dir with a
substring test on the full path. A parent folder whose name contains
"train", "valid" or "test" misrouted the files. Under a pytest
temporary folder (/tmp/pytest-of-<user>/...), valid.txt became the
test set and raw_valid_set stayed unset.

The reader now tests the file name only, as the research checkout on
kiowl does. The new test fails with the old rule and passes with the
new one. UMLS loads the same rows as before.
Resolve the conflicts with develop's PyArrow and Polars fixes, keeping both sides:
- read_with_polars: name the columns after reading, as develop does, since files may contain trailing fields; both sides read every column as a string.
- requirements.txt, setup.py: keep the split by Python version (PyArrow below 26 with the NumPy 1.26 pin before Python 3.13, NumPy 2 from 3.13), with develop's comment.
Since pytest 9.2, monkeypatch.setattr records the old value of an
attribute without a data descriptor and restores it with delattr. The
precision objects of PyTorch (for example _FP32Precision) serve
fp32_precision through __getattr__ and __setattr__. So the undo of the
autouse fixture in tests/test_inference_kernels.py raised AttributeError
at teardown and stopped CI. The fixture now saves each setting, sets
IEEE precision and restores the settings in a finally block, as
98b3ae7 does for the KGFM cache test.

Checked with pytest 9.2.0 and torch 2.14.1 (6 passed, 51 skipped on the
CPU, settings restored) and with pytest 8.3.4 and torch 2.5.1.
Resolve the conflict in KGE.answer_multi_hop_query by keeping this branch's query engine, which already combines sigmoid memberships by default, and adding develop's warning for raw logits with the product T-norm or a negation query, with its explanation in the docstring.

Develop's new tests in test_query_generator.py stubbed the atomic logits through KGE.predict, which the engine no longer calls. Their stub now provides the same logits through a table model, and their assertions are unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant