diff --git a/CLAUDE.md b/CLAUDE.md index 23727bafd..e8d5d5276 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -153,6 +153,7 @@ Requires Python >= 3.11. Test datasets: `wget https://files.dice-research.org/da - BPE models (`--byte_pair_encoding`) use subword token sequences as entity/relation indices. - Type hints required on new public functions (`Optional[T]`, explicit return types); Google-style docstrings. - Conventional commits (`feat:`, `fix:`, `docs:`, `refactor:`, `test:`, `chore:`, `perf:`); PRs target `develop`. +- Record codebase-quality findings and improvement ideas by opening a GitHub issue (`gh issue create`), not by adding tracking files to the repo. - `KGs.zip` and any loose root-level scratch scripts (e.g. `demo.py`, `example*.md`) are local artifacts, not canonical entry points — don't treat them as documented API. ## Specialized Subagents diff --git a/IMPROVEMENTS.md b/IMPROVEMENTS.md deleted file mode 100644 index 634e89784..000000000 --- a/IMPROVEMENTS.md +++ /dev/null @@ -1,47 +0,0 @@ -# dicee — Potential Improvements - -Living tracking doc for codebase-quality findings and their status. Not user-facing documentation — a working list for maintainers. - -## In progress / done - -### 1. `--read_only_few` loads the full Parquet file before slicing -**Status:** Fixed (this branch) -**Tracking:** [dice-group/dice-embeddings#410](https://github.com/dice-group/dice-embeddings/issues/410) - -`read_with_pandas()` in [dicee/read_preprocess_save_load_kg/util.py](dicee/read_preprocess_save_load_kg/util.py) called `pd.read_parquet(data_path)` unconditionally, then `.head(read_only_few)` — so the entire file was materialized in memory regardless of how few rows were requested. CSV/text inputs already used `nrows=` and didn't have this problem. - -Fix: added `_read_parquet_head()`, which uses `pyarrow.parquet.ParquetFile.iter_batches()` to stop pulling row groups as soon as enough rows have accumulated, only falling back to a full `pd.read_parquet` when `read_only_few` is unset. Verified with a synthetic 200k-row Parquet file: requesting 10 rows dropped peak traced memory from ~26.9 MB (full read) to ~0.2 MB (~135x), with identical row content to the old behavior. - -Regression test added: [tests/test_unit_read_parquet.py](tests/test_unit_read_parquet.py) generates a dummy Parquet file via a `tmp_path` fixture (no dependency on the `KGs/` dataset download) and checks: exact row count for `read_only_few`, row order/content matches `.head(n)` on the full file, oversized `read_only_few` returns all rows, `None` returns the full file. The actual regression guard for #410 mocks `pd.read_parquet` and asserts it is **not called** when `read_only_few` is set (only the row-group-bounded `_read_parquet_head` path runs) and **is called** when it's unset — a first attempt used `tracemalloc` peak-memory comparisons instead, which was flaky on small fixture files where fixed overhead dominated the signal in CI. All 6 tests pass, deterministically. - -### 2. `assert` used for runtime validation in public API paths -**Status:** Fixed (scoped to API boundaries) - -Converted the `assert` statements guarding user input at public API boundaries — `KGE.__init__`, `predict`, `predict_topk`, `to`, `answer_multi_hop_query`, `find_missing_triples`, `predict_literals` in [dicee/knowledge_graph_embeddings.py](dicee/knowledge_graph_embeddings.py), and the full validation path in [dicee/read_preprocess_save_load_kg/preprocess.py](dicee/read_preprocess_save_load_kg/preprocess.py) — to explicit `ValueError`/`TypeError`. Internal tensor-shape/invariant checks (not user input) were deliberately left as `assert`; ~246 remain across `dicee/`, mostly internal sanity checks in query-answering loops and model internals. Not literally "all asserts converted" — only the ones that guard values coming from outside the library. - -### 4. `print()` instead of `logging` throughout the library -**Status:** Fixed - -Migrated `print()` calls in trainers, models, evaluation code, and core utilities to per-module `logging.getLogger(__name__)`. Library consumers can now control verbosity, silence output, or redirect logs without monkeypatching. A handful of `print(...)` strings remain, but only inside docstrings/commented-out dead code (e.g. `models/clifford.py`, `trainer/torch_trainer.py`) — not live paths. - -### 5. `mypy` is non-blocking in CI with no ratchet -**Status:** Fixed - -`continue-on-error: true` is removed from the mypy step in [.github/workflows/github-actions-python-package.yml](.github/workflows/github-actions-python-package.yml). The step now runs mypy, compares the `Found N errors` count against a baseline recorded in [.github/mypy-baseline.txt](.github/mypy-baseline.txt) (1367, the count at the time this was added), and fails the build only if the count increases — new code can no longer add fresh type errors, while the existing 1367 stay non-blocking until someone pays them down deliberately (lowering the baseline file locks in each improvement). `pyproject.toml`'s `disallow_untyped_defs = false` and the "start lenient, tighten later" comment are unchanged; this is a regression gate, not a rewrite of the strictness config. Also removed two dead `ignore_missing_imports` entries (`gradio.*`, `owlready2.*`) that mypy itself flagged as "unused section(s)" — neither package is imported or declared as a dependency anywhere in the repo. - -## Open findings - -### 3. Core user-facing modules have no dedicated unit tests -**Status:** Partially resolved - -[dicee/knowledge_graph_embeddings.py](dicee/knowledge_graph_embeddings.py) (the main inference API) now has a dedicated unit test suite ([tests/test_unit_kge_inference.py](tests/test_unit_kge_inference.py), 32 tests covering input validation, device management, entity embedding extraction). `executer.py`, `static_funcs.py`, `query_generator.py`, and `knowledge_graph.py` are still only exercised indirectly through regression/integration tests. - -### 6. CI only tests a single Python version (3.11.14) -**Status:** Fixed - -`.github/workflows/github-actions-python-package.yml` now runs the matrix against Python 3.11, 3.12, and 3.13, matching the `python_requires >= 3.11` declaration in `pyproject.toml`. - -### 7. Accumulated "why do we need this?" TODOs -**Status:** Cataloged, not resolved - -42 TODO/FIXME markers in `dicee/` (45 including `tests/`), several of which flag genuine uncertainty about existing logic rather than pending work — e.g. `abstracts.py:721` (`# @TODO: Do we really need this ?!`), `knowledge_graph.py:132` (`# TODO:CD: Why do we need to create this inverse mapping at this point?`), and `query_generator.py:226` (`# @TODO: Why do we need a deep copy here ?`). Cataloged by priority in [docs/TODO_BACKLOG.md](docs/TODO_BACKLOG.md); none of the underlying uncertainty has actually been resolved yet — that still needs the original authors' input. diff --git a/benchmarks/regularization_sweep.py b/benchmarks/regularization_sweep.py new file mode 100644 index 000000000..375c4dd38 --- /dev/null +++ b/benchmarks/regularization_sweep.py @@ -0,0 +1,106 @@ +"""Sweep loss / label-smoothing / weight-averaging settings over datasets and models. + +Each experiment type combines a loss (BCE, label smoothing, ACLS) with an optional +weight-averaging scheme (none, ASWA, AMWA). One CSV row per run is appended to --out. + +Example: python benchmarks/regularization_sweep.py --datasets UMLS KINSHIP --models DistMult --gpu 0 +""" +import argparse +import csv +import os +from pathlib import Path +from typing import Dict + +DATASETS = ["UMLS", "KINSHIP", "NELL-995-h100", "FB15k-237", "WN18RR", "CoDEx-M"] +MODELS = ["ComplEx", "DistMult", "Pykeen_RotatE", "Pykeen_MuRE", "DeCaL", "Pykeen_TransH"] + +LOSSES = { + "BCE": {"loss_fn": "BCELoss", "label_smoothing_rate": 0.0}, + "LS": {"loss_fn": "BCELoss", "label_smoothing_rate": 0.1}, + "ACLS": {"loss_fn": "ACLS", "label_smoothing_rate": 0.0}, +} +AVERAGING = { + "": {"adaptive_swa": False}, + "ASWA": {"adaptive_swa": True}, + "AMWA": { + "adaptive_swa": False, + "amwa": True, + "amwa_start_epoch": 20, + "amwa_c_epochs": 10, + "amwa_monitor": "MRR", + "amwa_maximize": True, + "amwa_beta": None, + "amwa_beta_window": 10, + "amwa_beta_init": 1.0, + "amwa_beta_floor": 1e-12, + }, +} +EXP_TYPES = [f"{loss}_{avg}" if avg else loss for avg in AVERAGING for loss in LOSSES] + +FIELDNAMES = ["DB", "Model", "Type", "Seed", "Test-Hit@10", "Test-Hit@3", "Test-Hit@1", "Test-MRR"] + + +def exp_settings(exp_type: str) -> Dict: + """Return the Namespace overrides for an experiment type such as ``LS_ASWA``.""" + loss, _, avg = exp_type.partition("_") + return {**LOSSES[loss], **AVERAGING[avg]} + + +def parse_args() -> argparse.Namespace: + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--datasets", nargs="+", default=DATASETS) + p.add_argument("--models", nargs="+", default=MODELS) + p.add_argument("--exp_types", nargs="+", default=EXP_TYPES, choices=EXP_TYPES) + p.add_argument("--kg_root", type=Path, default=Path("KGs"), help="Directory containing the dataset folders.") + p.add_argument("--out", type=Path, default=Path("performance.csv")) + p.add_argument("--experiments_dir", type=Path, default=Path("Experiments")) + p.add_argument("--gpu", default="0", help="Value for CUDA_VISIBLE_DEVICES.") + p.add_argument("--seed", type=int, default=0) + return p.parse_args() + + +def main() -> None: + cli = parse_args() + os.environ["CUDA_VISIBLE_DEVICES"] = cli.gpu + # Imported after CUDA_VISIBLE_DEVICES is set so torch sees only the chosen GPU. + from dicee.config import Namespace + from dicee.executer import Execute + + cli.out.parent.mkdir(parents=True, exist_ok=True) + write_header = not cli.out.is_file() + with open(cli.out, "a", newline="", encoding="utf-8") as f: + writer = csv.DictWriter(f, fieldnames=FIELDNAMES) + if write_header: + writer.writeheader() + for db in cli.datasets: + for model in cli.models: + for exp_type in cli.exp_types: + args = Namespace() + for key, value in exp_settings(exp_type).items(): + setattr(args, key, value) + args.path_to_store_single_run = str(cli.experiments_dir / exp_type / db / model) + args.save_embeddings_as_csv = True + args.model = model + args.p, args.q = 0, 1 + args.optim = "Adam" + args.scoring_technique = "KvsAll" + args.dataset_dir = str(cli.kg_root / db) + args.num_epochs = 100 + args.embedding_dim = 32 + args.batch_size = 1024 + args.lr = 0.1 + args.eval_model = "test" + args.trainer = "PL" + args.random_seed = cli.seed + + test = Execute(args).start()["Test"] + writer.writerow({ + "DB": db, "Model": model, "Type": exp_type, "Seed": cli.seed, + "Test-Hit@10": test["H@10"], "Test-Hit@3": test["H@3"], + "Test-Hit@1": test["H@1"], "Test-MRR": test["MRR"], + }) + f.flush() + + +if __name__ == "__main__": + main() diff --git a/run_all.py b/run_all.py deleted file mode 100644 index 8d8d54c62..000000000 --- a/run_all.py +++ /dev/null @@ -1,146 +0,0 @@ -from dicee.executer import Execute -from dicee.config import Namespace -from dicee.knowledge_graph_embeddings import KGE -from pathlib import Path -import csv -import random -import os - -os.environ["CUDA_VISIBLE_DEVICES"] = "0" - -DATASETS = [ "UMLS", "KINSHIP", "NELL-995-h100", "FB15k-237", "WN18RR", "CoDEx-M" ] - -MODELS = [ "ComplEx", "DistMult", "Pykeen_RotatE", "Pykeen_MuRE", "DeCaL", "Pykeen_TransH" ] - -EXP_TYPES = [ "BCE", "LS", "ACLS", "BCE_ASWA", "LS_ASWA", "ACLS_ASWA", "BCE_AMWA", "LS_AMWA", "ACLS_AMWA" ] - - -OUT_CSV = Path(f"./performance.csv") -OUT_CSV.parent.mkdir(parents=True, exist_ok=True) - -file_exists = OUT_CSV.is_file() - -fieldnames = ["DB", "Model", "Type", "Seed", "Test-Hit@10", "Test-Hit@3", "Test-Hit@1", "Test-MRR"] - -with open(OUT_CSV, "a", newline="", encoding="utf-8") as f: - w = csv.DictWriter(f, fieldnames=fieldnames) - if not file_exists: - w.writeheader() - - for DB in DATASETS: - for MODEL in MODELS: - for EXP_TYPE in EXP_TYPES: - args = Namespace() - - # Baseline - if EXP_TYPE == "BCE": - args.loss_fn = "BCELoss" - args.label_smoothing_rate = 0.0 - args.adaptive_swa = False - - # Label Smoothing - if EXP_TYPE == "LS": - args.loss_fn = "BCELoss" - args.label_smoothing_rate = 0.1 - args.adaptive_swa = False - - # Adaptive and Conditional Label Smoothing - if EXP_TYPE == "ACLS": - args.loss_fn = "ACLS" - args.label_smoothing_rate = 0.0 - args.adaptive_swa = False - - # Baseline with ASWA - if EXP_TYPE == "BCE_ASWA": - args.loss_fn = "BCELoss" - args.label_smoothing_rate = 0.0 - args.adaptive_swa = True - - # Label Smoothing with ASWA - if EXP_TYPE == "LS_ASWA": - args.loss_fn = "BCELoss" - args.label_smoothing_rate = 0.1 - args.adaptive_swa = True - - # Adaptive and Conditional Label Smoothing with ASWA - if EXP_TYPE == "ACLS_ASWA": - args.loss_fn = "ACLS" - args.label_smoothing_rate = 0.0 - args.adaptive_swa = True - - # Baseline with Adaptive Momentum Weight Averaging (AMWA) - if EXP_TYPE == "BCE_AMWA": - args.loss_fn = "BCELoss" - args.adaptive_swa = False - args.amwa = True - args.amwa_start_epoch = 20 - args.amwa_c_epochs = 10 - args.amwa_monitor = "MRR" - args.amwa_maximize = True - args.amwa_beta = None - args.amwa_beta_window = 10 - args.amwa_beta_init = 1.0 - args.amwa_beta_floor = 1e-12 - - # Label Smoothing with AMWA - if EXP_TYPE == "LS_AMWA": - args.loss_fn = "BCELoss" - args.label_smoothing_rate = 0.1 - args.adaptive_swa = False - args.amwa = True - args.amwa_start_epoch = 20 - args.amwa_c_epochs = 10 - args.amwa_monitor = "MRR" - args.amwa_maximize = True - args.amwa_beta = None - args.amwa_beta_window = 10 - args.amwa_beta_init = 1.0 - args.amwa_beta_floor = 1e-12 - - # Adaptive and Conditional Label Smoothing (ACLS) with Adaptive Momentum Weight Averaging (AMWA) - if EXP_TYPE == "ACLS_AMWA": - args.loss_fn = "ACLS" - args.label_smoothing_rate = 0.0 - args.adaptive_swa = False - args.amwa = True - args.amwa_start_epoch = 20 - args.amwa_c_epochs = 10 - args.amwa_monitor = "MRR" - args.amwa_maximize = True - args.amwa_beta = None - args.amwa_beta_window = 10 - args.amwa_beta_init = 1.0 - args.amwa_beta_floor = 1e-12 - - - args.path_to_store_single_run = f"Experiments/{EXP_TYPE}/{DB}/{MODEL}" - args.save_embeddings_as_csv = True - - args.model = MODEL - args.p=0 - args.q=1 - args.optim = 'Adam' - args.scoring_technique = "KvsAll" - - args.dataset_dir = f"../KGs/{DB}" - - args.num_epochs = 100 - args.embedding_dim = 32 - args.batch_size = 1024 - args.lr = 0.1 - - args.eval_model = "test" - args.trainer = "PL" - - result = Execute(args).start() - - w.writerow({ - "DB": DB, - "Model": MODEL, - "Type": EXP_TYPE, - "Test-Hit@10": result["Test"]["H@10"], - "Test-Hit@3": result["Test"]["H@3"], - "Test-Hit@1": result["Test"]["H@1"], - "Test-MRR": result["Test"]["MRR"], - }) - f.flush()