Repository navigation
Research Alignment
In the era of Large Language Models (LLMs) and Transformer-based architectures, text preprocessing is no longer just "cleaning"—it is a critical stage of the data pipeline that directly impacts model performance and reproducibility.
Durak is designed to solve the Destructive Normalization Problem common in older Turkish NLP tools.
Many NLP libraries treat text processing as a one-way street: Input String -> Cleaned String. During this process, information regarding the original structure is lost.
Normalization drift: If a library removes double spaces or strips punctuation without tracking changes, the labels in your NER dataset (e.g., "Person entity starts at char 45") become invalid.
Linguistic noise: Regex-based lowercasing often corrupts Turkish data (e.g., DIYARBAKIR becoming diyarbakir instead of diyarbakır), introducing noise into the vocabulary embedding space.
Durak adopts the design philosophy found in modern tokenizers (like Hugging Face tokenizers) to ensure data integrity.
The Rust backend's tokenize_with_offsets function returns tokens alongside their exact start and end byte/character positions in the original raw text.
# Rust backend ensures these indices are accurate to the byte level
[
("İstanbul", 0, 8),
("geldi", 9, 14)
]This feature is essential for:
- NER & Span Labeling: Aligning tags from a raw dataset to tokenized inputs without misalignment errors.
- Question Answering: Mapping answer spans back to the original context.
We utilize the fast_normalize Rust implementation to guarantee standard Unicode Normalization Forms (NFC) and locale-aware case folding. This ensures that İ maps to i and I maps to ı consistently across operating systems and Python versions, eliminating "vocabulary split" issues where the same word appears as two different tokens due to encoding inconsistencies.
By compiling resources (stopword lists, suffix rules) directly into the library binary or managing them via strict versioning, Durak eliminates "environment drift." A pipeline run on a local laptop yields the exact same token stream as a run on a cloud training cluster.
If you use Durak in your academic work or research, please cite the following:
@software{karagoz2025durak,
author = {Karagöz, Fatih Burak},
title = {Durak: Turkish NLP Toolkit},
version = {0.3.0},
year = {2025},
url = {https://github.com/fbkaragoz/durak},
doi = {10.5281/zenodo.17477942}
}