spBELU-1K

README.md

@@ -288,6 +288,9 @@ but it expects that you pass through the entire translated test set.
    - `ja-mecab` tokenizes **Japanese** inputs using the [MeCab](https://pypi.org/project/mecab-python3) morphological analyzer
    - `ko-mecab` tokenizes **Korean** inputs using the [MeCab-ko](https://pypi.org/project/mecab-ko) morphological analyzer
    - `flores101` and `flores200` uses the SentencePiece model built from the Flores-101 and [Flores-200](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200) dataset, respectively. Note: the canonical .spm file will be automatically fetched if not found locally.
+   - 🔥🆕 `spBLEU-1K` is a newly introduced **multilingual** SentencePiece tokenizer, trained on **over 1,000** data sources, as proposed in the [Toucan](https://aclanthology.org/2024.findings-acl.781/) paper (ACL 2024).  For more details, see spBLEU's [GitHub repository](https://github.com/UBC-NLP/Toucan/tree/main/spBLEU-1K).
+
+
 - You can switch tokenizers using the `--tokenize` flag of sacreBLEU. Alternatively, if you provide language-pair strings
   using `--language-pair/-l`, `zh`, `ja-mecab` and `ko-mecab` tokenizers will be used if the target language is `zh` or `ja` or `ko`, respectively.
 - **Note that** there's no automatic language detection from the hypotheses so you need to make sure that you are correctly

sacrebleu/metrics/bleu.py

@@ -26,6 +26,8 @@
     'spm': 'tokenizer_spm.TokenizerSPM',
     'flores101': 'tokenizer_spm.Flores101Tokenizer',
     'flores200': 'tokenizer_spm.Flores200Tokenizer',
+    ### Added for spBLEU-1K tokenizer by AbdelRahim Elmadany
+    'spBLEU-1K':'tokenizer_spm.spBLEU1KTokenizer',
 }
 
 

sacrebleu/tokenizers/tokenizer_spm.py

@@ -24,6 +24,11 @@
         "url": "https://tinyurl.com/flores200sacrebleuspm",
         "signature": "flores200",
     },
+    ### Added for spBLEU-1K tokenizer by AbdelRahim Elmadany
+    "spBLEU-1K": {
+          "url": "https://www.dlnlp.ai/spBLEU-1K/spbleu-1k_tokenizer_spm.model",
+        "signature": "spBLEU-1K",
+    },
 }
 
 class TokenizerSPM(BaseTokenizer):
@@ -68,3 +73,8 @@ def __init__(self):
 class Flores101Tokenizer(TokenizerSPM):
     def __init__(self):
         super().__init__("flores101")
+
+### Added for spBLEU-1K tokenizer by AbdelRahim Elmadany
+class spBLEU1KTokenizer(TokenizerSPM):
+    def __init__(self):
+        super().__init__("spBLEU-1K")