Repository navigation
spBELU-1K - #282
spBELU-1K#282
Conversation
|
Thank you for your contribution. While I don't think we should include too many different tokenizers into sacrebleu (simple whitespace+punctuation tokenization is enough for all languages that separate words with spaces), having an alternative SPM model "covering 1,003 languages, including 614 African languages, 53 Indigenous American languages" seems like a good idea. |
|
Thanks, @martinpopel, for merging the spBLEU-1k! I plan to announce this soon. I was wondering when you plan to release the new version on PyPI. It would be great if users could simply run |
|
@mjpost would you want to release 2.5.2? (I have never done it, I think, but it seems I have the rights at PyPI, so I could try). |
| - `ko-mecab` tokenizes **Korean** inputs using the [MeCab-ko](https://pypi.org/project/mecab-ko) morphological analyzer | ||
| - `flores101` and `flores200` uses the SentencePiece model built from the Flores-101 and [Flores-200](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200) dataset, respectively. Note: the canonical .spm file will be automatically fetched if not found locally. | ||
| - 🔥🆕 `spBLEU-1K` is a newly introduced **multilingual** SentencePiece tokenizer, trained on **over 1,000** data sources, as proposed in the [Toucan](https://aclanthology.org/2024.findings-acl.781/) paper (ACL 2024). For more details, see spBLEU's [GitHub repository](https://github.com/UBC-NLP/Toucan/tree/main/spBLEU-1K). | ||
|
|
There was a problem hiding this comment.
Can you remove these two blank lines?
| 'spm': 'tokenizer_spm.TokenizerSPM', | ||
| 'flores101': 'tokenizer_spm.Flores101Tokenizer', | ||
| 'flores200': 'tokenizer_spm.Flores200Tokenizer', | ||
| ### Added for spBLEU-1K tokenizer by AbdelRahim Elmadany |
There was a problem hiding this comment.
I suggest you link to the paper: https://aclanthology.org/2024.findings-acl.781/
| "spBLEU-1K": { | ||
| "url": "https://www.dlnlp.ai/spBLEU-1K/spbleu-1k_tokenizer_spm.model", | ||
| "signature": "spBLEU-1K", | ||
| }, |
|
@martinpopel Sure, that's fine. If I remember, you can actually publish it by creating a tag. Then when you push, it is uploaded automatically. I always forget how to do this, though. |
|
We need to update the CHANGELOG, too. |
3 participants
spBLEU-1Kis a newly introduced multilingual SentencePiece tokenizer, trained on over 1,000 data sources, as proposed in the Toucan paper (ACL 2024). For more details, see spBLEU's GitHub repository.