Many conservative changes to make code run with both t4 and t5

[MihaiSurdeanu]

@kwalcock : this replaces the other PR from the branch deberta-large. I tried to be a bit more conservative here because deberta is known to be broken with transformers 5. Nevertheless, I ported over your workflow and tokenizer changes. Can you please check?

For the record: all transformer models work with transformers v4 here. With v5 all work except debertas.

[kwalcock]

@MihaiSurdeanu, remember that the pre-compiled Rust tokenizer has only been updated for Linux. It's possible that the Rust code hadn't changed much between transformers versions, but I should update for Windows and Mac. If other people are using these other operating systems, they might encounter some strangeness. It's possible that different builds would be needed depending on whether transformers 4 or 5 is being used. Is the encoder.onnx.data file not a problem right now?

[MihaiSurdeanu]

Thanks!

I'll get to the ONNX issue in detail tomorrow. It loads fine in the app here.

[kwalcock]

@MihaiSurdeanu, I made one very small change so that the unit tests pass on Scala 3.

[MihaiSurdeanu]

@kwalcock : I found this model which is very good:
https://huggingface.co/thomas-sounack/BioClinical-ModernBERT-base

Is there a tokenizer we need to add for it?
Thank you!

[kwalcock]

If I add to ScalaJniTokenizerApp the name "thomas-sounack/BioClinical-ModernBERT-base", it will grab the tokenizer file from https://huggingface.co/thomas-sounack/BioClinical-ModernBERT-base/blob/main/tokenizer.json and use it. That's probably OK for experimentation. For distribution, it would be nice if the system was not dependent on a network connection. In that case the file could be added to the resources so that the local copy in the jar file would be used instead. if you observe different behavior, like discrepancies between the python and rust version, please let me know.

…

On Sat, Mar 14, 2026 at 11:18 AM Mihai Surdeanu ***@***.***> wrote: *MihaiSurdeanu* left a comment (clulab/scala-transformers#72) <#72 (comment)> @kwalcock <https://github.com/kwalcock> : I found this model which is very good: https://huggingface.co/thomas-sounack/BioClinical-ModernBERT-base Is there a tokenizer we need to add for it? Thank you! — Reply to this email directly, view it on GitHub <#72 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCHCOUMEMJTKEH2NG3ELBT4QWO7FAVCNFSM6AAAAACWRSBGWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANRRGAZDIMZUGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[MihaiSurdeanu]

Thanks!

Should I add the tokenizer file to resources/ now?

[kwalcock]

@MihaiSurdeanu, if you'd like. I'm not certain whether you are still experimenting or already certain that it is destined for main. If I see the resource added, I can update some other files like SentencesTest.scala, TokenizersTest.scala and, TokenizerTest.scala, for example.

[MihaiSurdeanu]

Testing now... I should know soon.

[MihaiSurdeanu]

@kwalcock : I added the tokenizer for BioClinical-ModernBERT-base to this branch. Thank you for your help!

[MihaiSurdeanu]

@kwalcock : I think is fairly stable. I'd like to merge and release. Ok with you?

[kwalcock]

@MihaiSurdeanu, would you like to merge #74 first? I (or somebody) needs to check the state of the maven account. It may be that the sbt version needs to be updated before publishing and possibly some URLs changed in publish.sbt. That can happen in a different PR after this merge, however.

[MihaiSurdeanu]

@MihaiSurdeanu, would you like to merge #74 first? I (or somebody) needs to check the state of the maven account. It may be that the sbt version needs to be updated before publishing and possibly some URLs changed in publish.sbt. That can happen in a different PR after this merge, however.

Merged!
My Maven account should be fine (I published a few things). I can do the release once you're Ok with this PR.

[kwalcock]

@MihaiSurdeanu, that's great if the normal publishing procedure still works. Maybe they kept compatibility. Are you using Java 8 for releases? That was the intention in the past for maximum compatibility.

[MihaiSurdeanu]

I don't know what I'm using... How do I check? Just check the java version in the cmd line?

[kwalcock]

sbt just uses whatever it gets from the OS, so which java or java -version should tell.

[MihaiSurdeanu]

It looks like I'm on Java 11 (I remember installing that with sdkman I believe a while ago).