Allow building with thin module Docker containers

[mikegerber]

In #68 @bertsky :

But the real problem is that TF2 dependencies are lurking everywhere, so we will very soon have the unacceptable state that no catch-all venv (satisfying both TF1 and TF2 modules) is possible anymore. By then, a new solution needs to be in place, which (at least partially) isolates venvs from each other again.

[mikegerber]

I think OCR-D has to decide if you want a "have it all" environment. Then either separate venvs per processor or some kind of Docker setup could be the solution.

I'd go for a Docker solution because that also solves dependency issues outside of the Python world. And it is actually possible to do without over-engineering too hard. I.e. I use a run one-liner that just calls a container with the right UID and working directory in place.

[mikegerber]

docker run --rm -t  --user `id -u`:`id -g` --mount type=bind,src="$(pwd)",target=/data ocrd_example_processor

[bertsky]

I think OCR-D has to decide if you want a "have it all" environment. Then either separate venvs per processor or some kind of Docker setup could be the solution.

As long as we don't have REST integration, all-in-one is all we have. Isolation within that approach will always be up to demand and interdependencies. Whether we then delegate to local venvs or thin Docker containers is indeed another freedom of choice (which we ideally should also make a user choice here).

I'd go for a Docker solution because that also solves dependency issues outside of the Python world.

Yes, but it may also increase space and time cost unnecessarily under circumstances (depending on which modules are enabled and what platform the host is). So I'd really like that to be for the user to decide ultimately.

And it is actually possible to do without over-engineering too hard. I.e. I use a run one-liner that just calls a container with the right UID and working directory in place.

Yes of course, once we wrap the processor CLIs in another shell script layer, we can make that local venv or Docker run again. And for the latter we only need to ensure we pass on all the arguments. (Perhaps we could even avoid spinning up a new container with each invocation by using --attach somehow.)

The only thing that troubles me with delegating to thin module Docker containers is that we more or less surrender version control. It's really difficult to do that with Docker images solely based on digest numbers. But we could of course petition module providers to use the docker tag mechanism in a certain way.

[kba]

Unfortunately, we have now a situation where both tensorflow (tf2) and tensorflow-gpu (tf1) can be installed side-by-side, so scripts won't fail at startup anymore but when doing the customary import tensorflow as tf, this will lead to runtime errors if the script (or its dependencies!) expect tf1 API but actually get tf2 API.

After discussing with @bertsky I see no alternative to isolated venvs. Implementing this in the Makefile is tedious but doable. However for our Docker builds, we need to decide on a mechanism to create an entry-point "venv broker" script that activates the right environment or similar. I'm stumped on how we can sensibly support ocrd process in this scenario.

But the situation right now, with runtime errors instead of startup failures, is unacceptable, I see no alternative to package isolation. If anyone else does see an alternative, I'd be happy to hear.

[stweil]

Are there plans to upgrade models and software to TF 2? I think it would help to get an overview of all processors which still use TF 1 with an estimation whether and when they will run with TF 2 and who is responsable for that. And we should have all TF 1 based processors in their own group TF1_EXECUTABLES in the ocrd_all Makefile.

As soon as there is a separate group of TF 1 executables, the Makefile implementation could be straight forward by calling make recursively. We'd have the normal venv with all other processors. And we'd have a 2nd virtual environment, maybe simply a subdirectory of the other one, with the TF 1 processors. So all TF 1 processors would be in venv/tf1/bin. We'd have a generic executablevenv/bin/tf1 which can be linked to venv/bin/processor_x and which simply runs venv/tf1/bin/processor_x, passing all arguments and nearly all environment variables unmodified. Only PATH and VIRTUAL_ENV would be fixed.

[kba]

From the direct requirements:

project	tf1	tf2
cor-asv-ann	✗
ocrd_anybaseocr		✗
ocrd_calamari	✗
ocrd_keraslm	✗
sbb_textline_detector	✗

But we'll also have to check transitive dependencies.

[stweil]

ocrd_pc_segmentation depends indirectly on tensorflow<2.1.0,>=2.0.0. Again I have the problem that there is no prebuild tf2 for Python 3.8. That is not available for Python 3.8 and conflicts with tensorflow 2.2.0 which is available.

[stweil]

Because of the close relationship between Python version(s) and available prebuild Tensorflow versions we must be prepared that TF1 might require a different (= older) Python version than TF2. I already have that situation when I want to build with a recent Linux distribution, and because of a bug in Debian / Ubuntu it is currently not possible to create a Python 3.7 venv when Python 3.8 is installed, too. This is of course not relevant for the old Ubuntu which is currently our reference platform.

[stweil]

I'm stumped on how we can sensibly support ocrd process in this scenario.

I did not look into the details of the code. Does it spawn processes for the single steps (then it should work), or do all steps run in the same Python process?

[bertsky]

@stweil

Are there plans to upgrade models and software to TF 2? I think it would help to get an overview of all processors which still use TF 1 with an estimation whether and when they will run with TF 2 and who is responsable for that

The README is up-to-date w.r.t. that.

TF2 migration can be less or more effort, depending on what part of the API the module (or its dependent) relies. TF offers an upgrade script to rewrite the code to use tf.compat.v1 where necessary, but that is not always enough (automatic). True (native) TF2 migration also involves model retraining.

Plus TF 2.2 brings even more breaking changes (as you observed for ocrd_pc_segmentation already).

Since we have modules like ocrd_calamari (which depends on calamari_ocr which depends on TF1 with no visible migration activity) and ocrd_segment (which has a tool that depends on maskrcnn which depends on TF1 and is not maintained at the moment), this cannot be forced. (And forcing it would probably mean a lot of work for cor-asv-ann and ocrd_keraslm, too.)

As soon as there is a separate group of TF 1 executables, the Makefile implementation could be straight forward by calling make recursively. We'd have the normal venv with all other processors. And we'd have a 2nd virtual environment, maybe simply a subdirectory of the other one, with the TF 1 processors. So all TF 1 processors would be in venv/tf1/bin. We'd have a generic executablevenv/bin/tf1 which can be linked to venv/bin/processor_x and which simply runs venv/tf1/bin/processor_x, passing all arguments and nearly all environment variables unmodified. Only PATH and VIRTUAL_ENV would be fixed.

This would work, but why make an exception for TF? We have seen other conflicting dependencies already, and we know that pip is as bad as it gets in resolving these. See above for broader recipes, including delegation to thin Docker containers (possibly even behind REST in the future).

@kba

Implementing this in the Makefile is tedious but doable. However for our Docker builds, we need to decide on a mechanism to create an entry-point "venv broker" script that activates the right environment or similar. I'm stumped on how we can sensibly support ocrd process in this scenario.

The recipe is simple (and has already been discussed above): the top-level PATH directory (which will be a tiny venv merely for ocrd CLI, or /usr/local/bin in a Docker image) contains only shell scripts (created automatically at install-time) which then delegate to the right venv or Docker container. (Which modules are grouped together can be organised centrally in the Makefile.) Why shouldn't we allow venvs inside Docker images?

[kba]

Why shouldn't we allow venvs inside Docker images?

I was not clear: That can also be done with a mechanism like the one you describe. No reason not to use venv in a Docker image, on the contrary.

I'm stumped on how we can sensibly support ocrd process in this scenario.

After sleeping on it, I am indeed unstumped. I misremembered ocrd process to use run_processor but it obviously uses run_cli, so the processors are run with their own python instance (or bash or compiled code) which can be set at install time as we're discussing.

[stweil]

O what a mess! I now tried an installation of ocrd_all with a 2nd venv for sbb_textline_detector as an example of a TF1 processor. The build worked, no obvious conflicts, tensorflow-gpu 1.15.3 correctly installed for sbb_textline_detector. But it also installed Keras 2.4.3. And here is the result when running sbb_textline_detector --help:

ImportError: Keras requires TensorFlow 2.2 or higher. Install TensorFlow via `pip install tensorflow`

@mikegerber, I am afraid that requirements.txt should be more precise regarding the required Keras version.

[bertsky]

But it also installed Keras 2.4.3. And here is the result when running sbb_textline_detector --help:

ImportError: Keras requires TensorFlow 2.2 or higher. Install TensorFlow via `pip install tensorflow`

@mikegerber, I am afraid that requirements.txt should be more precise regarding the required Keras version.

@stweil please report to the respective repo (yes, probably keras < 2.4, maybe even keras < 2.3)!

[stweil]

please report to the respective repo

See qurator-spk/sbb_textline_detection#34

[stweil]

@bertsky, ocrd_keraslm must also limit the Keras version. It currently uses keras >= 2.3.1.