moon v2 and toolchains are using executables from proto/tools directory

[harlequin]

Since moon v2 there is a strange behaviour.

When installing a tool via moon v2 toolchains tier3 and executing the command, it will take the exe from the proto/tools directory instead of the proto/shims and proto/bin folder.

Example:

When using matlab (own proto plugin and moon toolchain) it will execute it matlab from C:\Users\xxx.proto\tools\matlab\R2022b\matlab.exe instead of C:\Users\xxx.proto\shims\matlab.exe

When installing python we have also 3 different incidents:-

1. Same incident as above, so the task is (Yes I know the syntax is incorrect ;))

  python-version:
    command: python -version        
    options:
      cache: false

Produces

python -version (in workspace)
Unknown option: -e
usage: C:\Users\xxx\.proto\tools\python\3.11.13\python.exe [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.

So my expectation would be to take it from shims, if not available take from bin (both paths are available in PATH)

2. Python shall take it from generated ".venv" folder

So esp. on the python toolchain it shall take the $workspaceRoot/.venv/[bin/Scripts]/python

3. The python toolchain can install via pip the dependencies from requirements.txt

So with the 2 incidents above the dependencies will be installed into C:\Users\xxx\.proto\tools\python\3.11.13 and not the .venv folder.

@milesj: Do you have an idea why we have now this strange behaivour?

[milesj]

This is by design. We know the exact version of the tool you are using, so we reference their executables directly. There is absolutely no need for the shims/bins.

This is also how it worked in v1:

• https://github.com/moonrepo/moon/blob/release-1.0/crates/toolchain-plugin/src/toolchain_registry_actions.rs#L95
• https://github.com/moonrepo/moon/blob/release-1.0/crates/toolchain-plugin/src/toolchain_plugin.rs#L125

As for python, the venv paths come before the proto ones in PATH, so they should take precedence. I'm not a python developer, so there may be more required here to get it working correctly.

[harlequin]

The shims technology is a nice way to define, for example, licenses.

So some applications can handle license information via env variables, which we have put as shims_env_var into the shims ...

So from our perspective there is (was) the need.

The python executable we can have a look into, to figure out what's going wrong

[harlequin]

I have investigated the incident and maybe I should explain the python use case...

The python toolchain is defined and also pip as dependency manager

Python will be installed, if not applicable, into C:/Users/xxx/.proto/tools/python/3.11.13/python.exe

requirements.txt is detected and the $workspace/.venv - Virtual Environment is created.

What python is doing in the background is to copy all essential python libs and executable into the $workspace/.venv

dependencies will be installed via pip

Here is the first issue, the call python -m pip install -r requirements.txt will take the binary from
C:/Users/xxx/.proto/tools/python/3.11.13 which causes that the dependencies will be installed into C:/Users/xxx/.proto/tools/python/3.11.13

The correct call has to be $workspace/.venv/Scripts/python.exe -m pip install ...

Python command/script usage in tasks from moon

Here the same issue as in dependencies will be installed via pip happens, because it takes the python executable from proto inventory and not the .venv executable.

Research

I have searched a little bit inside the moon source code and found this https://github.com/moonrepo/moon/blob/master/crates/task-runner/src/command_builder.rs#L59 maybe here the order shall be changed in order to support this.
Because, and yes you are correct, the python toolchain adapt the PATHS where to look, but this is included in the order AFTER the binary executable is detected.

@milesj: Are my assumptions correct? And then the most important would be how to influence via the toolchain to override the proto inventory with .venv location for the binary.
Additional, can you maybe explain why the proto shims and proto bins are put as last item in the function. Because only // Must be last! is written as comment. When this would be at first also the shims bins topic would be solved.

Would it make sense to change the order of inherits to

1. Include the paths when defined from toolchain plugin - Because toolchain shall always win
2. Include proto shims and bins - Because when a tool is installed it will create the shims and bins if the application shall be executable
3. Add the proto inventory detected tool version path