🤖 Merge 8.6.x-sync into master

src/reference/changes.rst

@@ -13,10 +13,27 @@ Cylc 8.6
    :class: hint
 
    :cylc-flow: `8.6 <https://github.com/cylc/cylc-flow/blob/8.6.x/CHANGES.md>`__
-   :cylc-ui: `2.11 <https://github.com/cylc/cylc-ui/blob/master/CHANGES.md#cylc-ui-2110-released-2025-11-27>`__
-   :cylc-uiserver: `1.8 <https://github.com/cylc/cylc-uiserver/blob/1.8.x/CHANGES.md>`__
+   :cylc-ui: `2.x <https://github.com/cylc/cylc-ui/blob/master/CHANGES.md>`__
+   :cylc-uiserver: `1.9 <https://github.com/cylc/cylc-uiserver/blob/1.9.x/CHANGES.md>`__
    :cylc-rose: `1.7 <https://github.com/cylc/cylc-rose/blob/1.7.x/CHANGES.md>`__
-   :rose: `2.6 <https://github.com/metomi/rose/blob/2.6.x/CHANGES.md>`__
+   :rose: `2.7 <https://github.com/metomi/rose/blob/2.7.x/CHANGES.md>`__
+
+
+Warnings during Jinja2 preprocessing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. versionadded:: cylc-flow 8.6.6
+
+Python warnings raised during Jinja2 preprocessing are now logged when running
+relevant Cylc commands (e.g. ``cylc validate`` and ``cylc play``).
+
+.. warning::
+
+   Cylc 8.7.0 will move from Jinja2 3.0 to 3.1, which removes a number of
+   deprecated features.
+   This should only affect you if you use :ref:`user-guide.jinja2.custom-extensions`.
+   Warnings for these deprecations are now logged, but were not previously.
+   See the `Jinja 3.1 release notes <https://jinja.palletsprojects.com/en/stable/changes/#version-3-1-0>`_.
 
 
 UI sidebar improvements

src/user-guide/writing-workflows/jinja2.rst

@@ -176,88 +176,140 @@ installed workflow at run time:
       {{ CYLC_WORKFLOW_RUN_DIR | default("not-defined") }}
 
 
-Environment Variables
----------------------
+.. _Jinja2Filters:
+
+Jinja2 Filters, Tests and Globals
+---------------------------------
+
+.. _Jinja2 Built-in Globals: https://jinja.palletsprojects.com/en/stable/templates/#list-of-global-functions
+.. _Jinja2 Built-in Filters: https://jinja.palletsprojects.com/en/stable/templates/#list-of-builtin-filters
+.. _Jinja2 Built-in Tests: https://jinja.palletsprojects.com/en/stable/templates/#builtin-tests
+
+Jinja2 provides "globals", "filters" and "tests" which can be helpful in
+workflow writing.
 
-Cylc automatically imports the parse-time environment to the template
-processor's global namespace (see :ref:`Jinja2Filters`),
-in a dictionary called ``environ``:
+Globals
+   Regular Python functions.
+
+   :Jinja2 builtins: `Jinja2 Built-in Globals`_
+   :Cylc builtins: :ref:`user-guide.jinja2.cylc-builtin-globals`
+   :Custom directory: :ref:`Jinja2Globals <user-guide.jinja2.custom-extensions>`
+Filters
+   Special functions which "chain" using the pipe character (``|``).
+
+   :Jinja2 builtins: `Jinja2 Built-in Filters`_
+   :Cylc builtins: :ref:`user-guide.jinja2.cylc-builtin-filters`
+   :Custom directory: :ref:`Jinja2Filters <user-guide.jinja2.custom-extensions>`
+Tests
+   Special functions which work with the ``is`` operator.
+
+   :Jinja2 builtins: `Jinja2 Built-in Tests`_
+   :Custom directory: :ref:`Jinja2Tests <user-guide.jinja2.custom-extensions>`
+
+For example, this :cylc:conf:`flow.cylc` file uses the
+:py:func:`pad <cylc.flow.jinja.filters.pad.pad>` filter to help write out
+task definitions:
 
 .. code-block:: cylc
 
-   #!Jinja2
-   #...
    [runtime]
-       [[root]]
-           [[[environment]]]
-               HOME_DIR_ON_WORKFLOW_HOST = {{environ['HOME']}}
+   {% for x in range(3) %}
+       [[task_{{ x | pad(3) }}]]
+           script = sleep {{ x }}
+   {% endfor %}
 
-.. important::
+The Jinja2 would be expanded like so:
 
-   The environment is read during configuration parsing. It is not the run time
-   job environment.
+.. code-block:: cylc
 
-.. _Jinja2Filters:
+   [runtime]
+       [[x_001]]
+           script = sleep 1
+       [[x_002]]
+           script = sleep 2
+       [[x_003]]
+           script = sleep 3
 
-Jinja2 Filters, Tests and Globals
----------------------------------
+In addition to the built-ins that Jinja2 and Cylc provide, you can also define
+your own custom filters (see :ref:`user-guide.jinja2.custom-extensions`).
 
-Jinja2 has three namespaces that separate "globals", "filters" and "tests".
 
- - Globals are template-wide variables and functions. Cylc extends this namespace
-   with the ``environ`` dictionary above, and
-   :ref:`raise <jinja2-raise>` and :ref:`assert <jinja2-assert>`
-   functions for raising exceptions to abort Cylc config parsing.
+.. _user-guide.jinja2.cylc-builtin-globals:
 
- - Filters can be used to modify variable values and are applied using pipe
-   notation. For example, the built-in ``trim`` filter strips leading
-   and trailing white space from a string:
+Cylc Built-in Globals
+^^^^^^^^^^^^^^^^^^^^^
 
-   .. code-block:: cylc
+.. list-table::
 
-      {% set MyString = "   dog   " %}
-      {{ MyString | trim() }}  # "dog"
+   * - :py:data:`environ`
+     - Access environment variables.
+   * - :py:func:`raise`
+     - Raise an error.
+   * - :py:func:`assert`
+     - Raise an error if a condition is not met.
 
- - Variable values can be tested using the ``is`` keyword followed by
-   the name of the test, e.g. ``{% if VARIABLE is defined %}``.
+.. _jinja2-environ:
 
-See `Jinja2 documentation <https://jinja.palletsprojects.com/en/stable/templates/>`_
-for available built-in globals, filters and tests.
+.. py:data:: environ
 
-.. _CustomJinja2Filters:
+   Provides access to environment variables.
 
-Custom Filters, Tests and Globals
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   Note, these are the "parse-time" environment variables - i.e, the environment
+   that is set when the workflow's :cylc:conf:`flow.cylc` file is processed.
+   This happens when a workflow is validated or started, not when jobs are
+   submitted. Jinja2 does not have access to dynamic environment variables
+   available to jobs.
 
-Cylc also supports custom Jinja2 filters, tests and globals.
-A custom filter or test is a single Python function in a source file
-with the same name as the function (plus ``.py`` extension).
-Likewise, a custom global is a single Python variable or function.
-These must be located in a subdirectory of the :term:`source directory` called
-``Jinja2Filters``, ``Jinja2Tests``, or ``Jinja2Globals`` respectively.
+   .. describe:: Jinja2 Examples:
 
-In the argument list of a filter or test function, the first argument is
-the variable value to be filtered or tested, and subsequent arguments can be
-whatever is needed.
+      .. code-block:: cylc
 
-.. seealso::
+         [runtime]
+             [[root]]
+                 [[[environment]]]
+                     HOME_DIR_ON_WORKFLOW_HOST = {{environ['HOME']}}
 
-   Jinja2 documentation:
+.. _jinja2-raise:
 
-   - `Custom Filters <https://jinja.palletsprojects.com/en/stable/api/#custom-filters>`_
-   - `Custom Tests <https://jinja.palletsprojects.com/en/stable/api/#custom-tests>`_
+.. py:function:: raise(error_message)
 
-.. _stdlib-imports-notice:
+   The ``raise`` function will result in an error containing the provided text.
 
-.. important::
+   Calling this will cause ``cylc validate`` to fail with the provided error
+   message and will prevent the workflow from being started. It's useful for
+   validating input template variables.
 
-   Only Python modules that are available in the environment used to run Cylc,
-   or the ``lib/python`` directory, can be imported inside custom globals, filters and tests.
-   You should avoid importing external modules that are not available in either
-   the standard library, Jinja2, Cylc, or Isodatetime,
-   as this could break between Cylc versions or when running on different systems.
+   .. describe:: Jinja2 Examples:
+
+      .. code-block:: cylc
+
+         {% if VARIABLE is not defined %}
+             {{ raise('VARIABLE must be defined for this workflow.') }}
+         {% endif %}
+
+.. _jinja2-assert:
+
+.. py:function:: assert(condition, error_message)
+
+   The ``assert`` function will raise an exception containing the text provided
+   in the second argument providing that the first argument evaluates as False.
+   The following example is equivalent to the "raise" example above.
+
+   Assertion errors will ``cylc validate`` to fail with the provided error
+   message and will prevent the workflow from being started. It's useful for
+   validating input template variables.
+
+   .. describe:: Jinja2 Examples:
+
+      .. code-block:: cylc
 
-Cylc provides several custom filters of its own:
+         {{ assert(VARIABLE is defined, 'VARIABLE must be defined for this workflow.') }}
+
+
+.. _user-guide.jinja2.cylc-builtin-filters:
+
+Cylc Built-in Filters
+^^^^^^^^^^^^^^^^^^^^^
 
 .. autosummary::
    :nosignatures:
@@ -273,6 +325,76 @@ Cylc provides several custom filters of its own:
 .. autofunction:: cylc.flow.jinja.filters.duration_as.duration_as
 
 
+.. _CustomJinja2Filters:
+.. _user-guide.jinja2.custom-extensions:
+
+Custom Jinja2 Extensions
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Custom Jinja2 globals, filters and tests can be defined within workflows.
+
+These extensions are Python modules containing a function with the same name
+as the module (e.g, a module called ``foo.py`` should contain a function called
+``foo``).
+
+Jinja2 globals go in the workflow :term:`source directory` in a subdirectory
+called ``Jinja2Globals``, filters in ``Jinja2Filters`` and tests in
+``Jinja2Tests``.
+
+This example defines one of each and demonstrates how to use them:
+
+.. code-block:: cylc
+   :caption: flow.cylc
+
+   #!Jinja2
+
+   # "globals" are regular Python functions
+   {{ square(5) }}
+
+   # filters are special functions which chain using the pipe character
+   {{ ('run', 1) | display_name }}
+
+   # tests are special functions which work with the "is" operator
+   {{ 42 is even }}
+
+.. code-block:: python
+   :caption: Jinja2Filters/display_name.py
+
+   def display_name(argument):
+       name, number = argument
+       return f'{name}_x{number:03d}'
+
+.. code-block:: python
+   :caption: Jinja2Globals/square.py
+
+   def square(number):
+       return number ** 2
+
+.. code-block:: python
+   :caption: Jinja2Tests/even.py
+
+   def even(number):
+       return number % 2 == 0
+
+
+.. seealso::
+
+   Jinja2 documentation:
+
+   - `Custom Filters <https://jinja.palletsprojects.com/en/stable/api/#custom-filters>`_
+   - `Custom Tests <https://jinja.palletsprojects.com/en/stable/api/#custom-tests>`_
+
+.. _stdlib-imports-notice:
+
+.. important::
+
+   Only Python modules that are available in the environment used to run Cylc,
+   or the ``lib/python`` directory, can be imported inside custom globals, filters and tests.
+   You should avoid importing external modules that are not available in either
+   the standard library, Jinja2, Cylc, or Isodatetime,
+   as this could break between Cylc versions or when running on different systems.
+
+
 Associative Arrays In Jinja2
 ----------------------------
 
@@ -506,47 +628,6 @@ For detail, see
 <https://jinja.palletsprojects.com/en/3.0.x/templates/#assignments>`_
 
 
-.. _Jinja2RaisingExceptions:
-
-Raising Exceptions
-------------------
-
-Cylc provides two functions for raising exceptions in Jinja2 code. These
-exceptions are raised when the :cylc:conf:`flow.cylc` file is loaded and will
-prevent a workflow from running.
-
-.. note::
-
-   These functions must be contained within ``{{`` Jinja2 print statements, not
-   ``{%`` code blocks.
-
-.. _jinja2-raise:
-
-Raise
-^^^^^
-
-The ``raise`` function will result in an error containing the provided text.
-
-.. code-block:: cylc
-
-   {% if not VARIABLE is defined %}
-       {{ raise('VARIABLE must be defined for this workflow.') }}
-   {% endif %}
-
-.. _jinja2-assert:
-
-Assert
-^^^^^^
-
-The ``assert`` function will raise an exception containing the text provided in
-the second argument providing that the first argument evaluates as False. The
-following example is equivalent to the "raise" example above.
-
-.. code-block:: cylc
-
-   {{ assert(VARIABLE is defined, 'VARIABLE must be defined for this workflow.') }}
-
-
 .. _jinja2.importing_python_modules:
 
 Importing Python modules