diff --git a/src/reference/changes.rst b/src/reference/changes.rst index 7d09920ad4..12d6ce0900 100644 --- a/src/reference/changes.rst +++ b/src/reference/changes.rst @@ -13,10 +13,27 @@ Cylc 8.6 :class: hint :cylc-flow: `8.6 `__ - :cylc-ui: `2.11 `__ - :cylc-uiserver: `1.8 `__ + :cylc-ui: `2.x `__ + :cylc-uiserver: `1.9 `__ :cylc-rose: `1.7 `__ - :rose: `2.6 `__ + :rose: `2.7 `__ + + +Warnings during Jinja2 preprocessing +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. versionadded:: cylc-flow 8.6.6 + +Python warnings raised during Jinja2 preprocessing are now logged when running +relevant Cylc commands (e.g. ``cylc validate`` and ``cylc play``). + +.. warning:: + + Cylc 8.7.0 will move from Jinja2 3.0 to 3.1, which removes a number of + deprecated features. + This should only affect you if you use :ref:`user-guide.jinja2.custom-extensions`. + Warnings for these deprecations are now logged, but were not previously. + See the `Jinja 3.1 release notes `_. UI sidebar improvements diff --git a/src/user-guide/writing-workflows/jinja2.rst b/src/user-guide/writing-workflows/jinja2.rst index d2d5157859..ce09e9e190 100644 --- a/src/user-guide/writing-workflows/jinja2.rst +++ b/src/user-guide/writing-workflows/jinja2.rst @@ -176,88 +176,140 @@ installed workflow at run time: {{ CYLC_WORKFLOW_RUN_DIR | default("not-defined") }} -Environment Variables ---------------------- +.. _Jinja2Filters: + +Jinja2 Filters, Tests and Globals +--------------------------------- + +.. _Jinja2 Built-in Globals: https://jinja.palletsprojects.com/en/stable/templates/#list-of-global-functions +.. _Jinja2 Built-in Filters: https://jinja.palletsprojects.com/en/stable/templates/#list-of-builtin-filters +.. _Jinja2 Built-in Tests: https://jinja.palletsprojects.com/en/stable/templates/#builtin-tests + +Jinja2 provides "globals", "filters" and "tests" which can be helpful in +workflow writing. -Cylc automatically imports the parse-time environment to the template -processor's global namespace (see :ref:`Jinja2Filters`), -in a dictionary called ``environ``: +Globals + Regular Python functions. + + :Jinja2 builtins: `Jinja2 Built-in Globals`_ + :Cylc builtins: :ref:`user-guide.jinja2.cylc-builtin-globals` + :Custom directory: :ref:`Jinja2Globals ` +Filters + Special functions which "chain" using the pipe character (``|``). + + :Jinja2 builtins: `Jinja2 Built-in Filters`_ + :Cylc builtins: :ref:`user-guide.jinja2.cylc-builtin-filters` + :Custom directory: :ref:`Jinja2Filters ` +Tests + Special functions which work with the ``is`` operator. + + :Jinja2 builtins: `Jinja2 Built-in Tests`_ + :Custom directory: :ref:`Jinja2Tests ` + +For example, this :cylc:conf:`flow.cylc` file uses the +:py:func:`pad ` filter to help write out +task definitions: .. code-block:: cylc - #!Jinja2 - #... [runtime] - [[root]] - [[[environment]]] - HOME_DIR_ON_WORKFLOW_HOST = {{environ['HOME']}} + {% for x in range(3) %} + [[task_{{ x | pad(3) }}]] + script = sleep {{ x }} + {% endfor %} -.. important:: +The Jinja2 would be expanded like so: - The environment is read during configuration parsing. It is not the run time - job environment. +.. code-block:: cylc -.. _Jinja2Filters: + [runtime] + [[x_001]] + script = sleep 1 + [[x_002]] + script = sleep 2 + [[x_003]] + script = sleep 3 -Jinja2 Filters, Tests and Globals ---------------------------------- +In addition to the built-ins that Jinja2 and Cylc provide, you can also define +your own custom filters (see :ref:`user-guide.jinja2.custom-extensions`). -Jinja2 has three namespaces that separate "globals", "filters" and "tests". - - Globals are template-wide variables and functions. Cylc extends this namespace - with the ``environ`` dictionary above, and - :ref:`raise ` and :ref:`assert ` - functions for raising exceptions to abort Cylc config parsing. +.. _user-guide.jinja2.cylc-builtin-globals: - - Filters can be used to modify variable values and are applied using pipe - notation. For example, the built-in ``trim`` filter strips leading - and trailing white space from a string: +Cylc Built-in Globals +^^^^^^^^^^^^^^^^^^^^^ - .. code-block:: cylc +.. list-table:: - {% set MyString = " dog " %} - {{ MyString | trim() }} # "dog" + * - :py:data:`environ` + - Access environment variables. + * - :py:func:`raise` + - Raise an error. + * - :py:func:`assert` + - Raise an error if a condition is not met. - - Variable values can be tested using the ``is`` keyword followed by - the name of the test, e.g. ``{% if VARIABLE is defined %}``. +.. _jinja2-environ: -See `Jinja2 documentation `_ -for available built-in globals, filters and tests. +.. py:data:: environ -.. _CustomJinja2Filters: + Provides access to environment variables. -Custom Filters, Tests and Globals -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Note, these are the "parse-time" environment variables - i.e, the environment + that is set when the workflow's :cylc:conf:`flow.cylc` file is processed. + This happens when a workflow is validated or started, not when jobs are + submitted. Jinja2 does not have access to dynamic environment variables + available to jobs. -Cylc also supports custom Jinja2 filters, tests and globals. -A custom filter or test is a single Python function in a source file -with the same name as the function (plus ``.py`` extension). -Likewise, a custom global is a single Python variable or function. -These must be located in a subdirectory of the :term:`source directory` called -``Jinja2Filters``, ``Jinja2Tests``, or ``Jinja2Globals`` respectively. + .. describe:: Jinja2 Examples: -In the argument list of a filter or test function, the first argument is -the variable value to be filtered or tested, and subsequent arguments can be -whatever is needed. + .. code-block:: cylc -.. seealso:: + [runtime] + [[root]] + [[[environment]]] + HOME_DIR_ON_WORKFLOW_HOST = {{environ['HOME']}} - Jinja2 documentation: +.. _jinja2-raise: - - `Custom Filters `_ - - `Custom Tests `_ +.. py:function:: raise(error_message) -.. _stdlib-imports-notice: + The ``raise`` function will result in an error containing the provided text. -.. important:: + Calling this will cause ``cylc validate`` to fail with the provided error + message and will prevent the workflow from being started. It's useful for + validating input template variables. - Only Python modules that are available in the environment used to run Cylc, - or the ``lib/python`` directory, can be imported inside custom globals, filters and tests. - You should avoid importing external modules that are not available in either - the standard library, Jinja2, Cylc, or Isodatetime, - as this could break between Cylc versions or when running on different systems. + .. describe:: Jinja2 Examples: + + .. code-block:: cylc + + {% if VARIABLE is not defined %} + {{ raise('VARIABLE must be defined for this workflow.') }} + {% endif %} + +.. _jinja2-assert: + +.. py:function:: assert(condition, error_message) + + The ``assert`` function will raise an exception containing the text provided + in the second argument providing that the first argument evaluates as False. + The following example is equivalent to the "raise" example above. + + Assertion errors will ``cylc validate`` to fail with the provided error + message and will prevent the workflow from being started. It's useful for + validating input template variables. + + .. describe:: Jinja2 Examples: + + .. code-block:: cylc -Cylc provides several custom filters of its own: + {{ assert(VARIABLE is defined, 'VARIABLE must be defined for this workflow.') }} + + +.. _user-guide.jinja2.cylc-builtin-filters: + +Cylc Built-in Filters +^^^^^^^^^^^^^^^^^^^^^ .. autosummary:: :nosignatures: @@ -273,6 +325,76 @@ Cylc provides several custom filters of its own: .. autofunction:: cylc.flow.jinja.filters.duration_as.duration_as +.. _CustomJinja2Filters: +.. _user-guide.jinja2.custom-extensions: + +Custom Jinja2 Extensions +^^^^^^^^^^^^^^^^^^^^^^^^ + +Custom Jinja2 globals, filters and tests can be defined within workflows. + +These extensions are Python modules containing a function with the same name +as the module (e.g, a module called ``foo.py`` should contain a function called +``foo``). + +Jinja2 globals go in the workflow :term:`source directory` in a subdirectory +called ``Jinja2Globals``, filters in ``Jinja2Filters`` and tests in +``Jinja2Tests``. + +This example defines one of each and demonstrates how to use them: + +.. code-block:: cylc + :caption: flow.cylc + + #!Jinja2 + + # "globals" are regular Python functions + {{ square(5) }} + + # filters are special functions which chain using the pipe character + {{ ('run', 1) | display_name }} + + # tests are special functions which work with the "is" operator + {{ 42 is even }} + +.. code-block:: python + :caption: Jinja2Filters/display_name.py + + def display_name(argument): + name, number = argument + return f'{name}_x{number:03d}' + +.. code-block:: python + :caption: Jinja2Globals/square.py + + def square(number): + return number ** 2 + +.. code-block:: python + :caption: Jinja2Tests/even.py + + def even(number): + return number % 2 == 0 + + +.. seealso:: + + Jinja2 documentation: + + - `Custom Filters `_ + - `Custom Tests `_ + +.. _stdlib-imports-notice: + +.. important:: + + Only Python modules that are available in the environment used to run Cylc, + or the ``lib/python`` directory, can be imported inside custom globals, filters and tests. + You should avoid importing external modules that are not available in either + the standard library, Jinja2, Cylc, or Isodatetime, + as this could break between Cylc versions or when running on different systems. + + Associative Arrays In Jinja2 ---------------------------- @@ -506,47 +628,6 @@ For detail, see `_ -.. _Jinja2RaisingExceptions: - -Raising Exceptions ------------------- - -Cylc provides two functions for raising exceptions in Jinja2 code. These -exceptions are raised when the :cylc:conf:`flow.cylc` file is loaded and will -prevent a workflow from running. - -.. note:: - - These functions must be contained within ``{{`` Jinja2 print statements, not - ``{%`` code blocks. - -.. _jinja2-raise: - -Raise -^^^^^ - -The ``raise`` function will result in an error containing the provided text. - -.. code-block:: cylc - - {% if not VARIABLE is defined %} - {{ raise('VARIABLE must be defined for this workflow.') }} - {% endif %} - -.. _jinja2-assert: - -Assert -^^^^^^ - -The ``assert`` function will raise an exception containing the text provided in -the second argument providing that the first argument evaluates as False. The -following example is equivalent to the "raise" example above. - -.. code-block:: cylc - - {{ assert(VARIABLE is defined, 'VARIABLE must be defined for this workflow.') }} - - .. _jinja2.importing_python_modules: Importing Python modules