New fixture for enabling TRACE levels and new function for checking opmon value strings

[bieryAtFnal]

Description

We currently don't have a way to enable a TRACE debug message from within an integtest. Part of the changes in this PR add the ability to do that.

Other changes in this PR add the ability to check the contents of opmon metric value strings.

The recent motivation for adding these functionalities was the desire to create a regression test that verified that we don't get garbled opmon strings when the shutdown of the system takes a long time. However, some elements of that regression test (e.g. the ability to introduce artificial delays into certain locations in the DAQ system) need to be understood better before they can be reviewed and merged. But, these changes can be reviewed now since they are not the subject of various concerns, and they will likely be useful in other situations.

These changes are correlated with ones in the daqconf repo, DUNE-DAQ/daqconf#632.

Here are sample instructions for testing that these changes work as intended:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260804_A9 ${DATE_PREFIX}FDDevArtificialDelays_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevArtificialDelays_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/appfwk.git -b kbiery/artificial_delay_sample
git clone https://github.com/DUNE-DAQ/confmodel.git -b kbiery/artificial_delay_sample
git clone https://github.com/DUNE-DAQ/daqconf.git -b kbiery/artificial_delay_sample
git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b kbiery/artificial_delay_sample
git clone https://github.com/DUNE-DAQ/ipm.git -b kbiery/artificial_delay_sample
git clone https://github.com/DUNE-DAQ/utilities.git -b kbiery/artificial_delay_sample

git clone https://github.com/DUNE-DAQ/opmonlib.git -b kbiery/graceful_termination_integtest

git clone https://github.com/DUNE-DAQ/appmodel.git -b develop
git clone https://github.com/DUNE-DAQ/datahandlinglibs.git -b develop
git clone https://github.com/DUNE-DAQ/dfmodules.git -b develop
git clone https://github.com/DUNE-DAQ/fdreadoutlibs.git -b develop
git clone https://github.com/DUNE-DAQ/fdreadoutmodules.git -b develop
git clone https://github.com/DUNE-DAQ/hdf5libs.git -b develop
git clone https://github.com/DUNE-DAQ/hsilibs.git -b develop
git clone https://github.com/DUNE-DAQ/iomanager.git -b develop
git clone https://github.com/DUNE-DAQ/trigger.git -b develop

cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b kbiery/metric_string_check
cd ..

. ./env.sh
dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh -r all -k 'graceful|listrev|disabled_tpg|tpstream'

echo ""
echo -e "\U1F535 \U2705 Note that the integtests ran successfully. \U2705 \U1F535"
echo ""
echo ""

Here are sample instructions for testing that these changes do not adversely affect existing functionality:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260804_A9 ${DATE_PREFIX}FDDevIntgTraceAndMetricString_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevIntgTraceAndMetricString_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqconf.git -b kbiery/artificial_delay_sample
git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b develop

cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b kbiery/metric_string_check
cd ..

. ./env.sh
dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh -r all -k '3df|listrev|disabled_tpg|tpstream|tpreplay'

echo ""
echo -e "\U1F535 \U2705 Note that the integtests ran successfully. \U2705 \U1F535"
echo ""
echo ""

Type of change

• New feature or enhancement (non-breaking change which adds functionality)

Testing checklist

• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

• Code is commented where needed, particularly in hard-to-understand areas