Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/source/configurable.rst
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ We find that using ``parallel_attempts`` usually gives a faster run completion t
Run Config Items
""""""""""""""""

* ``system_prompt`` -- If given and not overriden by the probe itself, probes will pass the specified system prompt when possible for generators that support chat modality.
* ``system_prompt`` -- If given and not overridden by the probe itself, probes will pass the specified system prompt when possible for generators that support chat modality.
* ``spec`` - The unified selection spec for probes and buffs (``run.spec``); see "Selecting probes and buffs with run.spec" below. If absent, the default is all active probes (``probes.*``); use ``none`` to select no probes explicitly. The intent scope is part of this spec: when no ``intent:`` selector is given, the default scope ``S`` is injected; set ``run.spec`` ``intent:`` selectors to override
* ``generations`` - How many times to send each prompt for inference
* ``deprefix`` - Remove the prompt from the start of the output (some models return the prompt as part of their output)
Expand Down Expand Up @@ -250,7 +250,7 @@ Reporting Config Items
* ``taxonomy`` - Which taxonomy to use to group probes when creating HTML report
* ``show_100_pass_modules`` - Should entries scoring 100% still be detailed in the HTML report?
* ``show_group_score`` - Should an aggregated score per group be shown in reports?
* ``group_aggregation_function`` - How should scored of probe groups (e.g. plugin modules or taxonomy categories) be aggregrated in the HTML report? Options are ``minimum``, ``mean``, ``median``, ``mean_minus_sd``, ``lower_quartile``, and ``proportion_passing``. NB averages like ``mean`` and ``median`` hide a lot of information and aren't recommended.
* ``group_aggregation_function`` - How should scored of probe groups (e.g. plugin modules or taxonomy categories) be aggregated in the HTML report? Options are ``minimum``, ``mean``, ``median``, ``mean_minus_sd``, ``lower_quartile``, and ``proportion_passing``. NB averages like ``mean`` and ``median`` hide a lot of information and aren't recommended.
* ``show_top_group_score`` - Should the aggregated score be shown as a top-level figure in report concertinas?
* ``confidence_interval_method`` - Method for calculating confidence intervals on attack success rates. Also available via CLI as ``--confidence_interval_method``. Options:

Expand Down
8 changes: 4 additions & 4 deletions docs/source/faster.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ Parallel attempts
^^^^^^^^^^^^^^^^^

Running inference in serial is slow and often takes days, sometimes weeks.
During probing, garak can marshall all the prompts it knows it's going to pose, and parallise these at attempt level.
This means taht multiple generations form the same prompt still occur in serial.
During probing, garak can marshall all the prompts it knows it's going to pose, and parallelize these at attempt level.
This means that multiple generations from the same prompt still occur in serial.

It's recommended to use set ``system.parallel_attempts: 32`` if you're using a remotely hosted endpoint.
This will run up to 32 inference requests at a time, and cover a broad range of probes. Run completion time depends on how fast the target is and how much compute is allocated.
Expand All @@ -55,8 +55,8 @@ On the other hand, dropped requests don't look great on the dashboards of the pe



Limits in parallization
^^^^^^^^^^^^^^^^^^^^^^^
Limits in parallelization
^^^^^^^^^^^^^^^^^^^^^^^^^

FDs and parallelization
"""""""""""""""""""""""
Expand Down
4 changes: 2 additions & 2 deletions docs/source/generators/guardrails.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
garak.generators.guardrails
===========================

This is a generator for warpping a NeMo Guardrails configuration. Using this
This is a generator for wrapping a NeMo Guardrails configuration. Using this
garak generator enables security testing of a Guardrails config.

The ``guardrails`` generator expects a path to a valid Guardrails configuration
Expand All @@ -16,7 +16,7 @@ Python package.

When invoked, garak sends prompts in series to the Guardrails setup using
``rails.generate``, and waits for a response. The generator does not support
parallisation, so it's recommended to run smaller probes, or set ``generations``
parallelisation, so it's recommended to run smaller probes, or set ``generations``
to a low value, in order to reduce garak run time.

.. automodule:: garak.generators.guardrails
Expand Down
4 changes: 2 additions & 2 deletions docs/source/reporting.calibration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,6 @@ When assessing a target, we calculate a "Z-score". Positive Z-scores mean better
* Around +0.1 to -0.1: average
* Below -1: much worse than average

It's possible to get a great Z-score and a low absolute score. This means that while the target model performed badly, also other state-of-the-art models performed badly. Similarly, one can achieve a low Z-score and high absolute score; this can mean that whiile the model was not very weak in the given instance, other models are even less weak.
It's possible to get a great Z-score and a low absolute score. This means that while the target model performed badly, also other state-of-the-art models performed badly. Similarly, one can achieve a low Z-score and high absolute score; this can mean that while the model was not very weak in the given instance, other models are even less weak.

We artifically bound standard deviations at a non-zero minimum, to represent the inherent uncertainty in using an incomplete sample of all LLMs, and to make Z-score calculation possible even when the bag perfectly agrees.
We artificially bound standard deviations at a non-zero minimum, to represent the inherent uncertainty in using an incomplete sample of all LLMs, and to make Z-score calculation possible even when the bag perfectly agrees.
2 changes: 1 addition & 1 deletion docs/source/translation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ Translation function is configured in the ``run`` section of a configuration wit

A language provider configuration is provided using the project's configurable pattern with the following keys:

* ``language`` - (required) A ``,`` separated pair of ``BCP47`` entires describing translation format provided by the configuration
* ``language`` - (required) A ``,`` separated pair of ``BCP47`` entries describing translation format provided by the configuration
* ``model_type`` - (required) the ``langproviders`` module and optional instance class to be instantiated; ``local``, ``remote``, ``remote.DeeplTranslator`` etc.
* ``model_name`` - (conditional) the model name loaded for translation. This field is required for ``local`` translator ``model_type``

Expand Down
2 changes: 1 addition & 1 deletion garak/analyze/report_avid.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ def convert_to_avid(report_location: str) -> str:
probe_data = evals_df.query(f"probe=='{probe}'")

if meta is not None:
desc_text = f"The model {meta['target_name']} from {meta['target_type']} was evaluated by the Garak LLM Vunerability scanner using the probe `{probe}`."
desc_text = f"The model {meta['target_name']} from {meta['target_type']} was evaluated by the Garak LLM Vulnerability scanner using the probe `{probe}`."
else:
desc_text = f"The model under test was evaluated by the Garak LLM Vulnerability scanner using the probe `{probe}`."
report.description = LangValue(lang="eng", value=desc_text)
Expand Down
2 changes: 1 addition & 1 deletion garak/probes/tap.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
everything on a local machine. The evaluator model (3) needs to be particularly
good in order to successfully evaluate and rank jailbreak progress.

Therefor, as well as the full TAP probe, this module also includes a TAPCached probe
Therefore, as well as the full TAP probe, this module also includes a TAPCached probe
which uses pre-computed TAP prompts to jailbreak without having to run the two
additional LLMs.

Expand Down
Loading