diff --git a/docs/source/configurable.rst b/docs/source/configurable.rst index a0a02456f..733b8307e 100644 --- a/docs/source/configurable.rst +++ b/docs/source/configurable.rst @@ -125,7 +125,7 @@ We find that using ``parallel_attempts`` usually gives a faster run completion t Run Config Items """""""""""""""" -* ``system_prompt`` -- If given and not overriden by the probe itself, probes will pass the specified system prompt when possible for generators that support chat modality. +* ``system_prompt`` -- If given and not overridden by the probe itself, probes will pass the specified system prompt when possible for generators that support chat modality. * ``spec`` - The unified selection spec for probes and buffs (``run.spec``); see "Selecting probes and buffs with run.spec" below. If absent, the default is all active probes (``probes.*``); use ``none`` to select no probes explicitly. The intent scope is part of this spec: when no ``intent:`` selector is given, the default scope ``S`` is injected; set ``run.spec`` ``intent:`` selectors to override * ``generations`` - How many times to send each prompt for inference * ``deprefix`` - Remove the prompt from the start of the output (some models return the prompt as part of their output) @@ -250,7 +250,7 @@ Reporting Config Items * ``taxonomy`` - Which taxonomy to use to group probes when creating HTML report * ``show_100_pass_modules`` - Should entries scoring 100% still be detailed in the HTML report? * ``show_group_score`` - Should an aggregated score per group be shown in reports? -* ``group_aggregation_function`` - How should scored of probe groups (e.g. plugin modules or taxonomy categories) be aggregrated in the HTML report? Options are ``minimum``, ``mean``, ``median``, ``mean_minus_sd``, ``lower_quartile``, and ``proportion_passing``. NB averages like ``mean`` and ``median`` hide a lot of information and aren't recommended. +* ``group_aggregation_function`` - How should scored of probe groups (e.g. plugin modules or taxonomy categories) be aggregated in the HTML report? Options are ``minimum``, ``mean``, ``median``, ``mean_minus_sd``, ``lower_quartile``, and ``proportion_passing``. NB averages like ``mean`` and ``median`` hide a lot of information and aren't recommended. * ``show_top_group_score`` - Should the aggregated score be shown as a top-level figure in report concertinas? * ``confidence_interval_method`` - Method for calculating confidence intervals on attack success rates. Also available via CLI as ``--confidence_interval_method``. Options: diff --git a/docs/source/faster.rst b/docs/source/faster.rst index 768886f43..700d93310 100644 --- a/docs/source/faster.rst +++ b/docs/source/faster.rst @@ -42,8 +42,8 @@ Parallel attempts ^^^^^^^^^^^^^^^^^ Running inference in serial is slow and often takes days, sometimes weeks. -During probing, garak can marshall all the prompts it knows it's going to pose, and parallise these at attempt level. -This means taht multiple generations form the same prompt still occur in serial. +During probing, garak can marshall all the prompts it knows it's going to pose, and parallelize these at attempt level. +This means that multiple generations from the same prompt still occur in serial. It's recommended to use set ``system.parallel_attempts: 32`` if you're using a remotely hosted endpoint. This will run up to 32 inference requests at a time, and cover a broad range of probes. Run completion time depends on how fast the target is and how much compute is allocated. @@ -55,8 +55,8 @@ On the other hand, dropped requests don't look great on the dashboards of the pe -Limits in parallization -^^^^^^^^^^^^^^^^^^^^^^^ +Limits in parallelization +^^^^^^^^^^^^^^^^^^^^^^^^^ FDs and parallelization """"""""""""""""""""""" diff --git a/docs/source/generators/guardrails.rst b/docs/source/generators/guardrails.rst index b0c5a74e0..c5f6ffeff 100644 --- a/docs/source/generators/guardrails.rst +++ b/docs/source/generators/guardrails.rst @@ -1,7 +1,7 @@ garak.generators.guardrails =========================== -This is a generator for warpping a NeMo Guardrails configuration. Using this +This is a generator for wrapping a NeMo Guardrails configuration. Using this garak generator enables security testing of a Guardrails config. The ``guardrails`` generator expects a path to a valid Guardrails configuration @@ -16,7 +16,7 @@ Python package. When invoked, garak sends prompts in series to the Guardrails setup using ``rails.generate``, and waits for a response. The generator does not support -parallisation, so it's recommended to run smaller probes, or set ``generations`` +parallelisation, so it's recommended to run smaller probes, or set ``generations`` to a low value, in order to reduce garak run time. .. automodule:: garak.generators.guardrails diff --git a/docs/source/reporting.calibration.rst b/docs/source/reporting.calibration.rst index 239e85ae3..de899b305 100644 --- a/docs/source/reporting.calibration.rst +++ b/docs/source/reporting.calibration.rst @@ -34,6 +34,6 @@ When assessing a target, we calculate a "Z-score". Positive Z-scores mean better * Around +0.1 to -0.1: average * Below -1: much worse than average -It's possible to get a great Z-score and a low absolute score. This means that while the target model performed badly, also other state-of-the-art models performed badly. Similarly, one can achieve a low Z-score and high absolute score; this can mean that whiile the model was not very weak in the given instance, other models are even less weak. +It's possible to get a great Z-score and a low absolute score. This means that while the target model performed badly, also other state-of-the-art models performed badly. Similarly, one can achieve a low Z-score and high absolute score; this can mean that while the model was not very weak in the given instance, other models are even less weak. -We artifically bound standard deviations at a non-zero minimum, to represent the inherent uncertainty in using an incomplete sample of all LLMs, and to make Z-score calculation possible even when the bag perfectly agrees. +We artificially bound standard deviations at a non-zero minimum, to represent the inherent uncertainty in using an incomplete sample of all LLMs, and to make Z-score calculation possible even when the bag perfectly agrees. diff --git a/docs/source/translation.rst b/docs/source/translation.rst index 60634a27e..6621f572a 100644 --- a/docs/source/translation.rst +++ b/docs/source/translation.rst @@ -79,7 +79,7 @@ Translation function is configured in the ``run`` section of a configuration wit A language provider configuration is provided using the project's configurable pattern with the following keys: -* ``language`` - (required) A ``,`` separated pair of ``BCP47`` entires describing translation format provided by the configuration +* ``language`` - (required) A ``,`` separated pair of ``BCP47`` entries describing translation format provided by the configuration * ``model_type`` - (required) the ``langproviders`` module and optional instance class to be instantiated; ``local``, ``remote``, ``remote.DeeplTranslator`` etc. * ``model_name`` - (conditional) the model name loaded for translation. This field is required for ``local`` translator ``model_type`` diff --git a/garak/analyze/report_avid.py b/garak/analyze/report_avid.py index 7448f1cfc..74bd15dd2 100644 --- a/garak/analyze/report_avid.py +++ b/garak/analyze/report_avid.py @@ -68,7 +68,7 @@ def convert_to_avid(report_location: str) -> str: probe_data = evals_df.query(f"probe=='{probe}'") if meta is not None: - desc_text = f"The model {meta['target_name']} from {meta['target_type']} was evaluated by the Garak LLM Vunerability scanner using the probe `{probe}`." + desc_text = f"The model {meta['target_name']} from {meta['target_type']} was evaluated by the Garak LLM Vulnerability scanner using the probe `{probe}`." else: desc_text = f"The model under test was evaluated by the Garak LLM Vulnerability scanner using the probe `{probe}`." report.description = LangValue(lang="eng", value=desc_text) diff --git a/garak/probes/tap.py b/garak/probes/tap.py index e5375cfd0..c2fcc6131 100644 --- a/garak/probes/tap.py +++ b/garak/probes/tap.py @@ -34,7 +34,7 @@ everything on a local machine. The evaluator model (3) needs to be particularly good in order to successfully evaluate and rank jailbreak progress. -Therefor, as well as the full TAP probe, this module also includes a TAPCached probe +Therefore, as well as the full TAP probe, this module also includes a TAPCached probe which uses pre-computed TAP prompts to jailbreak without having to run the two additional LLMs.