diff --git a/crowdsec-docs/docs/appsec/bot_detection/customization.md b/crowdsec-docs/docs/appsec/bot_detection/customization.md index 9103f1c58..3c463e175 100644 --- a/crowdsec-docs/docs/appsec/bot_detection/customization.md +++ b/crowdsec-docs/docs/appsec/bot_detection/customization.md @@ -221,6 +221,15 @@ Example file: The reverse-DNS confirmation used by `rdns` goes through the engine's DNS cache; see [`dns_cache`](/configuration/crowdsec_configuration.md#dns_cache) if you need to tune its TTL or size. +To check a definition without sending real traffic, use [`cscli bot match`](/cscli/cscli_bot_match.md). It runs the same matching against the `bots` files declared by your installed appsec-configs, or only by the ones you name, and tells you which definition matched and how: + +```bash +$ sudo cscli bot match --ip 66.249.66.1 --user-agent "Googlebot/2.1" mycorp/appsec-bot-challenge-known-bot +66.249.66.1 matches bot 'googlebot' (rdns: crawl-66-249-66-1.googlebot.com) from legit_bots/mybot.json +``` + +Reverse DNS is resolved from the machine you run it on, so run it where CrowdSec runs. + ### Block on a weak signal the default ignores The collection scores weak signals but weights them low on purpose, so none can cross the threshold alone. `accept_language` is worth 15 points against a default threshold of 75. It fires when the `Accept-Language` header disagrees with the browser's `navigator.language`: legitimate for some multilingual or proxied setups, unusual for a normal browser. If your traffic makes it worth enforcing, reject on it directly: diff --git a/crowdsec-docs/docs/appsec/bot_detection/hooks.md b/crowdsec-docs/docs/appsec/bot_detection/hooks.md index a14f3929f..7963f8f63 100644 --- a/crowdsec-docs/docs/appsec/bot_detection/hooks.md +++ b/crowdsec-docs/docs/appsec/bot_detection/hooks.md @@ -116,6 +116,8 @@ pre_eval: Once `ExemptFromChallenge(reason)` has flagged a request, `SendChallenge()` becomes a no-op for the rest of that request, so the exempted client is never challenged. +[`cscli bot match`](/cscli/cscli_bot_match.md) runs the same check from the command line, to find out whether a given IP and User-Agent would be recognized. + ### `ExemptFromChallenge` vs `GrantChallengeCookie` Both keep a client out of the challenge, but at different scopes: diff --git a/crowdsec-docs/docs/appsec/bot_detection/whats_included.md b/crowdsec-docs/docs/appsec/bot_detection/whats_included.md index 5e96207b5..ee8e1b1a6 100644 --- a/crowdsec-docs/docs/appsec/bot_detection/whats_included.md +++ b/crowdsec-docs/docs/appsec/bot_detection/whats_included.md @@ -138,7 +138,7 @@ $ sudo cscli alerts inspect 2019 -d +------------------+--------------------------------------------------------------+ ``` -`fingerprint_id` (the `fsid`) is stable per device profile, so it correlates submissions from the same client across IP changes. +`fingerprint_id` (the `fsid`) is stable per device profile, so it correlates submissions from the same client across IP changes. [`cscli bot fsid`](/cscli/cscli_bot_fsid.md) decodes it back into what it encodes: the bot-signal bitmasks, screen geometry, CPU and memory, language and timezone. The other signals are only kept as truncated hashes. The same alert in the Console, where most people will actually read it. Expanding the context shows the same fields as tags, with `score_reasons` spelling out how the client reached 115: diff --git a/crowdsec-docs/docs/cscli/cscli.md b/crowdsec-docs/docs/cscli/cscli.md index 64b108082..ec996c83e 100644 --- a/crowdsec-docs/docs/cscli/cscli.md +++ b/crowdsec-docs/docs/cscli/cscli.md @@ -35,6 +35,7 @@ cscli [flags] * [cscli allowlists](/cscli/cscli_allowlists.md) - Manage centralized allowlists * [cscli appsec-configs](/cscli/cscli_appsec-configs.md) - Manage hub appsec-configs * [cscli appsec-rules](/cscli/cscli_appsec-rules.md) - Manage hub appsec-rules +* [cscli bot](/cscli/cscli_bot.md) - Inspect bot detection: known-bot matching and fingerprint ids * [cscli bouncers](/cscli/cscli_bouncers.md) - Manage bouncers [requires local API] * [cscli capi](/cscli/cscli_capi.md) - Manage interaction with Central API (CAPI) * [cscli collections](/cscli/cscli_collections.md) - Manage hub collections diff --git a/crowdsec-docs/docs/cscli/cscli_bot.md b/crowdsec-docs/docs/cscli/cscli_bot.md new file mode 100644 index 000000000..04617a3af --- /dev/null +++ b/crowdsec-docs/docs/cscli/cscli_bot.md @@ -0,0 +1,37 @@ +--- +id: cscli_bot +title: cscli bot +--- +## cscli bot + +Inspect bot detection: known-bot matching and fingerprint ids + +``` +cscli bot [flags] +``` + +### Options + +``` + -h, --help help for bot +``` + +### Options inherited from parent commands + +``` + --color string Output color: yes, no, auto (default "auto") + -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") + --debug Set logging to debug + --error Set logging to error + --info Set logging to info + -o, --output string Output format: human, json, raw + --trace Set logging to trace + --warning Set logging to warning +``` + +### SEE ALSO + +* [cscli](/cscli/cscli.md) - cscli allows you to manage crowdsec +* [cscli bot fsid](/cscli/cscli_bot_fsid.md) - Decode a fingerprint id (FSID) back into a fingerprint +* [cscli bot match](/cscli/cscli_bot_match.md) - Check if a request would be recognized as a known bot + diff --git a/crowdsec-docs/docs/cscli/cscli_bot_fsid.md b/crowdsec-docs/docs/cscli/cscli_bot_fsid.md new file mode 100644 index 000000000..8bd046092 --- /dev/null +++ b/crowdsec-docs/docs/cscli/cscli_bot_fsid.md @@ -0,0 +1,53 @@ +--- +id: cscli_bot_fsid +title: cscli bot fsid +--- +## cscli bot fsid + +Decode a fingerprint id (FSID) back into a fingerprint + +### Synopsis + +Decode the FSID a challenged browser reports and rebuild the fingerprint object +from it. Only what the FSID encodes can be recovered: bitmasks, screen geometry, +cpu/memory, language and timezone. The remaining signals were folded into +truncated hashes and are shown as such. + +Use '-' to read the FSID from stdin. With -o json the reconstructed fingerprint +is printed instead of the summary. + +``` +cscli bot fsid [flags] +``` + +### Examples + +``` +cscli bot fsid FS1_000000000000000000000_00000h1f2e_1920x1080c8m8b1000h-3ab1_f... +cscli bot fsid -o json FS1_... +echo 'FS1_...' | cscli bot fsid - +``` + +### Options + +``` + -h, --help help for fsid +``` + +### Options inherited from parent commands + +``` + --color string Output color: yes, no, auto (default "auto") + -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") + --debug Set logging to debug + --error Set logging to error + --info Set logging to info + -o, --output string Output format: human, json, raw + --trace Set logging to trace + --warning Set logging to warning +``` + +### SEE ALSO + +* [cscli bot](/cscli/cscli_bot.md) - Inspect bot detection: known-bot matching and fingerprint ids + diff --git a/crowdsec-docs/docs/cscli/cscli_bot_match.md b/crowdsec-docs/docs/cscli/cscli_bot_match.md new file mode 100644 index 000000000..a9d9d4171 --- /dev/null +++ b/crowdsec-docs/docs/cscli/cscli_bot_match.md @@ -0,0 +1,51 @@ +--- +id: cscli_bot_match +title: cscli bot match +--- +## cscli bot match + +Check if a request would be recognized as a known bot + +### Synopsis + +Run the MatchKnownBot() expr helper against the 'bots' data files declared by +the given appsec-configs (all installed ones by default). Reverse DNS is resolved at +runtime, so the result depends on what the local resolver answers. + +``` +cscli bot match [appsec-config]... [flags] +``` + +### Examples + +``` +cscli bot match --ip 66.249.66.1 --user-agent "Googlebot/2.1" +cscli bot match --ip 1.2.3.4 --path /robots.txt crowdsecurity/appsec-default +``` + +### Options + +``` + -h, --help help for match + --ip string Source IP address of the request + --path string URI path of the request + --user-agent string User-Agent header of the request +``` + +### Options inherited from parent commands + +``` + --color string Output color: yes, no, auto (default "auto") + -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") + --debug Set logging to debug + --error Set logging to error + --info Set logging to info + -o, --output string Output format: human, json, raw + --trace Set logging to trace + --warning Set logging to warning +``` + +### SEE ALSO + +* [cscli bot](/cscli/cscli_bot.md) - Inspect bot detection: known-bot matching and fingerprint ids + diff --git a/crowdsec-docs/sidebars.ts b/crowdsec-docs/sidebars.ts index 86eb17188..8a67eb1e9 100644 --- a/crowdsec-docs/sidebars.ts +++ b/crowdsec-docs/sidebars.ts @@ -392,6 +392,15 @@ const sidebarsConfig: SidebarConfig = { "cscli/cscli_appsec-rules_upgrade", ], }, + { + type: "category", + label: "cscli bot", + link: { + type: "doc", + id: "cscli/cscli_bot", + }, + items: ["cscli/cscli_bot_fsid", "cscli/cscli_bot_match"], + }, { type: "category", label: "cscli bouncers", diff --git a/crowdsec-docs/versioned_docs/version-v1.8/appsec/bot_detection/customization.md b/crowdsec-docs/versioned_docs/version-v1.8/appsec/bot_detection/customization.md index ab4054a4e..3c463e175 100644 --- a/crowdsec-docs/versioned_docs/version-v1.8/appsec/bot_detection/customization.md +++ b/crowdsec-docs/versioned_docs/version-v1.8/appsec/bot_detection/customization.md @@ -2,7 +2,7 @@ id: customization title: "Bot Detection Customization & Recipes" sidebar_label: "Customization & recipes" -description: "Ready-to-use appsec-config customization for CrowdSec bot detection: challenge specific routes, tune scoring, and allow verified crawlers." +description: "Ready-to-use appsec-config overlays for CrowdSec bot detection: challenge specific routes, tune scoring, and allow verified crawlers without editing hub files." sidebar_position: 4 --- @@ -221,6 +221,15 @@ Example file: The reverse-DNS confirmation used by `rdns` goes through the engine's DNS cache; see [`dns_cache`](/configuration/crowdsec_configuration.md#dns_cache) if you need to tune its TTL or size. +To check a definition without sending real traffic, use [`cscli bot match`](/cscli/cscli_bot_match.md). It runs the same matching against the `bots` files declared by your installed appsec-configs, or only by the ones you name, and tells you which definition matched and how: + +```bash +$ sudo cscli bot match --ip 66.249.66.1 --user-agent "Googlebot/2.1" mycorp/appsec-bot-challenge-known-bot +66.249.66.1 matches bot 'googlebot' (rdns: crawl-66-249-66-1.googlebot.com) from legit_bots/mybot.json +``` + +Reverse DNS is resolved from the machine you run it on, so run it where CrowdSec runs. + ### Block on a weak signal the default ignores The collection scores weak signals but weights them low on purpose, so none can cross the threshold alone. `accept_language` is worth 15 points against a default threshold of 75. It fires when the `Accept-Language` header disagrees with the browser's `navigator.language`: legitimate for some multilingual or proxied setups, unusual for a normal browser. If your traffic makes it worth enforcing, reject on it directly: diff --git a/crowdsec-docs/versioned_docs/version-v1.8/appsec/bot_detection/hooks.md b/crowdsec-docs/versioned_docs/version-v1.8/appsec/bot_detection/hooks.md index a14f3929f..7963f8f63 100644 --- a/crowdsec-docs/versioned_docs/version-v1.8/appsec/bot_detection/hooks.md +++ b/crowdsec-docs/versioned_docs/version-v1.8/appsec/bot_detection/hooks.md @@ -116,6 +116,8 @@ pre_eval: Once `ExemptFromChallenge(reason)` has flagged a request, `SendChallenge()` becomes a no-op for the rest of that request, so the exempted client is never challenged. +[`cscli bot match`](/cscli/cscli_bot_match.md) runs the same check from the command line, to find out whether a given IP and User-Agent would be recognized. + ### `ExemptFromChallenge` vs `GrantChallengeCookie` Both keep a client out of the challenge, but at different scopes: diff --git a/crowdsec-docs/versioned_docs/version-v1.8/appsec/bot_detection/whats_included.md b/crowdsec-docs/versioned_docs/version-v1.8/appsec/bot_detection/whats_included.md index 422b78b35..ee8e1b1a6 100644 --- a/crowdsec-docs/versioned_docs/version-v1.8/appsec/bot_detection/whats_included.md +++ b/crowdsec-docs/versioned_docs/version-v1.8/appsec/bot_detection/whats_included.md @@ -2,7 +2,7 @@ id: whats_included title: "Bot Detection Default Configuration" sidebar_label: "Default Configuration" -description: "What the crowdsecurity/appsec-bot-challenge collection installs: the bot scoring and browser fingerprinting, with thresholds and the behavioral scenarios." +description: "What the crowdsecurity/appsec-bot-challenge collection installs: the two scoring and threshold appsec-configs, plus the behavioral scenarios it ships with." sidebar_position: 3 --- @@ -138,7 +138,7 @@ $ sudo cscli alerts inspect 2019 -d +------------------+--------------------------------------------------------------+ ``` -`fingerprint_id` (the `fsid`) is stable per device profile, so it correlates submissions from the same client across IP changes. +`fingerprint_id` (the `fsid`) is stable per device profile, so it correlates submissions from the same client across IP changes. [`cscli bot fsid`](/cscli/cscli_bot_fsid.md) decodes it back into what it encodes: the bot-signal bitmasks, screen geometry, CPU and memory, language and timezone. The other signals are only kept as truncated hashes. The same alert in the Console, where most people will actually read it. Expanding the context shows the same fields as tags, with `score_reasons` spelling out how the client reached 115: diff --git a/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli.md b/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli.md index 64b108082..ec996c83e 100644 --- a/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli.md +++ b/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli.md @@ -35,6 +35,7 @@ cscli [flags] * [cscli allowlists](/cscli/cscli_allowlists.md) - Manage centralized allowlists * [cscli appsec-configs](/cscli/cscli_appsec-configs.md) - Manage hub appsec-configs * [cscli appsec-rules](/cscli/cscli_appsec-rules.md) - Manage hub appsec-rules +* [cscli bot](/cscli/cscli_bot.md) - Inspect bot detection: known-bot matching and fingerprint ids * [cscli bouncers](/cscli/cscli_bouncers.md) - Manage bouncers [requires local API] * [cscli capi](/cscli/cscli_capi.md) - Manage interaction with Central API (CAPI) * [cscli collections](/cscli/cscli_collections.md) - Manage hub collections diff --git a/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli_bot.md b/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli_bot.md new file mode 100644 index 000000000..04617a3af --- /dev/null +++ b/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli_bot.md @@ -0,0 +1,37 @@ +--- +id: cscli_bot +title: cscli bot +--- +## cscli bot + +Inspect bot detection: known-bot matching and fingerprint ids + +``` +cscli bot [flags] +``` + +### Options + +``` + -h, --help help for bot +``` + +### Options inherited from parent commands + +``` + --color string Output color: yes, no, auto (default "auto") + -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") + --debug Set logging to debug + --error Set logging to error + --info Set logging to info + -o, --output string Output format: human, json, raw + --trace Set logging to trace + --warning Set logging to warning +``` + +### SEE ALSO + +* [cscli](/cscli/cscli.md) - cscli allows you to manage crowdsec +* [cscli bot fsid](/cscli/cscli_bot_fsid.md) - Decode a fingerprint id (FSID) back into a fingerprint +* [cscli bot match](/cscli/cscli_bot_match.md) - Check if a request would be recognized as a known bot + diff --git a/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli_bot_fsid.md b/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli_bot_fsid.md new file mode 100644 index 000000000..8bd046092 --- /dev/null +++ b/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli_bot_fsid.md @@ -0,0 +1,53 @@ +--- +id: cscli_bot_fsid +title: cscli bot fsid +--- +## cscli bot fsid + +Decode a fingerprint id (FSID) back into a fingerprint + +### Synopsis + +Decode the FSID a challenged browser reports and rebuild the fingerprint object +from it. Only what the FSID encodes can be recovered: bitmasks, screen geometry, +cpu/memory, language and timezone. The remaining signals were folded into +truncated hashes and are shown as such. + +Use '-' to read the FSID from stdin. With -o json the reconstructed fingerprint +is printed instead of the summary. + +``` +cscli bot fsid [flags] +``` + +### Examples + +``` +cscli bot fsid FS1_000000000000000000000_00000h1f2e_1920x1080c8m8b1000h-3ab1_f... +cscli bot fsid -o json FS1_... +echo 'FS1_...' | cscli bot fsid - +``` + +### Options + +``` + -h, --help help for fsid +``` + +### Options inherited from parent commands + +``` + --color string Output color: yes, no, auto (default "auto") + -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") + --debug Set logging to debug + --error Set logging to error + --info Set logging to info + -o, --output string Output format: human, json, raw + --trace Set logging to trace + --warning Set logging to warning +``` + +### SEE ALSO + +* [cscli bot](/cscli/cscli_bot.md) - Inspect bot detection: known-bot matching and fingerprint ids + diff --git a/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli_bot_match.md b/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli_bot_match.md new file mode 100644 index 000000000..a9d9d4171 --- /dev/null +++ b/crowdsec-docs/versioned_docs/version-v1.8/cscli/cscli_bot_match.md @@ -0,0 +1,51 @@ +--- +id: cscli_bot_match +title: cscli bot match +--- +## cscli bot match + +Check if a request would be recognized as a known bot + +### Synopsis + +Run the MatchKnownBot() expr helper against the 'bots' data files declared by +the given appsec-configs (all installed ones by default). Reverse DNS is resolved at +runtime, so the result depends on what the local resolver answers. + +``` +cscli bot match [appsec-config]... [flags] +``` + +### Examples + +``` +cscli bot match --ip 66.249.66.1 --user-agent "Googlebot/2.1" +cscli bot match --ip 1.2.3.4 --path /robots.txt crowdsecurity/appsec-default +``` + +### Options + +``` + -h, --help help for match + --ip string Source IP address of the request + --path string URI path of the request + --user-agent string User-Agent header of the request +``` + +### Options inherited from parent commands + +``` + --color string Output color: yes, no, auto (default "auto") + -c, --config string path to crowdsec config file (default "/etc/crowdsec/config.yaml") + --debug Set logging to debug + --error Set logging to error + --info Set logging to info + -o, --output string Output format: human, json, raw + --trace Set logging to trace + --warning Set logging to warning +``` + +### SEE ALSO + +* [cscli bot](/cscli/cscli_bot.md) - Inspect bot detection: known-bot matching and fingerprint ids + diff --git a/crowdsec-docs/versioned_sidebars/version-v1.8-sidebars.json b/crowdsec-docs/versioned_sidebars/version-v1.8-sidebars.json index d548c9bc7..642ef5886 100644 --- a/crowdsec-docs/versioned_sidebars/version-v1.8-sidebars.json +++ b/crowdsec-docs/versioned_sidebars/version-v1.8-sidebars.json @@ -399,6 +399,18 @@ "cscli/cscli_appsec-rules_upgrade" ] }, + { + "type": "category", + "label": "cscli bot", + "link": { + "type": "doc", + "id": "cscli/cscli_bot" + }, + "items": [ + "cscli/cscli_bot_fsid", + "cscli/cscli_bot_match" + ] + }, { "type": "category", "label": "cscli bouncers",