From 306450a4b6f78e08cfcaa10166778bad22ed476f Mon Sep 17 00:00:00 2001 From: Guillaume BOURGEOIS Date: Tue, 8 Sep 2026 00:35:27 +0200 Subject: [PATCH] fix(rapid7-insightidr): rule_keys_of_interest is an open vocabulary (v1.5.0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both mappers read that field through a hardcoded list of keys — hostname, process.cmd_line, parent_process.*, user — taken from the one alert this work was built against, a process_start_event detection. Every entry is a JSONata predicate on an exact key, so a rule emitting anything else matches none of them. A real alert from a second tenant showed what that costs. It is a custom ingress_auth rule, and its keys are: account locuz_hma@totogroup.net source_ip 183.82.103.101 geoip_country_code IN result SUCCESS Not one appears in the hardcoded list. The alert was fetched, inlined and mapped, and produced an incident as empty as the ones this integration set out to fix — with the account and the source IP sitting in the payload, unread. Two changes: - The whole list is flattened verbatim into `detection`, and rule_matching_keys into `detection_match`. Keys nobody anticipated now arrive intact, and they reach the AI analysis, which reads the mapped context and never `raw`. - account, source_ip, destination_ip, geoip_country_code and result gain explicit OCSF entries. The explicit mappings are not redundant with the flattening: a value under `detection.` is only ever seen by the IOC regex, so an account there would be indexed as an email, or not at all, rather than as a user. Typing an entity correctly still requires naming its field. evidences[] carries the flattened keys too, so the same holds for the second and third alert of an investigation, not only the first. Verified against both rule shapes end to end. The process rule yields the two hosts it did before; the authentication rule now yields the source IP and the account, where it previously yielded nothing. Co-Authored-By: Claude Opus 5 (1M context) --- integrations/rapid7-insightidr/manifest.yaml | 4 +-- .../mappers/get_investigations_full.yaml | 26 ++++++++++++++++++- .../mappers/search_alerts.yaml | 23 ++++++++++++++++ 3 files changed, 50 insertions(+), 3 deletions(-) diff --git a/integrations/rapid7-insightidr/manifest.yaml b/integrations/rapid7-insightidr/manifest.yaml index 59a35f7..ad2cc07 100644 --- a/integrations/rapid7-insightidr/manifest.yaml +++ b/integrations/rapid7-insightidr/manifest.yaml @@ -1,8 +1,8 @@ id: rapid7_insightidr name: Rapid7 InsightIDR -version: 1.4.1 +version: 1.5.0 description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search." -changelog: "1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." +changelog: "1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." category: siem # Per-instance configuration. The Insight platform region selects the API host diff --git a/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml b/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml index 6aad5c7..5411825 100644 --- a/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml +++ b/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml @@ -101,6 +101,29 @@ ocsf: # because rules are not consistent about which they emit. - { source_path: "alerts[0].rule_keys_of_interest[key='user'].values[0]", ocsf_field: "actor.user.name" } - { source_path: "alerts[0].rule_keys_of_interest[key='username'].values[0]", ocsf_field: "actor.user.name" } + # ── Every key the rule found interesting ────────────────────────── + # rule_keys_of_interest is an OPEN vocabulary: each detection rule names its own + # keys. A process_start_event rule emits hostname / process.cmd_line / + # parent_process.*; an ingress_auth rule emits account / source_ip / + # geoip_country_code / result and none of the above. The explicit entries below + # can therefore only ever cover the rules someone has already seen — on any + # other rule they all resolve to nothing and the incident is empty again, which + # is the exact failure this integration started from. + # + # So the whole list is also flattened, verbatim, into one object. Keys nobody + # anticipated arrive intact and are visible to the analyst and to the AI + # analysis, which reads the mapped context but never `raw`. The explicit + # mappings stay because they are what types an entity correctly — a value under + # `detection.` is only ever matched by the IOC regex, so an account there would + # never be indexed as a user. + - { source_path: "alerts[0].rule_keys_of_interest{key: values[0]}", ocsf_field: "detection" } + - { source_path: "alerts[0].rule_matching_keys{key: values[0]}", ocsf_field: "detection_match" } + # ── Keys seen on non-process rules ──────────────────────────────── + - { source_path: "alerts[0].rule_keys_of_interest[key='account'].values[0]", ocsf_field: "actor.user.account.name" } + - { source_path: "alerts[0].rule_keys_of_interest[key='source_ip'].values[0]", ocsf_field: "src_endpoint.ip" } + - { source_path: "alerts[0].rule_keys_of_interest[key='destination_ip'].values[0]", ocsf_field: "dst_endpoint.ip" } + - { source_path: "alerts[0].rule_keys_of_interest[key='geoip_country_code'].values[0]", ocsf_field: "src_endpoint.location.country" } + - { source_path: "alerts[0].rule_keys_of_interest[key='result'].values[0]", ocsf_field: "status_detail" } # ── Every alert, not just the first ─────────────────────────────── # An investigation aggregates alerts, and the fields above only ever read alerts[0]. # This entry lands the whole set as OCSF-shaped objects, so the host and the process of @@ -119,6 +142,7 @@ ocsf: "name": rule_keys_of_interest[key='process.name'].values[0], "cmd_line": rule_keys_of_interest[key='process.cmd_line'].values[0] }, - "attacks": { "technique": { "uid": rule.mitre_tcodes[$substring($, 0, 2) != 'TA'][0] } } + "attacks": { "technique": { "uid": rule.mitre_tcodes[$substring($, 0, 2) != 'TA'][0] } }, + "detection": rule_keys_of_interest{key: values[0]} } ocsf_field: "evidences" diff --git a/integrations/rapid7-insightidr/mappers/search_alerts.yaml b/integrations/rapid7-insightidr/mappers/search_alerts.yaml index 7699eff..036ce89 100644 --- a/integrations/rapid7-insightidr/mappers/search_alerts.yaml +++ b/integrations/rapid7-insightidr/mappers/search_alerts.yaml @@ -124,3 +124,26 @@ ocsf: - { source_path: "investigation.priority", ocsf_field: "finding_info.related_events.priority" } - { source_path: "investigation.disposition", ocsf_field: "finding_info.related_events.disposition" } - { source_path: "investigation.rrn", ocsf_field: "finding_info.related_events.uid" } + # ── Every key the rule found interesting ────────────────────────── + # rule_keys_of_interest is an OPEN vocabulary: each detection rule names its own + # keys. A process_start_event rule emits hostname / process.cmd_line / + # parent_process.*; an ingress_auth rule emits account / source_ip / + # geoip_country_code / result and none of the above. The explicit entries below + # can therefore only ever cover the rules someone has already seen — on any + # other rule they all resolve to nothing and the incident is empty again, which + # is the exact failure this integration started from. + # + # So the whole list is also flattened, verbatim, into one object. Keys nobody + # anticipated arrive intact and are visible to the analyst and to the AI + # analysis, which reads the mapped context but never `raw`. The explicit + # mappings stay because they are what types an entity correctly — a value under + # `detection.` is only ever matched by the IOC regex, so an account there would + # never be indexed as a user. + - { source_path: "rule_keys_of_interest{key: values[0]}", ocsf_field: "detection" } + - { source_path: "rule_matching_keys{key: values[0]}", ocsf_field: "detection_match" } + # ── Keys seen on non-process rules ──────────────────────────────── + - { source_path: "rule_keys_of_interest[key='account'].values[0]", ocsf_field: "actor.user.account.name" } + - { source_path: "rule_keys_of_interest[key='source_ip'].values[0]", ocsf_field: "src_endpoint.ip" } + - { source_path: "rule_keys_of_interest[key='destination_ip'].values[0]", ocsf_field: "dst_endpoint.ip" } + - { source_path: "rule_keys_of_interest[key='geoip_country_code'].values[0]", ocsf_field: "src_endpoint.location.country" } + - { source_path: "rule_keys_of_interest[key='result'].values[0]", ocsf_field: "status_detail" }