From e19530d3f369412486a6ebe59401c4dc00d8a186 Mon Sep 17 00:00:00 2001 From: Guillaume BOURGEOIS Date: Tue, 8 Sep 2026 16:19:08 +0200 Subject: [PATCH] fix(rapid7-insightidr): read the evidence in both of its shapes (v1.9.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit evidence_fields came back {} on a live run. The evidence was fetched, the run was clean — one call, 1.3s, no error — and the parser found nothing in it. `data` does not have one shape. The parser was written against an unparsed Fortigate source and looked only for {source_data, custom_data}. A PARSED source has no such wrapper: the event is the object. {user, account, source_ip, geoip_country_code, geoip_organization, result, service, user_agent, source_json{…}, r7_context{…}, detection_context{…}} Every one of those is what the incident was missing, sitting one json.loads away, and the parser walked past all of it and returned an empty dict. No error — which is how it survived a run that reported success. Now both are read as a union rather than one or the other: scalars at the top level are always taken, so a third shape still yields its plain fields instead of nothing, and the wrapped keys are overlaid when present. source_json is skipped — several kilobytes of vendor payload that stay in the raw line — while r7_context and detection_context are flattened one level, being Rapid7's own annotations and the same across parsed sources. They carry the account age in minutes, which is exactly what says whether a sign-in is from a new account. The Azure AD sign-in now maps to src_endpoint.ip, .isp_org and .location.country, actor.user.name/.uid/.account.name, http_request.user_agent and metadata.product.feature.name, and indexes the address and the account. Co-Authored-By: Claude Opus 5 (1M context) --- integrations/rapid7-insightidr/manifest.yaml | 4 +- .../mappers/get_investigations_full.yaml | 11 ++++ .../scripts/get_investigations_full.py | 60 +++++++++++++++---- 3 files changed, 62 insertions(+), 13 deletions(-) diff --git a/integrations/rapid7-insightidr/manifest.yaml b/integrations/rapid7-insightidr/manifest.yaml index 9d03470..80f3ee2 100644 --- a/integrations/rapid7-insightidr/manifest.yaml +++ b/integrations/rapid7-insightidr/manifest.yaml @@ -1,8 +1,8 @@ id: rapid7_insightidr name: Rapid7 InsightIDR -version: 1.9.1 +version: 1.9.2 description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search." -changelog: "1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." +changelog: "1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." category: siem # Per-instance configuration. The Insight platform region selects the API host diff --git a/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml b/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml index 2a117e5..b46e8cd 100644 --- a/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml +++ b/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml @@ -150,6 +150,17 @@ ocsf: # that names its fields the usual way. - { source_path: "alerts[0].evidence_fields", ocsf_field: "evidence" } - { source_path: "alerts[0].evidence_fields.srcip", ocsf_field: "src_endpoint.ip" } + # A parsed source states the same things under its own names. Both spellings + # are mapped rather than picked between: which one arrives depends on whether + # the log source was parsed by Rapid7, and one file has to serve both. + - { source_path: "alerts[0].evidence_fields.source_ip", ocsf_field: "src_endpoint.ip" } + - { source_path: "alerts[0].evidence_fields.account", ocsf_field: "actor.user.account.name" } + - { source_path: "alerts[0].evidence_fields.user_rrn", ocsf_field: "actor.user.uid" } + - { source_path: "alerts[0].evidence_fields.geoip_country_code", ocsf_field: "src_endpoint.location.country" } + - { source_path: "alerts[0].evidence_fields.geoip_organization", ocsf_field: "src_endpoint.isp_org" } + - { source_path: "alerts[0].evidence_fields.user_agent", ocsf_field: "http_request.user_agent" } + - { source_path: "alerts[0].evidence_fields.service", ocsf_field: "metadata.product.feature.name" } + - { source_path: "alerts[0].evidence_fields.result", ocsf_field: "status_detail" } - { source_path: "alerts[0].evidence_fields.dstip", ocsf_field: "dst_endpoint.ip" } - { source_path: "alerts[0].evidence_fields.srcport", ocsf_field: "src_endpoint.port" } - { source_path: "alerts[0].evidence_fields.dstport", ocsf_field: "dst_endpoint.port" } diff --git a/integrations/rapid7-insightidr/scripts/get_investigations_full.py b/integrations/rapid7-insightidr/scripts/get_investigations_full.py index b23ff2e..c834fee 100644 --- a/integrations/rapid7-insightidr/scripts/get_investigations_full.py +++ b/integrations/rapid7-insightidr/scripts/get_investigations_full.py @@ -132,17 +132,30 @@ def _clean(v): def _evidence_fields(ev): - """One evidence's log line as a flat, normalised dict. + """One evidence's event as a flat, normalised dict. - `data` is a JSON-encoded STRING holding {source_data, custom_data} — the - actual event behind the alert. JSONata cannot deserialise, so a mapper can - never reach into it; this is why the parsing has to happen here. + `data` is a JSON-encoded STRING. JSONata cannot deserialise, so a mapper can + never reach into it; this is why the parsing happens here. - custom_data is the vendor's normalised view, its keys prefixed per source - (Forti_user, Forti_srcip). The prefix is stripped so a mapper can name - `user` once rather than once per vendor. source_data is the raw line and is - overlaid on top: it carries far more keys (25 against 6 on the sample this - was written from) and it is not affected by the quoting artefact above. + It arrives in at least two shapes and the union of both is read, rather than + one or the other. The first version of this handled only the wrapped shape + and returned {} for everything else — silently, so an Azure AD sign-in + carrying its account, its source address and its geolocation produced an + incident with no observable at all. + + wrapped {source_data: "", custom_data: {Forti_user: …}} + An unparsed source. custom_data is the vendor's normalised view, + its keys prefixed per source; source_data is the raw line. + + flat {user: …, account: …, source_ip: …, geoip_country_code: …, + source_json: {…}, r7_context: {…}, detection_context: {…}} + A parsed source. The event IS the object. + + Scalars at the top level are always taken, so a third shape still yields its + plain fields instead of nothing. source_json is deliberately skipped — it is + the vendor's whole payload, several kilobytes of it, and it stays in the raw + line. r7_context and detection_context are flattened one level because they + are Rapid7's own annotations and are the same across parsed sources. """ raw = ev.get("data") if not isinstance(raw, str) or not raw.strip(): @@ -154,18 +167,43 @@ def _evidence_fields(ev): return {}, raw if not isinstance(blob, dict): return {}, raw + fields = {} + + # Flat shape: every scalar the event states about itself. + for k, v in blob.items(): + if k in ("source_data", "custom_data", "source_json"): + continue + if isinstance(v, bool) or isinstance(v, (int, float)): + fields[k.lower()] = v + elif isinstance(v, str): + fields[k.lower()] = _clean(v) + elif isinstance(v, dict) and k in ("r7_context", "detection_context"): + for sk, sv in v.items(): + if isinstance(sv, dict): + for ssk, ssv in sv.items(): + if not isinstance(ssv, (str, int, float)) or isinstance(ssv, bool): + continue + # detection_context nests user.user_age_mins, so a blind + # join yields user_user_age_mins. + name = ssk if ssk.startswith(sk + "_") else sk + "_" + ssk + fields[name.lower()] = _clean(ssv) if isinstance(ssv, str) else ssv + elif isinstance(sv, (str, int, float)): + fields[(k + "_" + sk).lower()] = _clean(sv) if isinstance(sv, str) else sv + + # Wrapped shape, overlaid on top when present. custom = blob.get("custom_data") if isinstance(custom, dict): for k, v in custom.items(): - if not isinstance(v, (str, int, float)): + if not isinstance(v, (str, int, float)) or isinstance(v, bool): continue key = k.split("_", 1)[1] if "_" in k else k - fields[key.lower()] = _clean(v) + fields[key.lower()] = _clean(v) if isinstance(v, str) else v source = blob.get("source_data") if isinstance(source, str): for k, v in _KV.findall(source): fields[k.lower()] = _clean(v) + return fields, (source if isinstance(source, str) else raw)