diff --git a/integrations/rapid7-insightidr/manifest.yaml b/integrations/rapid7-insightidr/manifest.yaml index 80f3ee2..3138b24 100644 --- a/integrations/rapid7-insightidr/manifest.yaml +++ b/integrations/rapid7-insightidr/manifest.yaml @@ -1,8 +1,8 @@ id: rapid7_insightidr name: Rapid7 InsightIDR -version: 1.9.2 +version: 1.10.0 description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search." -changelog: "1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." +changelog: "1.10.0 — Render every detection key as Markdown into custom:rapid7_summary, so a layout can show what the OCSF catalogue has no field for. Map both ends of an account action, read source_json top-level scalars, drop placeholder values, and force evidences to stay an array. | 1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." category: siem # Per-instance configuration. The Insight platform region selects the API host diff --git a/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml b/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml index b46e8cd..0416e50 100644 --- a/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml +++ b/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml @@ -39,6 +39,11 @@ field_mappings: "custom:rapid7_logset_id": "first_alert_logset_id" "custom:rapid7_rule_rrn": "first_alert_rule_rrn" "custom:rapid7_event_type": "first_alert_event_type" + # Every key the detection emitted, rendered as Markdown. Declare a custom field + # named rapid7_summary of type markdown and place it on the layout: it shows + # what no OCSF mapping can, because the layout's field picker only offers paths + # from the OCSF catalogue and this vocabulary is open-ended. + "custom:rapid7_summary": "summary_markdown" ocsf: # ── Finding: the investigation ──────────────────────────────────── - { source_path: "rrn", ocsf_field: "finding_info.uid" } @@ -177,6 +182,23 @@ ocsf: # offers, unlike `evidence` above, which only ever shows in the Context tab. - { source_path: "alerts[0].evidence_raw_line", ocsf_field: "raw_data" } - { source_path: "alerts[0].evidence_fields.msg ? alerts[0].evidence_fields.msg : alerts[0].evidence_fields.logdesc", ocsf_field: "message" } + # ── Who did it to whom ──────────────────────────────────────────── + # An ad_admin rule names an actor and a target account, and mapping only the + # actor loses the victims — on "Multiple User Accounts Were Disabled" the + # disabled accounts are the whole point of the alert. + - { source_path: "alerts[0].evidence_fields.target_account", ocsf_field: "user.name" } + - { source_path: "alerts[0].evidence_fields.target_user", ocsf_field: "user.full_name" } + - { source_path: "alerts[0].evidence_fields.target_user_domain", ocsf_field: "user.domain" } + - { source_path: "alerts[0].evidence_fields.source_user", ocsf_field: "actor.user.full_name" } + - { source_path: "alerts[0].evidence_fields.source_user_domain", ocsf_field: "actor.user.domain" } + - { source_path: "alerts[0].evidence_fields.source_account", ocsf_field: "actor.user.name" } + # The machine the event was recorded on. A Windows event puts it in + # source_json.computerName and nowhere else; source_asset is the parsed field + # when the source fills it, and is dropped when it says "unknown". + - { source_path: "alerts[0].evidence_fields.source_asset", ocsf_field: "device.hostname" } + - { source_path: "alerts[0].evidence_fields.computername", ocsf_field: "device.hostname" } + - { source_path: "alerts[0].evidence_fields.osversion", ocsf_field: "device.os.name" } + - { source_path: "alerts[0].evidence_fields.calleripaddress", ocsf_field: "src_endpoint.ip" } # ── Process tree ────────────────────────────────────────────────── # The layout block reads ctx.process_tree: N flat chains, root first, each # {alert_id, rule, host, user, nodes[]}. Rapid7 nests its tree through @@ -190,8 +212,11 @@ ocsf: # the SECOND and THIRD alert are indexed as entities too — the extractor walks arrays # and classifies on the path's suffix, so `.hostname` under here counts exactly as # `device.hostname` does above. + # The brackets force an array. Without them JSONata returns a bare OBJECT when + # the investigation holds a single alert and an array when it holds two, so the + # same key had two shapes depending on how busy the attacker was. - source_path: > - alerts.{ + [alerts.{ "uid": rrn, "title": title, "time": alerted_at, @@ -207,5 +232,5 @@ ocsf: "technique": rule.mitre_tcodes[$substring($, 0, 2) != 'TA'][0], "detection": rule_keys_of_interest{key: values[0]}, "evidence": evidence_fields - } + }] ocsf_field: "evidences" diff --git a/integrations/rapid7-insightidr/scripts/get_investigations_full.py b/integrations/rapid7-insightidr/scripts/get_investigations_full.py index c834fee..544ed94 100644 --- a/integrations/rapid7-insightidr/scripts/get_investigations_full.py +++ b/integrations/rapid7-insightidr/scripts/get_investigations_full.py @@ -118,6 +118,12 @@ def _collect_region_failures(out, sink): _KV = re.compile(r'([A-Za-z_][\w.\-]*)=("(?:[^"\\]|\\.)*"|\S+)') +# Values a source writes when it has nothing to say. Mapped as-is they become an +# asset named "unknown" in the entity index, shared by every incident that had no +# asset — a pivot that groups things by the absence of a fact. +_PLACEHOLDERS = {"unknown", "n/a", "na", "none", "null", "-", ""} + + def _clean(v): """Strip the quoting Rapid7's parser leaves behind. @@ -128,7 +134,8 @@ def _clean(v): v = str(v).strip() if len(v) >= 2 and v[0] == '"' and v[-1] == '"': v = v[1:-1] - return v.replace('\\"', '"').strip().strip('"') + v = v.replace('\\"', '"').strip().strip('"') + return "" if v.strip().lower() in _PLACEHOLDERS else v def _evidence_fields(ev): @@ -191,6 +198,23 @@ def _evidence_fields(ev): elif isinstance(sv, (str, int, float)): fields[(k + "_" + sk).lower()] = _clean(sv) if isinstance(sv, str) else sv + # source_json is the vendor's own payload. Its NESTED part is skipped — the + # Azure AD one runs to several kilobytes — but its top-level scalars are + # small, bounded and often the only place something essential appears: a + # Windows 4725 puts the domain controller in computerName and nowhere else, + # and an Azure sign-in puts the caller address in callerIpAddress. Existing + # keys win, so the event's own naming is never overwritten by the vendor's. + sj = blob.get("source_json") + if isinstance(sj, dict): + for k, v in sj.items(): + key = k.lower() + if key in fields: + continue + if isinstance(v, bool) or isinstance(v, (int, float)): + fields[key] = v + elif isinstance(v, str) and v.strip(): + fields[key] = _clean(v) + # Wrapped shape, overlaid on top when present. custom = blob.get("custom_data") if isinstance(custom, dict): @@ -204,6 +228,10 @@ def _evidence_fields(ev): for k, v in _KV.findall(source): fields[k.lower()] = _clean(v) + # A key whose value cleaned away to nothing is dropped rather than kept + # empty: an empty string is not nil, so it would still be written to its OCSF + # field, and device.hostname = "" asserts an asset with no name. + fields = {k: v for k, v in fields.items() if v != ""} return fields, (source if isinstance(source, str) else raw) @@ -296,6 +324,74 @@ def _tree_chains(payload, alert): return chains +# Keys already rendered elsewhere in the incident, or too noisy for a summary. +_MD_SKIP = {"entry_id", "timestamp"} +_MD_MAX_ROWS = 40 +_MD_MAX_VALUE = 300 + + +def _markdown_summary(inv, alerts): + """A readable rendering of what the detection actually says. + + The layout builder's field picker is a closed