diff --git a/integrations/rapid7-insightidr/manifest.yaml b/integrations/rapid7-insightidr/manifest.yaml index 3138b24..dca89df 100644 --- a/integrations/rapid7-insightidr/manifest.yaml +++ b/integrations/rapid7-insightidr/manifest.yaml @@ -1,8 +1,8 @@ id: rapid7_insightidr name: Rapid7 InsightIDR -version: 1.10.0 +version: 1.11.0 description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search." -changelog: "1.10.0 — Render every detection key as Markdown into custom:rapid7_summary, so a layout can show what the OCSF catalogue has no field for. Map both ends of an account action, read source_json top-level scalars, drop placeholder values, and force evidences to stay an array. | 1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." +changelog: "1.11.0 — Build one evidence row per alert evidence into custom:rapid7_evidence, for the table field: sortable and filterable where the Markdown summary only reads. | 1.10.0 — Render every detection key as Markdown into custom:rapid7_summary, so a layout can show what the OCSF catalogue has no field for. Map both ends of an account action, read source_json top-level scalars, drop placeholder values, and force evidences to stay an array. | 1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." category: siem # Per-instance configuration. The Insight platform region selects the API host diff --git a/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml b/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml index 0416e50..645c3ec 100644 --- a/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml +++ b/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml @@ -44,6 +44,11 @@ field_mappings: # what no OCSF mapping can, because the layout's field picker only offers paths # from the OCSF catalogue and this vocabulary is open-ended. "custom:rapid7_summary": "summary_markdown" + # One row per evidence, across every alert. Declare a custom field named + # rapid7_evidence of type table: it reads an array of objects as it stands, so + # nothing is reshaped here. It sits beside the Markdown summary rather than + # replacing it — the summary reads as a page, this one sorts and filters. + "custom:rapid7_evidence": "evidence_rows" ocsf: # ── Finding: the investigation ──────────────────────────────────── - { source_path: "rrn", ocsf_field: "finding_info.uid" } diff --git a/integrations/rapid7-insightidr/scripts/get_investigations_full.py b/integrations/rapid7-insightidr/scripts/get_investigations_full.py index 544ed94..056874f 100644 --- a/integrations/rapid7-insightidr/scripts/get_investigations_full.py +++ b/integrations/rapid7-insightidr/scripts/get_investigations_full.py @@ -330,6 +330,75 @@ _MD_MAX_ROWS = 40 _MD_MAX_VALUE = 300 +# The columns worth reading first, in this order. Everything else follows +# alphabetically. Column order is the ONLY ordering signal the table field has: +# it takes the union of keys in order of first appearance, so what the first row +# names first is what the analyst sees without scrolling. +_ROW_LEAD = [ + "alert", "time", "action", "result", "status", + "source_account", "source_user", "target_account", "target_user", + "hostname", "computername", "source_asset", "devname", + "source_ip", "srcip", "destination_ip", "dstip", + "user", "account", "process.name", "process.cmd_line", +] +_MAX_ROWS = 100 +# A table wider than this is scrolled, not read. The cap keeps the useful +# columns — the lead list, then whichever others are populated on the most rows +# — rather than whichever happened to appear first. +_MAX_COLUMNS = 30 + + +def _evidence_rows(alerts): + """One row per evidence, across every alert of the investigation. + + The table custom field reads an array of objects directly, so this needs no + reshaping on the mapper side. It exists beside summary_markdown rather than + instead of it: the summary reads as a page, this sorts and filters. + + Each row carries its alert's name, because an investigation aggregates + several and a row that does not say where it came from cannot be acted on. + """ + rows = [] + for a in alerts: + for ev in (a.get("evidences") or []): + f = ev.get("fields") or {} + if not f: + continue + row = {"alert": a.get("type") or a.get("title") or ""} + t = ev.get("evented_at") or f.get("timestamp") + if t: + row["time"] = str(t) + for k in _ROW_LEAD: + if k in f and k not in row: + row[k] = f[k] + for k in sorted(f): + if k not in row: + row[k] = f[k] + rows.append(row) + if len(rows) >= _MAX_ROWS: + break + if len(rows) >= _MAX_ROWS: + break + if not rows: + return [], 0 + + # Trim the width. Counted over every row, so a column present once out of + # eighty does not push out one present on all of them. + counts = {} + for r in rows: + for k in r: + counts[k] = counts.get(k, 0) + 1 + ordered = [k for k in _ROW_LEAD + ["alert", "time"] if k in counts] + ordered += sorted((k for k in counts if k not in ordered), + key=lambda k: (-counts[k], k)) + # `alert` and `time` lead whatever else survives. + keep = ["alert"] + [k for k in ordered if k != "alert"][:_MAX_COLUMNS - 1] + dropped = len(counts) - len(keep) + if dropped > 0: + rows = [{k: r[k] for k in keep if k in r} for r in rows] + return rows, max(dropped, 0) + + def _markdown_summary(inv, alerts): """A readable rendering of what the detection actually says. @@ -533,6 +602,11 @@ def run(): # nested where no mapper expression could reach past alerts[0]. if alerts: inv["summary_markdown"] = _markdown_summary(inv, alerts) + ev_rows, ev_dropped = _evidence_rows(alerts) + if ev_rows: + inv["evidence_rows"] = ev_rows + if ev_dropped: + inv["evidence_columns_dropped"] = ev_dropped rolled = [] for a in alerts: rolled.extend(a.get("process_tree") or [])