diff --git a/integrations/rapid7-insightidr/manifest.yaml b/integrations/rapid7-insightidr/manifest.yaml index 43c4d55..57b5108 100644 --- a/integrations/rapid7-insightidr/manifest.yaml +++ b/integrations/rapid7-insightidr/manifest.yaml @@ -1,8 +1,8 @@ id: rapid7_insightidr name: Rapid7 InsightIDR -version: 1.8.0 +version: 1.9.0 description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search." -changelog: "1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." +changelog: "1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." category: siem # Per-instance configuration. The Insight platform region selects the API host @@ -453,6 +453,7 @@ commands: field_ids: { type: string, description: "Comma-separated additional alert field ids to include (see rapid7-insight-idr-list-alert-fields)" } include_evidence: { type: string, description: "Also fetch each alert's evidence — the log line the alert was raised on, parsed and normalised into evidence_fields (true/false, default true). One extra call per alert." } max_evidence_per_alert: { type: string, description: "Maximum evidence items per alert (default 5)" } + include_process_tree: { type: string, description: "Fetch each alert process tree: auto (default, only endpoint event types that produce one), true (every alert), false (never). One extra call per alert asked. Chains are flattened root-first into ctx.process_tree, which the layout Process Tree block reads." } max_alerts: { type: string, description: "Ceiling on alert RRNs collected in one run (default 500)" } time_budget_seconds: { type: string, description: "Stop walking investigations after this many seconds and report truncated=true, rather than exceed the server's 90-second command budget and lose the whole run (default 60)" } index: { type: string, description: "0-based page index" } diff --git a/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml b/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml index a88c67d..2a117e5 100644 --- a/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml +++ b/integrations/rapid7-insightidr/mappers/get_investigations_full.yaml @@ -166,6 +166,13 @@ ocsf: # offers, unlike `evidence` above, which only ever shows in the Context tab. - { source_path: "alerts[0].evidence_raw_line", ocsf_field: "raw_data" } - { source_path: "alerts[0].evidence_fields.msg ? alerts[0].evidence_fields.msg : alerts[0].evidence_fields.logdesc", ocsf_field: "message" } + # ── Process tree ────────────────────────────────────────────────── + # The layout block reads ctx.process_tree: N flat chains, root first, each + # {alert_id, rule, host, user, nodes[]}. Rapid7 nests its tree through + # `children`, so the command flattens each root-to-leaf path into one chain + # and rolls every alert's chains up to the investigation — see + # get_investigations_full.py. Passed through verbatim; nothing to reshape here. + - { source_path: "process_tree", ocsf_field: "process_tree" } # ── Every alert, not just the first ─────────────────────────────── # An investigation aggregates alerts, and the fields above only ever read alerts[0]. # This entry lands the whole set as OCSF-shaped objects, so the host and the process of diff --git a/integrations/rapid7-insightidr/scripts/get_investigations_full.py b/integrations/rapid7-insightidr/scripts/get_investigations_full.py index 710f8ad..b23ff2e 100644 --- a/integrations/rapid7-insightidr/scripts/get_investigations_full.py +++ b/integrations/rapid7-insightidr/scripts/get_investigations_full.py @@ -169,6 +169,95 @@ def _evidence_fields(ev): return fields, (source if isinstance(source, str) else raw) +# Event types worth asking a process tree for. The endpoint sensor produces one; +# an authentication or firewall alert never will, and on a feed where those +# dominate, asking anyway is one wasted call per alert. include_process_tree=true +# overrides this and asks for every alert. +_PROCESS_EVENT_TYPES = { + "process_start_event", "advanced_malware", "virus", + "endpoint_activity", "file_modification", "local_service_installed", +} + +# Guards on a tree of unknown shape: a wide one would otherwise expand into more +# chains than anybody reads, and a cycle would not terminate at all. +_MAX_CHAINS = 12 +_MAX_DEPTH = 24 + + +def _flatten_tree(root, out, path=None, depth=0): + """Rapid7 nests a process tree through `children`; the layout block reads a + flat chain, root first. So each root-to-leaf path becomes one chain. + + The block marks the LAST node of a chain as the target. Rapid7 flags no node + as the one the alert fired on, so that mark means "end of this branch" here + and nothing stronger — inventing a target would assert something the sensor + never said. + """ + if not isinstance(root, dict) or depth > _MAX_DEPTH or len(out) >= _MAX_CHAINS: + return + name = root.get("name") + node = {} + if name: + node["name"] = str(name) + if root.get("cmdline"): + node["cmd_line"] = str(root["cmdline"]) + pid = root.get("pid") + if pid not in (None, ""): + try: + node["pid"] = int(str(pid)) + except ValueError: + pass + h = root.get("hashes") or {} + if isinstance(h, dict): + for k in ("md5", "sha256"): + if h.get(k): + node[k] = str(h[k]) + if root.get("start_time") not in (None, ""): + node["started_at"] = root["start_time"] + if root.get("username"): + node["_username"] = str(root["username"]) + chain = (path or []) + ([node] if node else []) + children = [c for c in (root.get("children") or []) if isinstance(c, dict)] + if not children: + if chain: + out.append(chain) + return + for c in children: + _flatten_tree(c, out, chain, depth + 1) + + +def _tree_chains(payload, alert): + """The process_tree block's own shape: N chains, each with its nodes.""" + chains = [] + for tree in (payload.get("process_trees") or []): + if not isinstance(tree, dict): + continue + paths = [] + _flatten_tree(tree.get("process_tree"), paths) + for nodes in paths: + user = "" + for n in nodes: + user = n.pop("_username", user) or user + if not nodes: + continue + chain = {"nodes": nodes, "alert_id": tree.get("alert_rrn") or alert.get("rrn")} + if user: + chain["user"] = user + if alert.get("type"): + chain["rule"] = alert["type"] + host = (alert.get("evidence_fields") or {}).get("devname") or (alert.get("evidence_fields") or {}).get("hostname") + if not host: + for kv in (alert.get("rule_keys_of_interest") or []): + if kv.get("key") == "hostname" and kv.get("values"): + host = kv["values"][0] + if host: + chain["host"] = host + chains.append(chain) + if len(chains) >= _MAX_CHAINS: + return chains + return chains + + def run(): _, _, multi = _cfg() inp = _inputs() @@ -270,12 +359,49 @@ def run(): a["evidence_fields"] = evs[0].get("fields") or {} a["evidence_raw_line"] = evs[0].get("raw_line") + # ⑤ Process trees. A third call per alert, so it is gated: "auto" asks only + # for the event types an endpoint sensor produces one for, "true" asks + # always, "false" never. Chains are attached to the alert AND to the + # investigation, because the layout block reads ctx.process_tree at the + # top level. + tree_mode = str(inp.get("include_process_tree") or "auto").strip().lower() + tree_calls, tree_errors = 0, 0 + if tree_mode != "false": + for rrn, a in by_rrn.items(): + if time.monotonic() - started > budget: + truncated = True + break + if tree_mode == "auto" and str(a.get("triggering_event_type") or "") not in _PROCESS_EVENT_TYPES: + continue + try: + page = request("POST", AT + "alerts/" + urllib.parse.quote(rrn, safe="") + "/process_trees/latest", + params={"index": "0", "size": "5"}) + tree_calls += 1 + except urllib.error.HTTPError as e: + # 404 is the normal answer for an alert that has no tree, not a + # fault: recorded only when it is something else. + if e.code != 404: + a["process_tree_error"] = "HTTP " + str(e.code) + tree_errors += 1 + continue + chains = _tree_chains(page, a) + if chains: + a["process_tree"] = chains + fetched = 0 for inv in investigations: rrns = rrns_by_inv.get(inv.get("rrn") or "", []) alerts = [by_rrn[r] for r in rrns if r in by_rrn] inv["alerts"] = alerts inv["alert_count"] = len(rrns) + # The block reads ctx.process_tree at the top level and draws one chain + # per alert, so every alert's chains are gathered here rather than left + # nested where no mapper expression could reach past alerts[0]. + rolled = [] + for a in alerts: + rolled.extend(a.get("process_tree") or []) + if rolled: + inv["process_tree"] = rolled[:_MAX_CHAINS] # Say it when the alert list is a sample: alert_count is what the # investigation actually has, len(alerts) what this run retrieved. if len(alerts) < len(rrns): @@ -316,6 +442,8 @@ def run(): "alerts_fetched": fetched, "evidence_calls": evidence_calls, "evidence_errors": evidence_errors, + "process_tree_calls": tree_calls, + "process_tree_errors": tree_errors, "alert_ids_without_rrn": skipped, "truncated": truncated, # HTTP 200 with a non-empty region_failures is a PARTIAL result: