feat(rapid7-insightidr): feed the layout Process Tree block (v1.9.0)
Rapid7 exposes a process tree per alert; the layout renderer has a block that
draws one. They do not speak the same shape.
The block reads ctx.process_tree as N FLAT chains, root first, each
{alert_id, rule, host, user, nodes[]} — it merges shared ancestors itself.
Rapid7 answers with a NESTED tree: one root entity and children[]. So the
command walks each root-to-leaf path into its own chain, maps the node fields
(cmdline -> cmd_line, hashes.md5 -> md5, start_time -> started_at, pid coerced
from the string Rapid7 sends), and rolls every alert's chains up onto the
investigation, since the block reads the key at the top level and no mapper
expression reaches past alerts[0].
Verified through the block's own reader: a tree of explorer -> cmd ->
{bcdedit, vssadmin} flattens to two chains and comes back as one tree with the
ancestors shared and both leaves marked.
That mark is worth stating plainly. The block marks the last node of a chain as
the target, and Rapid7 flags no node as the one the alert fired on — so here it
means "end of this branch" and nothing stronger. Naming a target would assert
something the sensor never said.
Gated, because this is a third call per alert on top of the evidence one:
include_process_tree=auto (the default) asks only for the event types an
endpoint sensor produces a tree for, true asks for every alert, false never.
An authentication or firewall alert never has one, and on a feed where those
dominate the default saves a call per alert. 404 is treated as "no tree", which
is what it means, and only other statuses are recorded as errors. Bounded at 12
chains and 24 levels so an unexpected shape cannot expand without limit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,8 +1,8 @@
|
||||
id: rapid7_insightidr
|
||||
name: Rapid7 InsightIDR
|
||||
version: 1.8.0
|
||||
version: 1.9.0
|
||||
description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search."
|
||||
changelog: "1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
|
||||
changelog: "1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
|
||||
category: siem
|
||||
|
||||
# Per-instance configuration. The Insight platform region selects the API host
|
||||
@@ -453,6 +453,7 @@ commands:
|
||||
field_ids: { type: string, description: "Comma-separated additional alert field ids to include (see rapid7-insight-idr-list-alert-fields)" }
|
||||
include_evidence: { type: string, description: "Also fetch each alert's evidence — the log line the alert was raised on, parsed and normalised into evidence_fields (true/false, default true). One extra call per alert." }
|
||||
max_evidence_per_alert: { type: string, description: "Maximum evidence items per alert (default 5)" }
|
||||
include_process_tree: { type: string, description: "Fetch each alert process tree: auto (default, only endpoint event types that produce one), true (every alert), false (never). One extra call per alert asked. Chains are flattened root-first into ctx.process_tree, which the layout Process Tree block reads." }
|
||||
max_alerts: { type: string, description: "Ceiling on alert RRNs collected in one run (default 500)" }
|
||||
time_budget_seconds: { type: string, description: "Stop walking investigations after this many seconds and report truncated=true, rather than exceed the server's 90-second command budget and lose the whole run (default 60)" }
|
||||
index: { type: string, description: "0-based page index" }
|
||||
|
||||
@@ -166,6 +166,13 @@ ocsf:
|
||||
# offers, unlike `evidence` above, which only ever shows in the Context tab.
|
||||
- { source_path: "alerts[0].evidence_raw_line", ocsf_field: "raw_data" }
|
||||
- { source_path: "alerts[0].evidence_fields.msg ? alerts[0].evidence_fields.msg : alerts[0].evidence_fields.logdesc", ocsf_field: "message" }
|
||||
# ── Process tree ──────────────────────────────────────────────────
|
||||
# The layout block reads ctx.process_tree: N flat chains, root first, each
|
||||
# {alert_id, rule, host, user, nodes[]}. Rapid7 nests its tree through
|
||||
# `children`, so the command flattens each root-to-leaf path into one chain
|
||||
# and rolls every alert's chains up to the investigation — see
|
||||
# get_investigations_full.py. Passed through verbatim; nothing to reshape here.
|
||||
- { source_path: "process_tree", ocsf_field: "process_tree" }
|
||||
# ── Every alert, not just the first ───────────────────────────────
|
||||
# An investigation aggregates alerts, and the fields above only ever read alerts[0].
|
||||
# This entry lands the whole set as OCSF-shaped objects, so the host and the process of
|
||||
|
||||
@@ -169,6 +169,95 @@ def _evidence_fields(ev):
|
||||
return fields, (source if isinstance(source, str) else raw)
|
||||
|
||||
|
||||
# Event types worth asking a process tree for. The endpoint sensor produces one;
|
||||
# an authentication or firewall alert never will, and on a feed where those
|
||||
# dominate, asking anyway is one wasted call per alert. include_process_tree=true
|
||||
# overrides this and asks for every alert.
|
||||
_PROCESS_EVENT_TYPES = {
|
||||
"process_start_event", "advanced_malware", "virus",
|
||||
"endpoint_activity", "file_modification", "local_service_installed",
|
||||
}
|
||||
|
||||
# Guards on a tree of unknown shape: a wide one would otherwise expand into more
|
||||
# chains than anybody reads, and a cycle would not terminate at all.
|
||||
_MAX_CHAINS = 12
|
||||
_MAX_DEPTH = 24
|
||||
|
||||
|
||||
def _flatten_tree(root, out, path=None, depth=0):
|
||||
"""Rapid7 nests a process tree through `children`; the layout block reads a
|
||||
flat chain, root first. So each root-to-leaf path becomes one chain.
|
||||
|
||||
The block marks the LAST node of a chain as the target. Rapid7 flags no node
|
||||
as the one the alert fired on, so that mark means "end of this branch" here
|
||||
and nothing stronger — inventing a target would assert something the sensor
|
||||
never said.
|
||||
"""
|
||||
if not isinstance(root, dict) or depth > _MAX_DEPTH or len(out) >= _MAX_CHAINS:
|
||||
return
|
||||
name = root.get("name")
|
||||
node = {}
|
||||
if name:
|
||||
node["name"] = str(name)
|
||||
if root.get("cmdline"):
|
||||
node["cmd_line"] = str(root["cmdline"])
|
||||
pid = root.get("pid")
|
||||
if pid not in (None, ""):
|
||||
try:
|
||||
node["pid"] = int(str(pid))
|
||||
except ValueError:
|
||||
pass
|
||||
h = root.get("hashes") or {}
|
||||
if isinstance(h, dict):
|
||||
for k in ("md5", "sha256"):
|
||||
if h.get(k):
|
||||
node[k] = str(h[k])
|
||||
if root.get("start_time") not in (None, ""):
|
||||
node["started_at"] = root["start_time"]
|
||||
if root.get("username"):
|
||||
node["_username"] = str(root["username"])
|
||||
chain = (path or []) + ([node] if node else [])
|
||||
children = [c for c in (root.get("children") or []) if isinstance(c, dict)]
|
||||
if not children:
|
||||
if chain:
|
||||
out.append(chain)
|
||||
return
|
||||
for c in children:
|
||||
_flatten_tree(c, out, chain, depth + 1)
|
||||
|
||||
|
||||
def _tree_chains(payload, alert):
|
||||
"""The process_tree block's own shape: N chains, each with its nodes."""
|
||||
chains = []
|
||||
for tree in (payload.get("process_trees") or []):
|
||||
if not isinstance(tree, dict):
|
||||
continue
|
||||
paths = []
|
||||
_flatten_tree(tree.get("process_tree"), paths)
|
||||
for nodes in paths:
|
||||
user = ""
|
||||
for n in nodes:
|
||||
user = n.pop("_username", user) or user
|
||||
if not nodes:
|
||||
continue
|
||||
chain = {"nodes": nodes, "alert_id": tree.get("alert_rrn") or alert.get("rrn")}
|
||||
if user:
|
||||
chain["user"] = user
|
||||
if alert.get("type"):
|
||||
chain["rule"] = alert["type"]
|
||||
host = (alert.get("evidence_fields") or {}).get("devname") or (alert.get("evidence_fields") or {}).get("hostname")
|
||||
if not host:
|
||||
for kv in (alert.get("rule_keys_of_interest") or []):
|
||||
if kv.get("key") == "hostname" and kv.get("values"):
|
||||
host = kv["values"][0]
|
||||
if host:
|
||||
chain["host"] = host
|
||||
chains.append(chain)
|
||||
if len(chains) >= _MAX_CHAINS:
|
||||
return chains
|
||||
return chains
|
||||
|
||||
|
||||
def run():
|
||||
_, _, multi = _cfg()
|
||||
inp = _inputs()
|
||||
@@ -270,12 +359,49 @@ def run():
|
||||
a["evidence_fields"] = evs[0].get("fields") or {}
|
||||
a["evidence_raw_line"] = evs[0].get("raw_line")
|
||||
|
||||
# ⑤ Process trees. A third call per alert, so it is gated: "auto" asks only
|
||||
# for the event types an endpoint sensor produces one for, "true" asks
|
||||
# always, "false" never. Chains are attached to the alert AND to the
|
||||
# investigation, because the layout block reads ctx.process_tree at the
|
||||
# top level.
|
||||
tree_mode = str(inp.get("include_process_tree") or "auto").strip().lower()
|
||||
tree_calls, tree_errors = 0, 0
|
||||
if tree_mode != "false":
|
||||
for rrn, a in by_rrn.items():
|
||||
if time.monotonic() - started > budget:
|
||||
truncated = True
|
||||
break
|
||||
if tree_mode == "auto" and str(a.get("triggering_event_type") or "") not in _PROCESS_EVENT_TYPES:
|
||||
continue
|
||||
try:
|
||||
page = request("POST", AT + "alerts/" + urllib.parse.quote(rrn, safe="") + "/process_trees/latest",
|
||||
params={"index": "0", "size": "5"})
|
||||
tree_calls += 1
|
||||
except urllib.error.HTTPError as e:
|
||||
# 404 is the normal answer for an alert that has no tree, not a
|
||||
# fault: recorded only when it is something else.
|
||||
if e.code != 404:
|
||||
a["process_tree_error"] = "HTTP " + str(e.code)
|
||||
tree_errors += 1
|
||||
continue
|
||||
chains = _tree_chains(page, a)
|
||||
if chains:
|
||||
a["process_tree"] = chains
|
||||
|
||||
fetched = 0
|
||||
for inv in investigations:
|
||||
rrns = rrns_by_inv.get(inv.get("rrn") or "", [])
|
||||
alerts = [by_rrn[r] for r in rrns if r in by_rrn]
|
||||
inv["alerts"] = alerts
|
||||
inv["alert_count"] = len(rrns)
|
||||
# The block reads ctx.process_tree at the top level and draws one chain
|
||||
# per alert, so every alert's chains are gathered here rather than left
|
||||
# nested where no mapper expression could reach past alerts[0].
|
||||
rolled = []
|
||||
for a in alerts:
|
||||
rolled.extend(a.get("process_tree") or [])
|
||||
if rolled:
|
||||
inv["process_tree"] = rolled[:_MAX_CHAINS]
|
||||
# Say it when the alert list is a sample: alert_count is what the
|
||||
# investigation actually has, len(alerts) what this run retrieved.
|
||||
if len(alerts) < len(rrns):
|
||||
@@ -316,6 +442,8 @@ def run():
|
||||
"alerts_fetched": fetched,
|
||||
"evidence_calls": evidence_calls,
|
||||
"evidence_errors": evidence_errors,
|
||||
"process_tree_calls": tree_calls,
|
||||
"process_tree_errors": tree_errors,
|
||||
"alert_ids_without_rrn": skipped,
|
||||
"truncated": truncated,
|
||||
# HTTP 200 with a non-empty region_failures is a PARTIAL result:
|
||||
|
||||
Reference in New Issue
Block a user