Compare commits

...

3 Commits

Author SHA1 Message Date
f3nris 2d6e097742 fix(rapid7-insightidr): explain the process-tree 404 (v1.9.1)
Running the command on an alert reports `{"error": "HTTP 404", "detail": ""}`.
The API returns 404 with an empty body, so the operator is told a number and
nothing else — and cannot tell a normal answer from a fault.

It is usually normal, and the URL is not at fault: it is built exactly as
get_alert_evidence builds its own, which works. Rapid7 uses the same 404 for
three unrelated situations and does not distinguish them:

  - the alert is not an endpoint detection, so no tree exists. This is the
    common case on a feed of ingress_auth, asset_auth or firewall events;

  - the tree exists in principle but was never generated. The endpoint returns
    the CACHED tree unless force_refresh is set, so an ungenerated tree answers
    404 while its data is there. Retrying with force_refresh=true is the first
    thing to try on an endpoint detection;

  - the key lacks the privilege.

All three are now named in the error, and the force_refresh input says what it
is really for rather than describing it only as a staleness control.

The ingestion path already handled this correctly — it treats 404 as "no tree"
and counts only other statuses as errors — so nothing changes there.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-08 15:56:23 +02:00
f3nris 1888648930 feat(rapid7-insightidr): feed the layout Process Tree block (v1.9.0)
Rapid7 exposes a process tree per alert; the layout renderer has a block that
draws one. They do not speak the same shape.

The block reads ctx.process_tree as N FLAT chains, root first, each
{alert_id, rule, host, user, nodes[]} — it merges shared ancestors itself.
Rapid7 answers with a NESTED tree: one root entity and children[]. So the
command walks each root-to-leaf path into its own chain, maps the node fields
(cmdline -> cmd_line, hashes.md5 -> md5, start_time -> started_at, pid coerced
from the string Rapid7 sends), and rolls every alert's chains up onto the
investigation, since the block reads the key at the top level and no mapper
expression reaches past alerts[0].

Verified through the block's own reader: a tree of explorer -> cmd ->
{bcdedit, vssadmin} flattens to two chains and comes back as one tree with the
ancestors shared and both leaves marked.

That mark is worth stating plainly. The block marks the last node of a chain as
the target, and Rapid7 flags no node as the one the alert fired on — so here it
means "end of this branch" and nothing stronger. Naming a target would assert
something the sensor never said.

Gated, because this is a third call per alert on top of the evidence one:
include_process_tree=auto (the default) asks only for the event types an
endpoint sensor produces a tree for, true asks for every alert, false never.
An authentication or firewall alert never has one, and on a feed where those
dominate the default saves a call per alert. 404 is treated as "no tree", which
is what it means, and only other statuses are recorded as errors. Bounded at 12
chains and 24 levels so an unexpected shape cannot expand without limit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-08 15:53:22 +02:00
f3nris 995a39ea66 feat(rapid7-insightidr): fetch and parse the evidence behind each alert (v1.8.0)
An alert names what fired. The evidence is what happened, and until now it was
never fetched — which is why an incident could show a rule name and nothing
else. On a Fortigate admin login the alert carried an empty
rule_keys_of_interest; the evidence carried the whole event.

It cannot be reached by mapping alone. The evidence's `data` is a
JSON-ENCODED STRING holding {source_data, custom_data}, and JSONata does not
deserialise, so no mapper expression can ever see inside it. The parsing has to
happen in the command, and that is the reason for this change more than the
extra call is.

Once parsed, two views of the same event:

  custom_data  the vendor's normalised fields, prefixed per source
               (Forti_user, Forti_srcip) — 6 keys on the sample
  source_data  the raw line in key="value" form — 25 keys

Both are merged into evidence_fields with the vendor prefix stripped, so a
mapper names `user` once instead of once per vendor, and source_data is overlaid
last because it carries more and is not affected by the quoting artefact below.

Values are cleaned, and that is not cosmetic: custom_data returns
'10.70.20.66"' with a trailing quote left by the vendor parser. Kept as-is it
yields an address matching no regex — an observable nobody can pivot on, and no
error to say why.

The line itself lands on raw_data and the source's own sentence on message,
both of which the layout builder offers. The Fortigate login now maps to
src_endpoint.ip, dst_endpoint.ip, actor.user.name, device.hostname/name/uid,
activity_name "login" and status_detail "success", and indexes four entities
where it indexed none.

Optional (include_evidence, default true) and capped (max_evidence_per_alert,
default 5), one call per alert under the run's existing time budget. An alert
whose evidence cannot be read keeps an evidence_error and still arrives;
evidence_calls and evidence_errors are reported in the run metadata.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-08 15:46:17 +02:00
4 changed files with 273 additions and 4 deletions
+6 -3
View File
@@ -1,8 +1,8 @@
id: rapid7_insightidr id: rapid7_insightidr
name: Rapid7 InsightIDR name: Rapid7 InsightIDR
version: 1.7.0 version: 1.9.1
description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search." description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search."
changelog: "1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." changelog: "1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
category: siem category: siem
# Per-instance configuration. The Insight platform region selects the API host # Per-instance configuration. The Insight platform region selects the API host
@@ -391,7 +391,7 @@ commands:
inputs_schema: inputs_schema:
properties: properties:
alert_rrn: { type: string, description: "Alert RRN" } alert_rrn: { type: string, description: "Alert RRN" }
force_refresh: { type: string, description: "Regenerate the tree instead of returning the cached one (true/false). Documented as expensive and slow — leave off unless the cached tree is stale." } force_refresh: { type: string, description: "Regenerate the tree instead of returning the cached one (true/false). Without it the endpoint returns only a CACHED tree, so an alert whose tree was never generated answers 404 even though the data exists — this is the first thing to try on a 404 for an endpoint detection. Documented as expensive and slow." }
branch: { type: string, description: "Branch number to generate the process tree with" } branch: { type: string, description: "Branch number to generate the process tree with" }
index: { type: string, description: "0-based page index" } index: { type: string, description: "0-based page index" }
page_size: { type: string, description: "Page size" } page_size: { type: string, description: "Page size" }
@@ -451,6 +451,9 @@ commands:
statuses: { type: string, description: "Comma-separated investigation statuses to include (open, investigating, closed). Leaving out closed skips what the source has already triaged." } statuses: { type: string, description: "Comma-separated investigation statuses to include (open, investigating, closed). Leaving out closed skips what the source has already triaged." }
priorities: { type: string, description: "Comma-separated investigation priorities to include (Unspecified, Low, Medium, High, Critical)" } priorities: { type: string, description: "Comma-separated investigation priorities to include (Unspecified, Low, Medium, High, Critical)" }
field_ids: { type: string, description: "Comma-separated additional alert field ids to include (see rapid7-insight-idr-list-alert-fields)" } field_ids: { type: string, description: "Comma-separated additional alert field ids to include (see rapid7-insight-idr-list-alert-fields)" }
include_evidence: { type: string, description: "Also fetch each alert's evidence — the log line the alert was raised on, parsed and normalised into evidence_fields (true/false, default true). One extra call per alert." }
max_evidence_per_alert: { type: string, description: "Maximum evidence items per alert (default 5)" }
include_process_tree: { type: string, description: "Fetch each alert process tree: auto (default, only endpoint event types that produce one), true (every alert), false (never). One extra call per alert asked. Chains are flattened root-first into ctx.process_tree, which the layout Process Tree block reads." }
max_alerts: { type: string, description: "Ceiling on alert RRNs collected in one run (default 500)" } max_alerts: { type: string, description: "Ceiling on alert RRNs collected in one run (default 500)" }
time_budget_seconds: { type: string, description: "Stop walking investigations after this many seconds and report truncated=true, rather than exceed the server's 90-second command budget and lose the whole run (default 60)" } time_budget_seconds: { type: string, description: "Stop walking investigations after this many seconds and report truncated=true, rather than exceed the server's 90-second command budget and lose the whole run (default 60)" }
index: { type: string, description: "0-based page index" } index: { type: string, description: "0-based page index" }
@@ -20,6 +20,8 @@ field_mappings:
# alert rule's default severity. # alert rule's default severity.
severity: "priority" severity: "priority"
source: "source" source: "source"
# The log line's own sentence beats a generic title when there is one.
description: "first_evidence_message"
# Rapid7's own triage state. It cannot reach the incident's status and verdict columns # Rapid7's own triage state. It cannot reach the incident's status and verdict columns
# — MapFields accepts only title, severity, source, description and custom:* — so it # — MapFields accepts only title, severity, source, description and custom:* — so it
# rides as custom fields, where an analyst at least sees that the source already closed # rides as custom fields, where an analyst at least sees that the source already closed
@@ -140,6 +142,37 @@ ocsf:
- { source_path: "alerts[0].rule_keys_of_interest[key='destination_ip'].values[0]", ocsf_field: "dst_endpoint.ip" } - { source_path: "alerts[0].rule_keys_of_interest[key='destination_ip'].values[0]", ocsf_field: "dst_endpoint.ip" }
- { source_path: "alerts[0].rule_keys_of_interest[key='geoip_country_code'].values[0]", ocsf_field: "src_endpoint.location.country" } - { source_path: "alerts[0].rule_keys_of_interest[key='geoip_country_code'].values[0]", ocsf_field: "src_endpoint.location.country" }
- { source_path: "alerts[0].rule_keys_of_interest[key='result'].values[0]", ocsf_field: "status_detail" } - { source_path: "alerts[0].rule_keys_of_interest[key='result'].values[0]", ocsf_field: "status_detail" }
# ── The evidence: the log line the alert was raised on ────────────
# An alert names what fired; the evidence is what actually happened. It lives
# in a JSON-encoded string the command parses and normalises into
# evidence_fields — see get_investigations_full.py. Vendor prefixes are
# stripped there (Forti_srcip -> srcip), so one entry serves every log source
# that names its fields the usual way.
- { source_path: "alerts[0].evidence_fields", ocsf_field: "evidence" }
- { source_path: "alerts[0].evidence_fields.srcip", ocsf_field: "src_endpoint.ip" }
- { source_path: "alerts[0].evidence_fields.dstip", ocsf_field: "dst_endpoint.ip" }
- { source_path: "alerts[0].evidence_fields.srcport", ocsf_field: "src_endpoint.port" }
- { source_path: "alerts[0].evidence_fields.dstport", ocsf_field: "dst_endpoint.port" }
- { source_path: "alerts[0].evidence_fields.user", ocsf_field: "actor.user.name" }
- { source_path: "alerts[0].evidence_fields.devname", ocsf_field: "device.name" }
- { source_path: "alerts[0].evidence_fields.devname", ocsf_field: "device.hostname" }
- { source_path: "alerts[0].evidence_fields.devid", ocsf_field: "device.uid" }
- { source_path: "alerts[0].evidence_fields.hostname", ocsf_field: "device.hostname" }
- { source_path: "alerts[0].evidence_fields.status", ocsf_field: "status_detail" }
# `action` is what the source system did ("login"), which says more than the
# investigation's own `source` ("ALERT"). Mapped after it so it wins.
- { source_path: "alerts[0].evidence_fields.action", ocsf_field: "activity_name" }
# The line itself, on the OCSF field meant for it — and one the layout builder
# offers, unlike `evidence` above, which only ever shows in the Context tab.
- { source_path: "alerts[0].evidence_raw_line", ocsf_field: "raw_data" }
- { source_path: "alerts[0].evidence_fields.msg ? alerts[0].evidence_fields.msg : alerts[0].evidence_fields.logdesc", ocsf_field: "message" }
# ── Process tree ──────────────────────────────────────────────────
# The layout block reads ctx.process_tree: N flat chains, root first, each
# {alert_id, rule, host, user, nodes[]}. Rapid7 nests its tree through
# `children`, so the command flattens each root-to-leaf path into one chain
# and rolls every alert's chains up to the investigation — see
# get_investigations_full.py. Passed through verbatim; nothing to reshape here.
- { source_path: "process_tree", ocsf_field: "process_tree" }
# ── Every alert, not just the first ─────────────────────────────── # ── Every alert, not just the first ───────────────────────────────
# An investigation aggregates alerts, and the fields above only ever read alerts[0]. # An investigation aggregates alerts, and the fields above only ever read alerts[0].
# This entry lands the whole set as OCSF-shaped objects, so the host and the process of # This entry lands the whole set as OCSF-shaped objects, so the host and the process of
@@ -161,6 +194,7 @@ ocsf:
"process": rule_keys_of_interest[key='process.name'].values[0], "process": rule_keys_of_interest[key='process.name'].values[0],
"cmd_line": rule_keys_of_interest[key='process.cmd_line'].values[0], "cmd_line": rule_keys_of_interest[key='process.cmd_line'].values[0],
"technique": rule.mitre_tcodes[$substring($, 0, 2) != 'TA'][0], "technique": rule.mitre_tcodes[$substring($, 0, 2) != 'TA'][0],
"detection": rule_keys_of_interest{key: values[0]} "detection": rule_keys_of_interest{key: values[0]},
"evidence": evidence_fields
} }
ocsf_field: "evidences" ocsf_field: "evidences"
@@ -62,6 +62,19 @@ try:
run() run()
except urllib.error.HTTPError as e: except urllib.error.HTTPError as e:
detail = e.read().decode("utf-8", "replace") detail = e.read().decode("utf-8", "replace")
if e.code == 404:
# The API returns 404 with an EMPTY body here, so without this the
# operator gets `{"error": "HTTP 404", "detail": ""}` and no way to tell
# a normal answer from a fault. It is usually normal: only an endpoint
# detection has a tree at all.
detail = (detail or "no process tree for this alert. Rapid7 answers 404 for three "
"different situations and does not distinguish them: (1) the alert is not "
"an endpoint detection, so no tree exists — check triggering_event_type "
"with rapid7-insight-idr-get-alert, and expect no tree for ingress_auth, "
"asset_auth, unparsed or firewall events; (2) the tree was never generated "
"— this endpoint returns the CACHED tree unless force_refresh is set, so "
"retry with force_refresh=true, which the API documents as slow; (3) the "
"API key lacks the privilege for it.")
if e.code in (401, 403): if e.code in (401, 403):
detail += (" — the Alert Triage API (/idr/at) is only available to Managed Detection and" detail += (" — the Alert Triage API (/idr/at) is only available to Managed Detection and"
" Response customers with access to the Alerts experience; a key that works on" " Response customers with access to the Alerts experience; a key that works on"
@@ -43,6 +43,7 @@ def request(method, path, params=None, body=None):
raw = r.read() raw = r.read()
return json.loads(raw) if raw else {} return json.loads(raw) if raw else {}
import re
import time import time
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
@@ -112,6 +113,151 @@ def _collect_region_failures(out, sink):
return out return out
# key="value" / key=value, the shape syslog-derived log lines take. Non-matching
# sources simply yield nothing, which is the right answer for them.
_KV = re.compile(r'([A-Za-z_][\w.\-]*)=("(?:[^"\\]|\\.)*"|\S+)')
def _clean(v):
"""Strip the quoting Rapid7's parser leaves behind.
custom_data values arrive as '10.70.20.66"' — a trailing quote from the
vendor parser. Left alone it produces an IP that matches no regex and an
entity nobody can pivot on, so the cleanup is not cosmetic.
"""
v = str(v).strip()
if len(v) >= 2 and v[0] == '"' and v[-1] == '"':
v = v[1:-1]
return v.replace('\\"', '"').strip().strip('"')
def _evidence_fields(ev):
"""One evidence's log line as a flat, normalised dict.
`data` is a JSON-encoded STRING holding {source_data, custom_data} — the
actual event behind the alert. JSONata cannot deserialise, so a mapper can
never reach into it; this is why the parsing has to happen here.
custom_data is the vendor's normalised view, its keys prefixed per source
(Forti_user, Forti_srcip). The prefix is stripped so a mapper can name
`user` once rather than once per vendor. source_data is the raw line and is
overlaid on top: it carries far more keys (25 against 6 on the sample this
was written from) and it is not affected by the quoting artefact above.
"""
raw = ev.get("data")
if not isinstance(raw, str) or not raw.strip():
return {}, None
try:
blob = json.loads(raw)
except ValueError:
# Not JSON: keep the string itself as the raw line rather than lose it.
return {}, raw
if not isinstance(blob, dict):
return {}, raw
fields = {}
custom = blob.get("custom_data")
if isinstance(custom, dict):
for k, v in custom.items():
if not isinstance(v, (str, int, float)):
continue
key = k.split("_", 1)[1] if "_" in k else k
fields[key.lower()] = _clean(v)
source = blob.get("source_data")
if isinstance(source, str):
for k, v in _KV.findall(source):
fields[k.lower()] = _clean(v)
return fields, (source if isinstance(source, str) else raw)
# Event types worth asking a process tree for. The endpoint sensor produces one;
# an authentication or firewall alert never will, and on a feed where those
# dominate, asking anyway is one wasted call per alert. include_process_tree=true
# overrides this and asks for every alert.
_PROCESS_EVENT_TYPES = {
"process_start_event", "advanced_malware", "virus",
"endpoint_activity", "file_modification", "local_service_installed",
}
# Guards on a tree of unknown shape: a wide one would otherwise expand into more
# chains than anybody reads, and a cycle would not terminate at all.
_MAX_CHAINS = 12
_MAX_DEPTH = 24
def _flatten_tree(root, out, path=None, depth=0):
"""Rapid7 nests a process tree through `children`; the layout block reads a
flat chain, root first. So each root-to-leaf path becomes one chain.
The block marks the LAST node of a chain as the target. Rapid7 flags no node
as the one the alert fired on, so that mark means "end of this branch" here
and nothing stronger — inventing a target would assert something the sensor
never said.
"""
if not isinstance(root, dict) or depth > _MAX_DEPTH or len(out) >= _MAX_CHAINS:
return
name = root.get("name")
node = {}
if name:
node["name"] = str(name)
if root.get("cmdline"):
node["cmd_line"] = str(root["cmdline"])
pid = root.get("pid")
if pid not in (None, ""):
try:
node["pid"] = int(str(pid))
except ValueError:
pass
h = root.get("hashes") or {}
if isinstance(h, dict):
for k in ("md5", "sha256"):
if h.get(k):
node[k] = str(h[k])
if root.get("start_time") not in (None, ""):
node["started_at"] = root["start_time"]
if root.get("username"):
node["_username"] = str(root["username"])
chain = (path or []) + ([node] if node else [])
children = [c for c in (root.get("children") or []) if isinstance(c, dict)]
if not children:
if chain:
out.append(chain)
return
for c in children:
_flatten_tree(c, out, chain, depth + 1)
def _tree_chains(payload, alert):
"""The process_tree block's own shape: N chains, each with its nodes."""
chains = []
for tree in (payload.get("process_trees") or []):
if not isinstance(tree, dict):
continue
paths = []
_flatten_tree(tree.get("process_tree"), paths)
for nodes in paths:
user = ""
for n in nodes:
user = n.pop("_username", user) or user
if not nodes:
continue
chain = {"nodes": nodes, "alert_id": tree.get("alert_rrn") or alert.get("rrn")}
if user:
chain["user"] = user
if alert.get("type"):
chain["rule"] = alert["type"]
host = (alert.get("evidence_fields") or {}).get("devname") or (alert.get("evidence_fields") or {}).get("hostname")
if not host:
for kv in (alert.get("rule_keys_of_interest") or []):
if kv.get("key") == "hostname" and kv.get("values"):
host = kv["values"][0]
if host:
chain["host"] = host
chains.append(chain)
if len(chains) >= _MAX_CHAINS:
return chains
return chains
def run(): def run():
_, _, multi = _cfg() _, _, multi = _cfg()
inp = _inputs() inp = _inputs()
@@ -185,12 +331,77 @@ def run():
if a.get("rrn"): if a.get("rrn"):
by_rrn[a["rrn"]] = a by_rrn[a["rrn"]] = a
# ④ The evidence behind each alert: the actual log line. An alert names what
# fired; the evidence is what happened. One call per alert — there is no
# bulk form — so it is optional, capped, and under the same time budget.
evidence_calls, evidence_errors = 0, 0
if str(inp.get("include_evidence") or "true").strip().lower() in ("1", "true", "yes"):
per_alert = int(inp.get("max_evidence_per_alert") or 5)
for rrn, a in by_rrn.items():
if time.monotonic() - started > budget:
truncated = True
break
try:
page = request("GET", AT + "alerts/" + urllib.parse.quote(rrn, safe="") + "/evidences",
params={"index": "0", "size": per_alert})
evidence_calls += 1
except urllib.error.HTTPError as e:
a["evidence_error"] = "HTTP " + str(e.code)
evidence_errors += 1
continue
evs = (page.get("evidences") or [])[:per_alert]
for ev in evs:
ev["fields"], ev["raw_line"] = _evidence_fields(ev)
a["evidences"] = evs
if evs:
# Lifted for the mapper: MapFields cannot index an array, and the
# ocsf section reads alerts[0] only.
a["evidence_fields"] = evs[0].get("fields") or {}
a["evidence_raw_line"] = evs[0].get("raw_line")
# ⑤ Process trees. A third call per alert, so it is gated: "auto" asks only
# for the event types an endpoint sensor produces one for, "true" asks
# always, "false" never. Chains are attached to the alert AND to the
# investigation, because the layout block reads ctx.process_tree at the
# top level.
tree_mode = str(inp.get("include_process_tree") or "auto").strip().lower()
tree_calls, tree_errors = 0, 0
if tree_mode != "false":
for rrn, a in by_rrn.items():
if time.monotonic() - started > budget:
truncated = True
break
if tree_mode == "auto" and str(a.get("triggering_event_type") or "") not in _PROCESS_EVENT_TYPES:
continue
try:
page = request("POST", AT + "alerts/" + urllib.parse.quote(rrn, safe="") + "/process_trees/latest",
params={"index": "0", "size": "5"})
tree_calls += 1
except urllib.error.HTTPError as e:
# 404 is the normal answer for an alert that has no tree, not a
# fault: recorded only when it is something else.
if e.code != 404:
a["process_tree_error"] = "HTTP " + str(e.code)
tree_errors += 1
continue
chains = _tree_chains(page, a)
if chains:
a["process_tree"] = chains
fetched = 0 fetched = 0
for inv in investigations: for inv in investigations:
rrns = rrns_by_inv.get(inv.get("rrn") or "", []) rrns = rrns_by_inv.get(inv.get("rrn") or "", [])
alerts = [by_rrn[r] for r in rrns if r in by_rrn] alerts = [by_rrn[r] for r in rrns if r in by_rrn]
inv["alerts"] = alerts inv["alerts"] = alerts
inv["alert_count"] = len(rrns) inv["alert_count"] = len(rrns)
# The block reads ctx.process_tree at the top level and draws one chain
# per alert, so every alert's chains are gathered here rather than left
# nested where no mapper expression could reach past alerts[0].
rolled = []
for a in alerts:
rolled.extend(a.get("process_tree") or [])
if rolled:
inv["process_tree"] = rolled[:_MAX_CHAINS]
# Say it when the alert list is a sample: alert_count is what the # Say it when the alert list is a sample: alert_count is what the
# investigation actually has, len(alerts) what this run retrieved. # investigation actually has, len(alerts) what this run retrieved.
if len(alerts) < len(rrns): if len(alerts) < len(rrns):
@@ -218,6 +429,10 @@ def run():
"first_alert_logset_id": log.get("logset_id"), "first_alert_logset_id": log.get("logset_id"),
"first_alert_log_entry_id": log.get("log_entry_id"), "first_alert_log_entry_id": log.get("log_entry_id"),
}) })
ef = a.get("evidence_fields") or {}
if ef:
inv["first_evidence_message"] = ef.get("msg") or ef.get("logdesc") or ef.get("message")
inv["first_evidence_raw_line"] = a.get("evidence_raw_line")
print(json.dumps({ print(json.dumps({
"data": investigations, "data": investigations,
@@ -225,6 +440,10 @@ def run():
"investigations_returned": len(investigations), "investigations_returned": len(investigations),
"alert_rrns_collected": len(order), "alert_rrns_collected": len(order),
"alerts_fetched": fetched, "alerts_fetched": fetched,
"evidence_calls": evidence_calls,
"evidence_errors": evidence_errors,
"process_tree_calls": tree_calls,
"process_tree_errors": tree_errors,
"alert_ids_without_rrn": skipped, "alert_ids_without_rrn": skipped,
"truncated": truncated, "truncated": truncated,
# HTTP 200 with a non-empty region_failures is a PARTIAL result: # HTTP 200 with a non-empty region_failures is a PARTIAL result: