Compare commits
3 Commits
45e9cbd269
...
2d6e097742
| Author | SHA1 | Date | |
|---|---|---|---|
| 2d6e097742 | |||
| 1888648930 | |||
| 995a39ea66 |
@@ -1,8 +1,8 @@
|
|||||||
id: rapid7_insightidr
|
id: rapid7_insightidr
|
||||||
name: Rapid7 InsightIDR
|
name: Rapid7 InsightIDR
|
||||||
version: 1.7.0
|
version: 1.9.1
|
||||||
description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search."
|
description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search."
|
||||||
changelog: "1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
|
changelog: "1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
|
||||||
category: siem
|
category: siem
|
||||||
|
|
||||||
# Per-instance configuration. The Insight platform region selects the API host
|
# Per-instance configuration. The Insight platform region selects the API host
|
||||||
@@ -391,7 +391,7 @@ commands:
|
|||||||
inputs_schema:
|
inputs_schema:
|
||||||
properties:
|
properties:
|
||||||
alert_rrn: { type: string, description: "Alert RRN" }
|
alert_rrn: { type: string, description: "Alert RRN" }
|
||||||
force_refresh: { type: string, description: "Regenerate the tree instead of returning the cached one (true/false). Documented as expensive and slow — leave off unless the cached tree is stale." }
|
force_refresh: { type: string, description: "Regenerate the tree instead of returning the cached one (true/false). Without it the endpoint returns only a CACHED tree, so an alert whose tree was never generated answers 404 even though the data exists — this is the first thing to try on a 404 for an endpoint detection. Documented as expensive and slow." }
|
||||||
branch: { type: string, description: "Branch number to generate the process tree with" }
|
branch: { type: string, description: "Branch number to generate the process tree with" }
|
||||||
index: { type: string, description: "0-based page index" }
|
index: { type: string, description: "0-based page index" }
|
||||||
page_size: { type: string, description: "Page size" }
|
page_size: { type: string, description: "Page size" }
|
||||||
@@ -451,6 +451,9 @@ commands:
|
|||||||
statuses: { type: string, description: "Comma-separated investigation statuses to include (open, investigating, closed). Leaving out closed skips what the source has already triaged." }
|
statuses: { type: string, description: "Comma-separated investigation statuses to include (open, investigating, closed). Leaving out closed skips what the source has already triaged." }
|
||||||
priorities: { type: string, description: "Comma-separated investigation priorities to include (Unspecified, Low, Medium, High, Critical)" }
|
priorities: { type: string, description: "Comma-separated investigation priorities to include (Unspecified, Low, Medium, High, Critical)" }
|
||||||
field_ids: { type: string, description: "Comma-separated additional alert field ids to include (see rapid7-insight-idr-list-alert-fields)" }
|
field_ids: { type: string, description: "Comma-separated additional alert field ids to include (see rapid7-insight-idr-list-alert-fields)" }
|
||||||
|
include_evidence: { type: string, description: "Also fetch each alert's evidence — the log line the alert was raised on, parsed and normalised into evidence_fields (true/false, default true). One extra call per alert." }
|
||||||
|
max_evidence_per_alert: { type: string, description: "Maximum evidence items per alert (default 5)" }
|
||||||
|
include_process_tree: { type: string, description: "Fetch each alert process tree: auto (default, only endpoint event types that produce one), true (every alert), false (never). One extra call per alert asked. Chains are flattened root-first into ctx.process_tree, which the layout Process Tree block reads." }
|
||||||
max_alerts: { type: string, description: "Ceiling on alert RRNs collected in one run (default 500)" }
|
max_alerts: { type: string, description: "Ceiling on alert RRNs collected in one run (default 500)" }
|
||||||
time_budget_seconds: { type: string, description: "Stop walking investigations after this many seconds and report truncated=true, rather than exceed the server's 90-second command budget and lose the whole run (default 60)" }
|
time_budget_seconds: { type: string, description: "Stop walking investigations after this many seconds and report truncated=true, rather than exceed the server's 90-second command budget and lose the whole run (default 60)" }
|
||||||
index: { type: string, description: "0-based page index" }
|
index: { type: string, description: "0-based page index" }
|
||||||
|
|||||||
@@ -20,6 +20,8 @@ field_mappings:
|
|||||||
# alert rule's default severity.
|
# alert rule's default severity.
|
||||||
severity: "priority"
|
severity: "priority"
|
||||||
source: "source"
|
source: "source"
|
||||||
|
# The log line's own sentence beats a generic title when there is one.
|
||||||
|
description: "first_evidence_message"
|
||||||
# Rapid7's own triage state. It cannot reach the incident's status and verdict columns
|
# Rapid7's own triage state. It cannot reach the incident's status and verdict columns
|
||||||
# — MapFields accepts only title, severity, source, description and custom:* — so it
|
# — MapFields accepts only title, severity, source, description and custom:* — so it
|
||||||
# rides as custom fields, where an analyst at least sees that the source already closed
|
# rides as custom fields, where an analyst at least sees that the source already closed
|
||||||
@@ -140,6 +142,37 @@ ocsf:
|
|||||||
- { source_path: "alerts[0].rule_keys_of_interest[key='destination_ip'].values[0]", ocsf_field: "dst_endpoint.ip" }
|
- { source_path: "alerts[0].rule_keys_of_interest[key='destination_ip'].values[0]", ocsf_field: "dst_endpoint.ip" }
|
||||||
- { source_path: "alerts[0].rule_keys_of_interest[key='geoip_country_code'].values[0]", ocsf_field: "src_endpoint.location.country" }
|
- { source_path: "alerts[0].rule_keys_of_interest[key='geoip_country_code'].values[0]", ocsf_field: "src_endpoint.location.country" }
|
||||||
- { source_path: "alerts[0].rule_keys_of_interest[key='result'].values[0]", ocsf_field: "status_detail" }
|
- { source_path: "alerts[0].rule_keys_of_interest[key='result'].values[0]", ocsf_field: "status_detail" }
|
||||||
|
# ── The evidence: the log line the alert was raised on ────────────
|
||||||
|
# An alert names what fired; the evidence is what actually happened. It lives
|
||||||
|
# in a JSON-encoded string the command parses and normalises into
|
||||||
|
# evidence_fields — see get_investigations_full.py. Vendor prefixes are
|
||||||
|
# stripped there (Forti_srcip -> srcip), so one entry serves every log source
|
||||||
|
# that names its fields the usual way.
|
||||||
|
- { source_path: "alerts[0].evidence_fields", ocsf_field: "evidence" }
|
||||||
|
- { source_path: "alerts[0].evidence_fields.srcip", ocsf_field: "src_endpoint.ip" }
|
||||||
|
- { source_path: "alerts[0].evidence_fields.dstip", ocsf_field: "dst_endpoint.ip" }
|
||||||
|
- { source_path: "alerts[0].evidence_fields.srcport", ocsf_field: "src_endpoint.port" }
|
||||||
|
- { source_path: "alerts[0].evidence_fields.dstport", ocsf_field: "dst_endpoint.port" }
|
||||||
|
- { source_path: "alerts[0].evidence_fields.user", ocsf_field: "actor.user.name" }
|
||||||
|
- { source_path: "alerts[0].evidence_fields.devname", ocsf_field: "device.name" }
|
||||||
|
- { source_path: "alerts[0].evidence_fields.devname", ocsf_field: "device.hostname" }
|
||||||
|
- { source_path: "alerts[0].evidence_fields.devid", ocsf_field: "device.uid" }
|
||||||
|
- { source_path: "alerts[0].evidence_fields.hostname", ocsf_field: "device.hostname" }
|
||||||
|
- { source_path: "alerts[0].evidence_fields.status", ocsf_field: "status_detail" }
|
||||||
|
# `action` is what the source system did ("login"), which says more than the
|
||||||
|
# investigation's own `source` ("ALERT"). Mapped after it so it wins.
|
||||||
|
- { source_path: "alerts[0].evidence_fields.action", ocsf_field: "activity_name" }
|
||||||
|
# The line itself, on the OCSF field meant for it — and one the layout builder
|
||||||
|
# offers, unlike `evidence` above, which only ever shows in the Context tab.
|
||||||
|
- { source_path: "alerts[0].evidence_raw_line", ocsf_field: "raw_data" }
|
||||||
|
- { source_path: "alerts[0].evidence_fields.msg ? alerts[0].evidence_fields.msg : alerts[0].evidence_fields.logdesc", ocsf_field: "message" }
|
||||||
|
# ── Process tree ──────────────────────────────────────────────────
|
||||||
|
# The layout block reads ctx.process_tree: N flat chains, root first, each
|
||||||
|
# {alert_id, rule, host, user, nodes[]}. Rapid7 nests its tree through
|
||||||
|
# `children`, so the command flattens each root-to-leaf path into one chain
|
||||||
|
# and rolls every alert's chains up to the investigation — see
|
||||||
|
# get_investigations_full.py. Passed through verbatim; nothing to reshape here.
|
||||||
|
- { source_path: "process_tree", ocsf_field: "process_tree" }
|
||||||
# ── Every alert, not just the first ───────────────────────────────
|
# ── Every alert, not just the first ───────────────────────────────
|
||||||
# An investigation aggregates alerts, and the fields above only ever read alerts[0].
|
# An investigation aggregates alerts, and the fields above only ever read alerts[0].
|
||||||
# This entry lands the whole set as OCSF-shaped objects, so the host and the process of
|
# This entry lands the whole set as OCSF-shaped objects, so the host and the process of
|
||||||
@@ -161,6 +194,7 @@ ocsf:
|
|||||||
"process": rule_keys_of_interest[key='process.name'].values[0],
|
"process": rule_keys_of_interest[key='process.name'].values[0],
|
||||||
"cmd_line": rule_keys_of_interest[key='process.cmd_line'].values[0],
|
"cmd_line": rule_keys_of_interest[key='process.cmd_line'].values[0],
|
||||||
"technique": rule.mitre_tcodes[$substring($, 0, 2) != 'TA'][0],
|
"technique": rule.mitre_tcodes[$substring($, 0, 2) != 'TA'][0],
|
||||||
"detection": rule_keys_of_interest{key: values[0]}
|
"detection": rule_keys_of_interest{key: values[0]},
|
||||||
|
"evidence": evidence_fields
|
||||||
}
|
}
|
||||||
ocsf_field: "evidences"
|
ocsf_field: "evidences"
|
||||||
|
|||||||
@@ -62,6 +62,19 @@ try:
|
|||||||
run()
|
run()
|
||||||
except urllib.error.HTTPError as e:
|
except urllib.error.HTTPError as e:
|
||||||
detail = e.read().decode("utf-8", "replace")
|
detail = e.read().decode("utf-8", "replace")
|
||||||
|
if e.code == 404:
|
||||||
|
# The API returns 404 with an EMPTY body here, so without this the
|
||||||
|
# operator gets `{"error": "HTTP 404", "detail": ""}` and no way to tell
|
||||||
|
# a normal answer from a fault. It is usually normal: only an endpoint
|
||||||
|
# detection has a tree at all.
|
||||||
|
detail = (detail or "no process tree for this alert. Rapid7 answers 404 for three "
|
||||||
|
"different situations and does not distinguish them: (1) the alert is not "
|
||||||
|
"an endpoint detection, so no tree exists — check triggering_event_type "
|
||||||
|
"with rapid7-insight-idr-get-alert, and expect no tree for ingress_auth, "
|
||||||
|
"asset_auth, unparsed or firewall events; (2) the tree was never generated "
|
||||||
|
"— this endpoint returns the CACHED tree unless force_refresh is set, so "
|
||||||
|
"retry with force_refresh=true, which the API documents as slow; (3) the "
|
||||||
|
"API key lacks the privilege for it.")
|
||||||
if e.code in (401, 403):
|
if e.code in (401, 403):
|
||||||
detail += (" — the Alert Triage API (/idr/at) is only available to Managed Detection and"
|
detail += (" — the Alert Triage API (/idr/at) is only available to Managed Detection and"
|
||||||
" Response customers with access to the Alerts experience; a key that works on"
|
" Response customers with access to the Alerts experience; a key that works on"
|
||||||
|
|||||||
@@ -43,6 +43,7 @@ def request(method, path, params=None, body=None):
|
|||||||
raw = r.read()
|
raw = r.read()
|
||||||
return json.loads(raw) if raw else {}
|
return json.loads(raw) if raw else {}
|
||||||
|
|
||||||
|
import re
|
||||||
import time
|
import time
|
||||||
from datetime import datetime, timedelta, timezone
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
@@ -112,6 +113,151 @@ def _collect_region_failures(out, sink):
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# key="value" / key=value, the shape syslog-derived log lines take. Non-matching
|
||||||
|
# sources simply yield nothing, which is the right answer for them.
|
||||||
|
_KV = re.compile(r'([A-Za-z_][\w.\-]*)=("(?:[^"\\]|\\.)*"|\S+)')
|
||||||
|
|
||||||
|
|
||||||
|
def _clean(v):
|
||||||
|
"""Strip the quoting Rapid7's parser leaves behind.
|
||||||
|
|
||||||
|
custom_data values arrive as '10.70.20.66"' — a trailing quote from the
|
||||||
|
vendor parser. Left alone it produces an IP that matches no regex and an
|
||||||
|
entity nobody can pivot on, so the cleanup is not cosmetic.
|
||||||
|
"""
|
||||||
|
v = str(v).strip()
|
||||||
|
if len(v) >= 2 and v[0] == '"' and v[-1] == '"':
|
||||||
|
v = v[1:-1]
|
||||||
|
return v.replace('\\"', '"').strip().strip('"')
|
||||||
|
|
||||||
|
|
||||||
|
def _evidence_fields(ev):
|
||||||
|
"""One evidence's log line as a flat, normalised dict.
|
||||||
|
|
||||||
|
`data` is a JSON-encoded STRING holding {source_data, custom_data} — the
|
||||||
|
actual event behind the alert. JSONata cannot deserialise, so a mapper can
|
||||||
|
never reach into it; this is why the parsing has to happen here.
|
||||||
|
|
||||||
|
custom_data is the vendor's normalised view, its keys prefixed per source
|
||||||
|
(Forti_user, Forti_srcip). The prefix is stripped so a mapper can name
|
||||||
|
`user` once rather than once per vendor. source_data is the raw line and is
|
||||||
|
overlaid on top: it carries far more keys (25 against 6 on the sample this
|
||||||
|
was written from) and it is not affected by the quoting artefact above.
|
||||||
|
"""
|
||||||
|
raw = ev.get("data")
|
||||||
|
if not isinstance(raw, str) or not raw.strip():
|
||||||
|
return {}, None
|
||||||
|
try:
|
||||||
|
blob = json.loads(raw)
|
||||||
|
except ValueError:
|
||||||
|
# Not JSON: keep the string itself as the raw line rather than lose it.
|
||||||
|
return {}, raw
|
||||||
|
if not isinstance(blob, dict):
|
||||||
|
return {}, raw
|
||||||
|
fields = {}
|
||||||
|
custom = blob.get("custom_data")
|
||||||
|
if isinstance(custom, dict):
|
||||||
|
for k, v in custom.items():
|
||||||
|
if not isinstance(v, (str, int, float)):
|
||||||
|
continue
|
||||||
|
key = k.split("_", 1)[1] if "_" in k else k
|
||||||
|
fields[key.lower()] = _clean(v)
|
||||||
|
source = blob.get("source_data")
|
||||||
|
if isinstance(source, str):
|
||||||
|
for k, v in _KV.findall(source):
|
||||||
|
fields[k.lower()] = _clean(v)
|
||||||
|
return fields, (source if isinstance(source, str) else raw)
|
||||||
|
|
||||||
|
|
||||||
|
# Event types worth asking a process tree for. The endpoint sensor produces one;
|
||||||
|
# an authentication or firewall alert never will, and on a feed where those
|
||||||
|
# dominate, asking anyway is one wasted call per alert. include_process_tree=true
|
||||||
|
# overrides this and asks for every alert.
|
||||||
|
_PROCESS_EVENT_TYPES = {
|
||||||
|
"process_start_event", "advanced_malware", "virus",
|
||||||
|
"endpoint_activity", "file_modification", "local_service_installed",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Guards on a tree of unknown shape: a wide one would otherwise expand into more
|
||||||
|
# chains than anybody reads, and a cycle would not terminate at all.
|
||||||
|
_MAX_CHAINS = 12
|
||||||
|
_MAX_DEPTH = 24
|
||||||
|
|
||||||
|
|
||||||
|
def _flatten_tree(root, out, path=None, depth=0):
|
||||||
|
"""Rapid7 nests a process tree through `children`; the layout block reads a
|
||||||
|
flat chain, root first. So each root-to-leaf path becomes one chain.
|
||||||
|
|
||||||
|
The block marks the LAST node of a chain as the target. Rapid7 flags no node
|
||||||
|
as the one the alert fired on, so that mark means "end of this branch" here
|
||||||
|
and nothing stronger — inventing a target would assert something the sensor
|
||||||
|
never said.
|
||||||
|
"""
|
||||||
|
if not isinstance(root, dict) or depth > _MAX_DEPTH or len(out) >= _MAX_CHAINS:
|
||||||
|
return
|
||||||
|
name = root.get("name")
|
||||||
|
node = {}
|
||||||
|
if name:
|
||||||
|
node["name"] = str(name)
|
||||||
|
if root.get("cmdline"):
|
||||||
|
node["cmd_line"] = str(root["cmdline"])
|
||||||
|
pid = root.get("pid")
|
||||||
|
if pid not in (None, ""):
|
||||||
|
try:
|
||||||
|
node["pid"] = int(str(pid))
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
h = root.get("hashes") or {}
|
||||||
|
if isinstance(h, dict):
|
||||||
|
for k in ("md5", "sha256"):
|
||||||
|
if h.get(k):
|
||||||
|
node[k] = str(h[k])
|
||||||
|
if root.get("start_time") not in (None, ""):
|
||||||
|
node["started_at"] = root["start_time"]
|
||||||
|
if root.get("username"):
|
||||||
|
node["_username"] = str(root["username"])
|
||||||
|
chain = (path or []) + ([node] if node else [])
|
||||||
|
children = [c for c in (root.get("children") or []) if isinstance(c, dict)]
|
||||||
|
if not children:
|
||||||
|
if chain:
|
||||||
|
out.append(chain)
|
||||||
|
return
|
||||||
|
for c in children:
|
||||||
|
_flatten_tree(c, out, chain, depth + 1)
|
||||||
|
|
||||||
|
|
||||||
|
def _tree_chains(payload, alert):
|
||||||
|
"""The process_tree block's own shape: N chains, each with its nodes."""
|
||||||
|
chains = []
|
||||||
|
for tree in (payload.get("process_trees") or []):
|
||||||
|
if not isinstance(tree, dict):
|
||||||
|
continue
|
||||||
|
paths = []
|
||||||
|
_flatten_tree(tree.get("process_tree"), paths)
|
||||||
|
for nodes in paths:
|
||||||
|
user = ""
|
||||||
|
for n in nodes:
|
||||||
|
user = n.pop("_username", user) or user
|
||||||
|
if not nodes:
|
||||||
|
continue
|
||||||
|
chain = {"nodes": nodes, "alert_id": tree.get("alert_rrn") or alert.get("rrn")}
|
||||||
|
if user:
|
||||||
|
chain["user"] = user
|
||||||
|
if alert.get("type"):
|
||||||
|
chain["rule"] = alert["type"]
|
||||||
|
host = (alert.get("evidence_fields") or {}).get("devname") or (alert.get("evidence_fields") or {}).get("hostname")
|
||||||
|
if not host:
|
||||||
|
for kv in (alert.get("rule_keys_of_interest") or []):
|
||||||
|
if kv.get("key") == "hostname" and kv.get("values"):
|
||||||
|
host = kv["values"][0]
|
||||||
|
if host:
|
||||||
|
chain["host"] = host
|
||||||
|
chains.append(chain)
|
||||||
|
if len(chains) >= _MAX_CHAINS:
|
||||||
|
return chains
|
||||||
|
return chains
|
||||||
|
|
||||||
|
|
||||||
def run():
|
def run():
|
||||||
_, _, multi = _cfg()
|
_, _, multi = _cfg()
|
||||||
inp = _inputs()
|
inp = _inputs()
|
||||||
@@ -185,12 +331,77 @@ def run():
|
|||||||
if a.get("rrn"):
|
if a.get("rrn"):
|
||||||
by_rrn[a["rrn"]] = a
|
by_rrn[a["rrn"]] = a
|
||||||
|
|
||||||
|
# ④ The evidence behind each alert: the actual log line. An alert names what
|
||||||
|
# fired; the evidence is what happened. One call per alert — there is no
|
||||||
|
# bulk form — so it is optional, capped, and under the same time budget.
|
||||||
|
evidence_calls, evidence_errors = 0, 0
|
||||||
|
if str(inp.get("include_evidence") or "true").strip().lower() in ("1", "true", "yes"):
|
||||||
|
per_alert = int(inp.get("max_evidence_per_alert") or 5)
|
||||||
|
for rrn, a in by_rrn.items():
|
||||||
|
if time.monotonic() - started > budget:
|
||||||
|
truncated = True
|
||||||
|
break
|
||||||
|
try:
|
||||||
|
page = request("GET", AT + "alerts/" + urllib.parse.quote(rrn, safe="") + "/evidences",
|
||||||
|
params={"index": "0", "size": per_alert})
|
||||||
|
evidence_calls += 1
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
a["evidence_error"] = "HTTP " + str(e.code)
|
||||||
|
evidence_errors += 1
|
||||||
|
continue
|
||||||
|
evs = (page.get("evidences") or [])[:per_alert]
|
||||||
|
for ev in evs:
|
||||||
|
ev["fields"], ev["raw_line"] = _evidence_fields(ev)
|
||||||
|
a["evidences"] = evs
|
||||||
|
if evs:
|
||||||
|
# Lifted for the mapper: MapFields cannot index an array, and the
|
||||||
|
# ocsf section reads alerts[0] only.
|
||||||
|
a["evidence_fields"] = evs[0].get("fields") or {}
|
||||||
|
a["evidence_raw_line"] = evs[0].get("raw_line")
|
||||||
|
|
||||||
|
# ⑤ Process trees. A third call per alert, so it is gated: "auto" asks only
|
||||||
|
# for the event types an endpoint sensor produces one for, "true" asks
|
||||||
|
# always, "false" never. Chains are attached to the alert AND to the
|
||||||
|
# investigation, because the layout block reads ctx.process_tree at the
|
||||||
|
# top level.
|
||||||
|
tree_mode = str(inp.get("include_process_tree") or "auto").strip().lower()
|
||||||
|
tree_calls, tree_errors = 0, 0
|
||||||
|
if tree_mode != "false":
|
||||||
|
for rrn, a in by_rrn.items():
|
||||||
|
if time.monotonic() - started > budget:
|
||||||
|
truncated = True
|
||||||
|
break
|
||||||
|
if tree_mode == "auto" and str(a.get("triggering_event_type") or "") not in _PROCESS_EVENT_TYPES:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
page = request("POST", AT + "alerts/" + urllib.parse.quote(rrn, safe="") + "/process_trees/latest",
|
||||||
|
params={"index": "0", "size": "5"})
|
||||||
|
tree_calls += 1
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
# 404 is the normal answer for an alert that has no tree, not a
|
||||||
|
# fault: recorded only when it is something else.
|
||||||
|
if e.code != 404:
|
||||||
|
a["process_tree_error"] = "HTTP " + str(e.code)
|
||||||
|
tree_errors += 1
|
||||||
|
continue
|
||||||
|
chains = _tree_chains(page, a)
|
||||||
|
if chains:
|
||||||
|
a["process_tree"] = chains
|
||||||
|
|
||||||
fetched = 0
|
fetched = 0
|
||||||
for inv in investigations:
|
for inv in investigations:
|
||||||
rrns = rrns_by_inv.get(inv.get("rrn") or "", [])
|
rrns = rrns_by_inv.get(inv.get("rrn") or "", [])
|
||||||
alerts = [by_rrn[r] for r in rrns if r in by_rrn]
|
alerts = [by_rrn[r] for r in rrns if r in by_rrn]
|
||||||
inv["alerts"] = alerts
|
inv["alerts"] = alerts
|
||||||
inv["alert_count"] = len(rrns)
|
inv["alert_count"] = len(rrns)
|
||||||
|
# The block reads ctx.process_tree at the top level and draws one chain
|
||||||
|
# per alert, so every alert's chains are gathered here rather than left
|
||||||
|
# nested where no mapper expression could reach past alerts[0].
|
||||||
|
rolled = []
|
||||||
|
for a in alerts:
|
||||||
|
rolled.extend(a.get("process_tree") or [])
|
||||||
|
if rolled:
|
||||||
|
inv["process_tree"] = rolled[:_MAX_CHAINS]
|
||||||
# Say it when the alert list is a sample: alert_count is what the
|
# Say it when the alert list is a sample: alert_count is what the
|
||||||
# investigation actually has, len(alerts) what this run retrieved.
|
# investigation actually has, len(alerts) what this run retrieved.
|
||||||
if len(alerts) < len(rrns):
|
if len(alerts) < len(rrns):
|
||||||
@@ -218,6 +429,10 @@ def run():
|
|||||||
"first_alert_logset_id": log.get("logset_id"),
|
"first_alert_logset_id": log.get("logset_id"),
|
||||||
"first_alert_log_entry_id": log.get("log_entry_id"),
|
"first_alert_log_entry_id": log.get("log_entry_id"),
|
||||||
})
|
})
|
||||||
|
ef = a.get("evidence_fields") or {}
|
||||||
|
if ef:
|
||||||
|
inv["first_evidence_message"] = ef.get("msg") or ef.get("logdesc") or ef.get("message")
|
||||||
|
inv["first_evidence_raw_line"] = a.get("evidence_raw_line")
|
||||||
|
|
||||||
print(json.dumps({
|
print(json.dumps({
|
||||||
"data": investigations,
|
"data": investigations,
|
||||||
@@ -225,6 +440,10 @@ def run():
|
|||||||
"investigations_returned": len(investigations),
|
"investigations_returned": len(investigations),
|
||||||
"alert_rrns_collected": len(order),
|
"alert_rrns_collected": len(order),
|
||||||
"alerts_fetched": fetched,
|
"alerts_fetched": fetched,
|
||||||
|
"evidence_calls": evidence_calls,
|
||||||
|
"evidence_errors": evidence_errors,
|
||||||
|
"process_tree_calls": tree_calls,
|
||||||
|
"process_tree_errors": tree_errors,
|
||||||
"alert_ids_without_rrn": skipped,
|
"alert_ids_without_rrn": skipped,
|
||||||
"truncated": truncated,
|
"truncated": truncated,
|
||||||
# HTTP 200 with a non-empty region_failures is a PARTIAL result:
|
# HTTP 200 with a non-empty region_failures is a PARTIAL result:
|
||||||
|
|||||||
Reference in New Issue
Block a user