f3nris
995a39ea66
An alert names what fired. The evidence is what happened, and until now it was
never fetched — which is why an incident could show a rule name and nothing
else. On a Fortigate admin login the alert carried an empty
rule_keys_of_interest; the evidence carried the whole event.
It cannot be reached by mapping alone. The evidence's `data` is a
JSON-ENCODED STRING holding {source_data, custom_data}, and JSONata does not
deserialise, so no mapper expression can ever see inside it. The parsing has to
happen in the command, and that is the reason for this change more than the
extra call is.
Once parsed, two views of the same event:
custom_data the vendor's normalised fields, prefixed per source
(Forti_user, Forti_srcip) — 6 keys on the sample
source_data the raw line in key="value" form — 25 keys
Both are merged into evidence_fields with the vendor prefix stripped, so a
mapper names `user` once instead of once per vendor, and source_data is overlaid
last because it carries more and is not affected by the quoting artefact below.
Values are cleaned, and that is not cosmetic: custom_data returns
'10.70.20.66"' with a trailing quote left by the vendor parser. Kept as-is it
yields an address matching no regex — an observable nobody can pivot on, and no
error to say why.
The line itself lands on raw_data and the source's own sentence on message,
both of which the layout builder offers. The Fortigate login now maps to
src_endpoint.ip, dst_endpoint.ip, actor.user.name, device.hostname/name/uid,
activity_name "login" and status_detail "success", and indexes four entities
where it indexed none.
Optional (include_evidence, default true) and capped (max_evidence_per_alert,
default 5), one call per alert under the run's existing time budget. An alert
whose evidence cannot be read keeps an evidence_error and still arrives;
evidence_calls and evidence_errors are reported in the run metadata.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
194 lines
16 KiB
YAML
194 lines
16 KiB
YAML
name: "Rapid7 InsightIDR Investigations (full) → OCSF"
|
|
description: |
|
|
Maps one Rapid7 investigation fetched with its alerts inlined
|
|
(get_investigations_full, results_path = data) to OCSF finding fields.
|
|
|
|
The plain investigation feed cannot carry a detection: idr/v2/investigations answers
|
|
with a title, a priority, a status, an assignee and timestamps, and nothing else. An
|
|
incident built from it indexes zero entities. This mapper reads the same investigation
|
|
plus the alerts the command inlined under `alerts`, where the detection content
|
|
actually lives.
|
|
|
|
Investigation values are mapped first and the first alert's equivalents after, so the
|
|
alert wins wherever the container says nothing. `alerts` is empty on an investigation
|
|
whose alerts could not be read (the command sets alerts_error) — every alert path then
|
|
resolves to nothing and is skipped, and the incident degrades to what the investigation
|
|
alone says rather than failing.
|
|
field_mappings:
|
|
title: "title"
|
|
# toSeverity maps CRITICAL→5, HIGH→4, MEDIUM→3, LOW→2; anything else falls back to the
|
|
# alert rule's default severity.
|
|
severity: "priority"
|
|
source: "source"
|
|
# The log line's own sentence beats a generic title when there is one.
|
|
description: "first_evidence_message"
|
|
# Rapid7's own triage state. It cannot reach the incident's status and verdict columns
|
|
# — MapFields accepts only title, severity, source, description and custom:* — so it
|
|
# rides as custom fields, where an analyst at least sees that the source already closed
|
|
# this and what it concluded.
|
|
"custom:rapid7_status": "status"
|
|
"custom:rapid7_disposition": "disposition"
|
|
# Rapid7's AI suggestion, exposed as a fact and never as a verdict.
|
|
# first_alert_* are lifted to the top level by the command on purpose: this
|
|
# mapper section resolves dotted paths only and cannot index an array, so
|
|
# `alerts[0].x` here would read as nothing — silently, leaving the custom field
|
|
# empty with no error. The ocsf section below is full JSONata and indexes freely.
|
|
"custom:rapid7_ai_disposition": "first_alert_ai_disposition"
|
|
"custom:rapid7_alert_count": "alert_count"
|
|
"custom:rapid7_log_id": "first_alert_log_id"
|
|
"custom:rapid7_logset_id": "first_alert_logset_id"
|
|
"custom:rapid7_rule_rrn": "first_alert_rule_rrn"
|
|
"custom:rapid7_event_type": "first_alert_event_type"
|
|
ocsf:
|
|
# ── Finding: the investigation ────────────────────────────────────
|
|
- { source_path: "rrn", ocsf_field: "finding_info.uid" }
|
|
- { source_path: "title", ocsf_field: "finding_info.title" }
|
|
- { source_path: "created_time", ocsf_field: "finding_info.created_time" }
|
|
- { source_path: "last_accessed", ocsf_field: "finding_info.modified_time" }
|
|
- { source_path: "first_alert_time", ocsf_field: "finding_info.first_seen_time" }
|
|
- { source_path: "latest_alert_time", ocsf_field: "finding_info.last_seen_time" }
|
|
- { source_path: "tags", ocsf_field: "metadata.labels" }
|
|
- { source_path: "alert_count", ocsf_field: "count" }
|
|
# ── Incident state ────────────────────────────────────────────────
|
|
- { source_path: "status", ocsf_field: "status" }
|
|
- { source_path: "priority", ocsf_field: "priority" }
|
|
- { source_path: "disposition", ocsf_field: "disposition" }
|
|
- { source_path: "source", ocsf_field: "activity_name" }
|
|
- { source_path: "organization_id", ocsf_field: "cloud.org.uid" }
|
|
- { source_path: "assignee.name", ocsf_field: "assignee.name" }
|
|
- { source_path: "assignee.email", ocsf_field: "assignee.email_addr" }
|
|
# ── Detection time: the MTTD anchor ───────────────────────────────
|
|
# Weakest first, strongest last. The investigation is created after the alert that
|
|
# opened it; the alert's own alerted_at is when the event happened in the source system.
|
|
- { source_path: "created_time", ocsf_field: "time" }
|
|
- { source_path: "first_alert_time", ocsf_field: "time" }
|
|
- { source_path: "alerts[0].alerted_at", ocsf_field: "time" }
|
|
# ── The detection rule, from the first alert ──────────────────────
|
|
# `rule.name` is documented but absent from live ABA alerts, so `type` — which repeats
|
|
# the rule's display name — is the honest fallback.
|
|
- { source_path: "alerts[0].type", ocsf_field: "finding_info.analytic.name" }
|
|
- { source_path: "alerts[0].rule.rrn", ocsf_field: "finding_info.analytic.uid" }
|
|
- { source_path: "alerts[0].rule.version_rrn", ocsf_field: "finding_info.analytic.version" }
|
|
- { source_path: "alerts[0].rrn", ocsf_field: "finding_info.uid_alt" }
|
|
# ── Product identity ──────────────────────────────────────────────
|
|
- { source_path: "'Rapid7 InsightIDR'", ocsf_field: "metadata.product.name" }
|
|
- { source_path: "'Rapid7'", ocsf_field: "metadata.product.vendor_name" }
|
|
- { source_path: "alerts[0].external_source", ocsf_field: "metadata.log_source" }
|
|
- { source_path: "alerts[0].triggering_event_type", ocsf_field: "metadata.event_code" }
|
|
- { source_path: "alerts[0].log_details[0].log_entry_id", ocsf_field: "metadata.uid" }
|
|
# ── MITRE ATT&CK ──────────────────────────────────────────────────
|
|
# mitre_tcodes mixes both kinds in one array: "TA0040" is a tactic, "T1490" a technique.
|
|
# Splitting on the "TA" prefix keeps a tactic out of the technique field, where it would
|
|
# name a technique that does not exist.
|
|
- { source_path: "alerts[0].rule.mitre_tcodes[$substring($, 0, 2) = 'TA'][0]", ocsf_field: "attacks.tactic.uid" }
|
|
- { source_path: "alerts[0].rule.mitre_tcodes[$substring($, 0, 2) != 'TA'][0]", ocsf_field: "attacks.technique.uid" }
|
|
# ── The affected host ─────────────────────────────────────────────
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='hostname'].values[0]", ocsf_field: "device.hostname" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='hostname'].values[0]", ocsf_field: "src_endpoint.hostname" }
|
|
# ── The process the alert is about ────────────────────────────────
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='process.name'].values[0]", ocsf_field: "process.name" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='process.name'].values[0]", ocsf_field: "process.file.name" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='process.cmd_line'].values[0]", ocsf_field: "process.cmd_line" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='process.exe_path'].values[0]", ocsf_field: "process.file.path" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='process.exe_file.hashes.md5'].values[0]", ocsf_field: "process.file.hashes.md5" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='process.exe_file.hashes.sha256'].values[0]", ocsf_field: "process.file.hashes.sha256" }
|
|
# ── Its parent, which says how it was launched ────────────────────
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='parent_process.name'].values[0]", ocsf_field: "actor.process.parent_process.name" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='parent_process.cmd_line'].values[0]", ocsf_field: "actor.process.parent_process.cmd_line" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='parent_process.exe_path'].values[0]", ocsf_field: "actor.process.parent_process.file.path" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='parent_process.exe_file.hashes.md5'].values[0]", ocsf_field: "actor.process.parent_process.file.hashes.md5" }
|
|
# ── The acting account ────────────────────────────────────────────
|
|
# This tenant's field catalogue has no canonical user namespace, so an account only
|
|
# appears when a rule names it among its keys of interest. Both spellings are mapped
|
|
# because rules are not consistent about which they emit.
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='user'].values[0]", ocsf_field: "actor.user.name" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='username'].values[0]", ocsf_field: "actor.user.name" }
|
|
# ── Lateral movement: source and destination ──────────────────────
|
|
# An asset_auth rule describes a connection, so it names BOTH ends and neither
|
|
# is "the host": source_asset_address / destination_asset_address, and
|
|
# source_account / destination_account. Mapping them to src_endpoint and
|
|
# dst_endpoint is not only the correct OCSF shape — it is also what makes them
|
|
# appear in the layout builder, whose field picker is a closed <select> over
|
|
# the OCSF catalogue. A path that is not in that catalogue cannot be placed on
|
|
# a layout at all, which is why `detection` below, useful as it is, only ever
|
|
# shows up in the Context tab.
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='source_asset_address'].values[0]", ocsf_field: "src_endpoint.hostname" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='destination_asset_address'].values[0]", ocsf_field: "dst_endpoint.hostname" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='destination_asset'].values[0]", ocsf_field: "dst_endpoint.hostname" }
|
|
# actor.user is who acted, user is who it was done TO. Keeping them apart is
|
|
# what lets an analyst read a lateral move in one line.
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='source_account'].values[0]", ocsf_field: "actor.user.name" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='destination_account'].values[0]", ocsf_field: "user.name" }
|
|
# ── Every key the rule found interesting ──────────────────────────
|
|
# rule_keys_of_interest is an OPEN vocabulary: each detection rule names its own
|
|
# keys. A process_start_event rule emits hostname / process.cmd_line /
|
|
# parent_process.*; an ingress_auth rule emits account / source_ip /
|
|
# geoip_country_code / result and none of the above. The explicit entries below
|
|
# can therefore only ever cover the rules someone has already seen — on any
|
|
# other rule they all resolve to nothing and the incident is empty again, which
|
|
# is the exact failure this integration started from.
|
|
#
|
|
# So the whole list is also flattened, verbatim, into one object. Keys nobody
|
|
# anticipated arrive intact and are visible to the analyst and to the AI
|
|
# analysis, which reads the mapped context but never `raw`. The explicit
|
|
# mappings stay because they are what types an entity correctly — a value under
|
|
# `detection.` is only ever matched by the IOC regex, so an account there would
|
|
# never be indexed as a user.
|
|
- { source_path: "alerts[0].rule_keys_of_interest{key: values[0]}", ocsf_field: "detection" }
|
|
- { source_path: "alerts[0].rule_matching_keys{key: values[0]}", ocsf_field: "detection_match" }
|
|
# ── Keys seen on non-process rules ────────────────────────────────
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='account'].values[0]", ocsf_field: "actor.user.account.name" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='source_ip'].values[0]", ocsf_field: "src_endpoint.ip" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='destination_ip'].values[0]", ocsf_field: "dst_endpoint.ip" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='geoip_country_code'].values[0]", ocsf_field: "src_endpoint.location.country" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='result'].values[0]", ocsf_field: "status_detail" }
|
|
# ── The evidence: the log line the alert was raised on ────────────
|
|
# An alert names what fired; the evidence is what actually happened. It lives
|
|
# in a JSON-encoded string the command parses and normalises into
|
|
# evidence_fields — see get_investigations_full.py. Vendor prefixes are
|
|
# stripped there (Forti_srcip -> srcip), so one entry serves every log source
|
|
# that names its fields the usual way.
|
|
- { source_path: "alerts[0].evidence_fields", ocsf_field: "evidence" }
|
|
- { source_path: "alerts[0].evidence_fields.srcip", ocsf_field: "src_endpoint.ip" }
|
|
- { source_path: "alerts[0].evidence_fields.dstip", ocsf_field: "dst_endpoint.ip" }
|
|
- { source_path: "alerts[0].evidence_fields.srcport", ocsf_field: "src_endpoint.port" }
|
|
- { source_path: "alerts[0].evidence_fields.dstport", ocsf_field: "dst_endpoint.port" }
|
|
- { source_path: "alerts[0].evidence_fields.user", ocsf_field: "actor.user.name" }
|
|
- { source_path: "alerts[0].evidence_fields.devname", ocsf_field: "device.name" }
|
|
- { source_path: "alerts[0].evidence_fields.devname", ocsf_field: "device.hostname" }
|
|
- { source_path: "alerts[0].evidence_fields.devid", ocsf_field: "device.uid" }
|
|
- { source_path: "alerts[0].evidence_fields.hostname", ocsf_field: "device.hostname" }
|
|
- { source_path: "alerts[0].evidence_fields.status", ocsf_field: "status_detail" }
|
|
# `action` is what the source system did ("login"), which says more than the
|
|
# investigation's own `source` ("ALERT"). Mapped after it so it wins.
|
|
- { source_path: "alerts[0].evidence_fields.action", ocsf_field: "activity_name" }
|
|
# The line itself, on the OCSF field meant for it — and one the layout builder
|
|
# offers, unlike `evidence` above, which only ever shows in the Context tab.
|
|
- { source_path: "alerts[0].evidence_raw_line", ocsf_field: "raw_data" }
|
|
- { source_path: "alerts[0].evidence_fields.msg ? alerts[0].evidence_fields.msg : alerts[0].evidence_fields.logdesc", ocsf_field: "message" }
|
|
# ── Every alert, not just the first ───────────────────────────────
|
|
# An investigation aggregates alerts, and the fields above only ever read alerts[0].
|
|
# This entry lands the whole set as OCSF-shaped objects, so the host and the process of
|
|
# the SECOND and THIRD alert are indexed as entities too — the extractor walks arrays
|
|
# and classifies on the path's suffix, so `.hostname` under here counts exactly as
|
|
# `device.hostname` does above.
|
|
- source_path: >
|
|
alerts.{
|
|
"uid": rrn,
|
|
"title": title,
|
|
"time": alerted_at,
|
|
"status": status,
|
|
"disposition": disposition,
|
|
"hostname": rule_keys_of_interest[key='hostname'].values[0],
|
|
"src_hostname": rule_keys_of_interest[key='source_asset_address'].values[0],
|
|
"dst_hostname": rule_keys_of_interest[key='destination_asset_address'].values[0],
|
|
"actor_user": rule_keys_of_interest[key='source_account'].values[0],
|
|
"target_user": rule_keys_of_interest[key='destination_account'].values[0],
|
|
"process": rule_keys_of_interest[key='process.name'].values[0],
|
|
"cmd_line": rule_keys_of_interest[key='process.cmd_line'].values[0],
|
|
"technique": rule.mitre_tcodes[$substring($, 0, 2) != 'TA'][0],
|
|
"detection": rule_keys_of_interest{key: values[0]},
|
|
"evidence": evidence_fields
|
|
}
|
|
ocsf_field: "evidences"
|