f3nris
e19530d3f3
evidence_fields came back {} on a live run. The evidence was fetched, the run
was clean — one call, 1.3s, no error — and the parser found nothing in it.
`data` does not have one shape. The parser was written against an unparsed
Fortigate source and looked only for {source_data, custom_data}. A PARSED
source has no such wrapper: the event is the object.
{user, account, source_ip, geoip_country_code, geoip_organization,
result, service, user_agent, source_json{…}, r7_context{…},
detection_context{…}}
Every one of those is what the incident was missing, sitting one json.loads
away, and the parser walked past all of it and returned an empty dict. No error
— which is how it survived a run that reported success.
Now both are read as a union rather than one or the other: scalars at the top
level are always taken, so a third shape still yields its plain fields instead
of nothing, and the wrapped keys are overlaid when present. source_json is
skipped — several kilobytes of vendor payload that stay in the raw line —
while r7_context and detection_context are flattened one level, being Rapid7's
own annotations and the same across parsed sources. They carry the account age
in minutes, which is exactly what says whether a sign-in is from a new account.
The Azure AD sign-in now maps to src_endpoint.ip, .isp_org and
.location.country, actor.user.name/.uid/.account.name, http_request.user_agent
and metadata.product.feature.name, and indexes the address and the account.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
212 lines
17 KiB
YAML
212 lines
17 KiB
YAML
name: "Rapid7 InsightIDR Investigations (full) → OCSF"
|
|
description: |
|
|
Maps one Rapid7 investigation fetched with its alerts inlined
|
|
(get_investigations_full, results_path = data) to OCSF finding fields.
|
|
|
|
The plain investigation feed cannot carry a detection: idr/v2/investigations answers
|
|
with a title, a priority, a status, an assignee and timestamps, and nothing else. An
|
|
incident built from it indexes zero entities. This mapper reads the same investigation
|
|
plus the alerts the command inlined under `alerts`, where the detection content
|
|
actually lives.
|
|
|
|
Investigation values are mapped first and the first alert's equivalents after, so the
|
|
alert wins wherever the container says nothing. `alerts` is empty on an investigation
|
|
whose alerts could not be read (the command sets alerts_error) — every alert path then
|
|
resolves to nothing and is skipped, and the incident degrades to what the investigation
|
|
alone says rather than failing.
|
|
field_mappings:
|
|
title: "title"
|
|
# toSeverity maps CRITICAL→5, HIGH→4, MEDIUM→3, LOW→2; anything else falls back to the
|
|
# alert rule's default severity.
|
|
severity: "priority"
|
|
source: "source"
|
|
# The log line's own sentence beats a generic title when there is one.
|
|
description: "first_evidence_message"
|
|
# Rapid7's own triage state. It cannot reach the incident's status and verdict columns
|
|
# — MapFields accepts only title, severity, source, description and custom:* — so it
|
|
# rides as custom fields, where an analyst at least sees that the source already closed
|
|
# this and what it concluded.
|
|
"custom:rapid7_status": "status"
|
|
"custom:rapid7_disposition": "disposition"
|
|
# Rapid7's AI suggestion, exposed as a fact and never as a verdict.
|
|
# first_alert_* are lifted to the top level by the command on purpose: this
|
|
# mapper section resolves dotted paths only and cannot index an array, so
|
|
# `alerts[0].x` here would read as nothing — silently, leaving the custom field
|
|
# empty with no error. The ocsf section below is full JSONata and indexes freely.
|
|
"custom:rapid7_ai_disposition": "first_alert_ai_disposition"
|
|
"custom:rapid7_alert_count": "alert_count"
|
|
"custom:rapid7_log_id": "first_alert_log_id"
|
|
"custom:rapid7_logset_id": "first_alert_logset_id"
|
|
"custom:rapid7_rule_rrn": "first_alert_rule_rrn"
|
|
"custom:rapid7_event_type": "first_alert_event_type"
|
|
ocsf:
|
|
# ── Finding: the investigation ────────────────────────────────────
|
|
- { source_path: "rrn", ocsf_field: "finding_info.uid" }
|
|
- { source_path: "title", ocsf_field: "finding_info.title" }
|
|
- { source_path: "created_time", ocsf_field: "finding_info.created_time" }
|
|
- { source_path: "last_accessed", ocsf_field: "finding_info.modified_time" }
|
|
- { source_path: "first_alert_time", ocsf_field: "finding_info.first_seen_time" }
|
|
- { source_path: "latest_alert_time", ocsf_field: "finding_info.last_seen_time" }
|
|
- { source_path: "tags", ocsf_field: "metadata.labels" }
|
|
- { source_path: "alert_count", ocsf_field: "count" }
|
|
# ── Incident state ────────────────────────────────────────────────
|
|
- { source_path: "status", ocsf_field: "status" }
|
|
- { source_path: "priority", ocsf_field: "priority" }
|
|
- { source_path: "disposition", ocsf_field: "disposition" }
|
|
- { source_path: "source", ocsf_field: "activity_name" }
|
|
- { source_path: "organization_id", ocsf_field: "cloud.org.uid" }
|
|
- { source_path: "assignee.name", ocsf_field: "assignee.name" }
|
|
- { source_path: "assignee.email", ocsf_field: "assignee.email_addr" }
|
|
# ── Detection time: the MTTD anchor ───────────────────────────────
|
|
# Weakest first, strongest last. The investigation is created after the alert that
|
|
# opened it; the alert's own alerted_at is when the event happened in the source system.
|
|
- { source_path: "created_time", ocsf_field: "time" }
|
|
- { source_path: "first_alert_time", ocsf_field: "time" }
|
|
- { source_path: "alerts[0].alerted_at", ocsf_field: "time" }
|
|
# ── The detection rule, from the first alert ──────────────────────
|
|
# `rule.name` is documented but absent from live ABA alerts, so `type` — which repeats
|
|
# the rule's display name — is the honest fallback.
|
|
- { source_path: "alerts[0].type", ocsf_field: "finding_info.analytic.name" }
|
|
- { source_path: "alerts[0].rule.rrn", ocsf_field: "finding_info.analytic.uid" }
|
|
- { source_path: "alerts[0].rule.version_rrn", ocsf_field: "finding_info.analytic.version" }
|
|
- { source_path: "alerts[0].rrn", ocsf_field: "finding_info.uid_alt" }
|
|
# ── Product identity ──────────────────────────────────────────────
|
|
- { source_path: "'Rapid7 InsightIDR'", ocsf_field: "metadata.product.name" }
|
|
- { source_path: "'Rapid7'", ocsf_field: "metadata.product.vendor_name" }
|
|
- { source_path: "alerts[0].external_source", ocsf_field: "metadata.log_source" }
|
|
- { source_path: "alerts[0].triggering_event_type", ocsf_field: "metadata.event_code" }
|
|
- { source_path: "alerts[0].log_details[0].log_entry_id", ocsf_field: "metadata.uid" }
|
|
# ── MITRE ATT&CK ──────────────────────────────────────────────────
|
|
# mitre_tcodes mixes both kinds in one array: "TA0040" is a tactic, "T1490" a technique.
|
|
# Splitting on the "TA" prefix keeps a tactic out of the technique field, where it would
|
|
# name a technique that does not exist.
|
|
- { source_path: "alerts[0].rule.mitre_tcodes[$substring($, 0, 2) = 'TA'][0]", ocsf_field: "attacks.tactic.uid" }
|
|
- { source_path: "alerts[0].rule.mitre_tcodes[$substring($, 0, 2) != 'TA'][0]", ocsf_field: "attacks.technique.uid" }
|
|
# ── The affected host ─────────────────────────────────────────────
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='hostname'].values[0]", ocsf_field: "device.hostname" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='hostname'].values[0]", ocsf_field: "src_endpoint.hostname" }
|
|
# ── The process the alert is about ────────────────────────────────
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='process.name'].values[0]", ocsf_field: "process.name" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='process.name'].values[0]", ocsf_field: "process.file.name" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='process.cmd_line'].values[0]", ocsf_field: "process.cmd_line" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='process.exe_path'].values[0]", ocsf_field: "process.file.path" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='process.exe_file.hashes.md5'].values[0]", ocsf_field: "process.file.hashes.md5" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='process.exe_file.hashes.sha256'].values[0]", ocsf_field: "process.file.hashes.sha256" }
|
|
# ── Its parent, which says how it was launched ────────────────────
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='parent_process.name'].values[0]", ocsf_field: "actor.process.parent_process.name" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='parent_process.cmd_line'].values[0]", ocsf_field: "actor.process.parent_process.cmd_line" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='parent_process.exe_path'].values[0]", ocsf_field: "actor.process.parent_process.file.path" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='parent_process.exe_file.hashes.md5'].values[0]", ocsf_field: "actor.process.parent_process.file.hashes.md5" }
|
|
# ── The acting account ────────────────────────────────────────────
|
|
# This tenant's field catalogue has no canonical user namespace, so an account only
|
|
# appears when a rule names it among its keys of interest. Both spellings are mapped
|
|
# because rules are not consistent about which they emit.
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='user'].values[0]", ocsf_field: "actor.user.name" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='username'].values[0]", ocsf_field: "actor.user.name" }
|
|
# ── Lateral movement: source and destination ──────────────────────
|
|
# An asset_auth rule describes a connection, so it names BOTH ends and neither
|
|
# is "the host": source_asset_address / destination_asset_address, and
|
|
# source_account / destination_account. Mapping them to src_endpoint and
|
|
# dst_endpoint is not only the correct OCSF shape — it is also what makes them
|
|
# appear in the layout builder, whose field picker is a closed <select> over
|
|
# the OCSF catalogue. A path that is not in that catalogue cannot be placed on
|
|
# a layout at all, which is why `detection` below, useful as it is, only ever
|
|
# shows up in the Context tab.
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='source_asset_address'].values[0]", ocsf_field: "src_endpoint.hostname" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='destination_asset_address'].values[0]", ocsf_field: "dst_endpoint.hostname" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='destination_asset'].values[0]", ocsf_field: "dst_endpoint.hostname" }
|
|
# actor.user is who acted, user is who it was done TO. Keeping them apart is
|
|
# what lets an analyst read a lateral move in one line.
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='source_account'].values[0]", ocsf_field: "actor.user.name" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='destination_account'].values[0]", ocsf_field: "user.name" }
|
|
# ── Every key the rule found interesting ──────────────────────────
|
|
# rule_keys_of_interest is an OPEN vocabulary: each detection rule names its own
|
|
# keys. A process_start_event rule emits hostname / process.cmd_line /
|
|
# parent_process.*; an ingress_auth rule emits account / source_ip /
|
|
# geoip_country_code / result and none of the above. The explicit entries below
|
|
# can therefore only ever cover the rules someone has already seen — on any
|
|
# other rule they all resolve to nothing and the incident is empty again, which
|
|
# is the exact failure this integration started from.
|
|
#
|
|
# So the whole list is also flattened, verbatim, into one object. Keys nobody
|
|
# anticipated arrive intact and are visible to the analyst and to the AI
|
|
# analysis, which reads the mapped context but never `raw`. The explicit
|
|
# mappings stay because they are what types an entity correctly — a value under
|
|
# `detection.` is only ever matched by the IOC regex, so an account there would
|
|
# never be indexed as a user.
|
|
- { source_path: "alerts[0].rule_keys_of_interest{key: values[0]}", ocsf_field: "detection" }
|
|
- { source_path: "alerts[0].rule_matching_keys{key: values[0]}", ocsf_field: "detection_match" }
|
|
# ── Keys seen on non-process rules ────────────────────────────────
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='account'].values[0]", ocsf_field: "actor.user.account.name" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='source_ip'].values[0]", ocsf_field: "src_endpoint.ip" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='destination_ip'].values[0]", ocsf_field: "dst_endpoint.ip" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='geoip_country_code'].values[0]", ocsf_field: "src_endpoint.location.country" }
|
|
- { source_path: "alerts[0].rule_keys_of_interest[key='result'].values[0]", ocsf_field: "status_detail" }
|
|
# ── The evidence: the log line the alert was raised on ────────────
|
|
# An alert names what fired; the evidence is what actually happened. It lives
|
|
# in a JSON-encoded string the command parses and normalises into
|
|
# evidence_fields — see get_investigations_full.py. Vendor prefixes are
|
|
# stripped there (Forti_srcip -> srcip), so one entry serves every log source
|
|
# that names its fields the usual way.
|
|
- { source_path: "alerts[0].evidence_fields", ocsf_field: "evidence" }
|
|
- { source_path: "alerts[0].evidence_fields.srcip", ocsf_field: "src_endpoint.ip" }
|
|
# A parsed source states the same things under its own names. Both spellings
|
|
# are mapped rather than picked between: which one arrives depends on whether
|
|
# the log source was parsed by Rapid7, and one file has to serve both.
|
|
- { source_path: "alerts[0].evidence_fields.source_ip", ocsf_field: "src_endpoint.ip" }
|
|
- { source_path: "alerts[0].evidence_fields.account", ocsf_field: "actor.user.account.name" }
|
|
- { source_path: "alerts[0].evidence_fields.user_rrn", ocsf_field: "actor.user.uid" }
|
|
- { source_path: "alerts[0].evidence_fields.geoip_country_code", ocsf_field: "src_endpoint.location.country" }
|
|
- { source_path: "alerts[0].evidence_fields.geoip_organization", ocsf_field: "src_endpoint.isp_org" }
|
|
- { source_path: "alerts[0].evidence_fields.user_agent", ocsf_field: "http_request.user_agent" }
|
|
- { source_path: "alerts[0].evidence_fields.service", ocsf_field: "metadata.product.feature.name" }
|
|
- { source_path: "alerts[0].evidence_fields.result", ocsf_field: "status_detail" }
|
|
- { source_path: "alerts[0].evidence_fields.dstip", ocsf_field: "dst_endpoint.ip" }
|
|
- { source_path: "alerts[0].evidence_fields.srcport", ocsf_field: "src_endpoint.port" }
|
|
- { source_path: "alerts[0].evidence_fields.dstport", ocsf_field: "dst_endpoint.port" }
|
|
- { source_path: "alerts[0].evidence_fields.user", ocsf_field: "actor.user.name" }
|
|
- { source_path: "alerts[0].evidence_fields.devname", ocsf_field: "device.name" }
|
|
- { source_path: "alerts[0].evidence_fields.devname", ocsf_field: "device.hostname" }
|
|
- { source_path: "alerts[0].evidence_fields.devid", ocsf_field: "device.uid" }
|
|
- { source_path: "alerts[0].evidence_fields.hostname", ocsf_field: "device.hostname" }
|
|
- { source_path: "alerts[0].evidence_fields.status", ocsf_field: "status_detail" }
|
|
# `action` is what the source system did ("login"), which says more than the
|
|
# investigation's own `source` ("ALERT"). Mapped after it so it wins.
|
|
- { source_path: "alerts[0].evidence_fields.action", ocsf_field: "activity_name" }
|
|
# The line itself, on the OCSF field meant for it — and one the layout builder
|
|
# offers, unlike `evidence` above, which only ever shows in the Context tab.
|
|
- { source_path: "alerts[0].evidence_raw_line", ocsf_field: "raw_data" }
|
|
- { source_path: "alerts[0].evidence_fields.msg ? alerts[0].evidence_fields.msg : alerts[0].evidence_fields.logdesc", ocsf_field: "message" }
|
|
# ── Process tree ──────────────────────────────────────────────────
|
|
# The layout block reads ctx.process_tree: N flat chains, root first, each
|
|
# {alert_id, rule, host, user, nodes[]}. Rapid7 nests its tree through
|
|
# `children`, so the command flattens each root-to-leaf path into one chain
|
|
# and rolls every alert's chains up to the investigation — see
|
|
# get_investigations_full.py. Passed through verbatim; nothing to reshape here.
|
|
- { source_path: "process_tree", ocsf_field: "process_tree" }
|
|
# ── Every alert, not just the first ───────────────────────────────
|
|
# An investigation aggregates alerts, and the fields above only ever read alerts[0].
|
|
# This entry lands the whole set as OCSF-shaped objects, so the host and the process of
|
|
# the SECOND and THIRD alert are indexed as entities too — the extractor walks arrays
|
|
# and classifies on the path's suffix, so `.hostname` under here counts exactly as
|
|
# `device.hostname` does above.
|
|
- source_path: >
|
|
alerts.{
|
|
"uid": rrn,
|
|
"title": title,
|
|
"time": alerted_at,
|
|
"status": status,
|
|
"disposition": disposition,
|
|
"hostname": rule_keys_of_interest[key='hostname'].values[0],
|
|
"src_hostname": rule_keys_of_interest[key='source_asset_address'].values[0],
|
|
"dst_hostname": rule_keys_of_interest[key='destination_asset_address'].values[0],
|
|
"actor_user": rule_keys_of_interest[key='source_account'].values[0],
|
|
"target_user": rule_keys_of_interest[key='destination_account'].values[0],
|
|
"process": rule_keys_of_interest[key='process.name'].values[0],
|
|
"cmd_line": rule_keys_of_interest[key='process.cmd_line'].values[0],
|
|
"technique": rule.mitre_tcodes[$substring($, 0, 2) != 'TA'][0],
|
|
"detection": rule_keys_of_interest{key: values[0]},
|
|
"evidence": evidence_fields
|
|
}
|
|
ocsf_field: "evidences"
|