An "RDP connection to DC" investigation arrived with device {}, process {} and
two indexed entities, both of them false — the 32-hex tails of its own alert
RRNs, which the IOC regex cannot tell from an MD5. Its four real values were
sitting in the payload, unmapped:
source_account vbalbakakis
destination_account vassilisadmin
source_asset_address sgazswvdip-01.toto.priv
destination_asset_address scazsvpdc01.toto.priv
The mappings assumed an alert has one host and one user, which holds for a
process_start_event and not for an auth rule: a connection names both ends and
neither is "the host". So source and destination now land on src_endpoint and
dst_endpoint, the acting account on actor.user and the account acted upon on
user — the distinction that lets a lateral move be read in one line.
That choice is also what makes the fields placeable. The layout builder's field
picker is a closed <select> over the OCSF catalogue in web/src/lib/ocsf.ts;
a path absent from it cannot be put on a layout at all. All four targets are in
that catalogue. `detection`, the open-vocabulary catch-all, is not and cannot
be — it keeps its value in the Context tab, which renders the whole tree, but a
layout cannot reference it, so anything meant for a layout has to be mapped to a
real OCSF field first.
evidences[] is flattened at the same time. It emitted "device": {}, "process":
{} and "attacks": {"technique": {}} whenever those keys were absent — three
empty objects per alert, filling the context and hiding what was populated.
Verified against the RDP investigation: four entities where there were none,
two hosts and two accounts, and no false hash once the incident type's
extraction policy is set to selected/host,ip,user.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both mappers read that field through a hardcoded list of keys — hostname,
process.cmd_line, parent_process.*, user — taken from the one alert this work
was built against, a process_start_event detection. Every entry is a JSONata
predicate on an exact key, so a rule emitting anything else matches none of
them.
A real alert from a second tenant showed what that costs. It is a custom
ingress_auth rule, and its keys are:
account locuz_hma@totogroup.net
source_ip 183.82.103.101
geoip_country_code IN
result SUCCESS
Not one appears in the hardcoded list. The alert was fetched, inlined and
mapped, and produced an incident as empty as the ones this integration set out
to fix — with the account and the source IP sitting in the payload, unread.
Two changes:
- The whole list is flattened verbatim into `detection`, and rule_matching_keys
into `detection_match`. Keys nobody anticipated now arrive intact, and they
reach the AI analysis, which reads the mapped context and never `raw`.
- account, source_ip, destination_ip, geoip_country_code and result gain
explicit OCSF entries. The explicit mappings are not redundant with the
flattening: a value under `detection.` is only ever seen by the IOC regex,
so an account there would be indexed as an email, or not at all, rather than
as a user. Typing an entity correctly still requires naming its field.
evidences[] carries the flattened keys too, so the same holds for the second and
third alert of an investigation, not only the first.
Verified against both rule shapes end to end. The process rule yields the two
hosts it did before; the authentication rule now yields the source IP and the
account, where it previously yielded nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous command returned the alerts as its result rows, which quietly
changed the unit: an investigation holding three alerts would have become three
incidents. The investigation is what an analyst opens and what the SOC tracks —
it stays the incident, and the alerts are its content.
get_investigations_full walks the same three hops and inverts the shape:
investigations under `data`, each carrying its alerts under `alerts` plus an
alert_count. results_path=data and dedup_key=rrn, so it is a drop-in replacement
for list_investigations in an existing ingestion rule — only the command name
and the mapper change.
A few first-alert values are also lifted to the top level, and that is not
cosmetic. MapFields resolves dotted paths only and cannot index an array, so
`alerts[0].ai_suggested_disposition` in a field_mappings entry reads as nothing
— silently, leaving the custom field empty with no error anywhere. Three custom
fields were empty for exactly that reason until an end-to-end run showed it. The
ocsf section has no such limit: it is evaluated by full JSONata and indexes
freely, which is why the host and process mappings worked while the custom
fields did not.
The mapper maps the investigation first and the first alert after, so the alert
wins wherever the container says nothing. It also lands every alert — not just
alerts[0] — as OCSF-shaped objects under `evidences`, because the extractor
walks arrays and classifies on the path suffix: an investigation aggregating
two alerts on two machines now indexes both hosts, where reading alerts[0]
alone would have indexed one and silently dropped the other.
Verified end to end against the investigation that started this, with a second
alert added on another host: field mapping, OCSF mapping and entity extraction.
Both hosts are indexed. With the incident type's extraction policy set to
selected/host,ip,user, the entity set is exactly those two — the alert RRN's
32-hex suffix, which the IOC regex cannot tell from an MD5, is filtered out
there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ingesting from search_alerts brings back every alert the tenant produces,
including the operational noise — two of the first three sampled were an agent
reporting it lacked Full Disk Access. What is worth an incident is what Rapid7
or the analyst folded into an investigation.
Filtering search_alerts on investigation_rrn does not work. The term is accepted
— no validation error, HTTP 200 — and matches nothing: zero alerts over fifteen
days, on a tenant where an alert carrying an investigation_rrn provably exists
in that window. The field is not searchable, and the API says so by returning an
empty page rather than by refusing the term. That failure mode is the reason not
to build an ingestion feed on it: a filter that silently stops matching drains
the feed to zero with no error anywhere, and nothing would ever say why.
search_investigation_alerts is guaranteed by construction instead:
1. the investigations in the window (incremental on their created_time)
2. their alert RRNs, one call per investigation — there is no bulk form
3. the alerts themselves via /alerts/ops/rrns, batched 100 at a time
So 1+N+1 calls, and an alert with no investigation is never even named. Its
failure modes are loud: an unreadable investigation is skipped and counted, and
HTTP errors surface as themselves.
Step 2 is the one that grows with the window, so it runs under a time budget
(60s, against the server's 90s ceiling) and reports truncated=true rather than
being killed with the whole run's alerts lost. A partial fetch that says it is
partial beats a silent zero — the same reasoning that rules out the filter.
The output shape matches search_alerts exactly, so the mapper committed in 1.2.0
serves both. It gains five entries for the parent investigation, which resolve
to nothing on the search_alerts feed and are skipped there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An incident built from a Rapid7 investigation indexed zero entities. Nothing to
pivot on, nothing to correlate, and an AI analysis with nothing to read. The
reason turned out not to be the mapper but the feed: an investigation is an
envelope, and neither it nor its /alerts list carries a host, an account, a
process or a MITRE code.
The Alert Triage API does, in rule_keys_of_interest. On the alert behind the
investigation that started this, that field holds:
hostname NBSGIT01-26
process.name bcdedit.exe
process.cmd_line bcdedit /set {default} recoveryenabled No
process.exe_path C:\WINDOWS\System32\bcdedit.exe
parent_process.name cmd.exe
parent_process.cmd_line "C:\WINDOWS\system32\cmd.exe"
parent_process.exe_file.hashes.md5 e463e36104d22a2479046a7d4f620816
with rule.mitre_tcodes ["TA0040", "T1490"] — Impact / Inhibit System Recovery,
which is what disabling Windows recovery actually is. None of it was reachable
before.
It is a list of {key, values} pairs rather than an object, so every entry reads
through a JSONata predicate — rule_keys_of_interest[key='hostname'].values[0].
A key a rule does not emit resolves to nothing and is skipped, so one file
covers every detection rule.
Two details worth keeping:
- mitre_tcodes mixes tactics and techniques in one array ("TA0040" beside
"T1490"). They are split on the "TA" prefix so a tactic never lands in the
technique field, where it would name a technique that does not exist.
- rule.name is documented but absent from live ABA alerts, so the analytic
name falls back to `type`, which repeats the rule's display name.
Rapid7's own status, disposition and AI-suggested disposition ride as custom
fields. They cannot reach the incident's status and verdict columns — MapFields
accepts only title, severity, source, description and custom:* — but an analyst
can at least see that the source already closed the alert and what it decided.
The AI suggestion is exposed as a fact, never as a verdict.
Validated end to end against a live alert: field mapping, OCSF mapping and
entity extraction, which now yields the host and the parent process hash where
it previously yielded nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
REST API integration for Rapid7 InsightIDR. Investigation ingestion
(list_investigations) with an exhaustive OCSF mapper and a bundled default
incident type, plus 18 commands across investigations (list/get/search/
create/update/assign/set-status/bulk-close), investigation alerts and Rapid7
product alerts, custom threat indicators (add/replace), log management and
LEQL log/log-set queries with downloads, and user directory search.
API v1/v2 selectable per instance (is_v2) and per command (api_version);
multi-customer query parameter supported on v2 calls.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
f3nris
Guillaume BOURGEOIS