An alert names what fired. The evidence is what happened, and until now it was
never fetched — which is why an incident could show a rule name and nothing
else. On a Fortigate admin login the alert carried an empty
rule_keys_of_interest; the evidence carried the whole event.
It cannot be reached by mapping alone. The evidence's `data` is a
JSON-ENCODED STRING holding {source_data, custom_data}, and JSONata does not
deserialise, so no mapper expression can ever see inside it. The parsing has to
happen in the command, and that is the reason for this change more than the
extra call is.
Once parsed, two views of the same event:
custom_data the vendor's normalised fields, prefixed per source
(Forti_user, Forti_srcip) — 6 keys on the sample
source_data the raw line in key="value" form — 25 keys
Both are merged into evidence_fields with the vendor prefix stripped, so a
mapper names `user` once instead of once per vendor, and source_data is overlaid
last because it carries more and is not affected by the quoting artefact below.
Values are cleaned, and that is not cosmetic: custom_data returns
'10.70.20.66"' with a trailing quote left by the vendor parser. Kept as-is it
yields an address matching no regex — an observable nobody can pivot on, and no
error to say why.
The line itself lands on raw_data and the source's own sentence on message,
both of which the layout builder offers. The Fortigate login now maps to
src_endpoint.ip, dst_endpoint.ip, actor.user.name, device.hostname/name/uid,
activity_name "login" and status_detail "success", and indexes four entities
where it indexed none.
Optional (include_evidence, default true) and capped (max_evidence_per_alert,
default 5), one call per alert under the run's existing time budget. An alert
whose evidence cannot be read keeps an evidence_error and still arrives;
evidence_calls and evidence_errors are reported in the run metadata.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An RRN states its region — rrn:alerts:eu:<org>:alert:… — while the base URL is
built from the instance's configured `region`. Following that thread found a
gap that matters more than the naming.
The Alert Triage API is multi-region behind one gateway, and eleven of its
endpoints can answer HTTP 200 with a region_failures list. The spec is explicit:
the presence of items in that field IS the partial failure. Three of the
commands here call such endpoints and all three read only the payload, so a
region the gateway could not reach produced fewer alerts and nothing else — no
error, no count, no log line. For an ingestion feed that is indistinguishable
from a quiet period, which is the one failure mode this integration has now hit
three separate times.
So region_failures is carried into the run metadata by every command that can
receive it, alongside the regions actually seen in the collected RRNs, the
configured region, and any mismatch between them.
The mismatch report is deliberately a report and not a correction. On a
single-region tenant the two always agree. On a multi-customer key, which spans
organizations, they need not — and rewriting the host per resource would be a
guess about how Rapid7 routes these calls, made without a tenant to verify it
against. Naming the discrepancy is what can be done honestly today; if it ever
fires, the metadata says so instead of the alerts silently going missing.
list_alert_fields already passed region_failures through untouched, since it
returns the API response as it stands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The run died with HTTP 400 and eight "The provided RRN: [...] is invalid",
every one a plain UUID. Two mistakes compounded.
The alert list under an investigation is not consistent about where the RRN
sits. On the ABA alert this work was built against, `id` held the RRN. On
others it holds a bare UUID. The code read `id` first and `rrn` only as a
fallback, so it picked the UUID whenever both were present.
Worse, the failure was total rather than partial. /alerts/ops/rrns validates the
whole list and rejects the entire request when one entry is malformed, so a
single bad identifier costs every alert in that batch — and with one batch, the
whole run. An investigation feed that stops dead because one alert is shaped
differently is not an acceptable failure mode.
So: read `rrn` first, and drop anything that is not RRN-shaped before the batch
is sent, counting it as alert_ids_without_rrn in the run metadata. The
investigation still arrives, its retrievable alerts still arrive, and what could
not be fetched is a number an operator can see rather than an absence nobody
would notice.
Both commands that walk this list are fixed; the selector is shared and was
checked against the two real shapes plus the empty and null cases.
This turns a total failure into a partial one. It does NOT establish that those
eight alerts are reachable at all: if they carry no RRN anywhere, they are not
in the Alert Triage system and no field_ids or batch call will produce them.
That needs the raw shape of one, which the next run of list_investigation_alerts
on the failing investigation will show.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous command returned the alerts as its result rows, which quietly
changed the unit: an investigation holding three alerts would have become three
incidents. The investigation is what an analyst opens and what the SOC tracks —
it stays the incident, and the alerts are its content.
get_investigations_full walks the same three hops and inverts the shape:
investigations under `data`, each carrying its alerts under `alerts` plus an
alert_count. results_path=data and dedup_key=rrn, so it is a drop-in replacement
for list_investigations in an existing ingestion rule — only the command name
and the mapper change.
A few first-alert values are also lifted to the top level, and that is not
cosmetic. MapFields resolves dotted paths only and cannot index an array, so
`alerts[0].ai_suggested_disposition` in a field_mappings entry reads as nothing
— silently, leaving the custom field empty with no error anywhere. Three custom
fields were empty for exactly that reason until an end-to-end run showed it. The
ocsf section has no such limit: it is evaluated by full JSONata and indexes
freely, which is why the host and process mappings worked while the custom
fields did not.
The mapper maps the investigation first and the first alert after, so the alert
wins wherever the container says nothing. It also lands every alert — not just
alerts[0] — as OCSF-shaped objects under `evidences`, because the extractor
walks arrays and classifies on the path suffix: an investigation aggregating
two alerts on two machines now indexes both hosts, where reading alerts[0]
alone would have indexed one and silently dropped the other.
Verified end to end against the investigation that started this, with a second
alert added on another host: field mapping, OCSF mapping and entity extraction.
Both hosts are indexed. With the incident type's extraction policy set to
selected/host,ip,user, the entity set is exactly those two — the alert RRN's
32-hex suffix, which the IOC regex cannot tell from an MD5, is filtered out
there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f3nris