ab63ead00ebf570b69e65b79251f48c3d648a72b
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 f3nris
|
9081bc66fa |
feat(rapid7-insightidr): evidence rows for the table field (v1.11.0)
The Markdown summary reads as a page. It cannot be sorted or filtered, and an investigation aggregating several alerts, each with several evidences, is exactly where those are wanted: "Multiple User Accounts Were Disabled" is a question about which accounts, and the answer is a column. So one row per evidence, across every alert, into custom:rapid7_evidence. The table field reads an array of objects as it stands, so nothing is reshaped on the mapper side. It sits beside the summary rather than replacing it — one is read, the other is queried. Two decisions worth stating. Each row carries its alert's name first. An investigation aggregates several, and a row that does not say where it came from cannot be acted on. Column order is load-bearing, because it is the only ordering signal the field has: it takes the union of keys in order of first appearance, so whatever the first row names first is what an analyst sees without scrolling. Known keys lead — action, result, the source and target pairs, the host, the addresses — and the rest follow alphabetically. Width is capped at 30 columns, decided by how many rows carry each key rather than by which appeared first: an investigation mixing rule types has a sparse union, and a column present on one row out of eighty would otherwise push out one present on all of them. Exercised on a 187-column case — the five keys common to every row all survived, and the count is reported as evidence_columns_dropped rather than the loss being silent. Verified against the account-disabled investigation: two evidences become two rows of twenty columns, `alert` first, the two disabled accounts in target_account — checked through the table field's own reader, not just the generator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
f3nris
|
b26257aaeb |
feat(rapid7-insightidr): render the detection as Markdown, and fix four gaps (v1.10.0)
An "Multiple User Accounts Were Disabled" investigation came back with one entity — the actor — while the layout stayed nearly empty. Four separate reasons, and the last one is why the first three keep recurring. evidences was an OBJECT, not an array. JSONata returns a bare object when the path yields one item, so the key had one shape on an investigation with a single alert and another with two. Bracketed to force an array. Only the actor was mapped. An ad_admin rule names who acted AND who it was done to, and on this alert the disabled accounts are the point of it. Target account, target user and both domains now map to the OCSF Target User group, the actor to Actor — the distinction that lets the sentence be read in one line. The domain controller was nowhere. A Windows event puts it in source_json.computerName and nowhere else, and source_json was skipped whole because the Azure AD one runs to kilobytes. Its top-level SCALARS are now read — bounded, and where computerName, osVersion and callerIpAddress live — while its nested part stays skipped. source_asset was the string "unknown". Mapped as-is it becomes an asset named "unknown" in the entity index, shared by every incident that had no asset: a pivot that groups things by the absence of a fact. Placeholders are dropped, and a key whose value cleans away to nothing is removed rather than kept empty, since "" is not nil and would still be written to its OCSF field. Five entities now, from one: both accounts, both display names, the DC. And the reason this kept happening. The layout builder's field picker is a closed <select> over the OCSF catalogue in web/src/lib/ocsf.ts, so a path absent from it cannot be placed on a layout at all — which is what `detection` and `evidence` are, and what every unanticipated key will be. Mapping each one to a real OCSF field is right where a field exists, but rule_keys_of_interest is an open vocabulary: four spellings turned up in as many days. So the whole of it is also rendered as a Markdown table into custom:rapid7_summary — keys of interest first, since the rule itself says those are what matter, then the evidence fields. Declare a custom field of type markdown with that name, put it on the layout, and a new rule type is readable the day it appears instead of after another mapping round. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
f3nris
|
e19530d3f3 |
fix(rapid7-insightidr): read the evidence in both of its shapes (v1.9.2)
evidence_fields came back {} on a live run. The evidence was fetched, the run
was clean — one call, 1.3s, no error — and the parser found nothing in it.
`data` does not have one shape. The parser was written against an unparsed
Fortigate source and looked only for {source_data, custom_data}. A PARSED
source has no such wrapper: the event is the object.
{user, account, source_ip, geoip_country_code, geoip_organization,
result, service, user_agent, source_json{…}, r7_context{…},
detection_context{…}}
Every one of those is what the incident was missing, sitting one json.loads
away, and the parser walked past all of it and returned an empty dict. No error
— which is how it survived a run that reported success.
Now both are read as a union rather than one or the other: scalars at the top
level are always taken, so a third shape still yields its plain fields instead
of nothing, and the wrapped keys are overlaid when present. source_json is
skipped — several kilobytes of vendor payload that stay in the raw line —
while r7_context and detection_context are flattened one level, being Rapid7's
own annotations and the same across parsed sources. They carry the account age
in minutes, which is exactly what says whether a sign-in is from a new account.
The Azure AD sign-in now maps to src_endpoint.ip, .isp_org and
.location.country, actor.user.name/.uid/.account.name, http_request.user_agent
and metadata.product.feature.name, and indexes the address and the account.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
f3nris
|
1888648930 |
feat(rapid7-insightidr): feed the layout Process Tree block (v1.9.0)
Rapid7 exposes a process tree per alert; the layout renderer has a block that
draws one. They do not speak the same shape.
The block reads ctx.process_tree as N FLAT chains, root first, each
{alert_id, rule, host, user, nodes[]} — it merges shared ancestors itself.
Rapid7 answers with a NESTED tree: one root entity and children[]. So the
command walks each root-to-leaf path into its own chain, maps the node fields
(cmdline -> cmd_line, hashes.md5 -> md5, start_time -> started_at, pid coerced
from the string Rapid7 sends), and rolls every alert's chains up onto the
investigation, since the block reads the key at the top level and no mapper
expression reaches past alerts[0].
Verified through the block's own reader: a tree of explorer -> cmd ->
{bcdedit, vssadmin} flattens to two chains and comes back as one tree with the
ancestors shared and both leaves marked.
That mark is worth stating plainly. The block marks the last node of a chain as
the target, and Rapid7 flags no node as the one the alert fired on — so here it
means "end of this branch" and nothing stronger. Naming a target would assert
something the sensor never said.
Gated, because this is a third call per alert on top of the evidence one:
include_process_tree=auto (the default) asks only for the event types an
endpoint sensor produces a tree for, true asks for every alert, false never.
An authentication or firewall alert never has one, and on a feed where those
dominate the default saves a call per alert. 404 is treated as "no tree", which
is what it means, and only other statuses are recorded as errors. Bounded at 12
chains and 24 levels so an unexpected shape cannot expand without limit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
f3nris
|
995a39ea66 |
feat(rapid7-insightidr): fetch and parse the evidence behind each alert (v1.8.0)
An alert names what fired. The evidence is what happened, and until now it was
never fetched — which is why an incident could show a rule name and nothing
else. On a Fortigate admin login the alert carried an empty
rule_keys_of_interest; the evidence carried the whole event.
It cannot be reached by mapping alone. The evidence's `data` is a
JSON-ENCODED STRING holding {source_data, custom_data}, and JSONata does not
deserialise, so no mapper expression can ever see inside it. The parsing has to
happen in the command, and that is the reason for this change more than the
extra call is.
Once parsed, two views of the same event:
custom_data the vendor's normalised fields, prefixed per source
(Forti_user, Forti_srcip) — 6 keys on the sample
source_data the raw line in key="value" form — 25 keys
Both are merged into evidence_fields with the vendor prefix stripped, so a
mapper names `user` once instead of once per vendor, and source_data is overlaid
last because it carries more and is not affected by the quoting artefact below.
Values are cleaned, and that is not cosmetic: custom_data returns
'10.70.20.66"' with a trailing quote left by the vendor parser. Kept as-is it
yields an address matching no regex — an observable nobody can pivot on, and no
error to say why.
The line itself lands on raw_data and the source's own sentence on message,
both of which the layout builder offers. The Fortigate login now maps to
src_endpoint.ip, dst_endpoint.ip, actor.user.name, device.hostname/name/uid,
activity_name "login" and status_detail "success", and indexes four entities
where it indexed none.
Optional (include_evidence, default true) and capped (max_evidence_per_alert,
default 5), one call per alert under the run's existing time budget. An alert
whose evidence cannot be read keeps an evidence_error and still arrives;
evidence_calls and evidence_errors are reported in the run metadata.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
f3nris
|
eeebbfed3a |
fix(rapid7-insightidr): a partial region result is not a quiet period (v1.6.0)
An RRN states its region — rrn:alerts:eu:<org>:alert:… — while the base URL is built from the instance's configured `region`. Following that thread found a gap that matters more than the naming. The Alert Triage API is multi-region behind one gateway, and eleven of its endpoints can answer HTTP 200 with a region_failures list. The spec is explicit: the presence of items in that field IS the partial failure. Three of the commands here call such endpoints and all three read only the payload, so a region the gateway could not reach produced fewer alerts and nothing else — no error, no count, no log line. For an ingestion feed that is indistinguishable from a quiet period, which is the one failure mode this integration has now hit three separate times. So region_failures is carried into the run metadata by every command that can receive it, alongside the regions actually seen in the collected RRNs, the configured region, and any mismatch between them. The mismatch report is deliberately a report and not a correction. On a single-region tenant the two always agree. On a multi-customer key, which spans organizations, they need not — and rewriting the host per resource would be a guess about how Rapid7 routes these calls, made without a tenant to verify it against. Naming the discrepancy is what can be done honestly today; if it ever fires, the metadata says so instead of the alerts silently going missing. list_alert_fields already passed region_failures through untouched, since it returns the API response as it stands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
f3nris
|
69e3a3e81c |
fix(rapid7-insightidr): do not send a bare UUID to /alerts/ops/rrns (v1.4.1)
The run died with HTTP 400 and eight "The provided RRN: [...] is invalid", every one a plain UUID. Two mistakes compounded. The alert list under an investigation is not consistent about where the RRN sits. On the ABA alert this work was built against, `id` held the RRN. On others it holds a bare UUID. The code read `id` first and `rrn` only as a fallback, so it picked the UUID whenever both were present. Worse, the failure was total rather than partial. /alerts/ops/rrns validates the whole list and rejects the entire request when one entry is malformed, so a single bad identifier costs every alert in that batch — and with one batch, the whole run. An investigation feed that stops dead because one alert is shaped differently is not an acceptable failure mode. So: read `rrn` first, and drop anything that is not RRN-shaped before the batch is sent, counting it as alert_ids_without_rrn in the run metadata. The investigation still arrives, its retrievable alerts still arrive, and what could not be fetched is a number an operator can see rather than an absence nobody would notice. Both commands that walk this list are fixed; the selector is shared and was checked against the two real shapes plus the empty and null cases. This turns a total failure into a partial one. It does NOT establish that those eight alerts are reachable at all: if they carry no RRN anywhere, they are not in the Alert Triage system and no field_ids or batch call will produce them. That needs the raw shape of one, which the next run of list_investigation_alerts on the failing investigation will show. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
f3nris
|
841ae1199f |
feat(rapid7-insightidr): the investigation, with its alerts inlined (v1.4.0)
The previous command returned the alerts as its result rows, which quietly changed the unit: an investigation holding three alerts would have become three incidents. The investigation is what an analyst opens and what the SOC tracks — it stays the incident, and the alerts are its content. get_investigations_full walks the same three hops and inverts the shape: investigations under `data`, each carrying its alerts under `alerts` plus an alert_count. results_path=data and dedup_key=rrn, so it is a drop-in replacement for list_investigations in an existing ingestion rule — only the command name and the mapper change. A few first-alert values are also lifted to the top level, and that is not cosmetic. MapFields resolves dotted paths only and cannot index an array, so `alerts[0].ai_suggested_disposition` in a field_mappings entry reads as nothing — silently, leaving the custom field empty with no error anywhere. Three custom fields were empty for exactly that reason until an end-to-end run showed it. The ocsf section has no such limit: it is evaluated by full JSONata and indexes freely, which is why the host and process mappings worked while the custom fields did not. The mapper maps the investigation first and the first alert after, so the alert wins wherever the container says nothing. It also lands every alert — not just alerts[0] — as OCSF-shaped objects under `evidences`, because the extractor walks arrays and classifies on the path suffix: an investigation aggregating two alerts on two machines now indexes both hosts, where reading alerts[0] alone would have indexed one and silently dropped the other. Verified end to end against the investigation that started this, with a second alert added on another host: field mapping, OCSF mapping and entity extraction. Both hosts are indexed. With the incident type's extraction policy set to selected/host,ip,user, the entity set is exactly those two — the alert RRN's 32-hex suffix, which the IOC regex cannot tell from an MD5, is filtered out there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |