feat(rapid7-insightidr): evidence rows for the table field (v1.11.0)

The Markdown summary reads as a page. It cannot be sorted or filtered, and an
investigation aggregating several alerts, each with several evidences, is
exactly where those are wanted: "Multiple User Accounts Were Disabled" is a
question about which accounts, and the answer is a column.

So one row per evidence, across every alert, into custom:rapid7_evidence. The
table field reads an array of objects as it stands, so nothing is reshaped on
the mapper side. It sits beside the summary rather than replacing it — one is
read, the other is queried.

Two decisions worth stating.

Each row carries its alert's name first. An investigation aggregates several,
and a row that does not say where it came from cannot be acted on.

Column order is load-bearing, because it is the only ordering signal the field
has: it takes the union of keys in order of first appearance, so whatever the
first row names first is what an analyst sees without scrolling. Known keys
lead — action, result, the source and target pairs, the host, the addresses —
and the rest follow alphabetically.

Width is capped at 30 columns, decided by how many rows carry each key rather
than by which appeared first: an investigation mixing rule types has a sparse
union, and a column present on one row out of eighty would otherwise push out
one present on all of them. Exercised on a 187-column case — the five keys
common to every row all survived, and the count is reported as
evidence_columns_dropped rather than the loss being silent.

Verified against the account-disabled investigation: two evidences become two
rows of twenty columns, `alert` first, the two disabled accounts in
target_account — checked through the table field's own reader, not just the
generator.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

integrations/rapid7-insightidr/manifest.yaml

@@ -1,8 +1,8 @@
 id: rapid7_insightidr
 name: Rapid7 InsightIDR
-version: 1.10.0
+version: 1.11.0
 description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search."
-changelog: "1.10.0 — Render every detection key as Markdown into custom:rapid7_summary, so a layout can show what the OCSF catalogue has no field for. Map both ends of an account action, read source_json top-level scalars, drop placeholder values, and force evidences to stay an array. | 1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
+changelog: "1.11.0 — Build one evidence row per alert evidence into custom:rapid7_evidence, for the table field: sortable and filterable where the Markdown summary only reads. | 1.10.0 — Render every detection key as Markdown into custom:rapid7_summary, so a layout can show what the OCSF catalogue has no field for. Map both ends of an account action, read source_json top-level scalars, drop placeholder values, and force evidences to stay an array. | 1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
 category: siem
 
 # Per-instance configuration. The Insight platform region selects the API host

integrations/rapid7-insightidr/mappers/get_investigations_full.yaml

@@ -44,6 +44,11 @@ field_mappings:
   # what no OCSF mapping can, because the layout's field picker only offers paths
   # from the OCSF catalogue and this vocabulary is open-ended.
   "custom:rapid7_summary": "summary_markdown"
+  # One row per evidence, across every alert. Declare a custom field named
+  # rapid7_evidence of type table: it reads an array of objects as it stands, so
+  # nothing is reshaped here. It sits beside the Markdown summary rather than
+  # replacing it — the summary reads as a page, this one sorts and filters.
+  "custom:rapid7_evidence": "evidence_rows"
 ocsf:
   # ── Finding: the investigation ────────────────────────────────────
   - { source_path: "rrn",                ocsf_field: "finding_info.uid" }

integrations/rapid7-insightidr/scripts/get_investigations_full.py

@@ -330,6 +330,75 @@ _MD_MAX_ROWS = 40
 _MD_MAX_VALUE = 300
 
 
+# The columns worth reading first, in this order. Everything else follows
+# alphabetically. Column order is the ONLY ordering signal the table field has:
+# it takes the union of keys in order of first appearance, so what the first row
+# names first is what the analyst sees without scrolling.
+_ROW_LEAD = [
+    "alert", "time", "action", "result", "status",
+    "source_account", "source_user", "target_account", "target_user",
+    "hostname", "computername", "source_asset", "devname",
+    "source_ip", "srcip", "destination_ip", "dstip",
+    "user", "account", "process.name", "process.cmd_line",
+]
+_MAX_ROWS = 100
+# A table wider than this is scrolled, not read. The cap keeps the useful
+# columns — the lead list, then whichever others are populated on the most rows
+# — rather than whichever happened to appear first.
+_MAX_COLUMNS = 30
+
+
+def _evidence_rows(alerts):
+    """One row per evidence, across every alert of the investigation.
+
+    The table custom field reads an array of objects directly, so this needs no
+    reshaping on the mapper side. It exists beside summary_markdown rather than
+    instead of it: the summary reads as a page, this sorts and filters.
+
+    Each row carries its alert's name, because an investigation aggregates
+    several and a row that does not say where it came from cannot be acted on.
+    """
+    rows = []
+    for a in alerts:
+        for ev in (a.get("evidences") or []):
+            f = ev.get("fields") or {}
+            if not f:
+                continue
+            row = {"alert": a.get("type") or a.get("title") or ""}
+            t = ev.get("evented_at") or f.get("timestamp")
+            if t:
+                row["time"] = str(t)
+            for k in _ROW_LEAD:
+                if k in f and k not in row:
+                    row[k] = f[k]
+            for k in sorted(f):
+                if k not in row:
+                    row[k] = f[k]
+            rows.append(row)
+            if len(rows) >= _MAX_ROWS:
+                break
+        if len(rows) >= _MAX_ROWS:
+            break
+    if not rows:
+        return [], 0
+
+    # Trim the width. Counted over every row, so a column present once out of
+    # eighty does not push out one present on all of them.
+    counts = {}
+    for r in rows:
+        for k in r:
+            counts[k] = counts.get(k, 0) + 1
+    ordered = [k for k in _ROW_LEAD + ["alert", "time"] if k in counts]
+    ordered += sorted((k for k in counts if k not in ordered),
+                      key=lambda k: (-counts[k], k))
+    # `alert` and `time` lead whatever else survives.
+    keep = ["alert"] + [k for k in ordered if k != "alert"][:_MAX_COLUMNS - 1]
+    dropped = len(counts) - len(keep)
+    if dropped > 0:
+        rows = [{k: r[k] for k in keep if k in r} for r in rows]
+    return rows, max(dropped, 0)
+
+
 def _markdown_summary(inv, alerts):
     """A readable rendering of what the detection actually says.
 
@@ -533,6 +602,11 @@ def run():
         # nested where no mapper expression could reach past alerts[0].
         if alerts:
             inv["summary_markdown"] = _markdown_summary(inv, alerts)
+            ev_rows, ev_dropped = _evidence_rows(alerts)
+            if ev_rows:
+                inv["evidence_rows"] = ev_rows
+                if ev_dropped:
+                    inv["evidence_columns_dropped"] = ev_dropped
         rolled = []
         for a in alerts:
             rolled.extend(a.get("process_tree") or [])