feat(rapid7-insightidr): evidence rows for the table field (v1.11.0)

The Markdown summary reads as a page. It cannot be sorted or filtered, and an
investigation aggregating several alerts, each with several evidences, is
exactly where those are wanted: "Multiple User Accounts Were Disabled" is a
question about which accounts, and the answer is a column.

So one row per evidence, across every alert, into custom:rapid7_evidence. The
table field reads an array of objects as it stands, so nothing is reshaped on
the mapper side. It sits beside the summary rather than replacing it — one is
read, the other is queried.

Two decisions worth stating.

Each row carries its alert's name first. An investigation aggregates several,
and a row that does not say where it came from cannot be acted on.

Column order is load-bearing, because it is the only ordering signal the field
has: it takes the union of keys in order of first appearance, so whatever the
first row names first is what an analyst sees without scrolling. Known keys
lead — action, result, the source and target pairs, the host, the addresses —
and the rest follow alphabetically.

Width is capped at 30 columns, decided by how many rows carry each key rather
than by which appeared first: an investigation mixing rule types has a sparse
union, and a column present on one row out of eighty would otherwise push out
one present on all of them. Exercised on a 187-column case — the five keys
common to every row all survived, and the count is reported as
evidence_columns_dropped rather than the loss being silent.

Verified against the account-disabled investigation: two evidences become two
rows of twenty columns, `alert` first, the two disabled accounts in
target_account — checked through the table field's own reader, not just the
generator.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

integrations/rapid7-insightidr/scripts/get_investigations_full.py

@@ -330,6 +330,75 @@ _MD_MAX_ROWS = 40
 _MD_MAX_VALUE = 300
 
 
+# The columns worth reading first, in this order. Everything else follows
+# alphabetically. Column order is the ONLY ordering signal the table field has:
+# it takes the union of keys in order of first appearance, so what the first row
+# names first is what the analyst sees without scrolling.
+_ROW_LEAD = [
+    "alert", "time", "action", "result", "status",
+    "source_account", "source_user", "target_account", "target_user",
+    "hostname", "computername", "source_asset", "devname",
+    "source_ip", "srcip", "destination_ip", "dstip",
+    "user", "account", "process.name", "process.cmd_line",
+]
+_MAX_ROWS = 100
+# A table wider than this is scrolled, not read. The cap keeps the useful
+# columns — the lead list, then whichever others are populated on the most rows
+# — rather than whichever happened to appear first.
+_MAX_COLUMNS = 30
+
+
+def _evidence_rows(alerts):
+    """One row per evidence, across every alert of the investigation.
+
+    The table custom field reads an array of objects directly, so this needs no
+    reshaping on the mapper side. It exists beside summary_markdown rather than
+    instead of it: the summary reads as a page, this sorts and filters.
+
+    Each row carries its alert's name, because an investigation aggregates
+    several and a row that does not say where it came from cannot be acted on.
+    """
+    rows = []
+    for a in alerts:
+        for ev in (a.get("evidences") or []):
+            f = ev.get("fields") or {}
+            if not f:
+                continue
+            row = {"alert": a.get("type") or a.get("title") or ""}
+            t = ev.get("evented_at") or f.get("timestamp")
+            if t:
+                row["time"] = str(t)
+            for k in _ROW_LEAD:
+                if k in f and k not in row:
+                    row[k] = f[k]
+            for k in sorted(f):
+                if k not in row:
+                    row[k] = f[k]
+            rows.append(row)
+            if len(rows) >= _MAX_ROWS:
+                break
+        if len(rows) >= _MAX_ROWS:
+            break
+    if not rows:
+        return [], 0
+
+    # Trim the width. Counted over every row, so a column present once out of
+    # eighty does not push out one present on all of them.
+    counts = {}
+    for r in rows:
+        for k in r:
+            counts[k] = counts.get(k, 0) + 1
+    ordered = [k for k in _ROW_LEAD + ["alert", "time"] if k in counts]
+    ordered += sorted((k for k in counts if k not in ordered),
+                      key=lambda k: (-counts[k], k))
+    # `alert` and `time` lead whatever else survives.
+    keep = ["alert"] + [k for k in ordered if k != "alert"][:_MAX_COLUMNS - 1]
+    dropped = len(counts) - len(keep)
+    if dropped > 0:
+        rows = [{k: r[k] for k in keep if k in r} for r in rows]
+    return rows, max(dropped, 0)
+
+
 def _markdown_summary(inv, alerts):
     """A readable rendering of what the detection actually says.
 
@@ -533,6 +602,11 @@ def run():
         # nested where no mapper expression could reach past alerts[0].
         if alerts:
             inv["summary_markdown"] = _markdown_summary(inv, alerts)
+            ev_rows, ev_dropped = _evidence_rows(alerts)
+            if ev_rows:
+                inv["evidence_rows"] = ev_rows
+                if ev_dropped:
+                    inv["evidence_columns_dropped"] = ev_dropped
         rolled = []
         for a in alerts:
             rolled.extend(a.get("process_tree") or [])