feat(rapid7-insightidr): render the detection as Markdown, and fix four gaps (v1.10.0)

An "Multiple User Accounts Were Disabled" investigation came back with one
entity — the actor — while the layout stayed nearly empty. Four separate
reasons, and the last one is why the first three keep recurring.

  evidences was an OBJECT, not an array. JSONata returns a bare object when the
  path yields one item, so the key had one shape on an investigation with a
  single alert and another with two. Bracketed to force an array.

  Only the actor was mapped. An ad_admin rule names who acted AND who it was
  done to, and on this alert the disabled accounts are the point of it. Target
  account, target user and both domains now map to the OCSF Target User group,
  the actor to Actor — the distinction that lets the sentence be read in one
  line.

  The domain controller was nowhere. A Windows event puts it in
  source_json.computerName and nowhere else, and source_json was skipped whole
  because the Azure AD one runs to kilobytes. Its top-level SCALARS are now
  read — bounded, and where computerName, osVersion and callerIpAddress live —
  while its nested part stays skipped.

  source_asset was the string "unknown". Mapped as-is it becomes an asset named
  "unknown" in the entity index, shared by every incident that had no asset: a
  pivot that groups things by the absence of a fact. Placeholders are dropped,
  and a key whose value cleans away to nothing is removed rather than kept
  empty, since "" is not nil and would still be written to its OCSF field.

Five entities now, from one: both accounts, both display names, the DC.

And the reason this kept happening. The layout builder's field picker is a
closed <select> over the OCSF catalogue in web/src/lib/ocsf.ts, so a path absent
from it cannot be placed on a layout at all — which is what `detection` and
`evidence` are, and what every unanticipated key will be. Mapping each one to a
real OCSF field is right where a field exists, but rule_keys_of_interest is an
open vocabulary: four spellings turned up in as many days.

So the whole of it is also rendered as a Markdown table into
custom:rapid7_summary — keys of interest first, since the rule itself says
those are what matter, then the evidence fields. Declare a custom field of type
markdown with that name, put it on the layout, and a new rule type is readable
the day it appears instead of after another mapping round.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

integrations/rapid7-insightidr/scripts/get_investigations_full.py

@@ -118,6 +118,12 @@ def _collect_region_failures(out, sink):
 _KV = re.compile(r'([A-Za-z_][\w.\-]*)=("(?:[^"\\]|\\.)*"|\S+)')
 
 
+# Values a source writes when it has nothing to say. Mapped as-is they become an
+# asset named "unknown" in the entity index, shared by every incident that had no
+# asset — a pivot that groups things by the absence of a fact.
+_PLACEHOLDERS = {"unknown", "n/a", "na", "none", "null", "-", ""}
+
+
 def _clean(v):
     """Strip the quoting Rapid7's parser leaves behind.
 
@@ -128,7 +134,8 @@ def _clean(v):
     v = str(v).strip()
     if len(v) >= 2 and v[0] == '"' and v[-1] == '"':
         v = v[1:-1]
-    return v.replace('\\"', '"').strip().strip('"')
+    v = v.replace('\\"', '"').strip().strip('"')
+    return "" if v.strip().lower() in _PLACEHOLDERS else v
 
 
 def _evidence_fields(ev):
@@ -191,6 +198,23 @@ def _evidence_fields(ev):
                 elif isinstance(sv, (str, int, float)):
                     fields[(k + "_" + sk).lower()] = _clean(sv) if isinstance(sv, str) else sv
 
+    # source_json is the vendor's own payload. Its NESTED part is skipped — the
+    # Azure AD one runs to several kilobytes — but its top-level scalars are
+    # small, bounded and often the only place something essential appears: a
+    # Windows 4725 puts the domain controller in computerName and nowhere else,
+    # and an Azure sign-in puts the caller address in callerIpAddress. Existing
+    # keys win, so the event's own naming is never overwritten by the vendor's.
+    sj = blob.get("source_json")
+    if isinstance(sj, dict):
+        for k, v in sj.items():
+            key = k.lower()
+            if key in fields:
+                continue
+            if isinstance(v, bool) or isinstance(v, (int, float)):
+                fields[key] = v
+            elif isinstance(v, str) and v.strip():
+                fields[key] = _clean(v)
+
     # Wrapped shape, overlaid on top when present.
     custom = blob.get("custom_data")
     if isinstance(custom, dict):
@@ -204,6 +228,10 @@ def _evidence_fields(ev):
         for k, v in _KV.findall(source):
             fields[k.lower()] = _clean(v)
 
+    # A key whose value cleaned away to nothing is dropped rather than kept
+    # empty: an empty string is not nil, so it would still be written to its OCSF
+    # field, and device.hostname = "" asserts an asset with no name.
+    fields = {k: v for k, v in fields.items() if v != ""}
     return fields, (source if isinstance(source, str) else raw)
 
 
@@ -296,6 +324,74 @@ def _tree_chains(payload, alert):
     return chains
 
 
+# Keys already rendered elsewhere in the incident, or too noisy for a summary.
+_MD_SKIP = {"entry_id", "timestamp"}
+_MD_MAX_ROWS = 40
+_MD_MAX_VALUE = 300
+
+
+def _markdown_summary(inv, alerts):
+    """A readable rendering of what the detection actually says.
+
+    The layout builder's field picker is a closed <select> over the OCSF
+    catalogue, so a path that is not in it — `detection`, `evidence`, anything
+    an unanticipated rule emits — cannot be placed on a layout at all. Mapping
+    each key to a real OCSF field is the right answer where one exists, but
+    rule_keys_of_interest is an open vocabulary and always will be: three
+    vocabularies turned up in as many days.
+
+    So the whole of it is also rendered as Markdown and mapped to a custom
+    field, which a layout CAN show. Nothing is lost to the analyst even when
+    nothing was mapped, and the mapping work stops being what stands between
+    a new rule type and a readable incident.
+    """
+    lines = []
+    for a in alerts:
+        title = a.get("type") or a.get("title") or "Alert"
+        lines.append("### " + str(title))
+        rule = a.get("rule") or {}
+        tcodes = [t for t in (rule.get("mitre_tcodes") or []) if t]
+        meta = []
+        if a.get("triggering_event_type"):
+            meta.append("`" + str(a["triggering_event_type"]) + "`")
+        if tcodes:
+            meta.append("ATT&CK " + ", ".join(str(t) for t in tcodes))
+        if a.get("status"):
+            meta.append("status " + str(a["status"]))
+        if a.get("disposition"):
+            meta.append("disposition " + str(a["disposition"]))
+        if meta:
+            lines.append(" · ".join(meta))
+        rows = []
+        seen = set()
+        # Keys of interest first: the rule itself says these are what matter.
+        for kv in (a.get("rule_keys_of_interest") or []):
+            k = kv.get("key")
+            vals = [str(v) for v in (kv.get("values") or []) if v not in (None, "")]
+            if k and vals and k not in seen:
+                seen.add(k)
+                rows.append((k, ", ".join(vals)))
+        for k, v in sorted((a.get("evidence_fields") or {}).items()):
+            if k in seen or k in _MD_SKIP or v in (None, ""):
+                continue
+            seen.add(k)
+            rows.append((k, str(v)))
+        if rows:
+            lines.append("")
+            lines.append("| Field | Value |")
+            lines.append("| --- | --- |")
+            for k, v in rows[:_MD_MAX_ROWS]:
+                if len(v) > _MD_MAX_VALUE:
+                    v = v[:_MD_MAX_VALUE] + "…"
+                # A pipe inside a value would split the row into a new column.
+                lines.append("| " + k.replace("|", "\\|") + " | " + v.replace("|", "\\|") + " |")
+            if len(rows) > _MD_MAX_ROWS:
+                lines.append("")
+                lines.append("_%d more field(s) not shown._" % (len(rows) - _MD_MAX_ROWS))
+        lines.append("")
+    return "\n".join(lines).strip()
+
+
 def run():
     _, _, multi = _cfg()
     inp = _inputs()
@@ -435,6 +531,8 @@ def run():
         # The block reads ctx.process_tree at the top level and draws one chain
         # per alert, so every alert's chains are gathered here rather than left
         # nested where no mapper expression could reach past alerts[0].
+        if alerts:
+            inv["summary_markdown"] = _markdown_summary(inv, alerts)
         rolled = []
         for a in alerts:
             rolled.extend(a.get("process_tree") or [])