feat(rapid7-insightidr): render the detection as Markdown, and fix four gaps (v1.10.0)

An "Multiple User Accounts Were Disabled" investigation came back with one
entity — the actor — while the layout stayed nearly empty. Four separate
reasons, and the last one is why the first three keep recurring.

  evidences was an OBJECT, not an array. JSONata returns a bare object when the
  path yields one item, so the key had one shape on an investigation with a
  single alert and another with two. Bracketed to force an array.

  Only the actor was mapped. An ad_admin rule names who acted AND who it was
  done to, and on this alert the disabled accounts are the point of it. Target
  account, target user and both domains now map to the OCSF Target User group,
  the actor to Actor — the distinction that lets the sentence be read in one
  line.

  The domain controller was nowhere. A Windows event puts it in
  source_json.computerName and nowhere else, and source_json was skipped whole
  because the Azure AD one runs to kilobytes. Its top-level SCALARS are now
  read — bounded, and where computerName, osVersion and callerIpAddress live —
  while its nested part stays skipped.

  source_asset was the string "unknown". Mapped as-is it becomes an asset named
  "unknown" in the entity index, shared by every incident that had no asset: a
  pivot that groups things by the absence of a fact. Placeholders are dropped,
  and a key whose value cleans away to nothing is removed rather than kept
  empty, since "" is not nil and would still be written to its OCSF field.

Five entities now, from one: both accounts, both display names, the DC.

And the reason this kept happening. The layout builder's field picker is a
closed <select> over the OCSF catalogue in web/src/lib/ocsf.ts, so a path absent
from it cannot be placed on a layout at all — which is what `detection` and
`evidence` are, and what every unanticipated key will be. Mapping each one to a
real OCSF field is right where a field exists, but rule_keys_of_interest is an
open vocabulary: four spellings turned up in as many days.

So the whole of it is also rendered as a Markdown table into
custom:rapid7_summary — keys of interest first, since the rule itself says
those are what matter, then the evidence fields. Declare a custom field of type
markdown with that name, put it on the layout, and a new rule type is readable
the day it appears instead of after another mapping round.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

integrations/rapid7-insightidr/manifest.yaml

@@ -1,8 +1,8 @@
 id: rapid7_insightidr
 name: Rapid7 InsightIDR
-version: 1.9.2
+version: 1.10.0
 description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search."
-changelog: "1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
+changelog: "1.10.0 — Render every detection key as Markdown into custom:rapid7_summary, so a layout can show what the OCSF catalogue has no field for. Map both ends of an account action, read source_json top-level scalars, drop placeholder values, and force evidences to stay an array. | 1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
 category: siem
 
 # Per-instance configuration. The Insight platform region selects the API host

integrations/rapid7-insightidr/mappers/get_investigations_full.yaml

@@ -39,6 +39,11 @@ field_mappings:
   "custom:rapid7_logset_id": "first_alert_logset_id"
   "custom:rapid7_rule_rrn": "first_alert_rule_rrn"
   "custom:rapid7_event_type": "first_alert_event_type"
+  # Every key the detection emitted, rendered as Markdown. Declare a custom field
+  # named rapid7_summary of type markdown and place it on the layout: it shows
+  # what no OCSF mapping can, because the layout's field picker only offers paths
+  # from the OCSF catalogue and this vocabulary is open-ended.
+  "custom:rapid7_summary": "summary_markdown"
 ocsf:
   # ── Finding: the investigation ────────────────────────────────────
   - { source_path: "rrn",                ocsf_field: "finding_info.uid" }
@@ -177,6 +182,23 @@ ocsf:
   # offers, unlike `evidence` above, which only ever shows in the Context tab.
   - { source_path: "alerts[0].evidence_raw_line",          ocsf_field: "raw_data" }
   - { source_path: "alerts[0].evidence_fields.msg ? alerts[0].evidence_fields.msg : alerts[0].evidence_fields.logdesc", ocsf_field: "message" }
+  # ── Who did it to whom ────────────────────────────────────────────
+  # An ad_admin rule names an actor and a target account, and mapping only the
+  # actor loses the victims — on "Multiple User Accounts Were Disabled" the
+  # disabled accounts are the whole point of the alert.
+  - { source_path: "alerts[0].evidence_fields.target_account",     ocsf_field: "user.name" }
+  - { source_path: "alerts[0].evidence_fields.target_user",        ocsf_field: "user.full_name" }
+  - { source_path: "alerts[0].evidence_fields.target_user_domain", ocsf_field: "user.domain" }
+  - { source_path: "alerts[0].evidence_fields.source_user",        ocsf_field: "actor.user.full_name" }
+  - { source_path: "alerts[0].evidence_fields.source_user_domain", ocsf_field: "actor.user.domain" }
+  - { source_path: "alerts[0].evidence_fields.source_account",     ocsf_field: "actor.user.name" }
+  # The machine the event was recorded on. A Windows event puts it in
+  # source_json.computerName and nowhere else; source_asset is the parsed field
+  # when the source fills it, and is dropped when it says "unknown".
+  - { source_path: "alerts[0].evidence_fields.source_asset",       ocsf_field: "device.hostname" }
+  - { source_path: "alerts[0].evidence_fields.computername",       ocsf_field: "device.hostname" }
+  - { source_path: "alerts[0].evidence_fields.osversion",          ocsf_field: "device.os.name" }
+  - { source_path: "alerts[0].evidence_fields.calleripaddress",    ocsf_field: "src_endpoint.ip" }
   # ── Process tree ──────────────────────────────────────────────────
   # The layout block reads ctx.process_tree: N flat chains, root first, each
   # {alert_id, rule, host, user, nodes[]}. Rapid7 nests its tree through
@@ -190,8 +212,11 @@ ocsf:
   # the SECOND and THIRD alert are indexed as entities too — the extractor walks arrays
   # and classifies on the path's suffix, so `.hostname` under here counts exactly as
   # `device.hostname` does above.
+  # The brackets force an array. Without them JSONata returns a bare OBJECT when
+  # the investigation holds a single alert and an array when it holds two, so the
+  # same key had two shapes depending on how busy the attacker was.
   - source_path: >
-      alerts.{
+      [alerts.{
         "uid": rrn,
         "title": title,
         "time": alerted_at,
@@ -207,5 +232,5 @@ ocsf:
         "technique": rule.mitre_tcodes[$substring($, 0, 2) != 'TA'][0],
         "detection": rule_keys_of_interest{key: values[0]},
         "evidence": evidence_fields
-      }
+      }]
     ocsf_field: "evidences"

integrations/rapid7-insightidr/scripts/get_investigations_full.py

@@ -118,6 +118,12 @@ def _collect_region_failures(out, sink):
 _KV = re.compile(r'([A-Za-z_][\w.\-]*)=("(?:[^"\\]|\\.)*"|\S+)')
 
 
+# Values a source writes when it has nothing to say. Mapped as-is they become an
+# asset named "unknown" in the entity index, shared by every incident that had no
+# asset — a pivot that groups things by the absence of a fact.
+_PLACEHOLDERS = {"unknown", "n/a", "na", "none", "null", "-", ""}
+
+
 def _clean(v):
     """Strip the quoting Rapid7's parser leaves behind.
 
@@ -128,7 +134,8 @@ def _clean(v):
     v = str(v).strip()
     if len(v) >= 2 and v[0] == '"' and v[-1] == '"':
         v = v[1:-1]
-    return v.replace('\\"', '"').strip().strip('"')
+    v = v.replace('\\"', '"').strip().strip('"')
+    return "" if v.strip().lower() in _PLACEHOLDERS else v
 
 
 def _evidence_fields(ev):
@@ -191,6 +198,23 @@ def _evidence_fields(ev):
                 elif isinstance(sv, (str, int, float)):
                     fields[(k + "_" + sk).lower()] = _clean(sv) if isinstance(sv, str) else sv
 
+    # source_json is the vendor's own payload. Its NESTED part is skipped — the
+    # Azure AD one runs to several kilobytes — but its top-level scalars are
+    # small, bounded and often the only place something essential appears: a
+    # Windows 4725 puts the domain controller in computerName and nowhere else,
+    # and an Azure sign-in puts the caller address in callerIpAddress. Existing
+    # keys win, so the event's own naming is never overwritten by the vendor's.
+    sj = blob.get("source_json")
+    if isinstance(sj, dict):
+        for k, v in sj.items():
+            key = k.lower()
+            if key in fields:
+                continue
+            if isinstance(v, bool) or isinstance(v, (int, float)):
+                fields[key] = v
+            elif isinstance(v, str) and v.strip():
+                fields[key] = _clean(v)
+
     # Wrapped shape, overlaid on top when present.
     custom = blob.get("custom_data")
     if isinstance(custom, dict):
@@ -204,6 +228,10 @@ def _evidence_fields(ev):
         for k, v in _KV.findall(source):
             fields[k.lower()] = _clean(v)
 
+    # A key whose value cleaned away to nothing is dropped rather than kept
+    # empty: an empty string is not nil, so it would still be written to its OCSF
+    # field, and device.hostname = "" asserts an asset with no name.
+    fields = {k: v for k, v in fields.items() if v != ""}
     return fields, (source if isinstance(source, str) else raw)
 
 
@@ -296,6 +324,74 @@ def _tree_chains(payload, alert):
     return chains
 
 
+# Keys already rendered elsewhere in the incident, or too noisy for a summary.
+_MD_SKIP = {"entry_id", "timestamp"}
+_MD_MAX_ROWS = 40
+_MD_MAX_VALUE = 300
+
+
+def _markdown_summary(inv, alerts):
+    """A readable rendering of what the detection actually says.
+
+    The layout builder's field picker is a closed <select> over the OCSF
+    catalogue, so a path that is not in it — `detection`, `evidence`, anything
+    an unanticipated rule emits — cannot be placed on a layout at all. Mapping
+    each key to a real OCSF field is the right answer where one exists, but
+    rule_keys_of_interest is an open vocabulary and always will be: three
+    vocabularies turned up in as many days.
+
+    So the whole of it is also rendered as Markdown and mapped to a custom
+    field, which a layout CAN show. Nothing is lost to the analyst even when
+    nothing was mapped, and the mapping work stops being what stands between
+    a new rule type and a readable incident.
+    """
+    lines = []
+    for a in alerts:
+        title = a.get("type") or a.get("title") or "Alert"
+        lines.append("### " + str(title))
+        rule = a.get("rule") or {}
+        tcodes = [t for t in (rule.get("mitre_tcodes") or []) if t]
+        meta = []
+        if a.get("triggering_event_type"):
+            meta.append("`" + str(a["triggering_event_type"]) + "`")
+        if tcodes:
+            meta.append("ATT&CK " + ", ".join(str(t) for t in tcodes))
+        if a.get("status"):
+            meta.append("status " + str(a["status"]))
+        if a.get("disposition"):
+            meta.append("disposition " + str(a["disposition"]))
+        if meta:
+            lines.append(" · ".join(meta))
+        rows = []
+        seen = set()
+        # Keys of interest first: the rule itself says these are what matter.
+        for kv in (a.get("rule_keys_of_interest") or []):
+            k = kv.get("key")
+            vals = [str(v) for v in (kv.get("values") or []) if v not in (None, "")]
+            if k and vals and k not in seen:
+                seen.add(k)
+                rows.append((k, ", ".join(vals)))
+        for k, v in sorted((a.get("evidence_fields") or {}).items()):
+            if k in seen or k in _MD_SKIP or v in (None, ""):
+                continue
+            seen.add(k)
+            rows.append((k, str(v)))
+        if rows:
+            lines.append("")
+            lines.append("| Field | Value |")
+            lines.append("| --- | --- |")
+            for k, v in rows[:_MD_MAX_ROWS]:
+                if len(v) > _MD_MAX_VALUE:
+                    v = v[:_MD_MAX_VALUE] + "…"
+                # A pipe inside a value would split the row into a new column.
+                lines.append("| " + k.replace("|", "\\|") + " | " + v.replace("|", "\\|") + " |")
+            if len(rows) > _MD_MAX_ROWS:
+                lines.append("")
+                lines.append("_%d more field(s) not shown._" % (len(rows) - _MD_MAX_ROWS))
+        lines.append("")
+    return "\n".join(lines).strip()
+
+
 def run():
     _, _, multi = _cfg()
     inp = _inputs()
@@ -435,6 +531,8 @@ def run():
         # The block reads ctx.process_tree at the top level and draws one chain
         # per alert, so every alert's chains are gathered here rather than left
         # nested where no mapper expression could reach past alerts[0].
+        if alerts:
+            inv["summary_markdown"] = _markdown_summary(inv, alerts)
         rolled = []
         for a in alerts:
             rolled.extend(a.get("process_tree") or [])