fix(rapid7-insightidr): read the evidence in both of its shapes (v1.9.2)

evidence_fields came back {} on a live run. The evidence was fetched, the run
was clean — one call, 1.3s, no error — and the parser found nothing in it.

`data` does not have one shape. The parser was written against an unparsed
Fortigate source and looked only for {source_data, custom_data}. A PARSED
source has no such wrapper: the event is the object.

  {user, account, source_ip, geoip_country_code, geoip_organization,
   result, service, user_agent, source_json{…}, r7_context{…},
   detection_context{…}}

Every one of those is what the incident was missing, sitting one json.loads
away, and the parser walked past all of it and returned an empty dict. No error
— which is how it survived a run that reported success.

Now both are read as a union rather than one or the other: scalars at the top
level are always taken, so a third shape still yields its plain fields instead
of nothing, and the wrapped keys are overlaid when present. source_json is
skipped — several kilobytes of vendor payload that stay in the raw line —
while r7_context and detection_context are flattened one level, being Rapid7's
own annotations and the same across parsed sources. They carry the account age
in minutes, which is exactly what says whether a sign-in is from a new account.

The Azure AD sign-in now maps to src_endpoint.ip, .isp_org and
.location.country, actor.user.name/.uid/.account.name, http_request.user_agent
and metadata.product.feature.name, and indexes the address and the account.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

integrations/rapid7-insightidr/manifest.yaml

@@ -1,8 +1,8 @@
 id: rapid7_insightidr
 name: Rapid7 InsightIDR
-version: 1.9.1
+version: 1.9.2
 description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search."
-changelog: "1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
+changelog: "1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
 category: siem
 
 # Per-instance configuration. The Insight platform region selects the API host

integrations/rapid7-insightidr/mappers/get_investigations_full.yaml

@@ -150,6 +150,17 @@ ocsf:
   # that names its fields the usual way.
   - { source_path: "alerts[0].evidence_fields",            ocsf_field: "evidence" }
   - { source_path: "alerts[0].evidence_fields.srcip",      ocsf_field: "src_endpoint.ip" }
+  # A parsed source states the same things under its own names. Both spellings
+  # are mapped rather than picked between: which one arrives depends on whether
+  # the log source was parsed by Rapid7, and one file has to serve both.
+  - { source_path: "alerts[0].evidence_fields.source_ip",          ocsf_field: "src_endpoint.ip" }
+  - { source_path: "alerts[0].evidence_fields.account",            ocsf_field: "actor.user.account.name" }
+  - { source_path: "alerts[0].evidence_fields.user_rrn",           ocsf_field: "actor.user.uid" }
+  - { source_path: "alerts[0].evidence_fields.geoip_country_code", ocsf_field: "src_endpoint.location.country" }
+  - { source_path: "alerts[0].evidence_fields.geoip_organization", ocsf_field: "src_endpoint.isp_org" }
+  - { source_path: "alerts[0].evidence_fields.user_agent",         ocsf_field: "http_request.user_agent" }
+  - { source_path: "alerts[0].evidence_fields.service",            ocsf_field: "metadata.product.feature.name" }
+  - { source_path: "alerts[0].evidence_fields.result",             ocsf_field: "status_detail" }
   - { source_path: "alerts[0].evidence_fields.dstip",      ocsf_field: "dst_endpoint.ip" }
   - { source_path: "alerts[0].evidence_fields.srcport",    ocsf_field: "src_endpoint.port" }
   - { source_path: "alerts[0].evidence_fields.dstport",    ocsf_field: "dst_endpoint.port" }

integrations/rapid7-insightidr/scripts/get_investigations_full.py

@@ -132,17 +132,30 @@ def _clean(v):
 
 
 def _evidence_fields(ev):
-    """One evidence's log line as a flat, normalised dict.
+    """One evidence's event as a flat, normalised dict.
 
-    `data` is a JSON-encoded STRING holding {source_data, custom_data} — the
+    `data` is a JSON-encoded STRING. JSONata cannot deserialise, so a mapper can
-    actual event behind the alert. JSONata cannot deserialise, so a mapper can
+    never reach into it; this is why the parsing happens here.
-    never reach into it; this is why the parsing has to happen here.
 
-    custom_data is the vendor's normalised view, its keys prefixed per source
+    It arrives in at least two shapes and the union of both is read, rather than
-    (Forti_user, Forti_srcip). The prefix is stripped so a mapper can name
+    one or the other. The first version of this handled only the wrapped shape
-    `user` once rather than once per vendor. source_data is the raw line and is
+    and returned {} for everything else — silently, so an Azure AD sign-in
-    overlaid on top: it carries far more keys (25 against 6 on the sample this
+    carrying its account, its source address and its geolocation produced an
-    was written from) and it is not affected by the quoting artefact above.
+    incident with no observable at all.
+
+      wrapped  {source_data: "<syslog line>", custom_data: {Forti_user: …}}
+               An unparsed source. custom_data is the vendor's normalised view,
+               its keys prefixed per source; source_data is the raw line.
+
+      flat     {user: …, account: …, source_ip: …, geoip_country_code: …,
+                source_json: {…}, r7_context: {…}, detection_context: {…}}
+               A parsed source. The event IS the object.
+
+    Scalars at the top level are always taken, so a third shape still yields its
+    plain fields instead of nothing. source_json is deliberately skipped — it is
+    the vendor's whole payload, several kilobytes of it, and it stays in the raw
+    line. r7_context and detection_context are flattened one level because they
+    are Rapid7's own annotations and are the same across parsed sources.
     """
     raw = ev.get("data")
     if not isinstance(raw, str) or not raw.strip():
@@ -154,18 +167,43 @@ def _evidence_fields(ev):
         return {}, raw
     if not isinstance(blob, dict):
         return {}, raw
+
     fields = {}
+
+    # Flat shape: every scalar the event states about itself.
+    for k, v in blob.items():
+        if k in ("source_data", "custom_data", "source_json"):
+            continue
+        if isinstance(v, bool) or isinstance(v, (int, float)):
+            fields[k.lower()] = v
+        elif isinstance(v, str):
+            fields[k.lower()] = _clean(v)
+        elif isinstance(v, dict) and k in ("r7_context", "detection_context"):
+            for sk, sv in v.items():
+                if isinstance(sv, dict):
+                    for ssk, ssv in sv.items():
+                        if not isinstance(ssv, (str, int, float)) or isinstance(ssv, bool):
+                            continue
+                        # detection_context nests user.user_age_mins, so a blind
+                        # join yields user_user_age_mins.
+                        name = ssk if ssk.startswith(sk + "_") else sk + "_" + ssk
+                        fields[name.lower()] = _clean(ssv) if isinstance(ssv, str) else ssv
+                elif isinstance(sv, (str, int, float)):
+                    fields[(k + "_" + sk).lower()] = _clean(sv) if isinstance(sv, str) else sv
+
+    # Wrapped shape, overlaid on top when present.
     custom = blob.get("custom_data")
     if isinstance(custom, dict):
         for k, v in custom.items():
-            if not isinstance(v, (str, int, float)):
+            if not isinstance(v, (str, int, float)) or isinstance(v, bool):
                 continue
             key = k.split("_", 1)[1] if "_" in k else k
-            fields[key.lower()] = _clean(v)
+            fields[key.lower()] = _clean(v) if isinstance(v, str) else v
     source = blob.get("source_data")
     if isinstance(source, str):
         for k, v in _KV.findall(source):
             fields[k.lower()] = _clean(v)
+
     return fields, (source if isinstance(source, str) else raw)