fix(rapid7-insightidr): read the evidence in both of its shapes (v1.9.2)

evidence_fields came back {} on a live run. The evidence was fetched, the run
was clean — one call, 1.3s, no error — and the parser found nothing in it.

`data` does not have one shape. The parser was written against an unparsed
Fortigate source and looked only for {source_data, custom_data}. A PARSED
source has no such wrapper: the event is the object.

  {user, account, source_ip, geoip_country_code, geoip_organization,
   result, service, user_agent, source_json{…}, r7_context{…},
   detection_context{…}}

Every one of those is what the incident was missing, sitting one json.loads
away, and the parser walked past all of it and returned an empty dict. No error
— which is how it survived a run that reported success.

Now both are read as a union rather than one or the other: scalars at the top
level are always taken, so a third shape still yields its plain fields instead
of nothing, and the wrapped keys are overlaid when present. source_json is
skipped — several kilobytes of vendor payload that stay in the raw line —
while r7_context and detection_context are flattened one level, being Rapid7's
own annotations and the same across parsed sources. They carry the account age
in minutes, which is exactly what says whether a sign-in is from a new account.

The Azure AD sign-in now maps to src_endpoint.ip, .isp_org and
.location.country, actor.user.name/.uid/.account.name, http_request.user_agent
and metadata.product.feature.name, and indexes the address and the account.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

integrations/rapid7-insightidr/scripts/get_investigations_full.py

@@ -132,17 +132,30 @@ def _clean(v):
 
 
 def _evidence_fields(ev):
-    """One evidence's log line as a flat, normalised dict.
+    """One evidence's event as a flat, normalised dict.
 
-    `data` is a JSON-encoded STRING holding {source_data, custom_data} — the
-    actual event behind the alert. JSONata cannot deserialise, so a mapper can
-    never reach into it; this is why the parsing has to happen here.
+    `data` is a JSON-encoded STRING. JSONata cannot deserialise, so a mapper can
+    never reach into it; this is why the parsing happens here.
 
-    custom_data is the vendor's normalised view, its keys prefixed per source
-    (Forti_user, Forti_srcip). The prefix is stripped so a mapper can name
-    `user` once rather than once per vendor. source_data is the raw line and is
-    overlaid on top: it carries far more keys (25 against 6 on the sample this
-    was written from) and it is not affected by the quoting artefact above.
+    It arrives in at least two shapes and the union of both is read, rather than
+    one or the other. The first version of this handled only the wrapped shape
+    and returned {} for everything else — silently, so an Azure AD sign-in
+    carrying its account, its source address and its geolocation produced an
+    incident with no observable at all.
+
+      wrapped  {source_data: "<syslog line>", custom_data: {Forti_user: …}}
+               An unparsed source. custom_data is the vendor's normalised view,
+               its keys prefixed per source; source_data is the raw line.
+
+      flat     {user: …, account: …, source_ip: …, geoip_country_code: …,
+                source_json: {…}, r7_context: {…}, detection_context: {…}}
+               A parsed source. The event IS the object.
+
+    Scalars at the top level are always taken, so a third shape still yields its
+    plain fields instead of nothing. source_json is deliberately skipped — it is
+    the vendor's whole payload, several kilobytes of it, and it stays in the raw
+    line. r7_context and detection_context are flattened one level because they
+    are Rapid7's own annotations and are the same across parsed sources.
     """
     raw = ev.get("data")
     if not isinstance(raw, str) or not raw.strip():
@@ -154,18 +167,43 @@ def _evidence_fields(ev):
         return {}, raw
     if not isinstance(blob, dict):
         return {}, raw
+
     fields = {}
+
+    # Flat shape: every scalar the event states about itself.
+    for k, v in blob.items():
+        if k in ("source_data", "custom_data", "source_json"):
+            continue
+        if isinstance(v, bool) or isinstance(v, (int, float)):
+            fields[k.lower()] = v
+        elif isinstance(v, str):
+            fields[k.lower()] = _clean(v)
+        elif isinstance(v, dict) and k in ("r7_context", "detection_context"):
+            for sk, sv in v.items():
+                if isinstance(sv, dict):
+                    for ssk, ssv in sv.items():
+                        if not isinstance(ssv, (str, int, float)) or isinstance(ssv, bool):
+                            continue
+                        # detection_context nests user.user_age_mins, so a blind
+                        # join yields user_user_age_mins.
+                        name = ssk if ssk.startswith(sk + "_") else sk + "_" + ssk
+                        fields[name.lower()] = _clean(ssv) if isinstance(ssv, str) else ssv
+                elif isinstance(sv, (str, int, float)):
+                    fields[(k + "_" + sk).lower()] = _clean(sv) if isinstance(sv, str) else sv
+
+    # Wrapped shape, overlaid on top when present.
     custom = blob.get("custom_data")
     if isinstance(custom, dict):
         for k, v in custom.items():
-            if not isinstance(v, (str, int, float)):
+            if not isinstance(v, (str, int, float)) or isinstance(v, bool):
                 continue
             key = k.split("_", 1)[1] if "_" in k else k
-            fields[key.lower()] = _clean(v)
+            fields[key.lower()] = _clean(v) if isinstance(v, str) else v
     source = blob.get("source_data")
     if isinstance(source, str):
         for k, v in _KV.findall(source):
             fields[k.lower()] = _clean(v)
+
     return fields, (source if isinstance(source, str) else raw)