diff --git a/integrations/rapid7-insightidr/manifest.yaml b/integrations/rapid7-insightidr/manifest.yaml index 1a5e0ce..da54202 100644 --- a/integrations/rapid7-insightidr/manifest.yaml +++ b/integrations/rapid7-insightidr/manifest.yaml @@ -1,6 +1,6 @@ id: rapid7_insightidr name: Rapid7 InsightIDR -version: 1.14.3 +version: 1.14.4 description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, investigation comments (add/list), custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search." changelog: "1.13.2 — query_log and query_log_set: trim the quotes and spaces a pasted log or log set ID drags along, and explain the 404. A trailing quote was URL-encoded into the path and the API answered a bare \"HTTP 404 Not Found\" that named nothing. Also send both time bounds on every run: a scheduled run passes only the watermark, and the first run nothing — the window now ends now and, without a start, opens 24 hours back. | 1.13.1 — test_connection: accept an InsightIDR Essential key. Essential answers the Investigations API with a 403 that names the package, so a valid key failed its test; the test now falls back to Log Search and says which commands that package leaves out — the LEQL feeds (query_log, query_log_set) are its way to ingest detections. | 1.13.0 — Add add_comment and list_comments: the investigation comment thread is where an analyst’s reasoning lives, and a playbook had no way to write into it or read it back. The Comments API is v1-only and addresses its target by RRN — the short investigation ID the other v1 endpoints accept is rejected with a 400 that names nothing, so both commands refuse it up front. | 1.12.0 — Make the LEQL queries ingestible: query_log and query_log_set gain an ingest block, a stable event_id for dedup, a bounded page walk and a detection_name stamp, and they now read start_time as ISO-8601, UNIX seconds or milliseconds alike. A tenant whose basic detection rules only notify (email, Slack) creates no investigation and no alert an API can read, so replaying the rule's own query on a schedule is the only way its detections reach a SOAR. | 1.11.1 — Strip the whitespace out of the comma-separated filter params (statuses, priorities, sources, tags) before the URL. \"HIGH, CRITICAL\" as typed sent \" CRITICAL\" to the API, which rejects the whole call — so a priority filter written naturally fetched nothing. | 1.11.0 —Build one evidence row per alert evidence into custom:rapid7_evidence, for the table field: sortable and filterable where the Markdown summary only reads. | 1.10.0 — Render every detection key as Markdown into custom:rapid7_summary, so a layout can show what the OCSF catalogue has no field for. Map both ends of an account action, read source_json top-level scalars, drop placeholder values, and force evidences to stay an array. | 1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." category: siem @@ -483,7 +483,7 @@ commands: - id: get_investigations_full name: rapid7-insight-idr-get-investigations-full - description: "Ingestion: investigations with their alerts inlined, so the INVESTIGATION stays the unit that becomes an incident. Walks the investigations in a time window, collects their alert RRNs and retrieves the alerts in batches from the Alert Triage API, then nests them under `alerts` with a few first-alert values lifted to the top level. Results path = data, dedup on rrn, incremental on start_time — a drop-in replacement for list_investigations, which carries no host, process or MITRE code. Use the get_investigations_full mapper." + description: "Ingestion: investigations with their alerts inlined, so the INVESTIGATION stays the unit that becomes an incident. Walks the investigations in a time window, collects their alert RRNs and retrieves the alerts in batches from the Alert Triage API, then nests them under `alerts` with a few first-alert values lifted to the top level. Results path = data, dedup on rrn, incremental on start_time — a drop-in replacement for list_investigations, which carries no host, process or MITRE code. Use the get_investigations_full mapper. Alerts with no RRN (behaviour/UBA alerts, which the Alert Triage API cannot fetch) are kept under alerts_without_rrn, counted in alert_count and rendered in summary_markdown, instead of being dropped." risk: read inputs_schema: properties: diff --git a/integrations/rapid7-insightidr/scripts/get_investigations_full.py b/integrations/rapid7-insightidr/scripts/get_investigations_full.py index d732022..d215f8d 100644 --- a/integrations/rapid7-insightidr/scripts/get_investigations_full.py +++ b/integrations/rapid7-insightidr/scripts/get_investigations_full.py @@ -399,6 +399,37 @@ def _evidence_rows(alerts): return rows, max(dropped, 0) +def _without_rrn(a): + """An investigation alert the Alert Triage API cannot read, as returned, + with what an agent or analyst needs to know about it.""" + out = dict(a) + out["evidence_api"] = False + out["evidence_hint"] = ("Behaviour (UBA) alert: InsightIDR exposes no actors or evidence for it through the " + "API. Its activity is in the authentication logs (Asset Authentication log set), " + "around its event times.") + return out + + +def _without_rrn_summary(alerts): + """Markdown for the alerts that have no RRN, so a layout or a reading shows + them instead of an empty investigation.""" + lines = [] + for a in alerts: + lines.append("### " + str(a.get("title") or a.get("alert_type") or "Behaviour alert")) + meta = [] + for k, label in (("alert_type", "type"), ("alert_source", "source"), + ("first_event_time", "first event"), ("latest_event_time", "latest event"), + ("created_time", "created")): + if a.get(k): + meta.append("%s %s" % (label, a[k])) + if meta: + lines.append(" · ".join(meta)) + lines.append("") + lines.append("_" + a["evidence_hint"] + "_") + lines.append("") + return "\n".join(lines).strip() + + def _markdown_summary(inv, alerts): """A readable rendering of what the detection actually says. @@ -495,6 +526,12 @@ def run(): # output with an alerts_error: losing the whole run over one of them, or # dropping it silently, would both be worse. rrns_by_inv, order, truncated, skipped = {}, [], False, 0 + # Alerts with no RRN — behaviour (UBA) alerts such as "attempted to remotely + # access N new assets". The Alert Triage API cannot fetch them, and they + # used to be dropped here: the investigation arrived with alerts: [] and + # alert_count: 0, and every reading of it worked from a title alone. They + # are kept as the investigation endpoint returned them. + no_rrn_by_inv = {} region_failures, regions_seen = [], set() for inv in investigations: inv_rrn = inv.get("rrn") or "" @@ -514,6 +551,7 @@ def run(): rrn = _alert_rrn(a) if rrn is None: skipped += 1 + no_rrn_by_inv.setdefault(inv_rrn, []).append(_without_rrn(a)) continue got.append(rrn) r = _rrn_region(rrn) @@ -598,7 +636,10 @@ def run(): rrns = rrns_by_inv.get(inv.get("rrn") or "", []) alerts = [by_rrn[r] for r in rrns if r in by_rrn] inv["alerts"] = alerts - inv["alert_count"] = len(rrns) + no_rrn = no_rrn_by_inv.get(inv.get("rrn") or "", []) + inv["alerts_without_rrn"] = no_rrn + # What the investigation holds, not what this run could fetch. + inv["alert_count"] = len(rrns) + len(no_rrn) # The block reads ctx.process_tree at the top level and draws one chain # per alert, so every alert's chains are gathered here rather than left # nested where no mapper expression could reach past alerts[0]. @@ -645,6 +686,18 @@ def run(): if ef: inv["first_evidence_message"] = ef.get("msg") or ef.get("logdesc") or ef.get("message") inv["first_evidence_raw_line"] = a.get("evidence_raw_line") + if no_rrn: + # Readable beside the detection alerts, or alone when they are all + # the investigation has — a UBA investigation usually is. + uba_md = _without_rrn_summary(no_rrn) + inv["summary_markdown"] = (inv["summary_markdown"] + "\n\n" + uba_md) if inv.get("summary_markdown") else uba_md + if not alerts: + u = no_rrn[0] + inv.update({ + "first_alert_type": u.get("alert_type") or u.get("title"), + "first_alert_source": u.get("alert_source"), + "first_alert_id": u.get("id"), + }) print(json.dumps({ "data": investigations,