diff --git a/integrations/rapid7-insightidr/agents/operator.yaml b/integrations/rapid7-insightidr/agents/operator.yaml index 994236d..d2b52e2 100644 --- a/integrations/rapid7-insightidr/agents/operator.yaml +++ b/integrations/rapid7-insightidr/agents/operator.yaml @@ -48,6 +48,10 @@ instructions: | one domain controller's set. * process / endpoint alerts: the Endpoint Agent log set. * Never File Access or File Modification for an authentication alert. + - ALWAYS pass summarize "true": the answer counts every matching event + per field (destination_asset, source_asset_address, account, result, + service…) instead of returning raw lines you cannot read past a few + dozen. Report the counts and say when "truncated" is true. - Keyword search, no field names: where(""), e.g. where("jdoe"). If it returns 0 events, try once more case-insensitively: where(/jdoe/i). Always parentheses; no groupby or @@ -56,7 +60,8 @@ instructions: | first seen time exactly as the data gives it, and the window is computed for you. A behaviour (UBA) alert aggregates hours of activity: hours_before 24, minutes_after 30 (the defaults). Other alerts: - hours_before 0.5. Report the "window" the answer returns, not your own. + hours_before 0.5. Report the "window" field of the answer word for + word, never your own reading of it. - "Invalid Query Syntax": report the exact query and error, no variants. - Keep at most 6 calls for logs. 5. ALWAYS call search_investigation(actor_user_name=) once, whatever diff --git a/integrations/rapid7-insightidr/manifest.yaml b/integrations/rapid7-insightidr/manifest.yaml index da54202..8d1e764 100644 --- a/integrations/rapid7-insightidr/manifest.yaml +++ b/integrations/rapid7-insightidr/manifest.yaml @@ -1,6 +1,6 @@ id: rapid7_insightidr name: Rapid7 InsightIDR -version: 1.14.4 +version: 1.14.5 description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, investigation comments (add/list), custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search." changelog: "1.13.2 — query_log and query_log_set: trim the quotes and spaces a pasted log or log set ID drags along, and explain the 404. A trailing quote was URL-encoded into the path and the API answered a bare \"HTTP 404 Not Found\" that named nothing. Also send both time bounds on every run: a scheduled run passes only the watermark, and the first run nothing — the window now ends now and, without a start, opens 24 hours back. | 1.13.1 — test_connection: accept an InsightIDR Essential key. Essential answers the Investigations API with a 403 that names the package, so a valid key failed its test; the test now falls back to Log Search and says which commands that package leaves out — the LEQL feeds (query_log, query_log_set) are its way to ingest detections. | 1.13.0 — Add add_comment and list_comments: the investigation comment thread is where an analyst’s reasoning lives, and a playbook had no way to write into it or read it back. The Comments API is v1-only and addresses its target by RRN — the short investigation ID the other v1 endpoints accept is rejected with a 400 that names nothing, so both commands refuse it up front. | 1.12.0 — Make the LEQL queries ingestible: query_log and query_log_set gain an ingest block, a stable event_id for dedup, a bounded page walk and a detection_name stamp, and they now read start_time as ISO-8601, UNIX seconds or milliseconds alike. A tenant whose basic detection rules only notify (email, Slack) creates no investigation and no alert an API can read, so replaying the rule's own query on a schedule is the only way its detections reach a SOAR. | 1.11.1 — Strip the whitespace out of the comma-separated filter params (statuses, priorities, sources, tags) before the URL. \"HIGH, CRITICAL\" as typed sent \" CRITICAL\" to the API, which rejects the whole call — so a priority filter written naturally fetched nothing. | 1.11.0 —Build one evidence row per alert evidence into custom:rapid7_evidence, for the table field: sortable and filterable where the Markdown summary only reads. | 1.10.0 — Render every detection key as Markdown into custom:rapid7_summary, so a layout can show what the OCSF catalogue has no field for. Map both ends of an account action, read source_json top-level scalars, drop placeholder values, and force evidences to stay an array. | 1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command." category: siem @@ -308,6 +308,7 @@ commands: anchor_time: { type: string, description: "The alert's time (ISO-8601, as the alert gives it). With it the window is computed here: hours_before it to minutes_after it, overriding start_time/end_time — no date arithmetic for the caller." } hours_before: { type: number, description: "With anchor_time: how many hours before it the window starts (default 24)." } minutes_after: { type: number, description: "With anchor_time: how many minutes after it the window ends (default 30)." } + summarize: { type: string, description: "true: return per-field counts (each field's most frequent values over every matching event, the first and last event time, and 5 sample events) instead of the raw events — what an agent should ask for. Raw events fill a model's context after a few dozen." } query: { type: string, description: "Valid LEQL query (no calculations: groupby/calculate are rejected by this endpoint)" } time_range: { type: string, description: "Relative time range (e.g. 1 week, 1 day). When set, start_time/end_time are not needed." } start_time: { type: string, description: "Lower bound: UNIX milliseconds, UNIX seconds or ISO-8601 — all three are accepted. Incremental fetch watermark." } diff --git a/integrations/rapid7-insightidr/scripts/query_log_set.py b/integrations/rapid7-insightidr/scripts/query_log_set.py index bac72b8..c9c3583 100644 --- a/integrations/rapid7-insightidr/scripts/query_log_set.py +++ b/integrations/rapid7-insightidr/scripts/query_log_set.py @@ -137,6 +137,78 @@ def _identify(events, fallback_log_id, detection_name): return out +# What a summary keeps: the values an analyst reads first, per field. +_TOP_VALUES = 8 +_MAX_FIELDS = 30 +_SAMPLES = 5 +# Rapid7's whole vendor payload, kilobytes per event: it stays out of a summary. +_SKIP_FIELDS = {"source_json", "source_data"} + + +def _fields(e): + """One Log Search event as flat fields. `message` is a JSON-encoded string + on parsed sources; nested objects are flattened one level (a.b).""" + msg = e.get("message") + data = None + if isinstance(msg, str): + try: + data = json.loads(msg) + except ValueError: + data = None + elif isinstance(msg, dict): + data = msg + if not isinstance(data, dict): + return {"message": msg} if msg not in (None, "") else {} + flat = {} + for k, v in data.items(): + if k in _SKIP_FIELDS: + continue + if isinstance(v, dict): + for k2, v2 in v.items(): + if not isinstance(v2, (dict, list)) and v2 not in (None, ""): + flat[k + "." + k2] = v2 + elif not isinstance(v, list) and v not in (None, ""): + flat[k] = v + return flat + + +def _summarize(events, limit): + """Counts instead of lines: per field, its most frequent values. + + An agent asked who a user logged on to does not need a thousand raw lines — + on a real run it got 1000 events, could read about ten, and reported the + accounts and hosts of those ten. Top values per field answer the question + over the whole window and fit in a model's context. + """ + counts = {} + first_ts, last_ts = None, None + for e in events: + ts = e.get("timestamp") + if isinstance(ts, (int, float)): + first_ts = ts if first_ts is None or ts < first_ts else first_ts + last_ts = ts if last_ts is None or ts > last_ts else last_ts + for k, v in _fields(e).items(): + counts.setdefault(k, {}) + key = str(v)[:200] + counts[k][key] = counts[k].get(key, 0) + 1 + # Fields that vary least first: they are the ones that describe the activity + # (result, service, destination) rather than identify a line (timestamps, ids). + ranked = sorted(counts.items(), key=lambda kv: (len(kv[1]), kv[0])) + top = {} + for k, vals in ranked[:_MAX_FIELDS]: + top[k] = {"distinct": len(vals), + "top": [[v, n] for v, n in sorted(vals.items(), key=lambda x: -x[1])[:_TOP_VALUES]]} + return { + "count": len(events), + "truncated": len(events) >= limit, + "first_event": _iso(first_ts) if first_ts is not None else None, + "last_event": _iso(last_ts) if last_ts is not None else None, + "fields": top, + "fields_omitted": max(0, len(counts) - _MAX_FIELDS), + "samples": [_fields(e) for e in events[:_SAMPLES]], + } + + def _log_sets(): """Every log set of the account, as (id, name).""" out = request("GET", "log_search/management/logsets") @@ -210,8 +282,14 @@ def run(): first = request("GET", "log_search/query/logsets/" + urllib.parse.quote(log_set_id, safe=""), params=params) # No fallback log id: a log set spans several logs and every event names its own. events = _identify(_collect(first, limit), "", inp.get("detection_name")) + window = "%s to %s" % (_iso(start_time), _iso(end_time)) + if str(inp.get("summarize") or "").strip().lower() in ("1", "true", "yes"): + out = _summarize(events, limit) + out.update({"from": start_time, "to": end_time, "window": window}) + print(json.dumps(out)) + return print(json.dumps({"events": events, "count": len(events), "from": start_time, "to": end_time, - "window": "%s to %s" % (_iso(start_time), _iso(end_time))})) + "window": window})) try: