rapid7-insightidr 1.12.0: ingest LEQL query results as alerts

A tenant whose basic detection rules only notify — email, Slack — creates no
investigation and no alert any API can read: /investigations stays empty and the
Alert Triage API is MDR-gated. The rule's LEQL query is the only artefact left,
so replay it on a schedule and turn every matching line into an alert.

query_log and query_log_set gain:

- an ingest block (results_path events, dedup_key event_id, watermark start_time);
- event_id = log_id + the line's sequence number, the line's own address in Log
  Search. It has to be stable across runs because the scheduler deliberately
  re-reads the last minute of the previous window on every tick, and anything
  derived from the fetch time would make that overlap a second incident. A hash
  of (log_id, timestamp, message) covers a source that does not number entries;
- start_time read as ISO-8601, UNIX seconds or milliseconds alike. The API only
  takes milliseconds, but which one the scheduler sends is an operator setting on
  the rule — the feed must not fetch the wrong window because of a dropdown;
- a bounded page walk (limit, default 1000): unattended every few minutes, a
  query matching more than intended would otherwise hold the run open long
  enough for the next tick to find it in flight and skip itself;
- detection_name, stamped on every event. The log line does not know which rule
  it is being replayed for, and without it an incident is titled by its own first
  characters where the source console shows an alert name.

Plus mappers/query_log{,_set}.yaml: the parsed event lives in `message` as a
JSON-encoded string, so account, addresses, asset and process come out through
$parseJson. A raw unparsed line is not JSON — those entries resolve to nothing
and are skipped, and custom:rapid7_log_line still carries the line verbatim.
Severity is left unmapped: a log line has no priority of its own, so the alert
rule's default applies.

Known limits, by construction: this endpoint rejects calculations, so a rule with
a threshold (N occurrences in M minutes) cannot be reproduced — only pattern
rules are 1:1 — and an inactivity rule never can, since it fires on the absence
of events. The query also becomes a copy: editing the rule in InsightIDR leaves
the feed reading the old one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UjtRtCzw6FEajDDHJ3mTQf

integrations/rapid7-insightidr/manifest.yaml

@@ -1,8 +1,8 @@
 id: rapid7_insightidr
 name: Rapid7 InsightIDR
-version: 1.11.1
+version: 1.12.0
 description: "Rapid7 InsightIDR (REST API) — investigation ingestion and full lifecycle (list/get/search/create/update/assign/set-status/bulk-close), investigation alerts and Rapid7 product alerts, custom threat indicators (add/replace), log management and LEQL log/log-set queries with downloads, and user directory search."
-changelog: "1.11.1 — Strip the whitespace out of the comma-separated filter params (statuses, priorities, sources, tags) before the URL. \"HIGH, CRITICAL\" as typed sent \" CRITICAL\" to the API, which rejects the whole call — so a priority filter written naturally fetched nothing. | 1.11.0 —Build one evidence row per alert evidence into custom:rapid7_evidence, for the table field: sortable and filterable where the Markdown summary only reads. | 1.10.0 — Render every detection key as Markdown into custom:rapid7_summary, so a layout can show what the OCSF catalogue has no field for. Map both ends of an account action, read source_json top-level scalars, drop placeholder values, and force evidences to stay an array. | 1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
+changelog: "1.12.0 — Make the LEQL queries ingestible: query_log and query_log_set gain an ingest block, a stable event_id for dedup, a bounded page walk and a detection_name stamp, and they now read start_time as ISO-8601, UNIX seconds or milliseconds alike. A tenant whose basic detection rules only notify (email, Slack) creates no investigation and no alert an API can read, so replaying the rule's own query on a schedule is the only way its detections reach a SOAR. | 1.11.1 — Strip the whitespace out of the comma-separated filter params (statuses, priorities, sources, tags) before the URL. \"HIGH, CRITICAL\" as typed sent \" CRITICAL\" to the API, which rejects the whole call — so a priority filter written naturally fetched nothing. | 1.11.0 —Build one evidence row per alert evidence into custom:rapid7_evidence, for the table field: sortable and filterable where the Markdown summary only reads. | 1.10.0 — Render every detection key as Markdown into custom:rapid7_summary, so a layout can show what the OCSF catalogue has no field for. Map both ends of an account action, read source_json top-level scalars, drop placeholder values, and force evidences to stay an array. | 1.9.2 — Read the evidence data in both of its shapes. A parsed source states its event flat, with no source_data/custom_data wrapper, and the parser returned {} for all of them. | 1.9.1 — get_alert_process_trees: explain the 404, which the API returns with an empty body for three different situations. | 1.9.0 — Fetch each alert process tree and flatten it into ctx.process_tree, the shape the layout Process Tree block reads. Gated on the event type by default, since an authentication alert never has one. | 1.8.0 — Fetch each alert evidence at ingestion and parse it: the log line behind the alert lives in a JSON-encoded string no mapper can reach, and it is where the source and destination addresses, the account and the device actually are. | 1.7.0 — Map source/destination assets and accounts to src_endpoint, dst_endpoint, actor.user and user. A connection rule names both ends and neither is \"the host\", so an RDP-to-DC alert produced no entity at all and nothing a layout could show. | 1.6.0 — Surface region_failures: the Alert Triage API answers HTTP 200 with a partial result when it cannot reach a region, and that list was being dropped. Also report the regions seen in the RRNs against the configured one. | 1.5.0 — Flatten the whole of rule_keys_of_interest into the context. It is an open vocabulary — a process rule and an authentication rule share no keys — so the hardcoded list only covered rules already seen and left every other alert empty. | 1.4.1 — Read the alert RRN from `rrn` before `id` and drop anything not RRN-shaped: the investigation alert list also returns bare UUIDs, and one of those rejects the entire /alerts/ops/rrns batch (HTTP 400), losing every alert in the run. | 1.4.0 — Add get_investigations_full: the investigation with its alerts inlined, so it stays the unit that becomes an incident. Drop-in for list_investigations (results_path data, dedup rrn). | 1.3.0 — Add search_investigation_alerts: ingest only the alerts that belong to an investigation, with their full detection content, in 1+N+1 calls. Filtering search_alerts on investigation_rrn does not work — the term is accepted and silently matches nothing. | 1.2.1 — search_alerts: report a malformed terms_json with the input named and echoed, instead of json's bare \"Expecting value: line 1 column 1\". | 1.2.0 — Add the search_alerts OCSF mapper: rule_keys_of_interest carries the host, the process, its command line, its parent and the file hashes, so an alert-based feed indexes entities where the investigation feed indexed none. Validated against a live ABA alert. | 1.1.1 — list_alert_fields: default path_depth to 1 and expose the search filter. The V2 fields endpoint refuses an unbounded walk of the tree (\"root path with path depth greater than 1 is not supported for V2 without a search filter\"), so the bare call returned HTTP 400. | 1.1.0 — Add the Alert Triage API (/idr/at): search alerts, get an alert, its evidence, its actors and its process trees, plus the tenant's alert field catalogue. These carry the detection content the investigation endpoints do not — MITRE T-codes, matched rule keys, assets and accounts, command lines. MDR-gated. | 1.0.0 — Initial release: investigation ingestion (list_investigations) with an exhaustive OCSF mapper, 18 commands across investigations, alerts, custom threats, logs/LEQL search and users. API v1/v2 selectable per instance and per command."
 category: siem
 
 # Per-instance configuration. The Insight platform region selects the API host
@@ -251,35 +251,47 @@ commands:
 
   - id: query_log
     name: rapid7-insight-idr-query-log
-    description: "Run a LEQL query against a single log, following pagination links to gather all events."
+    description: "Run a LEQL query against a single log, following pagination links to gather all events. Usable as an ingestion feed: point it at the LEQL of a basic detection rule and every matching line becomes an alert (results path = events, one incident per log entry)."
     risk: read
     inputs_schema:
       properties:
         log_id: { type: string, description: "Log key to query" }
-        query: { type: string, description: "Valid LEQL query (no calculations)" }
+        query: { type: string, description: "Valid LEQL query (no calculations: groupby/calculate are rejected by this endpoint)" }
         time_range: { type: string, description: "Relative time range (e.g. 1 week, 1 day). When set, start_time/end_time are not needed." }
-        start_time: { type: string, description: "Lower bound as a UNIX timestamp in milliseconds" }
-        end_time: { type: string, description: "Upper bound as a UNIX timestamp in milliseconds" }
+        start_time: { type: string, description: "Lower bound: UNIX milliseconds, UNIX seconds or ISO-8601 — all three are accepted. Incremental fetch watermark." }
+        end_time: { type: string, description: "Upper bound: UNIX milliseconds, UNIX seconds or ISO-8601" }
         logs_per_page: { type: string, description: "Maximum log entries per page (default 50)" }
         sequence_number: { type: string, description: "Earliest sequence number of a log entry to start from" }
+        limit: { type: number, description: "Maximum events to gather across pages (default 1000). Bounds an unattended run against a query that matches more than it was meant to." }
+        detection_name: { type: string, description: "Name of the detection rule this query stands for. Stamped on every event and used as the incident title — a log line does not know which rule it was replayed for." }
       required: [log_id, query]
     outputs_schema: { properties: {} }
+    ingest:
+      results_path: events
+      dedup_key: event_id
+      incremental_field: start_time
 
   - id: query_log_set
     name: rapid7-insight-idr-query-log-set
-    description: "Run a LEQL query against a log set, following pagination links to gather all events."
+    description: "Run a LEQL query against a log set, following pagination links to gather all events. Usable as an ingestion feed: point it at the LEQL of a basic detection rule and every matching line becomes an alert (results path = events, one incident per log entry)."
     risk: read
     inputs_schema:
       properties:
         log_set_id: { type: string, description: "Log set ID to query" }
-        query: { type: string, description: "Valid LEQL query (no calculations)" }
+        query: { type: string, description: "Valid LEQL query (no calculations: groupby/calculate are rejected by this endpoint)" }
         time_range: { type: string, description: "Relative time range (e.g. 1 week, 1 day). When set, start_time/end_time are not needed." }
-        start_time: { type: string, description: "Lower bound as a UNIX timestamp in milliseconds" }
-        end_time: { type: string, description: "Upper bound as a UNIX timestamp in milliseconds" }
+        start_time: { type: string, description: "Lower bound: UNIX milliseconds, UNIX seconds or ISO-8601 — all three are accepted. Incremental fetch watermark." }
+        end_time: { type: string, description: "Upper bound: UNIX milliseconds, UNIX seconds or ISO-8601" }
         logs_per_page: { type: string, description: "Maximum log entries per page (default 50)" }
         sequence_number: { type: string, description: "Earliest sequence number of a log entry to start from" }
+        limit: { type: number, description: "Maximum events to gather across pages (default 1000). Bounds an unattended run against a query that matches more than it was meant to." }
+        detection_name: { type: string, description: "Name of the detection rule this query stands for. Stamped on every event and used as the incident title — a log line does not know which rule it was replayed for." }
       required: [log_set_id, query]
     outputs_schema: { properties: {} }
+    ingest:
+      results_path: events
+      dedup_key: event_id
+      incremental_field: start_time
 
   - id: download_logs
     name: rapid7-insight-idr-download-logs