diff --git a/integrations/harfanglab/manifest.yaml b/integrations/harfanglab/manifest.yaml index d736d1c..a51a17b 100644 --- a/integrations/harfanglab/manifest.yaml +++ b/integrations/harfanglab/manifest.yaml @@ -1,8 +1,8 @@ id: harfanglab name: HarfangLab EDR -version: 1.7.2 +version: 1.7.3 description: "HarfangLab EDR — endpoint detection & response: alert/threat ingestion, endpoint enrichment, isolation, threat-intelligence (IOC/whitelist), telemetry hunting and forensic collection jobs." -changelog: "1.7.1 — get_security_events, get_alerts and get_threats no longer pass on the `filters` block HarfangLab's list endpoints echo on every call: the catalogue of every filterable field, about 50 KB, unrelated to the results. One alert weighs about 10 KB, so that block alone made up most of a small fetch. The get_security_events and get_alerts mappers give an incident the rule's message (msg) as its description — what the rule detects and what to check — instead of the bare alert type, which now stands in only on alerts without a message. | 1.7.0 — New threat_security_events command: returns every security event of a threat. Since HarfangLab merged alerts and security events, a threat's events are the alerts whose threat_key is the threat id — the filter the vendor's own XSOAR integration uses. get_alerts could already filter on it but stops at one page of 100; threat_security_events pages through up to max_results (default 1000) and returns the threat's `total` with a `truncated` flag, so a cut-off list never passes for the whole threat. threat_status, threat_level and alert_tag now offer their status and level as drop-downs, and threat_status says it is where a threat is closed or marked a false positive — HarfangLab has no separate verdict field and no assignment on a threat. | 1.6.0 — New endpoint_count command: counts agents matching filters without listing them. The HarfangLab agent collection is DRF-paginated, so its `count` field already carries the filtered total; endpoint_count asks for limit=1 and reads it, instead of pulling up to 10000 agent records back through the playbook the way endpoint_search does. Filters cover the documented agent query parameters (hostname, os, version, status, isolation_state, tenant, network), add lastseen/firstseen bounds, and leave a raw `filters` escape hatch for the rest. The API silently drops a parameter it does not recognise as a filter — which turns a typo into a plausible-looking fleet-wide total — so the fleet `total` is returned next to `count` to make that case visible. | 1.5.0 — Declare the risk tier on the 71 commands that carried none: 39 read, 24 safe_write, 8 destructive. An unset risk is read as destructive by the agent approval gate, so the 63 that only read or only annotate sat behind the same prompt as the ones that change the estate. deisolate_from_threat and unquarantine_file move from safe_write to destructive in the same pass: undoing a containment mid-incident is its own decision, and is tiered like the containment it lifts. | 1.4.1 — get_alerts now maps image_name, the one process path present on every alert subtype and the only one on those carrying neither a process object nor an event log (yara, binary, ioc). HarfangLab prefixes it with the matched threat name in brackets, so both halves are split out: the clean path feeds actor.process.file.path and its basename actor.process.name, and the threat name feeds malware.name — which now surfaces on every subtype rather than only on Defender event-log detections. Each half is guarded on the bracket being present so an unprefixed path is never written into malware.name. 1.4.0 — Mapper coverage from real payloads: get_alerts now maps the Windows event-log payload sigma alerts carry (alert_subtype = eventlog), which was entirely unmapped — Defender 1116/1117 yields the malware name, classification, severity and provider, the detected file, the process that touched it and the acting user, plus the log provider and event code. Also added the agent IP, DNS domain, device type, agent version and groups, the tenant, the alert subtype and threat type, the string severity and the event time. get_threats gained the threat level (previously never reaching the OCSF document at all), the event time and the top agent's reachability. 1.3.1 — Flagged get_alerts as ingestion-ready (ingest hints: results_path/dedup/incremental on from_date) so it appears in the recommended ingestion commands when creating an instance. 1.3.0 — Response & triage expansion: kill/dump process, on-demand AV/YARA scans, file/service/scheduled-task deletion, quarantine management, threat-level isolation, job cancel/relaunch/wait; alert search (get_alerts) and full alert/threat triage (tag, comments, notes, status, level, history, details); exhaustive alert OCSF mappers; reworked get_threats mapper (fixed MITRE tactics map handling, full top-agent/user/rule and counts coverage, rule-based incident title/description). 1.2.2 — Documentation wording cleanup. 1.2.1 — Corrected the get_threats OCSF mapper to native threat fields (top_agents/top_impacted_users/top_rules/mitre_tactics) instead of enriched-only arrays; added agent.osversion to get_security_events. 1.2.0 — Added ingestion commands get_security_events (alerts) and get_threats with bundled OCSF mappers and ingest hints (results_path/dedup/incremental). 1.1.1 — Added test_connection for the instance Test button. 1.1.0 — Command names prefixed with 'harfanglab-' (e.g. harfanglab-isolate-endpoint) for easier toolbox search; command IDs unchanged. 1.0.0 — Initial release: endpoint/agent management, isolation, policy assignment, IOC & whitelist management, security-event triage, telemetry hunting (processes, network, DNS, authentications, binaries, event logs), threat hunting by hash, and forensic collection jobs (pipes, prefetch, run keys, scheduled tasks, drivers, services, processes, network, sessions, WMI, IOC scan, artifacts, RAM dump) with their result retrieval commands. Compatible with HarfangLab EDR 2.13.7+." +changelog: "1.7.3 — Every command now retries a call HarfangLab answers with a transient error, up to three times after 1, 2 and 4 seconds. The API returns a 500 now and then on a call that passes when sent again, which failed an ingestion run or a playbook step for nothing. Reads (GET) are retried on 429, 500, 502, 503 and 504. Writes are retried only on 429, 502 and 503, answers that say the action was not run: a 500 or 504 on an isolation, a kill or a job creation may come after the action took effect, so it is reported rather than sent twice. | 1.7.1 — get_security_events, get_alerts and get_threats no longer pass on the `filters` block HarfangLab's list endpoints echo on every call: the catalogue of every filterable field, about 50 KB, unrelated to the results. One alert weighs about 10 KB, so that block alone made up most of a small fetch. The get_security_events and get_alerts mappers give an incident the rule's message (msg) as its description — what the rule detects and what to check — instead of the bare alert type, which now stands in only on alerts without a message. | 1.7.0 — New threat_security_events command: returns every security event of a threat. Since HarfangLab merged alerts and security events, a threat's events are the alerts whose threat_key is the threat id — the filter the vendor's own XSOAR integration uses. get_alerts could already filter on it but stops at one page of 100; threat_security_events pages through up to max_results (default 1000) and returns the threat's `total` with a `truncated` flag, so a cut-off list never passes for the whole threat. threat_status, threat_level and alert_tag now offer their status and level as drop-downs, and threat_status says it is where a threat is closed or marked a false positive — HarfangLab has no separate verdict field and no assignment on a threat. | 1.6.0 — New endpoint_count command: counts agents matching filters without listing them. The HarfangLab agent collection is DRF-paginated, so its `count` field already carries the filtered total; endpoint_count asks for limit=1 and reads it, instead of pulling up to 10000 agent records back through the playbook the way endpoint_search does. Filters cover the documented agent query parameters (hostname, os, version, status, isolation_state, tenant, network), add lastseen/firstseen bounds, and leave a raw `filters` escape hatch for the rest. The API silently drops a parameter it does not recognise as a filter — which turns a typo into a plausible-looking fleet-wide total — so the fleet `total` is returned next to `count` to make that case visible. | 1.5.0 — Declare the risk tier on the 71 commands that carried none: 39 read, 24 safe_write, 8 destructive. An unset risk is read as destructive by the agent approval gate, so the 63 that only read or only annotate sat behind the same prompt as the ones that change the estate. deisolate_from_threat and unquarantine_file move from safe_write to destructive in the same pass: undoing a containment mid-incident is its own decision, and is tiered like the containment it lifts. | 1.4.1 — get_alerts now maps image_name, the one process path present on every alert subtype and the only one on those carrying neither a process object nor an event log (yara, binary, ioc). HarfangLab prefixes it with the matched threat name in brackets, so both halves are split out: the clean path feeds actor.process.file.path and its basename actor.process.name, and the threat name feeds malware.name — which now surfaces on every subtype rather than only on Defender event-log detections. Each half is guarded on the bracket being present so an unprefixed path is never written into malware.name. 1.4.0 — Mapper coverage from real payloads: get_alerts now maps the Windows event-log payload sigma alerts carry (alert_subtype = eventlog), which was entirely unmapped — Defender 1116/1117 yields the malware name, classification, severity and provider, the detected file, the process that touched it and the acting user, plus the log provider and event code. Also added the agent IP, DNS domain, device type, agent version and groups, the tenant, the alert subtype and threat type, the string severity and the event time. get_threats gained the threat level (previously never reaching the OCSF document at all), the event time and the top agent's reachability. 1.3.1 — Flagged get_alerts as ingestion-ready (ingest hints: results_path/dedup/incremental on from_date) so it appears in the recommended ingestion commands when creating an instance. 1.3.0 — Response & triage expansion: kill/dump process, on-demand AV/YARA scans, file/service/scheduled-task deletion, quarantine management, threat-level isolation, job cancel/relaunch/wait; alert search (get_alerts) and full alert/threat triage (tag, comments, notes, status, level, history, details); exhaustive alert OCSF mappers; reworked get_threats mapper (fixed MITRE tactics map handling, full top-agent/user/rule and counts coverage, rule-based incident title/description). 1.2.2 — Documentation wording cleanup. 1.2.1 — Corrected the get_threats OCSF mapper to native threat fields (top_agents/top_impacted_users/top_rules/mitre_tactics) instead of enriched-only arrays; added agent.osversion to get_security_events. 1.2.0 — Added ingestion commands get_security_events (alerts) and get_threats with bundled OCSF mappers and ingest hints (results_path/dedup/incremental). 1.1.1 — Added test_connection for the instance Test button. 1.1.0 — Command names prefixed with 'harfanglab-' (e.g. harfanglab-isolate-endpoint) for easier toolbox search; command IDs unchanged. 1.0.0 — Initial release: endpoint/agent management, isolation, policy assignment, IOC & whitelist management, security-event triage, telemetry hunting (processes, network, DNS, authentications, binaries, event logs), threat hunting by hash, and forensic collection jobs (pipes, prefetch, run keys, scheduled tasks, drivers, services, processes, network, sessions, WMI, IOC scan, artifacts, RAM dump) with their result retrieval commands. Compatible with HarfangLab EDR 2.13.7+." category: endpoint # Per-instance configuration. Scripts use as the API base and call /api/... paths. diff --git a/integrations/harfanglab/scripts/add_ioc_to_source.py b/integrations/harfanglab/scripts/add_ioc_to_source.py index 822911f..04c7a87 100644 --- a/integrations/harfanglab/scripts/add_ioc_to_source.py +++ b/integrations/harfanglab/scripts/add_ioc_to_source.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/alert_comment.py b/integrations/harfanglab/scripts/alert_comment.py index 789326b..cf80845 100644 --- a/integrations/harfanglab/scripts/alert_comment.py +++ b/integrations/harfanglab/scripts/alert_comment.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/alert_details.py b/integrations/harfanglab/scripts/alert_details.py index f3fc973..2af2f10 100644 --- a/integrations/harfanglab/scripts/alert_details.py +++ b/integrations/harfanglab/scripts/alert_details.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/alert_history.py b/integrations/harfanglab/scripts/alert_history.py index 359a32d..b82acfe 100644 --- a/integrations/harfanglab/scripts/alert_history.py +++ b/integrations/harfanglab/scripts/alert_history.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/alert_tag.py b/integrations/harfanglab/scripts/alert_tag.py index 1619064..33845ed 100644 --- a/integrations/harfanglab/scripts/alert_tag.py +++ b/integrations/harfanglab/scripts/alert_tag.py @@ -1,16 +1,35 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error def csv(v): return [x.strip() for x in str(v or "").split(",") if x.strip()] +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) + + def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/api_call.py b/integrations/harfanglab/scripts/api_call.py index 05bb766..709edcb 100644 --- a/integrations/harfanglab/scripts/api_call.py +++ b/integrations/harfanglab/scripts/api_call.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/assign_policy_to_agent.py b/integrations/harfanglab/scripts/assign_policy_to_agent.py index 1eb8dd7..df6c44c 100644 --- a/integrations/harfanglab/scripts/assign_policy_to_agent.py +++ b/integrations/harfanglab/scripts/assign_policy_to_agent.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/av_scan.py b/integrations/harfanglab/scripts/av_scan.py index 7b0c37e..46cb6db 100644 --- a/integrations/harfanglab/scripts/av_scan.py +++ b/integrations/harfanglab/scripts/av_scan.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def csv(v): diff --git a/integrations/harfanglab/scripts/change_security_event_status.py b/integrations/harfanglab/scripts/change_security_event_status.py index 272903b..0baa54d 100644 --- a/integrations/harfanglab/scripts/change_security_event_status.py +++ b/integrations/harfanglab/scripts/change_security_event_status.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/deisolate_endpoint.py b/integrations/harfanglab/scripts/deisolate_endpoint.py index e2b849b..c4ee30d 100644 --- a/integrations/harfanglab/scripts/deisolate_endpoint.py +++ b/integrations/harfanglab/scripts/deisolate_endpoint.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/deisolate_from_threat.py b/integrations/harfanglab/scripts/deisolate_from_threat.py index 05e07a4..65dc2c4 100644 --- a/integrations/harfanglab/scripts/deisolate_from_threat.py +++ b/integrations/harfanglab/scripts/deisolate_from_threat.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/delete_file.py b/integrations/harfanglab/scripts/delete_file.py index 3368ed1..994e43a 100644 --- a/integrations/harfanglab/scripts/delete_file.py +++ b/integrations/harfanglab/scripts/delete_file.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/delete_ioc_from_source.py b/integrations/harfanglab/scripts/delete_ioc_from_source.py index 1dea707..54ab346 100644 --- a/integrations/harfanglab/scripts/delete_ioc_from_source.py +++ b/integrations/harfanglab/scripts/delete_ioc_from_source.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/delete_scheduled_task.py b/integrations/harfanglab/scripts/delete_scheduled_task.py index ddd6fd1..c78d76f 100644 --- a/integrations/harfanglab/scripts/delete_scheduled_task.py +++ b/integrations/harfanglab/scripts/delete_scheduled_task.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/delete_service.py b/integrations/harfanglab/scripts/delete_service.py index caba68c..d442211 100644 --- a/integrations/harfanglab/scripts/delete_service.py +++ b/integrations/harfanglab/scripts/delete_service.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/dump_process.py b/integrations/harfanglab/scripts/dump_process.py index d3a7b4d..57d9403 100644 --- a/integrations/harfanglab/scripts/dump_process.py +++ b/integrations/harfanglab/scripts/dump_process.py @@ -1,4 +1,4 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error FIELD_TYPES = { "name": 0, "path": 1, "username": 3, "commandline": 4, @@ -8,12 +8,31 @@ FIELD_TYPES = { OPERATORS = {"eq": 0, "not_eq": 1, "contains": 2, "not_contains": 3} +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) + + def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/endpoint_count.py b/integrations/harfanglab/scripts/endpoint_count.py index c1f741d..94100e7 100644 --- a/integrations/harfanglab/scripts/endpoint_count.py +++ b/integrations/harfanglab/scripts/endpoint_count.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/endpoint_search.py b/integrations/harfanglab/scripts/endpoint_search.py index 6f77d39..c613475 100644 --- a/integrations/harfanglab/scripts/endpoint_search.py +++ b/integrations/harfanglab/scripts/endpoint_search.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/get_alerts.py b/integrations/harfanglab/scripts/get_alerts.py index 98f4a9c..c6a4e51 100644 --- a/integrations/harfanglab/scripts/get_alerts.py +++ b/integrations/harfanglab/scripts/get_alerts.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/get_endpoint_info.py b/integrations/harfanglab/scripts/get_endpoint_info.py index 0e7ecb8..f625f1d 100644 --- a/integrations/harfanglab/scripts/get_endpoint_info.py +++ b/integrations/harfanglab/scripts/get_endpoint_info.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/get_security_events.py b/integrations/harfanglab/scripts/get_security_events.py index 7631512..4337d4c 100644 --- a/integrations/harfanglab/scripts/get_security_events.py +++ b/integrations/harfanglab/scripts/get_security_events.py @@ -1,13 +1,32 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error SEVERITIES = ["low", "medium", "high", "critical"] +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) + + def request(method, url, headers): - req = urllib.request.Request(url, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=90) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=90) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/get_threat.py b/integrations/harfanglab/scripts/get_threat.py index e1d6d05..43ef4ac 100644 --- a/integrations/harfanglab/scripts/get_threat.py +++ b/integrations/harfanglab/scripts/get_threat.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/get_threats.py b/integrations/harfanglab/scripts/get_threats.py index 5edecd1..1d8783d 100644 --- a/integrations/harfanglab/scripts/get_threats.py +++ b/integrations/harfanglab/scripts/get_threats.py @@ -1,13 +1,32 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error SEVERITIES = ["low", "medium", "high", "critical"] +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) + + def request(method, url, headers): - req = urllib.request.Request(url, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=90) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=90) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/hunt_search_hash.py b/integrations/harfanglab/scripts/hunt_search_hash.py index cc2f000..705f7e9 100644 --- a/integrations/harfanglab/scripts/hunt_search_hash.py +++ b/integrations/harfanglab/scripts/hunt_search_hash.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/hunt_search_runned_process_hash.py b/integrations/harfanglab/scripts/hunt_search_runned_process_hash.py index 3fe8949..f94b327 100644 --- a/integrations/harfanglab/scripts/hunt_search_runned_process_hash.py +++ b/integrations/harfanglab/scripts/hunt_search_runned_process_hash.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/hunt_search_running_process_hash.py b/integrations/harfanglab/scripts/hunt_search_running_process_hash.py index 5759500..8e8783a 100644 --- a/integrations/harfanglab/scripts/hunt_search_running_process_hash.py +++ b/integrations/harfanglab/scripts/hunt_search_running_process_hash.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/isolate_endpoint.py b/integrations/harfanglab/scripts/isolate_endpoint.py index 27f855f..2fe6ba4 100644 --- a/integrations/harfanglab/scripts/isolate_endpoint.py +++ b/integrations/harfanglab/scripts/isolate_endpoint.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/isolate_from_threat.py b/integrations/harfanglab/scripts/isolate_from_threat.py index 471aa4e..60b3fb7 100644 --- a/integrations/harfanglab/scripts/isolate_from_threat.py +++ b/integrations/harfanglab/scripts/isolate_from_threat.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/job_artifact_all.py b/integrations/harfanglab/scripts/job_artifact_all.py index d3dc989..d67278e 100644 --- a/integrations/harfanglab/scripts/job_artifact_all.py +++ b/integrations/harfanglab/scripts/job_artifact_all.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_artifact_downloadfile.py b/integrations/harfanglab/scripts/job_artifact_downloadfile.py index cb77bc7..b34e798 100644 --- a/integrations/harfanglab/scripts/job_artifact_downloadfile.py +++ b/integrations/harfanglab/scripts/job_artifact_downloadfile.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_artifact_evtx.py b/integrations/harfanglab/scripts/job_artifact_evtx.py index fe7e7fa..e279e97 100644 --- a/integrations/harfanglab/scripts/job_artifact_evtx.py +++ b/integrations/harfanglab/scripts/job_artifact_evtx.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_artifact_filesystem.py b/integrations/harfanglab/scripts/job_artifact_filesystem.py index f1bbc1b..93a296e 100644 --- a/integrations/harfanglab/scripts/job_artifact_filesystem.py +++ b/integrations/harfanglab/scripts/job_artifact_filesystem.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_artifact_hives.py b/integrations/harfanglab/scripts/job_artifact_hives.py index 6383a0a..fb59bea 100644 --- a/integrations/harfanglab/scripts/job_artifact_hives.py +++ b/integrations/harfanglab/scripts/job_artifact_hives.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_artifact_logs.py b/integrations/harfanglab/scripts/job_artifact_logs.py index 7bbebff..fb1a4a7 100644 --- a/integrations/harfanglab/scripts/job_artifact_logs.py +++ b/integrations/harfanglab/scripts/job_artifact_logs.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_artifact_mft.py b/integrations/harfanglab/scripts/job_artifact_mft.py index 1615d6e..24c375b 100644 --- a/integrations/harfanglab/scripts/job_artifact_mft.py +++ b/integrations/harfanglab/scripts/job_artifact_mft.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_artifact_ramdump.py b/integrations/harfanglab/scripts/job_artifact_ramdump.py index f47581d..9af79e4 100644 --- a/integrations/harfanglab/scripts/job_artifact_ramdump.py +++ b/integrations/harfanglab/scripts/job_artifact_ramdump.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_cancel.py b/integrations/harfanglab/scripts/job_cancel.py index ae05a84..28c19bf 100644 --- a/integrations/harfanglab/scripts/job_cancel.py +++ b/integrations/harfanglab/scripts/job_cancel.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/job_driverlist.py b/integrations/harfanglab/scripts/job_driverlist.py index 031abc4..26f0348 100644 --- a/integrations/harfanglab/scripts/job_driverlist.py +++ b/integrations/harfanglab/scripts/job_driverlist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_info.py b/integrations/harfanglab/scripts/job_info.py index 1df3d35..7e3bcd6 100644 --- a/integrations/harfanglab/scripts/job_info.py +++ b/integrations/harfanglab/scripts/job_info.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_ioc.py b/integrations/harfanglab/scripts/job_ioc.py index 77283eb..f23bf4a 100644 --- a/integrations/harfanglab/scripts/job_ioc.py +++ b/integrations/harfanglab/scripts/job_ioc.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_networkconnectionlist.py b/integrations/harfanglab/scripts/job_networkconnectionlist.py index 1de51d0..ba83dd6 100644 --- a/integrations/harfanglab/scripts/job_networkconnectionlist.py +++ b/integrations/harfanglab/scripts/job_networkconnectionlist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_networksharelist.py b/integrations/harfanglab/scripts/job_networksharelist.py index 65a6267..5a91903 100644 --- a/integrations/harfanglab/scripts/job_networksharelist.py +++ b/integrations/harfanglab/scripts/job_networksharelist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_persistencelist.py b/integrations/harfanglab/scripts/job_persistencelist.py index 8c11ca6..06bc8e3 100644 --- a/integrations/harfanglab/scripts/job_persistencelist.py +++ b/integrations/harfanglab/scripts/job_persistencelist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_pipelist.py b/integrations/harfanglab/scripts/job_pipelist.py index 2d7ed2f..6bb2c52 100644 --- a/integrations/harfanglab/scripts/job_pipelist.py +++ b/integrations/harfanglab/scripts/job_pipelist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_prefetchlist.py b/integrations/harfanglab/scripts/job_prefetchlist.py index 0bda952..c45d2f8 100644 --- a/integrations/harfanglab/scripts/job_prefetchlist.py +++ b/integrations/harfanglab/scripts/job_prefetchlist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_processlist.py b/integrations/harfanglab/scripts/job_processlist.py index f18983f..344b877 100644 --- a/integrations/harfanglab/scripts/job_processlist.py +++ b/integrations/harfanglab/scripts/job_processlist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_relaunch.py b/integrations/harfanglab/scripts/job_relaunch.py index df73f28..015a6f5 100644 --- a/integrations/harfanglab/scripts/job_relaunch.py +++ b/integrations/harfanglab/scripts/job_relaunch.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/job_runkeylist.py b/integrations/harfanglab/scripts/job_runkeylist.py index 61329fd..df1ba26 100644 --- a/integrations/harfanglab/scripts/job_runkeylist.py +++ b/integrations/harfanglab/scripts/job_runkeylist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_scheduledtasklist.py b/integrations/harfanglab/scripts/job_scheduledtasklist.py index 8854ec6..2b8ed68 100644 --- a/integrations/harfanglab/scripts/job_scheduledtasklist.py +++ b/integrations/harfanglab/scripts/job_scheduledtasklist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_servicelist.py b/integrations/harfanglab/scripts/job_servicelist.py index 61329fd..df1ba26 100644 --- a/integrations/harfanglab/scripts/job_servicelist.py +++ b/integrations/harfanglab/scripts/job_servicelist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_sessionlist.py b/integrations/harfanglab/scripts/job_sessionlist.py index 74e152f..dd0bcaf 100644 --- a/integrations/harfanglab/scripts/job_sessionlist.py +++ b/integrations/harfanglab/scripts/job_sessionlist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_startuplist.py b/integrations/harfanglab/scripts/job_startuplist.py index 2fddbf0..f0ecd48 100644 --- a/integrations/harfanglab/scripts/job_startuplist.py +++ b/integrations/harfanglab/scripts/job_startuplist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/job_wmilist.py b/integrations/harfanglab/scripts/job_wmilist.py index 2ce7692..b226a09 100644 --- a/integrations/harfanglab/scripts/job_wmilist.py +++ b/integrations/harfanglab/scripts/job_wmilist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/kill_process.py b/integrations/harfanglab/scripts/kill_process.py index 324f2b0..cde4c72 100644 --- a/integrations/harfanglab/scripts/kill_process.py +++ b/integrations/harfanglab/scripts/kill_process.py @@ -1,4 +1,4 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error FIELD_TYPES = { "name": 0, "path": 1, "username": 3, "commandline": 4, @@ -8,12 +8,31 @@ FIELD_TYPES = { OPERATORS = {"eq": 0, "not_eq": 1, "contains": 2, "not_contains": 3} +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) + + def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/quarantine_file.py b/integrations/harfanglab/scripts/quarantine_file.py index 0b1d7c2..a1f6cb5 100644 --- a/integrations/harfanglab/scripts/quarantine_file.py +++ b/integrations/harfanglab/scripts/quarantine_file.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/quarantine_request_upload.py b/integrations/harfanglab/scripts/quarantine_request_upload.py index 3de48fe..7a8d6d4 100644 --- a/integrations/harfanglab/scripts/quarantine_request_upload.py +++ b/integrations/harfanglab/scripts/quarantine_request_upload.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/quarantine_search.py b/integrations/harfanglab/scripts/quarantine_search.py index 4dd5627..204965f 100644 --- a/integrations/harfanglab/scripts/quarantine_search.py +++ b/integrations/harfanglab/scripts/quarantine_search.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_artifact_all.py b/integrations/harfanglab/scripts/result_artifact_all.py index 484b674..429be77 100644 --- a/integrations/harfanglab/scripts/result_artifact_all.py +++ b/integrations/harfanglab/scripts/result_artifact_all.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_artifact_downloadfile.py b/integrations/harfanglab/scripts/result_artifact_downloadfile.py index 484b674..429be77 100644 --- a/integrations/harfanglab/scripts/result_artifact_downloadfile.py +++ b/integrations/harfanglab/scripts/result_artifact_downloadfile.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_artifact_evtx.py b/integrations/harfanglab/scripts/result_artifact_evtx.py index 484b674..429be77 100644 --- a/integrations/harfanglab/scripts/result_artifact_evtx.py +++ b/integrations/harfanglab/scripts/result_artifact_evtx.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_artifact_filesystem.py b/integrations/harfanglab/scripts/result_artifact_filesystem.py index 484b674..429be77 100644 --- a/integrations/harfanglab/scripts/result_artifact_filesystem.py +++ b/integrations/harfanglab/scripts/result_artifact_filesystem.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_artifact_hives.py b/integrations/harfanglab/scripts/result_artifact_hives.py index 484b674..429be77 100644 --- a/integrations/harfanglab/scripts/result_artifact_hives.py +++ b/integrations/harfanglab/scripts/result_artifact_hives.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_artifact_logs.py b/integrations/harfanglab/scripts/result_artifact_logs.py index 484b674..429be77 100644 --- a/integrations/harfanglab/scripts/result_artifact_logs.py +++ b/integrations/harfanglab/scripts/result_artifact_logs.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_artifact_mft.py b/integrations/harfanglab/scripts/result_artifact_mft.py index 484b674..429be77 100644 --- a/integrations/harfanglab/scripts/result_artifact_mft.py +++ b/integrations/harfanglab/scripts/result_artifact_mft.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_artifact_ramdump.py b/integrations/harfanglab/scripts/result_artifact_ramdump.py index 484b674..429be77 100644 --- a/integrations/harfanglab/scripts/result_artifact_ramdump.py +++ b/integrations/harfanglab/scripts/result_artifact_ramdump.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_driverlist.py b/integrations/harfanglab/scripts/result_driverlist.py index 8680128..df9f9a9 100644 --- a/integrations/harfanglab/scripts/result_driverlist.py +++ b/integrations/harfanglab/scripts/result_driverlist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_ioc.py b/integrations/harfanglab/scripts/result_ioc.py index 4d109fa..d20491a 100644 --- a/integrations/harfanglab/scripts/result_ioc.py +++ b/integrations/harfanglab/scripts/result_ioc.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_networkconnectionlist.py b/integrations/harfanglab/scripts/result_networkconnectionlist.py index 23cc6a5..3885068 100644 --- a/integrations/harfanglab/scripts/result_networkconnectionlist.py +++ b/integrations/harfanglab/scripts/result_networkconnectionlist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_networksharelist.py b/integrations/harfanglab/scripts/result_networksharelist.py index 757ac10..bce109c 100644 --- a/integrations/harfanglab/scripts/result_networksharelist.py +++ b/integrations/harfanglab/scripts/result_networksharelist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_persistencelist.py b/integrations/harfanglab/scripts/result_persistencelist.py index 8618bda..e1e8f48 100644 --- a/integrations/harfanglab/scripts/result_persistencelist.py +++ b/integrations/harfanglab/scripts/result_persistencelist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_pipelist.py b/integrations/harfanglab/scripts/result_pipelist.py index b03ebab..1e55425 100644 --- a/integrations/harfanglab/scripts/result_pipelist.py +++ b/integrations/harfanglab/scripts/result_pipelist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_prefetchlist.py b/integrations/harfanglab/scripts/result_prefetchlist.py index 23e8bbf..79730be 100644 --- a/integrations/harfanglab/scripts/result_prefetchlist.py +++ b/integrations/harfanglab/scripts/result_prefetchlist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_processlist.py b/integrations/harfanglab/scripts/result_processlist.py index 23cc6a5..3885068 100644 --- a/integrations/harfanglab/scripts/result_processlist.py +++ b/integrations/harfanglab/scripts/result_processlist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_runkeylist.py b/integrations/harfanglab/scripts/result_runkeylist.py index 5ff0249..93e18d0 100644 --- a/integrations/harfanglab/scripts/result_runkeylist.py +++ b/integrations/harfanglab/scripts/result_runkeylist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_scheduledtasklist.py b/integrations/harfanglab/scripts/result_scheduledtasklist.py index e53dc41..22bf1a6 100644 --- a/integrations/harfanglab/scripts/result_scheduledtasklist.py +++ b/integrations/harfanglab/scripts/result_scheduledtasklist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_servicelist.py b/integrations/harfanglab/scripts/result_servicelist.py index 8545b0b..1bead3a 100644 --- a/integrations/harfanglab/scripts/result_servicelist.py +++ b/integrations/harfanglab/scripts/result_servicelist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_sessionlist.py b/integrations/harfanglab/scripts/result_sessionlist.py index b5689d5..05c5611 100644 --- a/integrations/harfanglab/scripts/result_sessionlist.py +++ b/integrations/harfanglab/scripts/result_sessionlist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_startuplist.py b/integrations/harfanglab/scripts/result_startuplist.py index 6895bc3..83b028b 100644 --- a/integrations/harfanglab/scripts/result_startuplist.py +++ b/integrations/harfanglab/scripts/result_startuplist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/result_wmilist.py b/integrations/harfanglab/scripts/result_wmilist.py index e9629c6..ca5a300 100644 --- a/integrations/harfanglab/scripts/result_wmilist.py +++ b/integrations/harfanglab/scripts/result_wmilist.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/telemetry_authentication_linux.py b/integrations/harfanglab/scripts/telemetry_authentication_linux.py index 9725cab..8d397aa 100644 --- a/integrations/harfanglab/scripts/telemetry_authentication_linux.py +++ b/integrations/harfanglab/scripts/telemetry_authentication_linux.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/telemetry_authentication_macos.py b/integrations/harfanglab/scripts/telemetry_authentication_macos.py index a40f42f..593f0d2 100644 --- a/integrations/harfanglab/scripts/telemetry_authentication_macos.py +++ b/integrations/harfanglab/scripts/telemetry_authentication_macos.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/telemetry_authentication_users.py b/integrations/harfanglab/scripts/telemetry_authentication_users.py index 1cc70d2..edc5dd8 100644 --- a/integrations/harfanglab/scripts/telemetry_authentication_users.py +++ b/integrations/harfanglab/scripts/telemetry_authentication_users.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/telemetry_authentication_windows.py b/integrations/harfanglab/scripts/telemetry_authentication_windows.py index 0f01f7a..292ec09 100644 --- a/integrations/harfanglab/scripts/telemetry_authentication_windows.py +++ b/integrations/harfanglab/scripts/telemetry_authentication_windows.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/telemetry_binary.py b/integrations/harfanglab/scripts/telemetry_binary.py index e08cdf7..1c9e555 100644 --- a/integrations/harfanglab/scripts/telemetry_binary.py +++ b/integrations/harfanglab/scripts/telemetry_binary.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/telemetry_dns.py b/integrations/harfanglab/scripts/telemetry_dns.py index 5fbf195..1f34080 100644 --- a/integrations/harfanglab/scripts/telemetry_dns.py +++ b/integrations/harfanglab/scripts/telemetry_dns.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/telemetry_eventlog.py b/integrations/harfanglab/scripts/telemetry_eventlog.py index c26270e..bb9d4c0 100644 --- a/integrations/harfanglab/scripts/telemetry_eventlog.py +++ b/integrations/harfanglab/scripts/telemetry_eventlog.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/telemetry_network.py b/integrations/harfanglab/scripts/telemetry_network.py index a9110d1..52240e7 100644 --- a/integrations/harfanglab/scripts/telemetry_network.py +++ b/integrations/harfanglab/scripts/telemetry_network.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/telemetry_process_graph.py b/integrations/harfanglab/scripts/telemetry_process_graph.py index c134872..838c011 100644 --- a/integrations/harfanglab/scripts/telemetry_process_graph.py +++ b/integrations/harfanglab/scripts/telemetry_process_graph.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/telemetry_processes.py b/integrations/harfanglab/scripts/telemetry_processes.py index a09b6f3..164c49a 100644 --- a/integrations/harfanglab/scripts/telemetry_processes.py +++ b/integrations/harfanglab/scripts/telemetry_processes.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/test_connection.py b/integrations/harfanglab/scripts/test_connection.py index 1110407..ec6583a 100644 --- a/integrations/harfanglab/scripts/test_connection.py +++ b/integrations/harfanglab/scripts/test_connection.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/threat_comment.py b/integrations/harfanglab/scripts/threat_comment.py index 0d6999b..978c86d 100644 --- a/integrations/harfanglab/scripts/threat_comment.py +++ b/integrations/harfanglab/scripts/threat_comment.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/threat_history.py b/integrations/harfanglab/scripts/threat_history.py index 1ba0cd4..9fb2893 100644 --- a/integrations/harfanglab/scripts/threat_history.py +++ b/integrations/harfanglab/scripts/threat_history.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/threat_level.py b/integrations/harfanglab/scripts/threat_level.py index 325fbe7..b9b983e 100644 --- a/integrations/harfanglab/scripts/threat_level.py +++ b/integrations/harfanglab/scripts/threat_level.py @@ -1,16 +1,35 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error def csv(v): return [x.strip() for x in str(v or "").split(",") if x.strip()] +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) + + def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/threat_note_set.py b/integrations/harfanglab/scripts/threat_note_set.py index 1edc534..f32bfcb 100644 --- a/integrations/harfanglab/scripts/threat_note_set.py +++ b/integrations/harfanglab/scripts/threat_note_set.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/threat_security_events.py b/integrations/harfanglab/scripts/threat_security_events.py index 3ae42da..d0d1935 100644 --- a/integrations/harfanglab/scripts/threat_security_events.py +++ b/integrations/harfanglab/scripts/threat_security_events.py @@ -1,14 +1,33 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error PAGE = 500 +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) + + def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/threat_status.py b/integrations/harfanglab/scripts/threat_status.py index a290202..77830cf 100644 --- a/integrations/harfanglab/scripts/threat_status.py +++ b/integrations/harfanglab/scripts/threat_status.py @@ -1,16 +1,35 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error def csv(v): return [x.strip() for x in str(v or "").split(",") if x.strip()] +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) + + def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/unquarantine_file.py b/integrations/harfanglab/scripts/unquarantine_file.py index 39d2e48..f638586 100644 --- a/integrations/harfanglab/scripts/unquarantine_file.py +++ b/integrations/harfanglab/scripts/unquarantine_file.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/wait_for_job.py b/integrations/harfanglab/scripts/wait_for_job.py index 64b2c8e..2bc6307 100644 --- a/integrations/harfanglab/scripts/wait_for_job.py +++ b/integrations/harfanglab/scripts/wait_for_job.py @@ -1,12 +1,31 @@ import json, os, sys, time, urllib.request, urllib.parse, urllib.error +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) + + def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def main(): diff --git a/integrations/harfanglab/scripts/whitelist_add.py b/integrations/harfanglab/scripts/whitelist_add.py index 4df3bb2..419309e 100644 --- a/integrations/harfanglab/scripts/whitelist_add.py +++ b/integrations/harfanglab/scripts/whitelist_add.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/whitelist_add_criterion.py b/integrations/harfanglab/scripts/whitelist_add_criterion.py index 8e9134d..bd15c26 100644 --- a/integrations/harfanglab/scripts/whitelist_add_criterion.py +++ b/integrations/harfanglab/scripts/whitelist_add_criterion.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/whitelist_delete.py b/integrations/harfanglab/scripts/whitelist_delete.py index 7387ee1..a1db86c 100644 --- a/integrations/harfanglab/scripts/whitelist_delete.py +++ b/integrations/harfanglab/scripts/whitelist_delete.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/whitelist_search.py b/integrations/harfanglab/scripts/whitelist_search.py index 820a016..f72a62f 100644 --- a/integrations/harfanglab/scripts/whitelist_search.py +++ b/integrations/harfanglab/scripts/whitelist_search.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def qs(params): diff --git a/integrations/harfanglab/scripts/yara_scan.py b/integrations/harfanglab/scripts/yara_scan.py index 3fcb9db..2cdd8cf 100644 --- a/integrations/harfanglab/scripts/yara_scan.py +++ b/integrations/harfanglab/scripts/yara_scan.py @@ -1,12 +1,31 @@ -import json, os, sys, urllib.request, urllib.parse, urllib.error +import json, os, sys, time, urllib.request, urllib.parse, urllib.error + + +RETRIES = 3 + + +def retryable(method, status): + # HarfangLab answers a 500 now and then on a call that passes when sent again. + # A read is safe to resend; a write only on an answer that says the action was + # not run (rate limit, gateway down), so an isolation or a job is never sent twice. + if status in (429, 502, 503): + return True + return method == "GET" and status in (500, 504) def request(method, url, headers, body=None): data = json.dumps(body).encode("utf-8") if body is not None else None - req = urllib.request.Request(url, data=data, headers=headers, method=method) - with urllib.request.urlopen(req, timeout=60) as resp: - raw = resp.read() - return json.loads(raw) if raw else {} + for attempt in range(RETRIES + 1): + req = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as e: + if attempt == RETRIES or not retryable(method, e.code): + raise + e.close() + time.sleep(2 ** attempt) def csv(v):