At Hunt & Hackett, our Threat Hunting (TH) service operates on the principle that no defense is perfect, and some sophisticated threats will inevitably bypass initial prevention and detection controls.
Our strategy involves a continuous feedback loop, whereby we investigate why past threats were missed to identify improvements in our defenses. These insights directly improve our customers' resiliency. To structure our threat hunts, we use a methodology derived from the Pyramid of Pain, a cybersecurity model that categorizes threat indicators based on how difficult they are for an adversary to change.
Image 1. Pyramid of Pain.
Our hunts focus on indicators at all levels of the pyramid, from simple hash values, IP addresses, and domains to the tactics, techniques, and procedures (TTPs) that are most disruptive for an attacker to alter, since it impacts the behaviour that they exhibit.
In this post, we're diving into how we hunt for Level 1 Indicators of Compromise (IOCs) - IPs, domains and hashes - at scale. A fair question might be, 'Why not just create a simple detection rule for every IOC?' The reality is that many low-fidelity indicators have a limited shelf life and lack the context needed to produce actionable detections. A rule-per-IOC approach at scale creates alert volume that overwhelms the SOC and drives false positives and tuning churn.
Instead, we hunt Level 1 IOCs selectively: only curated, high-confidence indicators, plus IR-triggered IOCs that are relevant in that timeframe. We perform threat hunts based on IOCs from continuous curation by the Hunt & Hackett Threat Intelligence team, to weed out the noise, and real-time triggers from our incident response team to hunt for what's relevant right now. This intelligent system runs alongside our traditional rules, which we still use for reliable, long-lived IOCs.
As part of our infrastructure architecture, each customer operates in a fully isolated environment with a dedicated SIEM instance. This tenant separation provides strong guarantees around security and confidentiality: customer telemetry is never co-mingled, access is tightly scoped, and data handling remains aligned with GDPR principles such as purpose limitation and data minimisation. It also reduces systemic risk by containing misconfigurations or compromise to a single environment, rather than creating cross-customer blast radius.
The trade-off is operational: because data is intentionally segmented per customer, our threat hunters cannot automatically run IOC hunts “globally” across all tenants in a single query. Hunting at scale therefore requires an orchestration layer that can execute the same validated hunt logic across isolated instances, without weakening the isolation and confidentiality guarantees that the architecture is designed to provide. Just as importantly, it standardizes hunt logic across tenants and produces repeatable, auditable evidence of what was searched, where, and when. This is critical for consistent operations and post-incident reporting.
Image 2. A simplified view of a customer environment.
In our SOC tech stack, we use Google Security Operations (formerly Chronicle) as our SIEM. When we identify a new IOC such as a sha256 hash, a malicious domain, or a suspicious command line, we hunt these IOCs across all customer environments. We do this by starting a "retrohunt" in Google Security Operations. A retrohunt allows us to apply a detection rule to historical data to find past threats. While powerful, running a retrohunt across multiple isolated, single-tenant environments was a highly manual and repetitive process. The detection rule for the retrohunt needs a list of the IOCs to search for. This is called a reference list.
For example:
| 1.1.1.1 8.8.8.8 9.9.9.9 |
And a YARA-L detection rule that uses this reference list might look like this:
rule threathunt_example {
meta:
author = "spearfish"
description = "rule automatically generated for cross customer IOC search"
severity = "Low"
events:
$e.principal.ip in %example_list OR
$e.target.ip in %example_list OR
$e.src.ip in %example_list OR
$e.dst.ip in %example_list
condition:
$e
} For each customer, our analysts had to:
At small scale this is manageable. Across multiple isolated tenants, it becomes time-consuming, inconsistent, and error-prone, especially when hunts are frequent or time-sensitive. Therefore, this was exactly the kind of task ripe for automation.
To address this challenge, we developed Spearhead, an internal tool designed to automate the entire multi-customer IOC hunting workflow. The primary goal of Spearhead is to make the analyst's workflow as simple as possible. Instead of manually creating reference lists and detection rules for each customer, an analyst simply provides the IOCs and the list of target customers. Spearhead takes care of the rest, automatically generating the necessary reference lists and detection rules, and then orchestrating the retrohunts across all specified customer instances. Because the hunt definition is generated from a single request, the logic is consistent across tenants and the execution can be logged end-to-end, making hunts repeatable and auditable.
We designed Spearhead to be a lightweight, scalable, and secure solution. Here’s a high-level overview of its architecture:
Let's walk through a typical workflow with Spearhead. Spearhead exposes a simple REST API. Here are the main endpoints:
POST /create_retrohunt: Creates a new hunt. GET /retrohunt/{threathunt_uuid}: Checks the status of a hunt. GET /retrohunt/{threathunt_uuid}/{customer}: Streams the results for a specific customer. The /create_retrohunt endpoint accepts a JSON body with the following structure:
{
"starttime": "2023-01-01T00:00:00Z",
"endtime": "2023-01-02T00:00:00Z",
"domains": ["evil.com", "malicious.net"],
"file_hashes_sha256": ["..."],
"ip_addresses": ["1.2.3.4"],
"urls": ["http://evil.com/malware.exe"],
"command_lines": ["powershell -e ..."],
"customers": ["customer-a", "customer-b"]
}
The JSON structure lets you bundle all known indicators for a threat - domains, IPs, file hashes and more - into one request. Spearhead then intelligently combines them into one large search, looking for a match against any of them.
Here’s how an analyst would use Spearhead:
POST /create_retrohunt endpoint. GET /retrohunt/{threathunt_uuid} endpoint. Once a hunt is complete, they can retrieve the results using the GET /retrohunt/{threathunt_uuid}/{customer} endpoint. This automated triage process provides the analyst with a comprehensive view of the alert, enabling them to quickly assess the impact and scope of the threat.
Currently, our Threat Intelligence team feeds an average of almost 100 new IOCs into Spearhead every week. But each hunt isn't limited to just the latest indicators, it carries forward all previously curated IOCs, meaning every retrohunt is a cumulative search across our full indicator set. This ensures that newly onboarded customers or freshly ingested telemetry is always evaluated against the complete history of validated threats.
This entire process is automated end-to-end. An analyst simply triggers a single endpoint. From there, Spearhead fetches the relevant IOCs from the TIP, generates the reference lists and detection rules, and kicks off the retrohunts across all customer tenants without further human involvement. The next time a human enters the loop is when a hit surfaces and an incident is automatically created in XSOAR for an analyst to pick up, validate, and act on.
We are constantly working to improve Spearhead. Currently, interacting with Spearhead requires some programming or technical knowledge to use the REST API directly, or by using the interactive /docs endpoint that FastAPI provides. While powerful, our future plans include building a dedicated, user-friendly frontend. This will provide a richer and more intuitive interface for our analysts, making Spearhead more accessible to non-programmers and further streamlining the hunting and triage process.
Another key feature we plan to implement is a notification system for completed hunts. Currently, an analyst needs to periodically poll the status of a hunt. To eliminate this manual step, we will be adding a mechanism to notify analysts when a hunt is complete. While implementing a notification system is not technically difficult, we need to carefully consider the best approach that fits our workflow, tooling, and security needs.
Spearhead is a prime example of how we at Hunt & Hackett leverage automation to improve the efficiency and effectiveness of our security operations. We've taken a lengthy, manual, and error-prone process that involved at least five steps per customer and automated it into a simple, three-step workflow powered by a REST API. An analyst can now initiate a hunt across all customers with a single API call, monitor the hunt with a second call, and retrieve the results with a third. By automating the tedious aspects of IOC hunting and triage across multiple single-tenant environments, we have freed up our analysts to focus on high-value tasks, reduced the risk of human error, and created a more scalable and consistent security operations workflow. This also standardizes hunt execution, making results consistent and audit-ready across the fleet.
When a retrohunt orchestrated by Spearhead surfaces a potential threat, it doesn’t stop at a technical finding. It initiates our response workflow: a case is created in our Security Operations Center (SOC), where an analyst validates the signal, assesses impact and scope, and determines the required next steps. From there, we work directly with the customer to contain and remediate, guiding actions through to closure and verifying that the intrusion is fully resolved, not just “acknowledged.”