Misclassification Adaptation in Cyberattack Detection

At Abnormal Security, we protect customers against cyberattacks hiding among billions of legitimate business emails. This requires a detection engine that adapts quickly to new attack methods while maintaining accuracy and explainability. Mistakes in this system can have serious consequences.

A key challenge in cyberattack detection is the issue of misclassification, where a legitimate message is mistakenly flagged as malicious or a threat slips through undetected. To address this, we use a structured, multi-layered system designed to evolve with emerging threats.

This blog explores how each of these layers functions, how they interact to minimize misclassifications, and how we maintain both security and usability for our customers.

Misclassifications typically fall into two categories. A false negative (FN) is a missed attack. These occur when threat actors discover new ways to bypass defenses. Once successful, attackers often reuse strategies at scale, making it critical to close these gaps quickly.

The other type of misclassification is a false positive (FP), where legitimate messages are blocked because they resemble attacks. For example, if our system flags Dropbox links in spoofed emails as malicious, but a new customer regularly uses similar links in their business, our system must adapt immediately to avoid disrupting their operations and impacting their experience.

Our detection system balances performance and adaptability through three layers:

• Signal layer: Enriches email data with features derived from API calls and database lookups.
• Model layer: Uses a neural network to classify messages based on features.
• Decision layer: Applies a rule engine over model scores and features to make a final decision.

Each layer offers unique strategies for addressing FNs and FPs.

Decision Layer

Manually overriding model decisions via pattern-specific blocklists and safelists is the simplest intervention approach. Although this method is interpretable and easy to edit, it can create technical debt over time, especially once we introduce automation. We therefore use this layer only as a last resort.

Model Layer

Retraining or fine-tuning our core machine learning models with new or customer-specific data helps improve overall performance. However, this approach is slow and insufficient for adapting to rare FN/FP trends. Even with more data, results can be unpredictable.

To mitigate this, we use an iterative process that integrates both the decision and model layers for a fast and sustainable response:

• Observe a misclassification.
• Modify the decision layer to adapt to this misclassification.
• The modified decision layer generates substantial training data for our core machine learning models.
• Retrain the core machine learning models with this new data to generate sustainable improvements.

Signal Layer

Signals can also adapt automatically to new patterns. For instance, we can design features that count how often messages matching a pattern appear in messages labeled “safe” or “attack” for a customer, sender, or recipient. When an FN or FP occurs, the signal adjusts, influencing the model’s next decision. This approach is fast, adaptable, and less disruptive than manual overrides since the model still makes the final call.

However, it is complex to implement. Decisions on aggregation keys (e.g., customer, sender) and patterns require explicit choices, and adding new ones involves retraining the model, which takes weeks.

By combining these approaches, we can respond effectively to evolving threats while maintaining customer trust.

Cyberattack detection is a constantly evolving challenge, requiring a balance between precision and adaptability. False positives and false negatives each pose distinct risks, demanding an approach that both reacts swiftly to threats and improves over time.

By combining decision-based interventions, machine learning refinements, and adaptive signals, we create a system that evolves alongside the tactics of cybercriminals. This multi-layered approach ensures that our customers remain protected while minimizing disruptions to legitimate business operations.

As a fast-growing company, we have lots of interesting engineering challenges to solve, just like this one. If these challenges interest you, and you want to further your growth as an engineer, we’re hiring! Learn more at our careers website.