chat
expand_more

Optimizing System Performance: How Operating Curves Drive Better Decisions

Explore how operating curves help optimize system performance by visualizing competing metrics, making trade-offs, and achieving efficient resource allocation.
January 17, 2025

Balancing competing metrics is a critical challenge in system optimization, especially when striving to improve performance. At Abnormal, we often face the task of making trade-offs between key goals—such as maximizing detection rates while minimizing false positives.

This blog explores how visualizing these trade-offs in a 2D plane using curves and surfaces can provide clearer insights. Rather than simply comparing isolated points, we’ll show how operating curves help us assess and optimize systems—whether we’re working with a single model or combining multiple subsystems.

Let’s dive into the mechanics of operating curves and how they guide better decision-making.

Understanding Competing Metrics in System Performance

At Abnormal, like most software companies, we measure the performance of our systems using key metrics. In many cases, two important metrics will compete with each other.

For example, in our detection systems, we want to detect as many attacks as possible (true positives) while minimizing the number of benign examples misclassified as attacks (false positives). One way to visualize this is by thinking of competing metrics in a two-dimensional plane, as in the diagram below:

Operating Curves 1 2 D Model of System

A system can be seen as a point in a 2D space with a “cost” and a “revenue” metric

To make things easier, we will call one metric the “revenue” metric and the other one the “cost” metric. Sometimes revenue and cost will be monetary but not necessarily. For instance, in the case of a detection system such as ours, “revenue” might be the rate of attacks correctly identified, while “cost” could be the number of false positives generated by the system.

Note: For simplicity, we focus on two metrics here, but in reality, there might be three or more. The same principles will apply broadly, as you can always examine pairs of metrics individually to return to the simpler case.

Now, let’s add a few more things to our graph.

Operating Curves 2 2 D Model of System v2

The green region is a “no-brainer” improvement; if we could move there, we would. No one would turn down higher revenue at a lower cost. Conversely, moving to the red region is clearly not a good idea because both metrics get worse.

That leaves us with the “?” regions. In these areas, improving one metric comes at the expense of the other. Therefore, deciding whether to move there requires more careful consideration.

Introducing Operating Curves

To be clear, it is not entirely accurate to represent the system as a single point on the 2D plane. Most systems actually allow us to control the trade-off between the two metrics using parameters (think of them as “levers” we can adjust).

For example, when we use a machine learning model to identify attacks, model scores above a certain threshold might be classified as attacks. The threshold acts as a parameter that controls the trade-off between recall (“revenue”) and precision (“cost”).

With this in mind, let’s update our graph:

Operating Curves 3 Parametric Curve

The operating point is now merely one point on a parametric curve, corresponding to one specific value of the parameter

By adjusting the value of the parameter, we create a parametric curve that shows the range of possible operating points for the system.

Note: In practice, there might be several hyper-parameters, which would lead to parametric surfaces rather than curves. Here we keep things simple and only consider one.

Comparing Operating Curves

We have seen how a system can be represented as a curve in the 2D plane. This framework becomes especially useful when comparing multiple curves with each other.

For instance, suppose we’re contemplating replacing Model A with a new Model B. Let’s say we have the following numbers:

Operating Curves 4 Table

At first glance, it’s difficult to determine whether Model B is better than Model A. Cost is higher, but revenue is also higher. However, if we were to draw operating curves for both models, it becomes much easier to see that B is better than A:

Operating Curves 5 Comparison

The general intuition is that the curve for Model B lies “above” the curve for Model A. Formally, for any point on the blue curve, there exists a corresponding point on the orange curve with both lower cost and higher revenue.

Another way to understand this is that by adjusting the parameter for Model B, we can either increase revenue without raising costs or reduce costs without sacrificing revenue (moving along the curve). This can be very useful as it is easier to compare one metric when the other is kept constant. For example, we could say that Model B improves revenue by X% at the same cost as Model A.

In practice, where we want to position ourselves on these curves will depend on business requirements. We might be more interested in reducing costs than increasing revenue, or vice versa.

So far, we’ve looked at fairly “regular” curves. In reality, however, we might face more complex scenarios. Let’s review three examples:

Operating Curves 6 Less Regular Curves

Three examples of less regular curves

  1. Crossing Curves: In the first scenario (left), the curves intersect. This is not necessarily a problem; it just means Model A is better when operating at lower costs, while Model B is better when operating at higher costs. The choice between the two will therefore depend on what cost/revenue we are trying to achieve. In the next section, we’ll explore how we can combine curves to maximize both.

  2. Better Revenue with Slightly Higher Cost: In the second scenario above (middle), we see that at the current operating points, Model A delivers more revenue at the same cost. However, due to the shape of the curves, by accepting a slightly higher cost, Model B could provide significantly more revenue. Thus, Model B might be a good option if that’s acceptable. This shows how powerful it can be to look at curves rather than single points. We would likely discard Model B fairly quickly if we only looked at the two points.

  3. Limitations in Achieving Goals: In some cases, it might not be possible to achieve a given cost or revenue. This is illustrated by the third scenario (right). While Model B is clearly better than Model A, because we are working with parametric curves, we are limited in the range of cost/revenue that we can explore. In this scenario, model B doesn’t have the potential to reach higher levels of revenue; thus, if we had a goal to achieve high revenue, we might still need Model A.

Operating Curves and Marginal Returns

What if, rather than choosing between Model A and Model B, we wanted to use both? After all, our system might consist of several subsystems, each detecting attacks (or whatever we might be interested in).

When working with cost/revenue metrics, we assume that, when using both models, the total cost (and revenue) is the sum of the costs (and revenues) of each model. This is quite a reasonable assumption in most cases.

Let’s explore this visually with an example:

Operating Curves 7 Operating Surface

Combining the operating curves of two subsystems results in a single “operating surface” for the whole system. Any point within the purple-shaded area can be reached by moving the subsystems along their respective curves.

In the example above, we show the curves for Model A and Model B, along with their operating points. When considering the entire system, we simply sum the cost and revenue from each model to obtain a new point.

Interestingly, with two free parameters to adjust (one for each model), the result is not just a parametric curve but a parametric surface This surface is visualized as the purple-shaded area in the graph.

Recall that in our 2D plane, one curve is considered better than another if it is located “above” (indicating higher revenue at lower cost). This means that the purple-shaded area can be reduced to a curve if we remove suboptimal points. Any point that isn’t on the upper edge is suboptimal because we can easily pick another point within the surface that has higher cost and revenue, as illustrated below:

Operating Curves 8 Trivial Optimality

With a constraint of trivial optimality, the purple surface reduces to a curve.

Now, one might naturally wonder how to obtain points on the optimal curve shown above, given that we still have two free parameters.

It is relatively straightforward to prove (left as an exercise to the reader 😉) that points on the purple curve correspond to pairs of points for Model A and Model B that have equal slopes. The intuition is that if the slopes were not equal, then we could “transfer” $1 of cost from the model with the lower slope to the one with the higher slope, resulting in a net increase in revenue.

As a picture is worth a thousand words, let’s look one last time at our 2D plane to see this phenomenon in action:

Operating Curves 9 System Optimality

Optimality for the whole system is reached by equalizing the marginal returns (slopes) of the subsystems

Indeed, the optimal curve for the whole system is found when we ensure the two points have the same slope. Formally, we say that both models have equal marginal returns. The word “marginal” is crucial here, as the average returns are not equal. In fact, here Model B delivers more than twice the return of Model A on average, but their marginal returns are equal.

This is because the shapes of the two curves are quite different. A common mistake is to think that Model A is “underperforming” compared to Model B, and therefore we should allocate more "budget" to Model B. However, as the graph clearly shows, this would lead to suboptimal returns overall.

Note: In this section, we made a few assumptions to keep things simple. The most important one is the curves exhibit diminishing marginal returns—i.e., the slope of the curves decreases as cost increases. This is true in many cases, but not always.

Achieving Optimal Performance Through Operating Curves

We hope this has been a helpful introduction to thinking of systems in the 2D plane using curves and surfaces, rather than looking at points in isolation. By visualizing the relationships between competing metrics, we can better understand the trade-offs at play.

When several subsystems contribute to the overall performance, we can find an optimal solution by adding a constraint that all subsystems have equal marginal returns. This approach ensures more efficient resource allocation—ultimately leading to better decision-making and improved system performance.

As a fast-growing company, we have lots of interesting engineering challenges to solve, just like this one. If these challenges interest you, and you want to further your growth as an engineer, we’re hiring! Learn more at our careers website.

Optimizing System Performance: How Operating Curves Drive Better Decisions

See Abnormal in Action

Get a Demo

Get the Latest Email Security Insights

Subscribe to our newsletter to receive updates on the latest attacks and new trends in the email threat landscape.

Get AI Protection for Your Human Interactions

Protect your organization from socially-engineered email attacks that target human behavior.
Request a Demo
Request a Demo

Related Posts

B Proofpoing Customer Story 10
Learn how a multinational travel center services provider blocked 1,180+ attacks missed by Proofpoint and reclaimed 450+ SOC hours per month by adding Abnormal.
Read More
B Operating Curves Blog
Explore how operating curves help optimize system performance by visualizing competing metrics, making trade-offs, and achieving efficient resource allocation.
Read More
B SOC Traits
Discover the traits and mindsets that define top SOC analysts, as explored in Season 1 of SOC Unlocked.
Read More
B Punycode Problem Blog
Explore how threat actors exploit Punycode in email attacks and learn how AI-driven solutions can protect against these threats.
Read More
B Product24
Discover how Abnormal transformed 2024 with groundbreaking AI innovations, enhanced cloud and email security solutions, and industry leadership, tackling evolving cyber threats while empowering organizations worldwide to stay secure.
Read More
B Top Reason to Attend Innovate 2025 Blog
Don’t miss your chance to hear from leading AI experts, Fortune 500 CISOs, threat intelligence leaders, and other exciting speakers at Abnormal Innovate.
Read More