Optimizing System Performance: How Operating Curves Drive Better Decisions
Balancing competing metrics is a critical challenge in system optimization, especially when striving to improve performance. At Abnormal, we often face the task of making trade-offs between key goals—such as maximizing detection rates while minimizing false positives.
This blog explores how visualizing these trade-offs in a 2D plane using curves and surfaces can provide clearer insights. Rather than simply comparing isolated points, we’ll show how operating curves help us assess and optimize systems—whether we’re working with a single model or combining multiple subsystems.
Let’s dive into the mechanics of operating curves and how they guide better decision-making.
Understanding Competing Metrics in System Performance
At Abnormal, like most software companies, we measure the performance of our systems using key metrics. In many cases, two important metrics will compete with each other.
For example, in our detection systems, we want to detect as many attacks as possible (true positives) while minimizing the number of benign examples misclassified as attacks (false positives). One way to visualize this is by thinking of competing metrics in a two-dimensional plane, as in the diagram below:
To make things easier, we will call one metric the “revenue” metric and the other one the “cost” metric. Sometimes revenue and cost will be monetary but not necessarily. For instance, in the case of a detection system such as ours, “revenue” might be the rate of attacks correctly identified, while “cost” could be the number of false positives generated by the system.
Note: For simplicity, we focus on two metrics here, but in reality, there might be three or more. The same principles will apply broadly, as you can always examine pairs of metrics individually to return to the simpler case.
Now, let’s add a few more things to our graph.
The green region is a “no-brainer” improvement; if we could move there, we would. No one would turn down higher revenue at a lower cost. Conversely, moving to the red region is clearly not a good idea because both metrics get worse.
That leaves us with the “?” regions. In these areas, improving one metric comes at the expense of the other. Therefore, deciding whether to move there requires more careful consideration.
Introducing Operating Curves
To be clear, it is not entirely accurate to represent the system as a single point on the 2D plane. Most systems actually allow us to control the trade-off between the two metrics using parameters (think of them as “levers” we can adjust).
For example, when we use a machine learning model to identify attacks, model scores above a certain threshold might be classified as attacks. The threshold acts as a parameter that controls the trade-off between recall (“revenue”) and precision (“cost”).
With this in mind, let’s update our graph:
By adjusting the value of the parameter, we create a parametric curve that shows the range of possible operating points for the system.
Note: In practice, there might be several hyper-parameters, which would lead to parametric surfaces rather than curves. Here we keep things simple and only consider one.
Comparing Operating Curves
We have seen how a system can be represented as a curve in the 2D plane. This framework becomes especially useful when comparing multiple curves with each other.
For instance, suppose we’re contemplating replacing Model A with a new Model B. Let’s say we have the following numbers:
At first glance, it’s difficult to determine whether Model B is better than Model A. Cost is higher, but revenue is also higher. However, if we were to draw operating curves for both models, it becomes much easier to see that B is better than A:
The general intuition is that the curve for Model B lies “above” the curve for Model A. Formally, for any point on the blue curve, there exists a corresponding point on the orange curve with both lower cost and higher revenue.
Another way to understand this is that by adjusting the parameter for Model B, we can either increase revenue without raising costs or reduce costs without sacrificing revenue (moving along the curve). This can be very useful as it is easier to compare one metric when the other is kept constant. For example, we could say that Model B improves revenue by X% at the same cost as Model A.
In practice, where we want to position ourselves on these curves will depend on business requirements. We might be more interested in reducing costs than increasing revenue, or vice versa.
So far, we’ve looked at fairly “regular” curves. In reality, however, we might face more complex scenarios. Let’s review three examples:
Crossing Curves: In the first scenario (left), the curves intersect. This is not necessarily a problem; it just means Model A is better when operating at lower costs, while Model B is better when operating at higher costs. The choice between the two will therefore depend on what cost/revenue we are trying to achieve. In the next section, we’ll explore how we can combine curves to maximize both.
Better Revenue with Slightly Higher Cost: In the second scenario above (middle), we see that at the current operating points, Model A delivers more revenue at the same cost. However, due to the shape of the curves, by accepting a slightly higher cost, Model B could provide significantly more revenue. Thus, Model B might be a good option if that’s acceptable. This shows how powerful it can be to look at curves rather than single points. We would likely discard Model B fairly quickly if we only looked at the two points.
Limitations in Achieving Goals: In some cases, it might not be possible to achieve a given cost or revenue. This is illustrated by the third scenario (right). While Model B is clearly better than Model A, because we are working with parametric curves, we are limited in the range of cost/revenue that we can explore. In this scenario, model B doesn’t have the potential to reach higher levels of revenue; thus, if we had a goal to achieve high revenue, we might still need Model A.
Operating Curves and Marginal Returns
What if, rather than choosing between Model A and Model B, we wanted to use both? After all, our system might consist of several subsystems, each detecting attacks (or whatever we might be interested in).
When working with cost/revenue metrics, we assume that, when using both models, the total cost (and revenue) is the sum of the costs (and revenues) of each model. This is quite a reasonable assumption in most cases.
Let’s explore this visually with an example:
In the example above, we show the curves for Model A and Model B, along with their operating points. When considering the entire system, we simply sum the cost and revenue from each model to obtain a new point.
Interestingly, with two free parameters to adjust (one for each model), the result is not just a parametric curve but a parametric surface This surface is visualized as the purple-shaded area in the graph.
Recall that in our 2D plane, one curve is considered better than another if it is located “above” (indicating higher revenue at lower cost). This means that the purple-shaded area can be reduced to a curve if we remove suboptimal points. Any point that isn’t on the upper edge is suboptimal because we can easily pick another point within the surface that has higher cost and revenue, as illustrated below:
Now, one might naturally wonder how to obtain points on the optimal curve shown above, given that we still have two free parameters.
It is relatively straightforward to prove (left as an exercise to the reader 😉) that points on the purple curve correspond to pairs of points for Model A and Model B that have equal slopes. The intuition is that if the slopes were not equal, then we could “transfer” $1 of cost from the model with the lower slope to the one with the higher slope, resulting in a net increase in revenue.
As a picture is worth a thousand words, let’s look one last time at our 2D plane to see this phenomenon in action:
Indeed, the optimal curve for the whole system is found when we ensure the two points have the same slope. Formally, we say that both models have equal marginal returns. The word “marginal” is crucial here, as the average returns are not equal. In fact, here Model B delivers more than twice the return of Model A on average, but their marginal returns are equal.
This is because the shapes of the two curves are quite different. A common mistake is to think that Model A is “underperforming” compared to Model B, and therefore we should allocate more "budget" to Model B. However, as the graph clearly shows, this would lead to suboptimal returns overall.
Note: In this section, we made a few assumptions to keep things simple. The most important one is the curves exhibit diminishing marginal returns—i.e., the slope of the curves decreases as cost increases. This is true in many cases, but not always.
Achieving Optimal Performance Through Operating Curves
We hope this has been a helpful introduction to thinking of systems in the 2D plane using curves and surfaces, rather than looking at points in isolation. By visualizing the relationships between competing metrics, we can better understand the trade-offs at play.
When several subsystems contribute to the overall performance, we can find an optimal solution by adding a constraint that all subsystems have equal marginal returns. This approach ensures more efficient resource allocation—ultimately leading to better decision-making and improved system performance.
As a fast-growing company, we have lots of interesting engineering challenges to solve, just like this one. If these challenges interest you, and you want to further your growth as an engineer, we’re hiring! Learn more at our careers website.