How to improve web performance: order and judgment

​

Overview

Performance is the property of how little time and how few resources a system needs to do the same work. This page covers how to improve it in web services, primarily application servers and databases. There are countless improvement techniques, but before any of them comes a shared set of judgments: why, how far, and in what order. This page is about those judgments, not the individual techniques. We start by articulating the goal, then move through classifying the problem, deciding how fast is fast enough, the order in which to consider fixes, and finally the principles of measurement and the basics of observability.

​

Why make it faster

Before considering concrete techniques, articulate the goal. The goal of performance work is not speed itself; most goals come down to one of three things. Decide first which goal the improvement serves. As described later, the goal determines which metrics to watch and which fixes make sense.

​

Classify the problem into three types

The problems addressed in performance work fall into three broad types. The symptoms determine both the metrics to watch and the direction of the fix, so identify which one you are solving first. Latency and throughput are visible from the outside: users or business operations are kept waiting. Resource load, by contrast, erodes stability headroom and cost without anyone noticing, and left alone it eventually surfaces as one of the first two problems — or as an outage. Latency is usually quantified not with the average response time but with percentiles: the speed of the “nth percent” request when all requests are sorted from fastest to slowest.

• p50 (50th percentile, the median) — the speed at the 50% mark from the fastest. The experience of a typical user
• p95 / p99 (tail latency) — the speed at the 95% and 99% marks. Read them as “5% (or 1%) of users are still having a slower experience than this”

Percentiles let you talk concretely about the bad side of the experience: “1 user in 100 is kept waiting more than 2 seconds (p99 = 2,000ms).” How much of that to tolerate differs by product and situation, and drawing that line is the goal setting covered in the next section. What you monitor is the trend of p95 and p99.

​

How fast is fast enough

With the vocabulary of quantification, you can also set targets. Performance tuning has no natural end, and cutting 100ms to 10ms costs far more than cutting 200ms to 100ms. That is exactly why you decide where to stop first. Agree on target values as SLOs (Service Level Objectives) in advance: “this API is a satisfying experience if p95 stays within 300ms,” “this batch is fine if it finishes within an hour and does not disrupt the next morning’s work.” When the target is met, the tuning has succeeded. Beyond that point the return on investment declines, and improvement without a stopping point keeps buying complexity that damages modifiability.

​

The hierarchy of fixes

Once the problem and the target are set, choose a fix. Fixes form a hierarchy by their balance of effect and cost. The default is to consider them from the top.

1. Reduce the requirement — Revisit the spec. Relax precision or completeness that nobody actually needs, so the heavy computation itself becomes unnecessary (detailed below).
2. Reduce the work — Delete unnecessary processing, thin out excessive frequency (polling, recomputation), eliminate N+1 queries, improve data structures and algorithms. This is waste removal: it takes implementation effort, but it makes the system faster without giving up any of its properties, such as functionality or consistency. Real systems almost always have room left here.
3. Remember the work — Caching and pre-aggregation. Store a computed result and just read it; one computation is reused many times, so the total amount of work itself goes down. Highly effective, but you take on a consistency debt (managing invalidation and recomputation), and missed invalidations are hard bugs to find.
4. Shift the work — Make it asynchronous, spread it across time. Move heavy processing out of the request and run it later, and the response can return without waiting for it to finish. Shift it to quiet hours, such as overnight, and peak load goes down too. Note that only the “when” changes; the total amount of work stays the same.
5. Add resources — Scale up, scale out. Adding capacity deliberately as capacity planning, in response to demand growth, is healthy. Weighing an engineer’s time against stability available right now, temporarily scaling up to buy time can be the right call. But beware: some bottlenecks — N+1 round trips, lock contention, algorithms with heavy computational complexity — are not solved by adding resources, and resources hide the cause while the fixed costs keep growing.

​

The option to revisit the spec: the “90 percent solution”

When the bottleneck has been identified, pause before asking how to solve it technically, and ask: does this heavy computation really need to keep running under a perfect spec? The UNIX Philosophy offers the guideline of aiming for the “90 percent solution”: chasing perfection makes complexity and processing costs jump, while a simple solution that meets nine-tenths of the requirement improves efficiency dramatically. It is not rare for the pressure on performance to come from a spec whose strictness neither the user experience nor the business actually demands. Making the heavy computation unnecessary beats making the program run faster: the return on investment is higher and the implementation stays simpler. Before starting on an optimization, discuss with the people who decide the spec whether it can be relaxed a little. This is an option that produces the largest results for the smallest effort.

​

What you trade away

When the spec cannot be relaxed, or relaxing it is not enough, you gain speed by trading something away. What this trade-off costs differs by fix.

• Caching ↔ consistency (can the screen show a value a few minutes old?)
• Pre-aggregation ↔ flexibility (aggregation axes become fixed; incompatible with free-form conditions)
• Denormalization ↔ update cost and inconsistency risk
• Going asynchronous ↔ immediacy (can completion come later? Retry and recovery on failure also need designing)
• Scaling up or out ↔ continuously accruing infrastructure cost

Choose by which price the spec can afford to pay. Also note that a price can arise unintentionally: even what was meant as waste removal, if it ends up needing a convoluted implementation, has traded away readability and modifiability.

​

Premature optimization and hard-to-reverse decisions

The flow so far has been: understand the problem and the target, then fix in order. The flip side is not writing optimization code before the need is demonstrated. Code can be fixed later, so write it plainly first; touching it after measurement shows the need is soon enough. Hard-to-reverse decisions, however, are the exception. How data is stored (table design, data granularity) and the shape of externally published APIs (whether there is pagination, whether bulk fetching is possible) cost incomparably more to fix after dependencies have piled up, so think about performance from the start. Concretely, ask at design time: “does this design still hold when the data grows 10x or 100x?”

​

Start with measurement, not guesses

Performance optimization has a basic property: speeding up anything other than the bottleneck barely changes the end-to-end time. When a database query occupies 8 seconds of a 10-second flow, tuning a 1-second loop to its limit saves at most 1 second. Halving the 8-second query, on the other hand, brings the total to 6 seconds by itself. Locating the bottleneck objectively, with data, is the first and most important step. Once you start fixing, change one thing at a time and re-measure. With several simultaneous changes, you can no longer tell what worked. Removing one bottleneck also brings the next one into view somewhere else: locating and removing bottlenecks is not a one-shot task but a cycle repeated with measurement until the target is met.

​

Observability basics

Executing “measurement, not guesses” requires the ability to observe the system’s internal state. Observability is built from three kinds of data used together. Raise the resolution of observation in that order: metrics for the overall trend, traces to locate the path, logs for the details. For collecting and exporting these three kinds of data (telemetry), OpenTelemetry, driven by the CNCF, has become the industry standard. Each APM vendor once required its own agent and vendor lock-in was a real cost, but instrument with the standard API/SDK and you can switch or combine backends (Datadog and New Relic, each cloud’s standard services, OSS options such as Jaeger, Prometheus, and the Grafana stack). Even in an environment where none of this is in place yet, you can start observing with minimal means: write response times into the access log, and enable the database’s slow query log.

​

APM and profilers

APM (Application Performance Monitoring) and profilers are what make bottleneck identification efficient with the collected data. APM — A tool centered on collecting and visualizing traces. Many products also show aggregates derived per endpoint, such as latency (p95, etc.) and error rates. Suited to judging where the bottleneck is within a whole web request (N+1 query bursts, waits on external APIs, and so on). Profilers — Track code-level CPU time and memory allocation. Suited to evaluating efficiency inside the code: which function is consuming the CPU. Every language has standard tools. In practice, the efficient investigation is hierarchical: first narrow down with APM (traces) to the responsible component or database query, then move to code-level profilers or log analysis as needed.

​

Using the CPU, or waiting

When analyzing observation data, the first property to identify is whether the delay comes from the CPU being saturated or from waiting on something. If the service is slow while the CPU sits idle, it is waiting on something. Most often that is I/O such as the database or the network, but it can also be jammed on lock contention. This single distinction avoids wasted trial and error and moves you toward the cause.

​

Where to start looking

Where to start is decided by the problem type identified earlier. For latency problems — a specific operation is slow — narrow down in this order, raising the resolution as you go. For resource-load problems — everything gradually heavier — hunt the top recurring consumers, starting from resource saturation and database-wide statistics. For throughput problems — a batch that does not finish — apply the same narrowing to the whole job: find the dominant span in the job’s time breakdown, then descend into individual queries and processing.

​

Related pages