> ## Documentation Index
> Fetch the complete documentation index at: https://lib.findy.co.jp/llms.txt
> Use this file to discover all available pages before exploring further.
# How to improve web performance: order and judgment
> The judgment that comes before individual techniques: why to improve web service performance, how fast is fast enough, and in what order to consider fixes.
## Overview
Performance is the property of how little time and how few resources a system needs to do the same work. This page covers how to improve it in web services, primarily application servers and databases.
There are countless improvement techniques, but before any of them comes a shared set of judgments: why, how far, and in what order. This page is about those judgments, not the individual techniques.
We start by articulating the goal, then move through classifying the problem, deciding how fast is fast enough, the order in which to consider fixes, and finally the principles of measurement and the basics of observability.
## Why make it faster
Before considering concrete techniques, articulate the goal. The goal of performance work is not speed itself; most goals come down to one of three things.
Fix problems users see directly: slow interactions and timeouts.
Relieve sustained high load and restore headroom against incidents.
Keep growing infrastructure costs under control.
Decide first which goal the improvement serves. As described later, the goal determines which metrics to watch and which fixes make sense.
## Classify the problem into three types
The problems addressed in performance work fall into three broad types. The symptoms determine both the metrics to watch and the direction of the fix, so identify which one you are solving first.
| Type | Symptom | Typical direction of the fix |
| ------------- | --------------------------------------------------------- | ------------------------------------------- |
| Latency | A specific operation is slow or times out | Find and remove the single dominant point |
| Throughput | Work does not finish within its window (batch jobs, etc.) | Parallelize, batch, reduce the work |
| Resource load | Resource utilization stays high; costs keep rising | Find and reduce the top recurring consumers |
Latency and throughput are visible from the outside: users or business operations are kept waiting. Resource load, by contrast, erodes stability headroom and cost without anyone noticing, and left alone it eventually surfaces as one of the first two problems — or as an outage.
Latency is usually quantified not with the average response time but with percentiles: the speed of the "nth percent" request when all requests are sorted from fastest to slowest.
* **p50 (50th percentile, the median)** — the speed at the 50% mark from the fastest. The experience of a typical user
* **p95 / p99 (tail latency)** — the speed at the 95% and 99% marks. Read them as "5% (or 1%) of users are still having a slower experience than this"
Percentiles let you talk concretely about the bad side of the experience: "1 user in 100 is kept waiting more than 2 seconds (p99 = 2,000ms)." How much of that to tolerate differs by product and situation, and drawing that line is the goal setting covered in the next section. What you monitor is the trend of p95 and p99.
The average is avoided because it hides a minority of extreme delays. In an API where 1 request in 10 takes 1 second and the other 9 return almost instantly, the average is about 100ms — yet not a single user experienced that speed. The maximum, on the other hand, is at the mercy of rare outliers. Percentiles cut by rank, providing a stable reference point between an average that dilutes the few slow requests and a maximum that swings on outliers.
## How fast is fast enough
With the vocabulary of quantification, you can also set targets. Performance tuning has no natural end, and cutting 100ms to 10ms costs far more than cutting 200ms to 100ms. That is exactly why you decide where to stop first.
Agree on target values as SLOs (Service Level Objectives) in advance: "this API is a satisfying experience if p95 stays within 300ms," "this batch is fine if it finishes within an hour and does not disrupt the next morning's work."
When the target is met, the tuning has succeeded. Beyond that point the return on investment declines, and improvement without a stopping point keeps buying complexity that damages [modifiability](/development/modifiability).
Agreed targets also serve as pass criteria for load testing. Before launch when there is no real traffic yet, or ahead of a major change or an event expected to bring more traffic, you can apply load and confirm the headroom against the target.
## The hierarchy of fixes
Once the problem and the target are set, choose a fix. Fixes form a hierarchy by their balance of effect and cost. The default is to consider them from the top.
1. **Reduce the requirement** — Revisit the spec. Relax precision or completeness that nobody actually needs, so the heavy computation itself becomes unnecessary (detailed below).
2. **Reduce the work** — Delete unnecessary processing, thin out excessive frequency (polling, recomputation), eliminate N+1 queries, improve data structures and algorithms. This is waste removal: it takes implementation effort, but it makes the system faster without giving up any of its properties, such as functionality or consistency. Real systems almost always have room left here.
3. **Remember the work** — Caching and pre-aggregation. Store a computed result and just read it; one computation is reused many times, so the total amount of work itself goes down. Highly effective, but you take on a consistency debt (managing invalidation and recomputation), and missed invalidations are hard bugs to find.
4. **Shift the work** — Make it asynchronous, spread it across time. Move heavy processing out of the request and run it later, and the response can return without waiting for it to finish. Shift it to quiet hours, such as overnight, and peak load goes down too. Note that only the "when" changes; the total amount of work stays the same.
5. **Add resources** — Scale up, scale out. Adding capacity deliberately as capacity planning, in response to demand growth, is healthy. Weighing an engineer's time against stability available right now, temporarily scaling up to buy time can be the right call. But beware: some bottlenecks — N+1 round trips, lock contention, algorithms with heavy computational complexity — are not solved by adding resources, and resources hide the cause while the fixed costs keep growing.
### The option to revisit the spec: the "90 percent solution"
When the bottleneck has been identified, pause before asking how to solve it technically, and ask: does this heavy computation really need to keep running under a perfect spec?
*The UNIX Philosophy* offers the guideline of aiming for the "90 percent solution": chasing perfection makes complexity and processing costs jump, while a simple solution that meets nine-tenths of the requirement improves efficiency dramatically. It is not rare for the pressure on performance to come from a spec whose strictness neither the user experience nor the business actually demands.
| Solving it with technology | Solving it by revisiting the spec (the 90 percent solution) |
| -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `COUNT(*)` on a huge table is slow, so build and synchronize a hand-rolled counter | Give up exact real-time counts: display an approximation like "1,000+", or accept a cache delayed by a few minutes |
| Opening the dashboard joins and aggregates the entire history with one giant SQL query | Limit the default view to the last 30 days, with older data behind an explicit date-range search |
| Every timeline render recomputes a complex algorithm in real time | Precompute it every few minutes with an asynchronous background worker |
Making the heavy computation unnecessary beats making the program run faster: the return on investment is higher and the implementation stays simpler. Before starting on an optimization, discuss with the people who decide the spec whether it can be relaxed a little. This is an option that produces the largest results for the smallest effort.
### What you trade away
When the spec cannot be relaxed, or relaxing it is not enough, you gain speed by trading something away. What this trade-off costs differs by fix.
* Caching ↔ consistency (can the screen show a value a few minutes old?)
* Pre-aggregation ↔ flexibility (aggregation axes become fixed; incompatible with free-form conditions)
* Denormalization ↔ update cost and inconsistency risk
* Going asynchronous ↔ immediacy (can completion come later? Retry and recovery on failure also need designing)
* Scaling up or out ↔ continuously accruing infrastructure cost
Choose by which price the spec can afford to pay. Also note that a price can arise unintentionally: even what was meant as waste removal, if it ends up needing a convoluted implementation, has traded away readability and modifiability.
## Premature optimization and hard-to-reverse decisions
The flow so far has been: understand the problem and the target, then fix in order. The flip side is not writing optimization code before the need is demonstrated. Code can be fixed later, so write it plainly first; touching it after measurement shows the need is soon enough.
Hard-to-reverse decisions, however, are the exception. How data is stored (table design, data granularity) and the shape of externally published APIs (whether there is pagination, whether bulk fetching is possible) cost incomparably more to fix after dependencies have piled up, so think about performance from the start. Concretely, ask at design time: "does this design still hold when the data grows 10x or 100x?"
## Start with measurement, not guesses
Performance optimization has a basic property: speeding up anything other than the bottleneck barely changes the end-to-end time. When a database query occupies 8 seconds of a 10-second flow, tuning a 1-second loop to its limit saves at most 1 second. Halving the 8-second query, on the other hand, brings the total to 6 seconds by itself.
Locating the bottleneck objectively, with data, is the first and most important step. Once you start fixing, change one thing at a time and re-measure. With several simultaneous changes, you can no longer tell what worked.
Removing one bottleneck also brings the next one into view somewhere else: locating and removing bottlenecks is not a one-shot task but a cycle repeated with measurement until the target is met.
## Observability basics
Executing "measurement, not guesses" requires the ability to observe the system's internal state. Observability is built from three kinds of data used together.
| Data | What it is | Role |
| ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| Metrics | Numbers aggregated over time: CPU utilization, memory, request counts, latency (p95/p99), database connections | Knowing the normal baseline, detecting changes and trends, before/after release comparisons, input data for capacity planning, the basis for alerts |
| Traces | The steps a request passed through before returning a response (application code, external API calls, database queries, etc.) and the time spent in each | Pinpointing the bottleneck: "which database query accounts for 1.8 of this request's 2 seconds" |
| Logs | Records of events and errors at specific times, with context (user IDs, parameters) | Digging into the located anomaly: under which conditions and parameters the delay or error occurred |
Raise the resolution of observation in that order: metrics for the overall trend, traces to locate the path, logs for the details.
For collecting and exporting these three kinds of data (telemetry), OpenTelemetry, driven by the CNCF, has become the industry standard. Each APM vendor once required its own agent and vendor lock-in was a real cost, but instrument with the standard API/SDK and you can switch or combine backends (Datadog and New Relic, each cloud's standard services, OSS options such as Jaeger, Prometheus, and the Grafana stack).
Even in an environment where none of this is in place yet, you can start observing with minimal means: write response times into the access log, and enable the database's slow query log.
### APM and profilers
APM (Application Performance Monitoring) and profilers are what make bottleneck identification efficient with the collected data.
**APM** — A tool centered on collecting and visualizing traces. Many products also show aggregates derived per endpoint, such as latency (p95, etc.) and error rates. Suited to judging where the bottleneck is within a whole web request (N+1 query bursts, waits on external APIs, and so on).
**Profilers** — Track code-level CPU time and memory allocation. Suited to evaluating efficiency inside the code: which function is consuming the CPU. Every language has standard tools.
In practice, the efficient investigation is hierarchical: first narrow down with APM (traces) to the responsible component or database query, then move to code-level profilers or log analysis as needed.
Below the application there is also a world of OS- and kernel-level profilers (Linux's `perf`, eBPF-based tools, and so on). It is beyond the scope of this page, but *Systems Performance* covers digging to that depth in detail.
### Using the CPU, or waiting
When analyzing observation data, the first property to identify is whether the delay comes from the CPU being saturated or from waiting on something.
| Kind | Typical causes | Observable signature | Direction of the fix |
| --------- | --------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------ |
| CPU-bound | Heavy computation, encryption, wasteful loops, JSON serialization | CPU utilization pinned near 100% | Improve algorithms, parallelize, scale up |
| Waiting | Waiting on database queries or external APIs, disk and network I/O, lock contention, connection pool exhaustion | Response time is long while CPU utilization stays low | Improve queries, add indexes, cache, go asynchronous, resolve contention |
If the service is slow while the CPU sits idle, it is waiting on something. Most often that is I/O such as the database or the network, but it can also be jammed on lock contention. This single distinction avoids wasted trial and error and moves you toward the cause.
### Where to start looking
Where to start is decided by the problem type identified earlier. For latency problems — a specific operation is slow — narrow down in this order, raising the resolution as you go.
Use APM traces to find where the request spends its time. The single dominant point usually shows up here.
If a query is the main cause, check database-wide statistics for how much load that query accounts for and what it is waiting on.
Use EXPLAIN to determine from the execution plan why the query is slow.
For resource-load problems — everything gradually heavier — hunt the top recurring consumers, starting from resource saturation and database-wide statistics. For throughput problems — a batch that does not finish — apply the same narrowing to the whole job: find the dominant span in the job's time breakdown, then descend into individual queries and processing.
## Related pages
How to speed up the database, the most frequent bottleneck, starting from one principle: minimize storage reads.
The property of accepting change at low cost and low risk. This is what runaway optimization damages.
Performance improvement is a behavior change users notice. The discipline of not mixing it with structure changes in the same pull request.