One computer can only pretend to be so many users before it becomes the slowest part of the test.

The plain meaning

JMeter is a tool that imitates many users visiting a system so that you can measure how the system behaves under crowd pressure. Each imitation user is a thread, a small independent worker inside the program. A computer has a limited amount of processor power, memory and network capacity, so it can run only so many threads before it starts to struggle. Distributed testing means using several computers together, so the crowd you create is the sum of what all of them can produce.

Why a single machine eventually fails you

When the machine that generates load runs out of processor or memory, it slows down. Slow generation looks like slow response, because the clock keeps running while the machine is too busy to send the next request or read the last reply. The numbers then describe the weakness of your test computer, not the weakness of the system you meant to examine. Spreading the work across several machines keeps each of them comfortable, so the timings stay honest.

Other reasons to spread the load

Besides raw capacity, there is a second reason: the network. A single machine has a single network card and a single route to the system under test. Many real users come from many places and many routes. Several generators, possibly in different locations, can show whether the network path itself is a limit, and can imitate geographic spread more truthfully.

A handful of words carry the whole subject; learn them once and everything else follows.

The shape is a simple star: one centre, many arms, and one target.

Three groups of machines

A distributed setup has three groups. The first is the controller, a single machine used by the tester. The second is the group of agents, each running a waiting program. The third is the system under test, which receives traffic from the agents directly. This last point surprises many beginners: the load does not pass through the controller. The controller only sends instructions and receives reports. The traffic itself flows from each agent straight to the target.

flowchart TD
  C[Controller machine] -->|plan and commands| A1[Agent 1]
  C -->|plan and commands| A2[Agent 2]
  C -->|plan and commands| A3[Agent 3]
  A1 -->|results| C
  A2 -->|results| C
  A3 -->|results| C
  A1 ==>|test traffic| T[System under test]
  A2 ==>|test traffic| T
  A3 ==>|test traffic| T
        

Inside each machine

The controller holds the test plan file, the program interface or command line, and the listeners that show results. Each agent holds the same tool installation and a small background service that waits for orders. When an order arrives, the agent starts its own copy of the test engine, runs the threads, and streams sample results back.

The controller and the agents hold a short, structured conversation at the beginning, the middle and the end.

The opening

Each agent starts a background service that listens on a network port. When the tester starts a distributed run, the controller looks up the list of agents, connects to each one, and sends the entire test plan across the network. Each agent receives the plan, prepares its own engine, and then waits for the go signal. The controller sends that signal to every agent in quick succession so the crowd forms at nearly the same moment.

The middle

While the test runs, each agent follows the plan independently. It does not ask the controller for permission before each request. It simply works, and periodically sends batches of sample results back to the controller so the listeners there can show progress. If the controller disappears, the agents continue until their plan ends, though their results have nowhere to go.

The ending

When the plan finishes, or the tester presses stop, the controller tells the agents to stop their engines. Each agent ends its threads, sends any remaining results, and returns to waiting for the next order. Optionally the controller can also tell the background services to exit completely.

sequenceDiagram
  participant T as Tester
  participant C as Controller
  participant A as Agent
  participant S as System under test
  T->>C: Start distributed run
  C->>A: Connect and send test plan
  A-->>C: Plan received and ready
  C->>A: Start signal
  A->>S: Requests from virtual users
  S-->>A: Responses
  A-->>C: Batches of sample results
  T->>C: Stop or plan ends
  C->>A: Stop engine
  A-->>C: Final results
        

The most common beginner surprise is the size of the crowd that appears.

Every agent runs the whole plan

The controller does not divide the plan into slices. Each agent receives the complete plan and runs it in full. If the thread group says one hundred users, and you have five agents, the total crowd is five hundred users. Loop counts, durations and rate controls inside the plan are also applied on every agent separately.

Planning the numbers

To reach a target crowd, divide the target by the number of agents and put the result in the plan. If you need one thousand users across four agents, set the group to two hundred and fifty. If the agents differ in power, you may prefer a plan that reads the thread count from a property, so each agent can be given its own share at start time.

Extra machines add effort, so use them when the evidence calls for them.

Signs that a single generator is overloaded

Watch the generator while the test runs. If the processor stays near its ceiling for long stretches, if memory climbs towards its limit and garbage collection pauses grow, or if the network card approaches its bandwidth, the machine is stressed. Another sign is that response times rise as you add users on the generator but the system under test shows plenty of spare capacity in its own monitoring. A third sign is that the achieved request rate stops growing even though you keep adding threads.

Most distributed failures are caused by small mismatches between machines, not by the tool.

An agent is simply an installed copy of the tool with its waiting service switched on.

The steps in words

The controller needs to know who the agents are and how to talk to them.

The list of agents

The tool has a settings file with a property that holds a comma-separated list of agent addresses. Entering the addresses there lets the controller start every agent with one command or one menu click. You may also supply the list at the moment you start, which overrides the file and lets you choose a subset for a particular run.

Matching security and settings

The controller must hold the same key store as the agents when secure communication is used. It also needs the same settings for any special ports. Where the controller sits behind a firewall, or on a machine with several network cards, it needs to announce the address that agents can use to reach it back with results.

Leaving the controller light

In a distributed run, the controller does not generate the load unless you specifically arrange it. That is a feature: its resources are then free to receive and show results from all agents. Keep the controller well supplied with memory, and keep the number of listeners attached to it modest.

You can start from the graphical window or from the command line; both do the same conversation.

From the graphical window

In the window, a Run menu offers a remote start option for all agents, one for a single named agent, and matching stop and shutdown options. This route is useful while you are learning and checking that every agent responds. It is not the route for large measured runs, because the window consumes memory and processing on the controller.

From the command line

For serious runs, start the controller in non-graphical mode with the plan file and a result file. Add the option that says to start all configured agents, or the option that names specific agents. Another option asks the tool to shut the agent services down when the test ends. A separate option sets a property on all agents at once, which is how you pass values such as thread counts to every machine.

Most “it just will not connect” stories end in a closed door somewhere between machines.

Two directions of conversation

The controller must reach each agent to send the plan and commands. Each agent must also reach the controller to send results back. Both directions need open doors. A firewall that allows the first but blocks the second produces the confusing situation where the test appears to start and then no results arrive.

Fixed and random ports

The communication method uses a registry port, which by default is a well-known number, and then additional ports for the actual objects exchanged. By default those additional ports are chosen at random, which is awkward for firewalls that need exact rules. The tool lets you pin these to fixed numbers through settings, on both the agent side and the controller side, so that the firewall can allow precisely those ports and nothing else.

Addresses that travel

A machine with several network addresses may announce the wrong one. Containers and cloud machines have private addresses inside and public addresses outside. If an agent announces an address the controller cannot reach, the first connection works but later calls fail. Each machine should be told the address that other machines should use to find it.

An agent obeys whoever speaks to it correctly, so the channel must be protected.

Why this matters

An agent is a program that accepts instructions from the network and runs a test plan, which can contain scripts. If anyone on the network could send it a plan, they could make it run arbitrary actions. For this reason the tool secures the channel by default with encrypted communication that uses a shared key store file, and recent releases expect that file to exist.

The key store idea

A key store is a file that holds the identity material used for encryption. A helper script supplied with the tool creates one. You then copy that same file to the controller and to every agent. Machines that hold the file can talk to each other; machines without it are refused. Treat the file with the same care as a password.

Turning it off

The tool allows the secure channel to be disabled through a setting, which is sometimes done in a small closed laboratory network to avoid certificate work. Do this only where the network is isolated and trusted, and never on a network others can reach. Both sides must agree: if one end is secure and the other is not, they cannot connect.

Settings can live on the controller, on an agent, or be pushed to all agents at once.

Three places a setting can live

A setting in the controller’s settings file affects only the controller. A setting in an agent’s settings file affects only that agent. A setting supplied with the global option on the controller’s command line is sent to every agent at start. Knowing which of the three you need avoids many puzzles, such as why a thread count changed on the controller but not on the agents.

Plans that read properties

A well-built plan reads its main numbers from properties through the property function, with sensible defaults. The tester then changes the crowd size, the duration, or the target address without editing the plan file. In a distributed run, the global option carries those numbers to every agent in one step.

A shared plan with separate files on each machine is the source of both power and mistakes.

The files must be on the agents

When the controller sends the plan, it sends the plan only. It does not send the data files the plan refers to, such as lists of user names and passwords. Each agent reads the file from its own disk at the path written in the plan. If the file is missing on an agent, the test elements that use it fail there, though they work on the controller.

The duplicate-data trap

If every agent holds an identical copy of the file and reads from the top, then agent one and agent two both begin with the same first account at the same moment. Many systems reject two simultaneous logins of one account, or lock it, so the test then measures login failures, not capacity. The cure is to give each agent its own slice of the data.

Every sample an agent produces must travel to the controller, and how it travels changes the load on both.

Why this matters

At high rates, agents produce thousands of sample results per second. Sending each one individually across the network, at the moment it is created, can slow the agent and flood the controller. The tool offers several sending modes that trade immediacy for efficiency.

The conductor can be overwhelmed by the sheer volume of reports from the players.

How it happens

All results from all agents are funnelled into the single controller. Ten agents, each producing a thousand results every second, send ten thousand records per second towards one machine. If the controller must also draw tables and graphs, it may fall behind. When it falls behind, agents may slow down as their send queues fill, and the combined result becomes late and incomplete.

flowchart LR
  A1[Agent 1 results] --> Q[Controller intake]
  A2[Agent 2 results] --> Q
  A3[Agent 3 results] --> Q
  A4[Agent N results] --> Q
  Q --> L{Listeners and writing}
  L -->|too heavy| X[Controller falls behind]
  L -->|light| Y[Compact result file]
        

Keeping it healthy

When many machines act together, the clock becomes part of the design.

Clocks that must agree

Every sample is stamped with the time on the agent that produced it. When results from several agents are combined, the stamps are merged on one timeline. If one agent’s clock is minutes off, its results appear in the wrong place, and charts of throughput or response over time look strange. A network time service on every machine fixes this cheaply.

Ramp-up is per agent

A ramp-up period tells each group to bring in its users gradually. With several agents, each ramps up independently, so the total crowd grows as the sum of several staircases. That is usually what you want, but if agents start at slightly different moments, the combined shape is a little jagged. Allow a settling period before you begin reading measurements.

Staggered starts and uneven agents

Agents with unequal power should receive unequal shares, or the weaker agent finishes late and distorts the end of the run. If you need controlled timing, such as all users reaching a certain step together, use a synchronising element in the plan, and remember that it coordinates threads within one agent, not between agents.

Capacity planning for the load generators is its own small discipline.

Three pieces of arithmetic

First, find how many users one agent can run comfortably for your particular plan. A plan with big responses, heavy scripts or encrypted traffic is much more demanding than a plan with tiny requests. Second, divide the total users you need by that comfortable number, then round up and add spare room. Third, check that the combined network traffic does not exceed the limits of the network path between the agents and the target.

What makes an agent heavier

The built-in controller-agent setup is not the only way to use many machines.

What the alternative looks like

Instead of one controller commanding many agents, you start the same plan separately on each machine, in non-graphical mode, using an ordinary script or scheduling tool. Each machine writes its own result file. After the run, you merge the result files into one and analyse them together. There is no persistent connection between machines during the test.

Choosing between them

The built-in method is attractive when you want one trigger and a live combined view and the network is simple. The independent method is attractive in locked-down networks, in cloud setups with orchestration tools, and for very large runs where the controller would be overloaded. Many teams use the independent method for their serious runs and the built-in method for quick experiments.

Symptoms usually point at one of a few causes; learn the map and troubleshooting becomes calm.

A calm routine

Agents are an ideal fit for machines that you create for a test and delete afterwards.

Why the cloud suits agents

An agent holds nothing precious. It is an installed tool, a key store, some supporting files and a waiting service. You can capture all of that as a machine image or a container image, start twenty copies for an afternoon, and delete them when the test is over. You pay only for the hours used, and you can place agents in regions near your real users.

Things that need care

You cannot trust a result without knowing the health of the machines that produced it.

Two things to watch

Watch the agents to prove that the test machines were comfortable. Watch the system under test to learn how it behaved. Without the first, you cannot tell whether slowness was real. Without the second, you will know that responses were slow but not why.

Recording the evidence

Keep monitoring data for the agents next to the result files for each run. When someone later questions a result, you can show that every agent stayed below a comfortable level, which makes the result defensible.

A test is only finished when the machines are back in a known state.

Stop versus shutdown

Stopping a test tells the agents to end their threads, finishing current requests gracefully where it can. Shutting down is a gentler request still, asking threads to complete their present iteration. Exiting the agent services entirely is a third action, which closes the waiting program on each agent. Knowing the difference avoids both abrupt endings and agents left running by accident.

A crowd of agents is a powerful tool that must be pointed carefully.

Protect the agents

Agents run test plans on request, so keep them off public networks, restrict who can reach them, keep the key store private and use the secure channel. When machines are rented in the cloud, apply the same rules you apply to any server: patched software, minimal open ports and removal after use.

Protect the target and the neighbours

Many agents together can produce a flood. Do not aim them at systems you do not own or have no permission to test. A shared environment may host other teams, so schedule heavy tests and tell the owners. Cloud providers and third-party services often require advance notice or forbid load testing at all.

Protect the data

Result files and logs may contain addresses, user names and response fragments. Store them securely and delete them when no longer needed. Use test accounts, never real customer accounts, and keep credentials out of shared plan files by supplying them through data files or properties.

A few repeatable habits make distributed runs boring in the best way.

These habits are tempting under time pressure and costly afterwards.

Distribution solves a real problem and brings real complexity with it.

Run through these points before every distributed run.

Five situations show how the ideas fit together in practice.

Short answers to the questions beginners ask most often.

In the usual remote start, no. Only the agents run the threads. The controller conducts and collects results, which keeps it free for that job.

One thousand, because each agent runs the whole plan. Divide your target by the number of agents when you design the plan.

No. It sends the plan only. Data files, plugins and certificates must already be on each agent, at the paths the plan uses.

Generally yes, as long as the tool and Java versions match and file paths in the plan work on every machine. Keeping machines identical is still simpler and safer.

Most often the agents cannot connect back to the controller, because of a firewall, address translation or a wrongly announced address. Check the return direction.

You should, unless the network is fully isolated and trusted. If you keep it on, every machine must share the same key store file.

For learning and small checks, yes. For large measured runs, use the command line, because the window and live listeners can overload the controller.

Make the plan read the thread count from a property and set that property in each agent’s own settings, instead of passing one global value to all.

Nearly, but not exactly. The start order goes agent by agent over the network, so there may be small gaps. Ramp-up periods and steady-state measurement hide them for most tests.

Watch its processor, memory, pause times and network during the run. Sustained high values, or timings that worsen without any change in the target’s health, point to an overloaded agent.

No. You can run the plan independently on each machine and merge the result files afterwards. This avoids network setup and the controller bottleneck, at the cost of a combined live view.

Yes, and it is a popular choice. Take care with announced addresses, network rules, the agent list for each run and cleanup of machines afterwards.

Such elements coordinate only the threads on their own agent. They cannot see threads on other machines.

Confirm the agent service is running, the address in the list is right, the port is open from the controller, and the key store matches. Then read the agent’s own log.

Distributed testing lets many ordinary machines act as one large crowd, with a controller that conducts and agents that perform. Success depends less on clever tricks than on discipline: identical machines, open and secure channels, divided crowds, separate data, light controllers and honest evidence that the generators were healthy. Follow these habits and the numbers you report describe the system you tested, not the machines you tested it with.