Running a test plan from the command line, with no window, only the work.

The simple definition

A GUI, short for graphical user interface, is the window with menus, trees and buttons that you see when you open JMeter. A command line is a text prompt where you type an instruction and the computer carries it out. Non-GUI mode means you start JMeter from the command line, give it a saved test plan, and it runs the plan without ever drawing the window. Many people also call it command-line mode, CLI mode or headless mode.

You still create the plan in the window. Non-GUI mode is only about how the plan is run. The plan is a file, usually with the extension .jmx, and the same file can run in either mode. This is why the workflow is so comfortable: design in the dining room, cook in the kitchen.

What you get in return

Without the window, JMeter does not spend memory on drawing trees, updating tables and keeping results visible on the screen. All that effort goes to creating load and recording timings. The result is that one machine can generate more users, and the numbers it measures are more honest.

The window is helpful for design, and costly during a real run.

When JMeter runs inside its window, every listener, tree and table that is open keeps data in memory and redraws itself as results arrive. At a small scale this is harmless. At a large scale it becomes the biggest consumer of resources on the machine. Here is what goes wrong.

flowchart LR
    A["Run inside the window"] --> B["Listeners store and draw every result"]
    B --> C["Heavy memory and CPU use"]
    C --> D["Generator becomes the bottleneck"]
    D --> E["Results measure the tool, not the service"]
        

The warning JMeter gives you

When you start the window, JMeter prints a message in its console that says, in short, that the graphical mode is for creating and debugging and that real load tests should use the command line. The message exists because the tool’s own authors saw the same problem many times.

Same plan, same engine, very different jobs.

The test engine, the part that creates virtual users and sends requests, is the same in both modes. What differs is everything around it. The table shows the main differences.

Different jobs, not rival tools

Do not think of one mode as good and the other as bad. Each has a job. The window is unbeatable for building plans, testing paths and checking a single reply. The command line is unbeatable for running at scale and for repeating the same run again and again. A mature team uses both, in a clear order: design first, run second.

A repeatable path from idea to trustworthy results.

A good performance test follows a clear path. Each stage has a purpose and a place where it is done. Following the path keeps your results reliable and your team in agreement about how a number was produced.

flowchart TD
    A["Design the plan in the window"] --> B["Debug with one user and a small loop"]
    B --> C["Parameterise: users, ramp-up, duration as properties"]
    C --> D["Remove heavy listeners and save the plan"]
    D --> E["Run from the command line with a results file"]
    E --> F["Generate the HTML report"]
    F --> G["Analyse, compare and share"]
    G --> H{"Need another run?"}
    H -->|"Yes"| C
    H -->|"No"| I["Archive plan, results and report"]
        

A short command with a few clear parts, each with one job.

The basic command starts the JMeter launcher, which is a small script in the tool’s folder, and then adds options. Each option starts with a dash and a letter. Most of the time you need only a few of them. Reading a command from left to right tells you the whole story of the run.

Reading it like a sentence

A full command reads like a sentence: “Run JMeter without a window, use this plan, write results to this file, write the tool log to that file, and make a dashboard in this folder at the end.” If you can say the sentence, you understand the command, and if a colleague asks what your run did, the sentence is the answer.

Why a script beats memory

Commands grow longer as plans mature. A realistic command may carry a dozen properties, several file paths and a report folder. Typing it by hand invites small mistakes, such as a missing space or a wrong folder name, and nobody remembers last month’s exact settings. A script file records every choice, can be reviewed by a colleague, and can be stored in version control beside the plan. When a result needs explaining, the script is the evidence of what was run, and when it needs repeating, the script is the instruction.

A small toolbox that covers almost every real test.

JMeter has many options, and you will use a handful daily. The table below lists the ones every tester should know. Exact spelling and behaviour can differ slightly between versions, so check the help text of your own installation by asking the launcher for its help.

The core four

If you remember only four options, remember the non-GUI flag, the test file, the results file and the report option pair. Everything else is added when a need appears.

The results file is the permanent memory of your test. Everything else is built from it.

During a non-GUI run, nobody watches tables fill up. The record of what happened is the results file, often called a JTL file after its common extension. Each line describes one request: when it started, how long it took, which request it was, whether it passed, what the status code was and more. A report, a chart or a comparison is only as good as this file, so it is worth understanding.

Typical columns

What you control

A set of properties decides which columns are saved, and whether results are written as text with commas or as XML. Text with commas is the common choice, because it is compact, fast to write and understood by the report generator and by many analysis tools. Saving extra data such as full response bodies makes the file very large, and it also slows the run. Save only what you need.

flowchart LR
    A["Virtual users send requests"] --> B["Engine records one line per request"]
    B --> C["Results file"]
    C --> D["HTML dashboard"]
    C --> E["Charts and analysis tools"]
    C --> F["Comparison between runs"]
        

One plan, many runs, no need to edit the file each time.

A property is a named setting that JMeter holds for the whole process. You can set properties on the command line, and the plan can read them. This lets you run the same plan with ten users in the morning and five hundred users in the evening without opening the file. It is the most important habit for clean automation.

Reading a property inside the plan

The plan reads properties with a built-in function that takes a name and a default value. The default is used when the command line does not provide a value. This makes the plan run sensibly in the window, where nobody passes a value, and run with chosen values on the command line.

Making the shape of the load easy to change from the outside.

The shape of a load test is made of a few numbers: how many virtual users, how fast they start, how long they run and how long they pause between steps. Each number belongs in a property, so that the same plan can serve smoke tests, normal-load tests, peak-load tests and endurance tests.

Why ramp-up matters

Starting all users at the same second creates an artificial storm that real life rarely produces. A ramp-up spreads the starts across a period, so the system warms up in a natural way. For analysis, you can also ignore the ramp-up period in the report and look only at the stable part of the run.

flowchart LR
    A["Command line values: users, ramp-up, duration"] --> B["Properties read by the plan"]
    B --> C["Thread group settings"]
    C --> D["Shape of the load"]
    D --> E["Smoke, load, stress, endurance or spike run"]
        

A ready-made report that turns the results file into charts and tables.

JMeter can create a web page called the dashboard report from a results file. It shows summary tables, response times over time, throughput, error percentages, percentiles and more. You can create it at the end of a run, by adding the report options to the command, or later from an existing results file.

What the dashboard shows

Rules to remember

The output folder must be new or empty, or the report generation refuses to continue. The results file must contain the columns the report needs, and the default settings usually provide them. Percentiles are more honest than averages, because a few very slow requests can hide behind a pleasant average.

Without a window, the console is your live dashboard.

When a test runs in non-GUI mode, JMeter prints a short summary line to the console every so often. This is produced by a built-in element called the summariser. Each line shows how many requests were sent in the latest period and in total, the average, minimum and maximum response times, the number of errors and the rate of requests. By default the line appears about every thirty seconds, and a property can change that interval.

Using it as an early warning

You do not need a graph to see trouble. If errors jump from zero to five percent in one interval, the plus line shows it at once. If the request rate falls while the user count stays the same, the server may be slowing down. Watching the console for the first few minutes of every large run costs nothing and catches many mistakes early, such as a wrong host name or an expired token.

The log tells you what the tool itself thinks, which is different from what the server says.

JMeter keeps a log of its own activity, usually in a file named after the tool, or in the file you choose with the log option. The log records start and stop messages, warnings about missing files, errors from scripts, problems with connections and more. The results file records what happened to requests. The log records what happened to the tool. You need both.

Reading the log after a problem

When a run behaves strangely, open the log and search from the end. Look for words such as error, exception, warning, refused and timeout. Problems that appear at the start of the file, such as a plan that could not find a data file, often explain what happened much later. Keep the log together with the results file, using a name that matches the run.

flowchart TD
    A["Something looks wrong"] --> B{"Where is the clue?"}
    B -->|"Slow or failing requests"| C["Results file and report"]
    B -->|"Plan did not start or stopped early"| D["Tool log"]
    B -->|"Machine looks overloaded"| E["Operating system monitoring"]
    C --> F["Form a theory"]
    D --> F
    E --> F
        

JMeter runs on Java, and Java needs a clear memory budget.

JMeter is a Java program. Java programs run inside a JVM, a Java Virtual Machine, which has a fixed memory area called the heap. Objects such as virtual users, variables and stored results live in the heap. If the heap is too small, the program slows down as the garbage collector works harder and, in the worst case, stops with an out-of-memory error. If the heap is too large for the machine, the operating system may start swapping memory to disk, which is far worse.

Setting the heap

The launcher script defines a default heap size, and most installations allow you to override it through an environment variable before starting the tool. Check the comments in your launcher script for the exact variable name in your version. A common approach is to set the minimum and maximum heap to the same value, so the memory is reserved at start and does not resize during the run.

A sensible rule of thumb

Leave enough memory for the operating system and for anything else on the machine. Give the heap a generous but safe share of what remains. Then measure: run a trial at the planned load and watch memory use. A generator that sits at a moderate level for the whole run is healthy. One that climbs steadily is holding on to data it should not.

A well-tuned machine produces trustworthy numbers.

Load generation puts pressure on the operating system. Each virtual user may hold open connections and files. The system has limits on how many it will allow, and the defaults are often too small for a heavy test. When the limits are reached, errors appear that look like server failures but are really generator failures.

Place the generator wisely

Put the load generator close to the system under test, in the same network zone, unless the test is about distance. A generator on a slow office connection would measure the office link. For tests that must show the user experience from different places, use generators in those places on purpose, and label the results by location.

flowchart LR
    A["JMeter process"] --> B["Operating system limits"]
    B --> C["Network link"]
    C --> D["System under test"]
    B -.->|"Too small: errors that look like server errors"| E["False failures"]
    C -.->|"Too slow: delays that look like server delays"| F["False slowness"]
        

A long test must not die when your connection does.

Many load generators are servers in a data centre or in the cloud. You reach them through a secure shell connection, which is a text session over the network. If the session closes, for example because your laptop sleeps or your network drops, programs started inside it may be stopped. A two-hour test that dies at minute ninety is a painful lesson.

Ways to keep the test alive

Checking on a run from outside

Once the test is in the background, you still want to know how it is going. Reading the tail of the console file, or of the results file, shows progress. A real-time dashboard fed by a backend listener, which a later chapter describes, is an even better option because it shows the trend without logging in.

There is a polite way to stop and a sudden way. Choose on purpose.

Sometimes a test must end before its planned time: the system under test is struggling, the data is wrong, or someone typed a wrong value. Without a window, you cannot press a stop button, so JMeter offers other ways. Each has a different effect on your results, and knowing the difference protects your data.

The control scripts

JMeter ships small scripts in its bin folder that send a shutdown or a stop message to a running non-GUI test on the same machine. The running process listens on a local port for these messages. Check the file names and the port setting for your version and platform, and test them once on a small run before you need them in an emergency.

flowchart TD
    A["Need to end the test early"] --> B{"Is the system under test in danger?"}
    B -->|"Yes"| C["Immediate stop"]
    B -->|"No"| D["Graceful shutdown"]
    D --> E["Users finish current iteration"]
    C --> F["Some requests cut off"]
    E --> G["Report can be built from a clean file"]
    F --> G
        

When one machine is not enough, several machines work together under one controller.

A single generator can only create so much load. For bigger tests, JMeter supports a distributed setup. One machine is the controller and starts the test. Several others are servers, also called workers or agents, that each run the same plan and create their share of the load. The controller collects results and builds the report.

flowchart TD
    C["Controller: starts the test, collects results"] --> W1["Worker 1: runs the plan"]
    C --> W2["Worker 2: runs the plan"]
    C --> W3["Worker 3: runs the plan"]
    W1 --> S["System under test"]
    W2 --> S
    W3 --> S
    W1 -.->|"results"| C
    W2 -.->|"results"| C
    W3 -.->|"results"| C
        

The key facts

When to use it

Use distributed mode when a single generator reaches its limit of CPU, memory or network before it reaches the target load. First measure one machine’s limit. Then add workers one at a time and check that total throughput grows in a straight line. If it does not, the bottleneck has moved elsewhere, perhaps to the controller, the network or the system under test.

A container packs the tool and its settings into a portable box.

A container is a lightweight package that holds an application and everything it needs: the right Java version, the JMeter files, plugins and settings. You start it with one command, on any machine that supports containers. This solves the classic problem of “it works on my machine”, because every run uses exactly the same environment.

What to put where

Performance checks that run on every change catch problems while they are cheap to fix.

A pipeline is an automatic series of steps that builds, tests and delivers software. Because the non-GUI mode is a plain command, any pipeline can run it. Teams add a performance stage after functional tests. The stage starts the test, waits for the end, builds the report, checks the numbers against agreed limits and stores the results.

flowchart LR
    A["Code change"] --> B["Build"]
    B --> C["Unit and functional tests"]
    C --> D["Deploy to test environment"]
    D --> E["JMeter non-GUI run"]
    E --> F["Report and checks"]
    F --> G{"Within limits?"}
    G -->|"Yes"| H["Promote"]
    G -->|"No"| I["Fail and notify"]
        

Two kinds of pipeline tests

Make results visible

Store the plan, the command, the results file and the report as pipeline artifacts, so anyone can open them later. Show the key numbers in the pipeline summary: error rate, ninety-fifth percentile and throughput. A test that nobody can see is a test that nobody trusts.

A pipeline needs a clear yes or no. The tool alone does not always give one.

When a command finishes, it returns a number called the exit code. Zero normally means success. Pipelines read this number to decide whether to continue. For JMeter, a finished run usually returns zero even when many requests failed, because the run itself completed. The exit code tells you that the tool ran, not that the system performed well.

Adding your own gate

To make a pipeline fail on poor performance, add a step after the run that reads the results and compares them with limits. The limits come from agreement with the business, for example an error rate under one percent and a ninety-fifth percentile under two seconds for the main journey. If a limit is broken, the step returns a failing code and the pipeline stops.

Include a validity check

A gate that only checks the server’s numbers can pass a broken test. If the generator was overloaded, or the test ended early, a green result means nothing. Add checks that the run lasted as planned, the expected number of users were active and the request count is in the expected range.

See the test in real time, from anywhere, without opening any window on the generator.

The Backend Listener is an element that sends results, in small batches, to an external system while the test runs. Common targets are time-series databases and monitoring platforms, which then draw live charts on a shared dashboard. This gives the whole team a view of the test, from any browser, without anyone logging in to the generator.

flowchart LR
    A["JMeter non-GUI run"] --> B["Backend Listener sends batches"]
    B --> C["Time-series store"]
    C --> D["Live dashboard"]
    D --> E["Team watches trends and alerts"]
    A --> F["Results file kept as the permanent record"]
        

What a live dashboard adds

Costs and cautions

Sending results costs some CPU, memory and network. Keep the batch size and the sending interval reasonable, and avoid sending extra detail you will never chart. If the monitoring system is slow or unreachable, the listener must not slow the test. Watch for warnings in the log and keep the results file as the reliable record, because live systems may sample or drop data.

Listeners are useful in the window and costly in a real run.

A listener is an element that receives results and shows or saves them. In the window, listeners such as the results tree and the table help you debug. In a command-line run, the same listeners still collect data, even though nobody sees it, and they use memory. Before a real run, review every listener in the plan.

A simple way to manage this

Group view-only listeners under a clear parent or name them with a prefix such as “DEBUG”. Before each real run, disable all of them in one step. Better still, keep a separate copy of the plan for real runs where those listeners are simply absent. Either way, the rule is the same: only elements that serve the real run stay enabled.

Most “it worked in the window” problems are about where files are found.

Plans often read data files, such as lists of users or search terms. In the window, the tool finds them from the plan’s folder or from the path you typed. On another machine, in a container or inside a pipeline, the working folder may be different, and the same path may point to nothing. A missing file does not always stop the run. It may only log a warning and use empty values.

Safe habits for paths

Data per worker

In distributed runs, each worker reads its own copy. If every worker has the same list of users, many virtual users will log in with the same account. Split the list so that each worker has a different part, or use a data approach that guarantees unique values. Otherwise the test may fail on duplicate logins or may touch the same records and distort results.

flowchart TD
    A["Full list of 3000 users"] --> B["Part 1: 1000 users to worker 1"]
    A --> C["Part 2: 1000 users to worker 2"]
    A --> D["Part 3: 1000 users to worker 3"]
        

When the window works and the command line does not, the difference is almost always in the surroundings.

The plan is the same, but the environment is not. The properties may differ, the working folder may differ, the heap may be different, the network may be different and listeners are absent. A calm checklist finds the cause faster than guessing.

The two-step trick

First, run the plan on the command line with one user and one loop, on the same machine where you designed it. If it works, the plan is fine and the surroundings of the other machine are the cause. Second, run the same tiny test on the other machine. Differences between the two tiny runs point at the surroundings, and you can fix them one by one.

A measuring tool must not disturb what it measures.

Every number in a report depends on the generator being healthy. If the machine is overloaded, it sends requests late and records replies late, and the recorded response time includes the generator’s own delay. This effect is called coordinated omission in its worst form, where slow periods are under-reported because blocked users send fewer requests. You cannot remove the effect completely, but you can detect and limit it.

Signs of an unhealthy generator

Calibration runs

Before trusting a setup, run the same plan against a very fast, simple target, such as a small static page, at the planned user count. If response times are very small and steady, the generator is healthy at that load. If they are already large, the generator is the limit, and no real test at that size would be reliable. Repeat after any change of machine, tool version or plan size.

flowchart TD
    A["Calibration run against a fast target"] --> B{"Times small and steady?"}
    B -->|"Yes"| C["Generator is healthy at this load"]
    B -->|"No"| D["Reduce users per machine or add workers"]
    D --> A
    C --> E["Run the real test and keep watching CPU and memory"]
        

A load test is a controlled attack on your own system. Handle it with care.

Non-GUI runs often happen on shared servers and in pipelines, where many people can read logs and files. At the same time, a load test can send enormous traffic to a system. Security here has two sides: protecting the data and credentials the test uses, and protecting the systems the test touches.

A safety habit that pays for itself

Build a guard into the plan or the command script that checks the target host against an allowed list and refuses to start when it is not on the list. A single such guard prevents the most painful class of mistakes, the test that hits the wrong place.

Habits that turn a one-off run into a repeatable practice.

Patterns are proven ways of arranging work. They save time and reduce arguments, because the team agrees on how things are done. These are the patterns that serve non-GUI testing best.

The run folder pattern in detail

Create a folder per run whose name tells the story: the date, the time, the scenario and perhaps the build number. Inside, keep the plan copy, the command script, the properties file, the results file, the tool log, the console output and the report folder. When someone asks about last month’s test, you open one folder and everything is there.

flowchart TD
    A["Run folder named by date, scenario and build"] --> B["Plan copy"]
    A --> C["Command script and properties"]
    A --> D["Results file"]
    A --> E["Tool log and console output"]
    A --> F["HTML report folder"]
    A --> G["Notes: goal, observations, decisions"]
        

Habits that look harmless and spoil large tests.

An anti-pattern is a way of working that seems fine and keeps causing trouble. Learning to spot these is the fastest way to improve your practice. Each box names the problem, explains the harm and shows the better way.

The habits of experienced performance engineers, in one list.

Best practices are not secrets. They are small habits, repeated until they feel natural. Together they make runs faster to prepare, safer to start and easier to trust.

Four familiar situations and how non-GUI execution supports each of them.

Before you plan the test, plan the tester.

One of the first questions in any large test is how many virtual users a single machine can carry. There is no universal answer, because the number depends on the plan, the size of replies, the scripts used, the think times and the hardware. The honest way to find the answer is to measure it, step by step, with your own plan. This is called capacity planning for the generator.

A simple stepping method

Real users pause, read and decide. A test that never pauses is a different thing.

A think time is a pause between steps that imitates a person reading a page or filling a form. Without think times, each virtual user sends the next request the moment the last reply arrives, and a small number of users can create a huge request rate. That may be useful for a stress test, and it does not match normal use. How you shape the pauses decides what kind of load your test really creates.

Closed and open workloads

In a closed model, a fixed set of users loops through the journey. When the system slows down, users wait longer, and the request rate falls on its own. In an open model, new arrivals come at a set rate whatever the state of the system, like real visitors on a public website. Many tests are closed by default, because thread groups are closed. If your real traffic is open, consider arrival-rate approaches, or a throughput target, so that a slow system faces the same pressure and not less.

flowchart LR
    A["Closed model: fixed users loop"] --> B["System slows down"]
    B --> C["Users wait longer, request rate falls"]
    D["Open model: arrivals at a set rate"] --> E["System slows down"]
    E --> F["Requests keep arriving, queues grow"]
        

A single report is a snapshot. Comparison turns it into knowledge.

The dashboard from one run tells you how the system behaved once. The real value comes from comparing runs: this release against the last, this configuration against that one, today against last month. To compare fairly, the runs must be alike in everything except the one thing you changed.

Make comparisons fair

What to compare

Keep a short table of the main numbers for every run in a shared place, with the date, the build and a link to the run folder. A simple history table makes slow decay visible. A service that gets two percent slower with each release can look fine release by release and be badly slower after a year.

A good run starts before the command and ends after the report.

The command is only a small part of a performance test. The environment under test must be ready, the data must be right, and the aftermath must be cleaned. Many failed tests are lost not in the tool but in the surroundings.

Data that grows during the test

Tests that create data, such as orders or messages, change the environment as they run. After thousands of orders, queries may become slower for reasons that have nothing to do with the new release. Plan how much data your test creates, and either clean it after each run or reset the environment to a known state. Without this step, each run starts from a different place, and comparisons lose their meaning.

flowchart LR
    A["Known state"] --> B["Run test: creates data"]
    B --> C["Changed state"]
    C --> D["Clean up or reset"]
    D --> A
        

A calm, step-by-step approach finds the cause of most problems in minutes.

Command-line runs fail in a small number of familiar ways. The table lists the common symptoms with their usual causes. Work from the top, and change one thing at a time.

The five-question routine

flowchart TD
    A["Run behaves badly"] --> B{"Did the command start the plan?"}
    B -->|"No"| C["Check paths, Java, permissions"]
    B -->|"Yes"| D{"Are requests reaching the right target?"}
    D -->|"No"| E["Check host, port, proxy, certificates"]
    D -->|"Yes"| F{"Is the generator healthy?"}
    F -->|"No"| G["Fix memory, limits, CPU, add workers"]
    F -->|"Yes"| H{"Are plan settings and data correct?"}
    H -->|"No"| I["Fix properties and data files"]
    H -->|"Yes"| J["Look at the system under test"]
        

Most problems end at the first or second question. A typing mistake in a path, a host name that points to the wrong place and a data file that is missing explain a large share of failed first runs. Only when these are ruled out is it worth looking at the system under test.

A short routine before every run, and a longer one every few months.

Before a big test, spend ten minutes on a checklist. It is the cheapest insurance in performance testing, because it protects hours of preparation and an environment that other people are waiting for.

Maintenance every few months

Short, plain answers to the questions testers ask most often.

It is a way of running a saved test plan from the command line without opening the graphical window. It uses fewer resources and is meant for real load tests and automation.

The window and its listeners use memory and CPU that the load threads need. The generator becomes the bottleneck, and the measured times include the tool’s own delay.

The non-GUI flag, the test file, the results file and, for a report, the report flag with an output folder. Properties, logs and remote options are added when needed.

Read them from properties inside the plan, with safe defaults, and pass the real values on the command line or through a property file.

Add the report flag and an output folder to the run command, or build the report later from an existing results file. The output folder must be new or empty.

Use the shutdown script for a graceful end, where users finish their current work, or the stop script for an immediate end. Forcing the process to end is a last resort and may leave incomplete results.

Use distributed mode: one controller starts the test and several workers each run the full plan. Remember that users multiply by the number of workers, and that data files are not shared automatically.

Run it inside a terminal multiplexer, or in the background in a way that ignores hang-ups, or through a pipeline agent. Test the method with a short run first.

The tool usually exits with success when it finishes, whatever the error rate. Add a step that reads the results and fails the pipeline when limits are broken.

Enough for the plan and users, but not more than the machine can spare. Set minimum and maximum equal, measure a trial run, and watch for steady growth, which signals data being kept.

Yes. The console prints a summary at regular intervals, and a backend listener can send results to a monitoring system for a live dashboard. Keep the results file as the permanent record.

Monitor its CPU, memory and network, and run a calibration test against a fast target at the planned load. Small, steady times show the generator is healthy.

The short version, for the day you only have two minutes.

Non-GUI mode is how JMeter is meant to be used for real load. The window is for building and debugging, and the command line is for running. With a small set of options, properties for every changing value, a unique run folder, a results file and a report, you get tests that are fast, repeatable and easy to automate.