Xray Test Plan vs Test Execution: What's the Difference?

The shortest useful answer: an Xray Test Plan defines and tracks the testing scope for a milestone. An Xray Test Execution records an actual run of tests against a specific build and environment. Plans answer “what did we commit to cover for 1.9 / UAT?” Executions answer “what happened when we ran these tests on staging build 482?”

If you remember only that split, Xray dashboards become readable. If you collapse the two, every UAT board eventually lies.

This article compares Xray’s four core objects—Test, Test Set, Test Plan, and Test Execution—then walks a realistic regression/UAT example. It is a comparison guide, not a first-run tutorial. For installing the mental model with a first Test, use the Xray tutorial. For how to operate a run, use Xray Test Execution.

Quick comparison

A Test is a specification. A Set is a playlist. A Plan is the syllabus for the term. An Execution is a dated exam sitting.

What is an Xray Test?

An Xray Test is a Jira issue that stores the reusable specification: preconditions, steps, and expected results (or an automated test mapping). It should be stable enough to run against many builds.

Healthy Tests:

• Cover one behavior
• Can be executed without Slack archaeology
• Link back to the story or requirement they protect
• Do not get rewritten to store “failed yesterday”

If you need the lifecycle of authoring, reviewing, and maintaining cases in Jira, that is Jira test case management—not a Plan vs Execution problem.

What is a Test Set?

An Xray Test Set is a Jira issue that holds an ordered collection of Tests. It is the reusable suite definition: smoke, checkout regression, payments high-risk, UAT core journeys.

The set is not a run. Membership should change when the suite intent changes—not every time you execute. Design detail lives in Xray Test Sets.

Rule of thumb: if you cannot name the set’s purpose in one sentence, it is a dump, not a Set.

What is a Test Plan?

An Xray Test Plan is a planning and progress container for a milestone. Teams typically use it for a release, a UAT window, or a named quality cycle.

A Plan usually tracks:

• Which Tests (often via Sets) are in scope for the milestone
• Overall progress as linked Executions complete
• A single place stakeholders can ask “are we done with UAT for 1.9?”

A Plan is not a list of step results. It should not be the place you click Pass/Fail on individual steps for a specific build. That belongs on Executions.

You do not always need a Plan on day one. Many teams run Tests, Sets, and Executions first, then add Plans when a release needs a rollup across smoke, regression, and UAT executions. The Xray tutorial makes the same recommendation.

Xray Cloud and Data Center UIs differ. Treat vendor docs as the source for exact buttons; the object roles stay stable.

What is a Test Execution?

An Xray Test Execution is a Jira issue that records a concrete run:

• Which Tests were included
• Who ran them
• Which environment and build
• Per-test (and often per-step) status
• Defects found during that run
• Optional link back to the Test Plan so progress rolls up

Executions have dates. Tests should not. If the same Execution issue is edited across three builds, you have destroyed comparability. Create a new Execution for a new build, or follow a documented retest flow that still preserves history. Operating detail: Xray Test Execution and the broader Jira test execution guide.

Test Plan vs Test Execution

This is the comparison Search Console-style queries usually want—including UAT dashboard confusion.

UAT dashboard implication

UAT stakeholders usually want two different charts:

1. Plan-level: What percent of the agreed UAT scope has a result for this cycle?
2. Execution-level: Did yesterday’s staging build pass the UAT pack?

If you use one Execution as the UAT dashboard for the whole cycle, you will either overwrite results when the build changes, or show a blend of old and new builds. If you skip the Plan and only have Executions, engineering can still work, but product/UAT leads lose a single scope object to point at.

A clean UAT pattern:

• One Test Plan named for the UAT cycle and version
• A UAT Test Set (or a small number of journey Sets) that defines reusable membership
• Separate Test Executions per build (or per environment), linked to the Plan

The dashboard then has a place for “scope” and a place for “this build.”

Test Set vs Test Plan

Teams also mix these two.

You select a Set into a Plan (or into an Execution). You do not replace a Plan with a Set. A Set has no honest answer to “are we finished with 1.9 UAT?” because 1.9 is a moment in time and the Set is a library object.

Example: regression and UAT for a release

Scenario: your team is shipping 1.9. Scope includes a checkout promo feature plus the usual regression risk around cart, auth, and payments. Product also wants a UAT pass on staging before production.

Objects you might create

Sequence

1. Impact analysis — which existing Tests are touched by 1.9 besides the new promo cases?
2. Update Tests — new promo cases linked to the story; existing cart Tests updated only if behavior changed.
3. Confirm Sets — smoke stays small; checkout regression includes the new promo Tests; UAT core is the stakeholder-facing journey list (usually smaller than full regression).
4. Create the Plan — add the Sets (or Tests) that 1.9 actually commits to. Do not dump the entire catalog.
5. Run smoke Execution on build 480 — gate “is this build testable?”
6. Run UAT Execution on build 482 — stakeholders care about this one.
7. Run regression Execution on the same build (or a later one)—broader risk pack.
8. Failures — file defects, fix, then retest on a new Execution for the new build, still linked to the same Plan.
9. Release decision — Plan progress plus remaining BLOCKED/FAIL items, not a vibe.

That is also the spine of regression testing in Jira. The point here is only which Xray object owns which question.

Common mistakes

Using a Test Plan as an Execution

People click through a Plan as if it were a run. Progress then cannot be tied to a build. When UAT is re-run on build 483, yesterday’s results vanish or collide.

Using one Test Execution for the whole release

“1.9 testing” as a single issue mixes smoke, UAT, and late fixes. Status stops meaning anything. Split by scope and build.

Treating a Test Set as the UAT dashboard

Sets do not store a dated UAT verdict. They only store membership.

Putting results into the Test issue

Editing steps to say “failed in UAT” destroys the specification. Put the fail on the Execution and link a defect.

Plan with no exit criteria

“1.9 UAT” with 400 Tests and no definition of done becomes a junk drawer. Scope the Plan to what you will actually sign off.

Skipping BLOCKED

Environment-down items left as TODO make Plan percent-complete look better than reality.

Practical recommendation

1. Tests for specifications; never for run history.
2. Sets for packs you will schedule more than once.
3. Executions for every serious build/environment/scope slice.
4. Plans when someone needs a milestone rollup (release or UAT)—not as a beginner requirement.
5. Name Executions with version, scope, and environment.
6. Link Executions to the Plan so UAT dashboards can show both scope and latest-run evidence.
7. Retest fixes on a new build’s Execution.

If you are still mixing native Jira Tasks with Xray objects, decide the model first: Jira vs Xray. Broader process context: Jira test management guide.

Related reading

• Xray tutorial
• Xray Test Execution
• Xray Test Sets
• Jira test case management
• Regression testing in Jira
• Xray best practices

Conclusion

Xray Test Plan and Test Execution are not synonyms. The Plan is milestone scope and progress. The Execution is a dated run with results, environment, and defects. Test Sets sit beside both as reusable membership. Tests sit underneath as the specification.

Get that vocabulary right before you design a UAT dashboard. The board can only be as honest as the objects you put on it.

Improve your QA workflow faster

QA Workflow Assistant helps teams draft structured Tests from Jira stories for human review, then deliver approved coverage into Jira Cloud and Xray as Tests, Test Sets, and Test Executions. It does not replace Xray planning, and it does not execute tests for you.

→ Try QA Workflow Assistant

See Pricing and Docs for integration details.