Xray Test Execution: How to Run and Track Tests in Jira

Xray Test Execution is where planning stops being theoretical. A Test Set or Test Plan can define scope beautifully and still fail your release if nobody runs the right tests against the right build with results you can defend. Test Execution is the Xray issue type that records that run: which tests, who ran them, in which environment, on which build, with which outcomes and defects.

This article walks through creating executions, adding tests and sets, assigning work, stepping through results, handling failures, and keeping Plan ↔ Execution relationships honest. It complements the broader Jira test management guide. For suite design see Xray Test Sets; for project conventions see Xray best practices; for end-to-end scheduling patterns see the Jira test execution guide.

What a Test Execution issue is

In Xray, a Test Execution is a Jira issue that represents a concrete run of one or more Tests. Think of it as a work order plus a results ledger.

A healthy execution usually captures:

• The list of Tests included (often sourced from a Test Set or Test Plan)
• Assignees (overall or per test, depending on how your team works)
• Environment (staging, QA, pre-prod, production-like)
• Build or revision identifier
• Planned start/end or sprint context
• Per-test and per-step statuses as work progresses
• Links to defects found during the run
• Traceability back to the Test Plan for the release

It is not a permanent suite definition. If you find yourself editing the same execution for three sprints, you wanted a Test Set or Plan change, not an immortal execution.

Execution vs Set vs Plan (quick map)

Getting this triangle right is half of Xray literacy. Mislabeling an execution as “the regression suite” is how catalogs rot. If the Plan vs Execution distinction is the part that is unclear—especially for UAT dashboards—read Xray Test Plan vs Test Execution.

Creating a Test Execution

UI labels differ slightly between Xray Cloud and Data Center, but the workflow is stable.

From a Test Set

1. Open the Test Set (for example your smoke set from Xray Test Sets).
2. Use the action to create a Test Execution from the set.
3. Confirm the Tests were copied/linked into the execution in the intended order.
4. Fill environment, build, and description fields before anyone starts clicking Pass.

From a Test Plan

1. Open the release Test Plan.
2. Create a Test Execution for a subset or for the full planned scope, following your team’s slicing rules (smoke first, then critical, then feature packs).
3. Ensure the execution remains associated with the plan so progress rolls up.

From scratch

Creating an empty execution and adding Tests manually is valid for hotfixes and exploratory-driven confirmation runs. It is a poor default for release regression because membership will be inconsistent between builds.

Naming that survives an audit

Prefer:

• TE — Smoke — staging — build 184
• TE — Checkout feature — QA — 2026-08-21

Avoid:

• Test execution
• Sprint stuff
• Final final run 2

Include build and environment in the summary or description every time. Future incident reviews will thank you.

Adding tests and Test Sets to an execution

Adding individual Tests

Add Tests when:

• A late story needs targeted validation
• A defect fix requires a small confirmation pack
• You are extending an in-flight run with one missing case (sparingly)

Keep titles and steps review-ready before adding. Weak cases waste execution time; fix authoring with how to create test cases in Jira and how to write QA test cases rather than improvising during the run.

Adding a Test Set

Adding a set is the standard path for reusable scope. Benefits:

• Consistent membership across builds
• Ordered lists for efficient manual flow
• Clear negotiation (“smoke is TESTSET-12, not whatever we remembered”)

After adding, spot-check that obsolete Tests did not sneak in. Suite hygiene still belongs to set owners.

Ad-hoc vs planned membership

Assigning work

Assignment models that work in practice:

Single owner per execution

Best for small smoke runs. One tester owns the ledger, asks for help only on blocked items.

Split by tester within one execution

Useful when a feature pack is large. Assign groups of Tests to different people. Agree on environment and build first so you are not comparing results from mismatched deploys.

Split by multiple executions

Often cleaner for reporting: TE — Auth — build 184 and TE — Payments — build 184 under the same Plan. Parallelizes cleanly and avoids merge conflicts in comments and statuses.

RACI-style clarity for a run

Ambiguity here produces “I thought you ran payments.”

Running steps and recording results

Before the first Pass/Fail

Confirm:

1. Correct environment URL and credentials
2. Build hash / version matches what release claims
3. Test data preconditions exist (accounts, feature flags, seed data)
4. Known outages are announced so false failures do not flood Jira

Executing a manual test

Open the Test inside the execution. Work step by step:

• Mark steps passed as you go, or mark the test after completing all steps—pick a team convention and stick to it
• Attach screenshots or logs on failures while context is fresh
• Capture actual results in the failure comment, not only “doesn’t work”
• Escalate environment blockers distinctly from product defects

Statuses you will actually use

Xray supports a set of execution statuses; projects sometimes customize them. Common meaningful states:

The failure mode to avoid: using Passed for “looked fine while multitasking” and Failed for “I got confused.” Statuses are evidence. Treat them that way.

Step-level results help on long flows; test-level status may be enough for tiny smoke checks. Consistency beats perfectionism—document the convention in your Xray tutorial onboarding notes.

Defects: linking failures without chaos

When a test fails: reproduce once if cheap; create or link a Bug with environment, build, steps, expected vs actual; link it to the failed test/execution; leave the test Failed until verified; retest in a new execution rather than rewriting history casually.

A linked defect should answer: build/environment, account/data, exact steps, expected vs observed, and whether it is new, regression, or known. Vague bugs attached to precise tests still waste engineering time.

Environments and builds

Why these fields are not optional

Without environment and build:

• You cannot compare two runs
• You cannot tell whether a failure is already fixed
• You cannot support release decisions with evidence

Put build identifiers in the execution summary or a dedicated custom field. Use the same format the pipeline emits (git SHA, CI build number, or artifact version).

Environment matrix example

Same build, different environments, different questions. Do not merge them into one ambiguous run.

Data and flag drift

Many “Xray flakes” are environment drift: feature flags differ, message queues lag, or third-party sandboxes reset. Call that out in blocked reasons instead of failing product tests for infrastructure lies.

Test Plan relationship

The Plan is the scoreboard for the milestone. Executions are the plays.

Healthy pattern:

1. Plan includes Sets and/or Tests for the release
2. Executions are created from that planned scope against successive builds
3. Progress, coverage, and remaining risk are read from the Plan
4. Late scope changes update the Plan deliberately (and the Sets if reuse changed)

Unhealthy pattern:

• Executions exist with no Plan link for a named release
• Plan membership diverges silently from what people actually run
• Stakeholders watch dashboards that no longer match tester behavior

For scheduling alternatives and ceremony timing, see the Jira test execution guide.

Automation notes (high level)

Xray can ingest automated results via REST, CI plugins, or report imports depending on your stack. Keep results in an execution context (job, build, environment); map tests to maintained Xray Test issues; never treat a flaky CI job as manual sign-off; align smoke automation with the same intent as your manual smoke set; treat import failures as pipeline defects.

Automation changes who presses Pass—not the need for statuses, defect links, and Plan rollup. Structure cases with a QA test case template and keep regression test cases intentional rather than “everything in CI.”

Common mistakes in Xray Test Execution

1. Reusing one execution across many builds. Results become incomprehensible. New build → new execution (or a clear retest protocol your team documents).
2. Empty environment/build fields. Evidence without coordinates is rumor.
3. Silent skips. Leaving tests Todo at release time without Plan acknowledgment hides risk.
4. Passing blocked tests. Sympathy is not a status.
5. Filing defects with no execution link. Breaks the investigative trail.
6. Mega-executions with 300 tests and five owners. Prefer sliced executions under one Plan.
7. Changing set membership mid-flight and assuming old executions update magically. Understand what your Xray version does; communicate changes.
8. No distinction between product fail and environment fail. Both look red; only one should page developers.
9. Ad-hoc scope for every release. Ignores Test Sets and invents membership under pressure.
10. Ignoring smoke-first order. Deep regression on a build that cannot log in wastes the day. Revisit smoke testing vs sanity testing.

Best practices

• Name executions with scope + env + build
• Prefer creating from Sets/Plans
• Smoke first, then risk, then feature depth
• Assign with explicit ownership
• Record actual results on failures immediately
• Link defects with build/env context
• Keep Plans as the release source of truth
• Retest fixes in a fresh, traceable way
• Align case quality before the run (how to write QA test cases)
• Follow team-wide Xray best practices

Example walkthrough: staging build 184

Context: Release candidate build 184 deployed to staging. Plan PLAN-22 already includes smoke and checkout feature sets.

1. Create TE — Smoke — staging — build 184 from the smoke Test Set; assign Alex.
2. Alex confirms flags and test users, runs smoke in order, all pass in 20 minutes.
3. Create TE — Checkout — staging — build 184 from the checkout set; split assignments between Alex and Sam.
4. Sam hits a payment failure on step 4 of TEST-441; files BUG-908 with HAR + screenshot; marks the test Failed; leaves adjacent tests Running/Todo per team rule.
5. Dev deploys build 185. Team creates new executions for smoke + the failed checkout slice rather than rewriting 184 history.
6. Plan PLAN-22 shows remaining failed/retest items; release manager reads Plan status, not chat lore.

That is Test Execution doing its job: evidence, not vibes.

Conclusion

Xray Test Execution turns suite definitions into auditable results. Create runs from Sets and Plans, label them with environment and build, assign clearly, record honest statuses, and link defects with context. Keep Plans as the milestone scoreboard and Sets as reusable membership—executions should stay ephemeral and precise. For the surrounding system, return to the Jira test management guide and deepen suite design with Xray Test Sets.

FAQ

What is an Xray Test Execution?

It is a Jira issue that records a specific run of Tests—including membership, assignees, environment, build, statuses, and linked defects—usually tied to a Test Plan.

Should I create a new Test Execution for every build?

Yes in most teams. Mixing multiple builds into one execution destroys comparability. Use a documented retest flow if you must continue an existing issue.

Can one execution include multiple Test Sets?

Often yes—your Plan may compose several sets into one run, or you may create separate executions per set. Prefer clarity over stuffing unrelated scopes together.

How do Test Executions relate to Test Plans?

Plans define and track milestone scope and progress. Executions provide the detailed results that roll up into that progress when linked correctly.

What status should I use when the environment is down?

Use Blocked (or your project’s equivalent), not Failed or Passed. Failed should mean the product under test did not meet the expected result.

How should automation report into Xray?

Import or publish results into execution contexts keyed by build and environment, mapped to maintained Test issues—not as anonymous one-off dumps.

Who should own signing off an execution?

Typically the QA lead is accountable for result integrity; individual testers remain responsible for the tests they ran. Release owners use Plan-level status for go/no-go.

Improve your QA workflow faster

QA Workflow Assistant helps QA engineers generate structured test cases from Jira stories, improve coverage, and organize testing workflows.

→ Try QA Workflow Assistant

See Pricing and Docs for plans and Jira/Xray integration details.