[matthewgrossman]

Summary

This PR adds a test runner that reads a plan (which test suites run in which environments), boots each environment as a VM, runs the suites there, and fails the lane if any expected test didn't pass. On CI, the cpu-tests plan runs one smoke test on Ubuntu with rootful Docker. The plan is the only list of what runs where: the CI matrix is built from it.

Most of the runner and the tmachine Session API come from @elezar's draft #4629.

Stack

1. feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI #4773 (this PR): the runner, plans and environments, the tmachine Session API, optional test IDs and testkit-spec, and a smoke suite on one Docker environment. Its CI lane is advisory.
2. test(suites): port the universal conformance suites onto testkit and run them from the plan #4774: ports the conformance suites that every driver must pass (CLI, lifecycle, file transfer, persistent workspace) to testkit and runs them on Docker, Podman and k3s. It will also move the conformance-integration lane onto the runner. Nothing is deleted.
3. Next: ships openshell-conformance, one program that runs the conformance suite against any gateway. The host e2e lanes move to it, and the old CLI conformance suite is deleted.
4. Next: moves the features and drivers suites (Keycloak providers, OCI image, Podman user namespaces) onto plans and deletes crates/openshell-conformance.

Deferred from this PR

To later PRs in the stack:

• More environments and suites. Only one environment and one smoke test run here. test(suites): port the universal conformance suites onto testkit and run them from the plan #4774 adds the ported conformance suites on Podman and k3s.
• Gating. The plan-cpu-tests lane is advisory and not part of the Core E2E result. test(suites): port the universal conformance suites onto testkit and run them from the plan #4774 moves conformance-integration onto the runner.
• Environment-specific inputs such as Keycloak's OIDC login or Podman's reference UID map. These are decided in PR 4, before the features and drivers suites go into a plan.
• arm64. The environment is a copy of the amd64 entry, but CI has no arm64 runner with KVM yet.
• Runner robustness: a hung SSH or QMP connection, and a signal that arrives while the runner is busy with blocking work.

Dropped from #4629, not planned:

• requires, capabilities and waivers. A test that fails in one environment fails the lane. #[ignore = "<issue link>"] takes a broken test out everywhere, and test(suites): port the universal conformance suites onto testkit and run them from the plan #4774 adds a check that conformance selects the same suites in every environment.
• The catalog and matrix subcommands. --dry-run shows what a plan runs, and --emit-github-matrix covers CI.
• The target file. Tests find the gateway the way the CLI does, through OPENSHELL_GATEWAY and OPENSHELL_BIN.

Out of scope for the whole stack: version compatibility guarantees between the conformance suite and gateways, and running tests from other repositories.

Related Issue

Part of #3954

Changes

• tests/runner (openshell-test-runner): plan resolution, environments in tests/environments, JUnit result checking (a missing, skipped, or unselected test fails the lane), strict mode, and --emit-github-matrix. --env, --suite and --keep narrow a local run and keep the VM for debugging. The catalog leaves #[ignore] tests out of what a lane expects, and refuses glob workspace members it can't expand.
• tests/plans/cpu-tests.yaml: the plan. tests/environments/docker.yaml defines the one amd64 Docker environment it uses.
• tmachine: the Session API; SSH, scp, and exec timeouts; down() never panics; QEMU is cleaned up on SIGTERM; the SSH port is TMACHINE_SSH_PORT. An openshell-test-runner flake package.
• testkit, moved here from test(testkit): add the testkit library for end-to-end suites #4724 because the runner is its only consumer: an optional #[testkit::test(id = "t-…")], and a testkit-spec crate that the macro and the runner's catalog share. It refuses tests inside inline modules and tests under #[cfg]/#[cfg_attr] (or a file-level #![cfg]), because the catalog lists tests from source and can't tell whether they compile. A test without an ID is unchanged.
• tests/suites/conformance/sandbox: the smoke suite. mise run e2e:testkit runs it against a local Docker gateway.
• .github/workflows/e2e-suites.yml (reusable) and a plan-cpu-tests job in Branch E2E (label test:e2e): one job per environment from the plan, running nix run .#openshell-test-runner -- test --plan ... --env <id> --strict, with JUnit and logs uploaded. It is not part of the Core E2E result yet.
• The lockfile alignment check, enrolling the testkit and conformance workspaces. fmt, clippy, and nextest for the runner, tmachine, and suite in Branch Checks.
• TESTING.md: #[ignore = "<issue link>"] is the only way to take a broken or flaky test out of a lane.

Testing

• Checks appropriate to the affected code and behavior pass
  • Locally: fmt and clippy for testkit, the runner, tmachine and the conformance suite; the runner (50 unit, 2 integration), tmachine (8), and testkit (18) tests; the lockfile alignment check; the plan dry run; the SPDX header check.
  • The runner checks JUnit parsing against a real report from the nextest version the guest installs, and strict mode from both --strict and OPENSHELL_TEST_STRICT.
  • Branch E2E on be103d680 (run): plan-cpu-tests passed on tmachine.ubuntu2404-amd64.docker-rootful.deb with smoke: create_exec_delete selected from the plan.
• Unit tests added/updated
• E2E tests added/updated

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (TESTING.md, AGENTS.md, CONTRIBUTING.md, tests/plans/README.md, tests/testkit/README.md)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4773.docs.buildwithfern.com/openshell

[drew]

The source catalog does not account for conditional attributes, so a healthy lane can fail JUnit verification. For example:

/// Disabled on every target.
#[cfg(any())]
#[testkit::test]
async fn disabled(cx: &mut Context) -> testkit::Result { /* ... */ }

I reproduced the catalog including disabled and result verification reporting it missing, although Cargo correctly excludes it. Platform-specific #[cfg(...)] tests have the same problem, and #[cfg_attr(..., ignore)] is not recognized by the literal #[ignore] check either.

Could we resolve these attributes for the guest build, or explicitly reject unsupported conditional declarations with a clear error, rather than adding them to the expected test set? Reproduced on 1da9c3b832bdae01be7f4022bd9e030a84b2dbe4; the relevant catalog and parser behavior remains present at 4781a500c1f8e4064c1c64e8b37c963fa4a46902.

[matthewgrossman]

Thanks, good catch. Rather than evaluate cfg for the guest target, testkit-spec now refuses it: a #[cfg] or #[cfg_attr] on a #[testkit::test] function, or a file-level #![cfg] in a file that has tests, is an error from both the macro and the catalog scan, so the catalog and the build can't disagree. A test that only applies to some environments belongs in a features or drivers suite, and #[ignore = "<issue>"] remains the way to skip one everywhere. Tests are in testkit-spec (rejects_conditional_tests, scan_rejects_conditional_files_with_tests), and tests/testkit/README.md documents the rule. Fixed in 1b80fad.