feat: support optional resource config IDs

.agents/skills/changesets/SKILL.md

@@ -14,7 +14,7 @@ for the index format and release handling.
 
 | Reference                              | When to read                                                                                                 |
 | :------------------------------------- | :----------------------------------------------------------------------------------------------------------- |
-| [Migrations](references/migrations.md) | Documenting breaking changes, deprecations, or replacements, and finalizing migration versions for a release |
+| [Migrations](../../../.github/skills/changesets/references/migrations.md) | Documenting breaking changes, deprecations, or replacements, and finalizing migration versions for a release |
 
 Add a changeset when a pull request changes the public API or user-visible
 behavior of the published `@primer/agent-eval` package. Public API includes:
@@ -50,6 +50,10 @@ For breaking changes, state what consumers need to change. Do not describe
 implementation details, internal refactors, file changes, or the development
 process unless they directly affect consumers.
 
+Pin migration-guide links in changesets to a full commit SHA containing the
+guide (`blob/<sha>/...`), never `blob/main/...` or another branch name, so future
+moves cannot break published release notes.
+
 Prefer:
 
 > Add support for filtering scenarios by multiple tags.

.changeset/cool-readers-glow.md

@@ -0,0 +1,5 @@
+---
+'@primer/agent-eval': minor
+---
+
+Add optional resource IDs and derive IDs from relative paths when omitted. Pin existing IDs to preserve plans and result history; see the [migration guide](https://github.com/primer/agent-eval/blob/6a2874cc88476ee69798baa3148ca3895fe45c9f/packages/agent-eval/docs/migrations/resource-identity.md).

.github/skills/changesets/references/migrations.md

@@ -6,9 +6,9 @@ Author concise guides with the implementation, even when no changeset is needed.
 
 ## Contents
 
-- [Index](#index)
-- [Guides](#guides)
-- [Release handling](#release-handling)
+- [Index](#index), L:13
+- [Guides](#guides), L:24
+- [Release handling](#release-handling), L:43
 
 ## Index
 
@@ -31,6 +31,10 @@ Include only what is needed to perform and verify the migration:
   configuration. Use npm/npx for consumer commands.
 - Verification steps using existing project checks or observable outcomes.
 
+Describe public behavior and what consumers need to change, not internal
+mechanisms. For derived values, explain their dependencies and stability rather
+than algorithms or internal representations.
+
 Add explicit migration steps or compatibility notes only when the examples
 are insufficient. Verify examples against the relevant releases and link
 supporting evidence inline rather than adding a separate evidence section.
@@ -43,4 +47,7 @@ While the package is pre-1.0, use the next minor version from
 (for example, `0.16.0` → `0.17.0`). Revisit this policy at 1.0 when the semver
 policy changes.
 Preserve released examples and stable guide filenames. Link the guide from the
-changeset when one is needed.
+changeset when one is needed, using a full commit SHA containing that version
+of the guide: `https://github.com/<owner>/<repo>/blob/<sha>/<path>`. Do not link
+to `main` or another branch: the guide may move or change after release.
+Commit the guide first, then use that commit's SHA in the changeset link.

AGENTS.md

@@ -12,6 +12,11 @@
   `--filter` option to target individual workspaces
 - README files are public-facing and should only contain information about
   using the package. Keep contributor and agent instructions elsewhere
+- The top-level `docs/` directory documents public API concepts, abstractions,
+  and user-facing flows. Explain what behavior depends on and when it changes,
+  not its internal mechanism. For example, document which paths determine an
+  ID and how renames affect it, not hash algorithms, input prefixes, or encoding.
+  Keep implementation details in contributor or agent references when needed.
 - Do not create changesets for updates to features, fixes, or other changes
   that have not been released yet
 - For backwards-incompatible public API or behavior changes, deprecations, or

docs/benchmarks.md

@@ -14,6 +14,16 @@ parts of icon usage that you care about.
 Configuration for benchmarks live in the `/benchmarks` folder. You can add a
 benchmark by adding a file to this folder and using `defineConfig` from `@primer/agent-eval/benchmark`.
 
+Set an optional `id` to keep a benchmark's identity stable across file renames
+and moves. Otherwise, its ID depends on the file's path relative to the
+configured benchmarks directory, excluding the file extension. Changing the
+display `name` or moving the entire benchmarks directory does not change that ID.
+
+Explicit IDs must be nonblank and unique among discovered benchmarks. You can
+still select benchmarks by filename without the extension. See the
+[resource identity migration](../packages/agent-eval/docs/migrations/resource-identity.md)
+when reusing existing plans or result history.
+
 ```ts
 // benchmarks/example.ts
 import {defineConfig} from '@primer/agent-eval/benchmark'

docs/experiments.md

@@ -19,6 +19,16 @@ which approach performed best given the specific scenario, model, and treatment.
 Configuration for experiments live in the `/experiments` folder. You can add a
 experiment by adding a file to this folder and using `defineConfig` from `@primer/agent-eval/experiment`.
 
+Set an optional `id` to keep an experiment's identity stable across file renames
+and moves. Otherwise, its ID depends on the file's path relative to the
+configured experiments directory, excluding the file extension. Changing the
+display `name` or moving the entire experiments directory does not change that ID.
+
+Explicit IDs must be nonblank and unique among discovered experiments. You can
+still select experiments by filename without the extension. See the
+[resource identity migration](../packages/agent-eval/docs/migrations/resource-identity.md)
+when reusing existing plans or result history.
+
 ```ts
 // experiments/example.ts
 import {defineConfig} from '@primer/agent-eval/experiment'

docs/scenarios.md

@@ -10,6 +10,17 @@ As an example, you may want to evaluate how well an agent adds a search feature
 
 You can add a scenario by creating a folder in `/scenarios` with a `package.json`, the files the agent needs to start the task, and a `scenario.config.ts` file. Use `defineConfig` from `@primer/agent-eval/scenario` to configure the scenario.
 
+Set an optional `id` to keep a scenario's identity stable across directory
+renames and moves. Otherwise, its ID depends on the scenario directory's path
+relative to the configured scenarios directory. Moving the entire scenarios
+directory does not change that ID.
+
+Explicit IDs must be nonblank and unique among discovered scenarios. You can
+still select scenarios by directory name. Inline `{path, name?}` references resolve to the same identity
+as string references; `name` no longer overrides the ID. See the
+[resource identity migration](../packages/agent-eval/docs/migrations/resource-identity.md)
+when reusing existing plans or result history.
+
 ```ts
 // scenarios/001-agent-scenario/scenario.config.ts
 import {defineConfig} from '@primer/agent-eval/scenario'

packages/agent-eval/docs/AGENTS.md

@@ -22,7 +22,7 @@ not a prerequisite for every operation.
 | Read results or diagnose agent behavior                     | [Trials and results](references/trials-and-results.md), [experiment reports](references/experiments.md#reports) |
 | Start the unstable, feature-flagged UI or find CLI help     | [CLI](references/cli.md)                                                                                        |
 | Enable or disable unstable features                         | [Feature flags](references/feature-flags.md)                                                                    |
-| Upgrade existing APIs, configuration, or workflows          | [Migrations](migrations/README.md)                                                                              |
+| Upgrade existing APIs, configuration, or workflows          | [Migrations](migrations/README.md), [resource identity](migrations/resource-identity.md)                        |
 
 For an evidence-driven investigation of why a model behaved a certain way, use
 the `agent-eval-investigate` skill (installed separately with

packages/agent-eval/docs/migrations/README.md

@@ -5,9 +5,10 @@ guides to migrate and verify your usage.
 
 ## Guides
 
-| Version introduced | Entrypoint           | export(s)                          | description                                                                       |
-| :----------------- | :------------------- | :--------------------------------- | :-------------------------------------------------------------------------------- |
-| 0.13.0             | `@primer/agent-eval` | `getExperiment`, `listExperiments` | [Supply a benchmarks directory when loading experiments](experiment-discovery.md) |
+| Version introduced | Entrypoint                                                                                     | export(s)                          | description                                                                            |
+| :----------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------- | :------------------------------------------------------------------------------------- |
+| 0.19.0             | `@primer/agent-eval/benchmark`, `@primer/agent-eval/experiment`, `@primer/agent-eval/scenario` | `defineConfig`                     | [Pin resource IDs to preserve existing plans and result history](resource-identity.md) |
+| 0.13.0             | `@primer/agent-eval`                                                                           | `getExperiment`, `listExperiments` | [Supply a benchmarks directory when loading experiments](experiment-discovery.md)      |
 
 This index is not a complete history of breaking changes before this section
 was introduced. Consult the package changelog for releases not covered here.

packages/agent-eval/docs/migrations/resource-identity.md

@@ -0,0 +1,55 @@
+# Resource identity
+
+In 0.19.0, benchmarks, experiments, and scenarios accept an optional nonblank
+`id`. Without it, identity depends on the path relative to the configured
+resource root. Benchmark and experiment paths omit the
+module extension; scenario paths identify the directory.
+
+Filename and directory selectors continue to work. Explicit IDs are used
+unchanged and must be unique within each resource kind's discovery scope.
+Moving a resource root preserves derived IDs; moving a resource within that
+root changes its derived ID.
+
+## Preserve existing identity
+
+Before (0.18.1, `scenarios/login/scenario.config.ts`, ID `login`):
+
+```ts
+import {defineConfig} from '@primer/agent-eval/scenario'
+
+export default defineConfig({
+  prompt: 'Add a login form.',
+})
+```
+
+After (0.19.0, same ID even if the directory is renamed):
+
+```ts
+import {defineConfig} from '@primer/agent-eval/scenario'
+
+export default defineConfig({
+  id: 'login',
+  prompt: 'Add a login form.',
+})
+```
+
+To reuse existing plans or continue result history, set `id` to the previously
+recorded ID on every referenced resource: the benchmark or experiment and its
+scenarios. Before 0.19.0, benchmark and experiment IDs are filenames without
+extensions; scenario IDs are directory names or inline `name` overrides.
+Inline `name` no longer overrides identity; pin the scenario's config `id`
+instead. A single scenario now has the same identity through string and inline
+references.
+
+If preserving history is unnecessary, omit `id` and regenerate plans after
+upgrading. Existing result files remain readable, but old and new IDs represent
+separate histories.
+
+## Verify
+
+Create a plan with
+`npx agent-eval experiment plan create comparison --output-path ./comparison-plan.json`.
+Check its top-level `id` and trial `scenarioId` values against the IDs you intend
+to preserve. After renaming a pinned resource, update path references and create
+another plan; those IDs must remain unchanged. The CLI continues to select
+resources by their filename or directory name as well as their resolved ID.

packages/agent-eval/docs/references/benchmarks.md

@@ -10,10 +10,18 @@ time. Use an [experiment](experiments.md) to compare multiple interventions.
 | File and import      | `benchmarks/<id>.ts`; `defineConfig` from `@primer/agent-eval/benchmark` |
 | Export               | Named `benchmark` or default                                             |
 | Required fields      | `name`, `description`, `models`, `capabilities`                          |
-| Optional fields      | `setup`, `runners`                                                       |
+| Optional fields      | `id`, `setup`, `runners`                                                 |
 | CLI identifier       | Filename without extension, not display `name`                           |
 | Automatic treatments | `Control` and `Benchmark`                                                |
 
+Set `id` to a nonblank string to pin identity across renames. Otherwise, the ID
+depends on the file's path relative to the configured benchmarks directory,
+excluding its extension. Changing the display name or moving the entire
+benchmarks directory does not change the ID. Explicit IDs are used unchanged.
+IDs must be unique among discovered benchmarks. CLI selection and
+experiment references accept the filename without extension or the resolved ID;
+resolved IDs take precedence.
+
 Benchmark-level `setup` runs only for `Benchmark`. Capability-level setup
 runs for **both** treatments, before treatment setup. Put neutral prerequisites
 in capability setup and the resource being measured in benchmark setup.

packages/agent-eval/docs/references/experiments.md

@@ -11,12 +11,20 @@ plugins, or execution backends on the same tasks. Use a
 | File and import     | `experiments/<id>.ts`; `defineConfig` from `@primer/agent-eval/experiment`           |
 | Export              | Use a default export; named `experiment` also supported                              |
 | Required fields     | `name`, `description`, `models`, `treatments`, and either `scenarios` or `benchmark` |
-| Optional fields     | `setup`, `runners`                                                                   |
+| Optional fields     | `id`, `setup`, `runners`                                                             |
 | CLI identifier      | Filename without extension, not display `name`                                       |
 | Scenario references | Folder IDs; alternatively `{path, name?}`                                            |
 | Benchmark reference | Benchmark filename ID; selects its capabilities and scenarios                        |
 | Runner default      | `copilot-cli`                                                                        |
 
+Set `id` to a nonblank string to pin identity across renames. Otherwise, the ID
+depends on the file's path relative to the configured experiments directory,
+excluding its extension. Changing the display name or moving the entire
+experiments directory does not change the ID. Explicit IDs are used unchanged.
+IDs must be unique among discovered experiments. CLI selection
+accepts the filename without extension or the resolved ID; resolved IDs take
+precedence.
+
 Top-level `setup` runs for **all trials, including control**. Put only neutral
 prerequisites there. Treatment setup runs afterward and installs the intervention.
 Do not put the tested resource in shared setup or the fixture.
@@ -114,7 +122,8 @@ treatments. A scenario shared by two capabilities runs once per capability.
 runner and resource confounds their effects; see [models and runners](models-and-runners.md).
 
 For inline scenario references, `path` resolves from the host working directory,
-not the experiment file. `name` optionally overrides the scenario ID.
+not the experiment file. Set `id` in the scenario's config to pin identity;
+inline `name` does not override it.
 
 Repeat close comparisons in separate output directories; there is no repetition
 field. Inspect per-scenario regressions and use held-out tasks before claiming

packages/agent-eval/docs/references/plans.md

@@ -23,8 +23,13 @@ resource contents. Execution reloads local configurations and resolves the
 saved identifiers. Preserve the source revision, dependency versions, resource
 contents, and directory layout with the plan.
 
-Configuration IDs come from filenames. Capability and treatment IDs depend on
-their names. Renaming or removing referenced entities can invalidate a plan.
+Benchmark, experiment, and scenario IDs use explicit config `id` values when
+provided. Otherwise, IDs depend on the path relative to the
+configured resource directory (without the module extension for benchmarks and
+experiments). Pin IDs before moving or renaming resources to keep plans
+resolvable. Capability and treatment IDs still depend on their names.
+Renaming or removing referenced entities can invalidate a plan. See
+[resource identity migration](../migrations/resource-identity.md) for existing plans.
 
 ## Create and run
 

packages/agent-eval/docs/references/scenarios.md

@@ -5,15 +5,28 @@ evaluation. Benchmarks and experiments reuse the same scenario across treatments
 
 ## Contract
 
-| Item              | Rule                                                              |
-| :---------------- | :---------------------------------------------------------------- |
-| Required files    | `package.json` and `scenario.config.ts`                           |
-| Import and export | `defineConfig` from `@primer/agent-eval/scenario`; default export |
-| Required field    | `prompt`, passed to the implementation agent                      |
-| Optional fields   | `description`, `tags`, `checks`, `judges`, `image`, `setup`       |
-| Defaults          | Empty tags, checks, and judges                                    |
-| Identity          | Directory name by default                                         |
-| Evaluation files  | Declare in check/judge `files`; restored during evaluation        |
+| Item              | Rule                                                                |
+| :---------------- | :------------------------------------------------------------------ |
+| Required files    | `package.json` and `scenario.config.ts`                             |
+| Import and export | `defineConfig` from `@primer/agent-eval/scenario`; default export   |
+| Required field    | `prompt`, passed to the implementation agent                        |
+| Optional fields   | `id`, `description`, `tags`, `checks`, `judges`, `image`, `setup`   |
+| Defaults          | Empty tags, checks, and judges                                      |
+| Identity          | Explicit config `id`, otherwise derived from the relative directory |
+| Evaluation files  | Declare in check/judge `files`; restored during evaluation          |
+
+Set `id` to a nonblank string to pin identity across renames. Otherwise, the ID
+depends on the scenario directory's path relative to the configured scenarios
+directory. Moving the entire scenarios directory does not change the ID.
+The ID depends on the directory, not the config filename. Explicit IDs are used
+unchanged. IDs must be unique among discovered scenarios.
+
+CLI selection and string references accept the directory name or resolved ID;
+resolved IDs take precedence. Inline `{path, name?}` references use the
+same identity as string references; `name` no longer overrides identity. Paths
+outside the configured scenarios root use a relative path beginning with `../`;
+set an explicit config `id` when that layout may change. Discovery still checks
+immediate child directories, not nested fixtures.
 
 Add [checks](checks.md), [judges](judges.md), or both before treating a run as a
 quality evaluation. Empty graders are valid configuration but provide no

packages/agent-eval/src/benchmark/config.ts

@@ -44,6 +44,14 @@ const CapabilityConfigSchema = z.object({
 })
 
 const BenchmarkConfigSchema = z.object({
+  id: z.optional(
+    z.string().check(
+      z.minLength(1),
+      z.refine(value => {
+        return value.trim().length > 0
+      }),
+    ),
+  ),
   name: z.string(),
   description: z.string(),
   models: z.array(ModelVariantConfigSchema),