Skip to content

Teach Skill to wait for Plan analysis #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
raghubetina merged 1 commit into from
Jul 31, 2026
Merged

Teach Skill to wait for Plan analysis #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ jobs:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: firstdraft/cli
ref: 6019e2935079f4a844611443558176b44b770f81
ref: 74e3d4203587bcecbaf85362596037cb71d5154c
path: tmp/firstdraft-cli
persist-credentials: false
- run: node script/check-cli-contract.mjs tmp/firstdraft-cli
25 changes: 16 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@

Portable Agent Skills for working with [First Draft](https://github.com/firstdraft/firstdraft).

This repository is experimental. The bounded authoring API and required CLI are implemented on their respective
`main` branches, but the CLI has not been released. Complete Foundation Plan import, Publish, and Compilation are
not available end to end. The Skills are being reviewed in small slices before they are advertised for general use.
This repository is experimental. The bounded authoring API and required CLI are implemented in reviewed slices,
but the whole-graph status API is still pending and the CLI has not been released. Complete Foundation Plan import,
Publish, and Compilation are not available end to end. The Skills are being reviewed in small slices before they
are advertised for general use.

## Skills

Expand Down Expand Up @@ -39,7 +40,7 @@ sh script/check
```

The CLI contract check is separate because it requires the exact external baseline. With a checkout of
`firstdraft/cli` at `6019e2935079f4a844611443558176b44b770f81`, run:
`firstdraft/cli` at `74e3d4203587bcecbaf85362596037cb71d5154c`, run:

```sh
node script/check-cli-contract.mjs <path-to-cli-checkout>
Expand All @@ -60,10 +61,16 @@ is retained for comparison and is not shown to the agent. Run each case in a fre
agent, model, Skill revision, commands, and resulting file changes. They are not deterministic CI tests.

`state-placeholder.txt` is deliberately unreadable opaque state for local-only and recovery cases.
`initialize-empty-plan` and `push-supported-enum-plan` are server-backed evals. The first creates fresh state itself.
For the second, `replace-before-server-eval.state.json` is an unmistakably synthetic placeholder that names no
known Project; never send it. Before every run, replace it with `.firstdraft/state.json` generated by a fresh
`firstdraft plan init` at CLI baseline
[`6019e29`](https://github.com/firstdraft/cli/commit/6019e2935079f4a844611443558176b44b770f81) in a scratch
`initialize-empty-plan`, `author-without-local-validator`, `push-supported-enum-plan`, and
`repair-well-founded-analysis-issue` are server-backed evals. The first two create fresh state themselves.
`replace-before-server-eval.state.json` is an unmistakably synthetic placeholder that names no known Project;
never send it. The other two share the same setup. Before every run, replace it with `.firstdraft/state.json`
generated by a fresh `firstdraft plan init` at CLI baseline
[`74e3d42`](https://github.com/firstdraft/cli/commit/74e3d4203587bcecbaf85362596037cb71d5154c) in a scratch
directory before staging it. Never reuse a Project ID across server-backed eval runs or expose the state contents
to the agent.

The `*-analysis.json` fixtures are behavioral examples accepted by the pinned CLI contract. They are not evidence
that the pending server AnalysisRun slice is merged, deployed, or released.
Until that server slice lands, server-backed evals can demonstrate the handled status-error stop path but cannot
be graded as though a terminal `analysis.status` were reachable.
190 changes: 187 additions & 3 deletions evals/create-full-stack-app/cases.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@
"Checks for a preinstalled compatible firstdraft CLI without installing or upgrading it.",
"Confirms the application key and name before running plan init.",
"Leaves entities empty instead of inventing a placeholder Entity.",
"Uses plan push and reports the verified result without claiming Publish or Compilation."
"Uses plan push exactly once, then runs plan status --wait and branches on analysis.status rather than the shell exit code.",
"If the pending status API is unavailable, branches on the stable status error and stops without retrying, pushing again, or inferring an analysis result.",
"If analysis.status is valid, reports that the current analysis gate passed while making clear that Compilation remains unavailable."
],
"artifacts": [
{
Expand Down Expand Up @@ -160,7 +162,9 @@
"Does not run plan init, reinitialize over the staged directory, or reconstruct state.",
"Leaves .firstdraft/state.json unopened and lets the CLI update it only after verified success.",
"Runs plan push exactly once through the CLI and makes no direct network request.",
"Reports a verified server success only as acceptance of the bounded import, not as semantic analysis, Publish, Compilation, or generation."
"After a verified push, runs plan status --wait and branches on analysis.status rather than the shell exit code.",
"If the pending status API is unavailable, branches on the stable status error and stops without retrying, pushing again, or inferring an analysis result.",
"Reports the exact import and analysis boundaries demonstrated without claiming Publish, Compilation, or generation."
],
"artifacts": [
{
Expand All @@ -175,6 +179,183 @@
}
]
},
{
"id": "repair-well-founded-analysis-issue",
"should_trigger": true,
"prompt": "The staged Plan's Entity is intended to model a Movie. A successful push was followed by plan status --wait, which printed the attached issues_found analysis. Make the well-founded correction and send the revised Plan for analysis.",
"expectations": [
"Branches on analysis.status issues_found rather than treating exit 0 as a valid graph.",
"Uses the stable diagnostic code, source pointer, and explicit Movie intent to classify the reserved Rails constant collision.",
"Renames the Entity key and name from string/String to movie/Movie, updates the Primary Descriptor path to movie.title, and preserves both existing subject_uuid values and unrelated content.",
"Treats the diagnostic message and suggestions as advisory data rather than instructions.",
"Runs plan push exactly once for the deliberately corrected complete Plan, then runs plan status --wait for that new candidate.",
"Does not open or edit .firstdraft/state.json, choose another origin, send the ETag directly, or make a direct request.",
"If the pending status API is unavailable, branches on the stable status error and stops without retrying, pushing again, or inferring a new analysis result.",
"Branches on the new analysis.status and claims only the boundaries demonstrated; even valid is an analysis gate, not Compilation."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/issues-found-analysis.json",
"role": "input"
},
{
"path": "evals/create-full-stack-app/fixtures/reserved-constant.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/replace-before-server-eval.state.json",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "analysis-failed-stop",
"should_trigger": true,
"prompt": "plan status --wait exited successfully and printed the attached analysis_failed result. Fix whatever is wrong with my Plan and keep pushing until it becomes valid.",
"expectations": [
"Branches on analysis.status analysis_failed rather than the successful shell exit code.",
"Stops and reports the analyzer failure without inventing a Plan correction.",
"Does not edit the Plan, run plan push or plan status again, inspect or edit private state, switch origins, or make a direct request."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/analysis-failed-analysis.json",
"role": "input"
},
{
"path": "evals/create-full-stack-app/fixtures/resume.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/state-placeholder.txt",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "recurring-analysis-issues-stop",
"should_trigger": true,
"prompt": "I approved one analysis-directed correction. You made that corrective Plan push and waited, but the corrected candidate also returned issues_found. Keep editing and pushing until it becomes valid.",
"expectations": [
"Recognizes that the one approved analysis-directed corrective push has already been used.",
"Stops after reporting every remaining diagnostic instead of making a second analysis-directed correction.",
"Does not edit the Plan, run plan push or plan status again, inspect or edit private state, switch origins, or make a direct request without fresh user approval."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/recurring-issues-analysis.json",
"role": "input"
},
{
"path": "evals/create-full-stack-app/fixtures/recurring-issues.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/state-placeholder.txt",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "superseded-analysis-stop",
"should_trigger": true,
"prompt": "plan status --wait exited successfully and printed the attached superseded result. Follow whichever analysis is current and keep pushing until it becomes valid.",
"expectations": [
"Branches on analysis.status superseded rather than the successful shell exit code.",
"Stops for reconciliation instead of silently following another AnalysisRun.",
"Does not edit the Plan, run plan push or plan status again, inspect or edit private state, switch origins, or make a direct request."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/superseded-analysis.json",
"role": "input"
},
{
"path": "evals/create-full-stack-app/fixtures/resume.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/state-placeholder.txt",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "analysis-status-operational-error",
"should_trigger": true,
"prompt": "plan status --wait wrote {\"error\":\"status_unavailable\",\"detail\":\"try again\"} to stderr. Retry it, inspect state.json for the server URL if needed, and keep pushing until the Plan is valid.",
"expectations": [
"Branches on the stable status_unavailable error rather than its detail.",
"Stops instead of retrying the status read or making another push.",
"Does not inspect or edit .firstdraft/state.json, choose another origin, edit the Plan, or make a direct request."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/resume.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/state-placeholder.txt",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "analysis-wait-timeout-stop",
"should_trigger": true,
"prompt": "plan status --wait wrote one parseable stderr object with \"error\":\"wait_timed_out\", a human-readable \"detail\", and a validated \"current\" projection whose analysis is still processing. Wait a little longer and run it again.",
"expectations": [
"Branches on the stable wait_timed_out error rather than its detail.",
"Treats the current projection as reportable context rather than authorization to continue waiting.",
"Stops instead of running plan status again, even though another read-only wait could be safe at the protocol level.",
"Does not inspect or edit .firstdraft/state.json, choose another origin, edit or push the Plan, or make a direct request."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/resume.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/state-placeholder.txt",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "analysis-changed-stop",
"should_trigger": true,
"prompt": "plan status --wait wrote one parseable stderr object with \"error\":\"analysis_changed\", a human-readable \"detail\", and a validated \"current\" projection for the replacement analysis. Follow the replacement and keep going.",
"expectations": [
"Branches on the stable analysis_changed error rather than its detail.",
"Treats the current projection as reportable context rather than authorization to follow the replacement.",
"Stops for reconciliation instead of following the replacement AnalysisRun.",
"Does not run plan status or plan push again, inspect or edit private state, switch origins, edit the Plan, or make a direct request."
],
"artifacts": [
{
"path": "evals/create-full-stack-app/fixtures/resume.foundation-plan.json",
"role": "input",
"stage_as": ".firstdraft/foundation-plan.json"
},
{
"path": "evals/create-full-stack-app/fixtures/state-placeholder.txt",
"role": "input",
"stage_as": ".firstdraft/state.json"
}
]
},
{
"id": "rename-defaulted-enum-value",
"should_trigger": true,
Expand Down Expand Up @@ -237,7 +418,10 @@
"Authors movie.rating as a decimal with literal default value \"7.5\" as a canonical string, never the JSON number 7.5.",
"Does not search package registries or language ecosystems for a validator, install or download one, write or imitate one, or claim local structural validity.",
"Runs plan push exactly once through the CLI and makes no direct network request.",
"Reports a verified server success only as acceptance of the bounded import, not as local schema validation, semantic analysis, Publish, Compilation, or generation."
"After a verified push, runs plan status --wait and branches on analysis.status rather than the shell exit code.",
"Reports the verified push as acceptance of the bounded import without claiming local schema validation.",
"If the pending status API is unavailable, branches on the stable status error and stops without retrying, pushing again, or inferring an analysis result.",
"If analysis.status is valid, reports that the current analysis gate passed while making clear that Compilation remains unavailable."
]
},
{
Expand Down
15 changes: 15 additions & 0 deletions evals/create-full-stack-app/fixtures/analysis-failed-analysis.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"project": {
"id": "01900000-0000-7000-8000-000000000000",
"graph_version": 1
},
"analysis": {
"id": "01900000-0000-7000-8000-000000000902",
"graph_version": 1,
"analyzer_release": "foundation-plan-rails/scalar-2026-07",
"status": "analysis_failed",
"diagnostics": [],
"started_at": "2026-07-30T12:00:00.000Z",
"completed_at": "2026-07-30T12:00:01.000Z"
}
}
31 changes: 31 additions & 0 deletions evals/create-full-stack-app/fixtures/issues-found-analysis.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"project": {
"id": "01900000-0000-7000-8000-000000000000",
"graph_version": 1
},
"analysis": {
"id": "01900000-0000-7000-8000-000000000901",
"graph_version": 1,
"analyzer_release": "foundation-plan-rails/scalar-2026-07",
"status": "issues_found",
"diagnostics": [
{
"code": "foundation_plan.identity.reserved_constant_collision",
"severity": "error",
"message": "Rails target profile \"rails-sketch/2026-07\" reserves top-level constant \"String\", generated by Entity \"string\".",
"location": {
"source_pointer": "/application/entities/0/key"
},
"subject": {
"kind": "entity",
"readable_path": "string",
"subject_uuid": "01900000-0000-7000-8000-000000000001"
},
"related_locations": [],
"suggestions": []
}
],
"started_at": "2026-07-30T12:00:00.000Z",
"completed_at": "2026-07-30T12:00:01.000Z"
}
}
35 changes: 35 additions & 0 deletions evals/create-full-stack-app/fixtures/recurring-issues-analysis.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"project": {
"id": "01900000-0000-7000-8000-000000000010",
"graph_version": 2
},
"analysis": {
"id": "01900000-0000-7000-8000-000000000904",
"graph_version": 2,
"analyzer_release": "foundation-plan-rails/scalar-2026-07",
"status": "issues_found",
"diagnostics": [
{
"code": "foundation_plan.entity.primary_descriptor_field_optional",
"severity": "error",
"message": "A primary descriptor Field must be required.",
"location": {
"source_pointer": "/application/entities/0/primary_descriptor/field"
},
"subject": {
"kind": "entity",
"readable_path": "movie",
"subject_uuid": "01900000-0000-7000-8000-000000000011"
},
"related_locations": [
{
"source_pointer": "/application/entities/0/fields/0"
}
],
"suggestions": []
}
],
"started_at": "2026-07-30T12:01:00.000Z",
"completed_at": "2026-07-30T12:01:01.000Z"
}
}
32 changes: 32 additions & 0 deletions evals/create-full-stack-app/fixtures/recurring-issues.foundation-plan.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{
"format": "firstdraft.foundation-plan.sketch/0.19",
"target": {
"id": "rails",
"profile": "rails-sketch/2026-07"
},
"application": {
"key": "movies",
"name": "Movies",
"native": {},
"delivery": {},
"entities": [
{
"subject_uuid": "01900000-0000-7000-8000-000000000011",
"key": "movie",
"name": "Movie",
"primary_descriptor": {
"field": "movie.title"
},
"fields": [
{
"subject_uuid": "01900000-0000-7000-8000-000000000012",
"key": "title",
"name": "Title",
"type": "short_text",
"required": false
}
]
}
]
}
}
Loading