Job record and export: the realized workflow beside every run, and exporting a job - #56
Merged
Conversation
Every run writes its realized workflow beside its manifest, a server job records its run id, MCP can read the realized workflow, and export_job gathers one job into a git-ready directory and a zip. Recoverability stays with docs/proposals/resume.md, which this record serves. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
realize_workflow folds the run's arguments into the variable defaults, writes the seed the run resolved, inlines stored prompt text and rewrites 'output:.../latest/...' to the run id it picked. asset:, constant:, previous_result: and builtin: are kept; a local sub-workflow path is kept and digested into the annotations the manifest will carry. Never mutates its input and never fails a run: a reference that will not resolve is left exactly as written, so the engine raises its own error. Also exports strings_with_prefix, a walker a later task (export) reuses to find asset:/output: references in a realized workflow; _pin shares the same underlying tree walk (_map_strings) rather than duplicating it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Workflow.run realizes the definition once the seed and the run directory have settled and writes it as workflow.json before the first step, so a crash or a cancel still leaves it. The manifest's workflow block gains 'realized', 'prompts' and 'sub_workflows'. Best effort throughout: a failed realization warns and the run continues. A sub-workflow inherits the parent's directory and writes none of its own; the flat layout has no run directory and so writes none either. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Workflow.run emits a run_start event carrying the run id, the workflow
identity and the run directory relative to the output root. The Job records
both, jobs.sqlite gains run_id and run_dir (ALTER when absent, NULL for
older rows), and JobManager.realized reads workflow.json back out of the
job's own output root, confined to it.
GET /api/jobs/{id}/workflow now answers {id, definition, realized}: the
realized copy when there is one, the submitted definition otherwise.
run_start now fires before workflow_start, so test_events.py's sequence
assertion is updated to match.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A read-only tool over GET /api/jobs/{id}/workflow. 'realized: true' means
every mutable input is pinned; false means the job predates run tracking.
The 'next' sentence points at save_workflow and run_workflow, which is how
an inline run worth keeping gets a name.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
POST /api/jobs/{id}/export gathers the realized workflow, the run's
manifest, the job row, every asset: and output: file the workflow named and
every file the manifest lists into <workspace>/exports/<job id>/, with a
generated README saying what the run was, where each input came from, how
to run it again, and that the media belongs in Git LFS. 404 for an unknown
job, 409 for a live one or an existing export without overwrite.
job.json takes one fixed key set whether the job was read live or out of
history, so an export is the same record either way; the asset: and output:
references are found with realize.strings_with_prefix rather than a third
tree walk.
GET /exports/{id}.zip streams the same tree, built on request. 'exports'
joins the reserved workspace names, so the folder is never mistaken for a
workspace.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A handler over POST /api/jobs/{id}/export returning the directory, the zip
URL, the file list with sizes and the three JSON documents inline. The
'where' sentence says the directory is on the machine running the server -
the lesson download_output taught.
DwClient.post_json gains an optional params argument, since the route's
overwrite flag is a query parameter beside the workspace selector.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The zip route built its temp archive with the unlink attached only to a successful response, so a failure mid-write left the file behind. It now builds inside try/except BaseException like archive_outputs does, and answers with a FileResponse, which also sets Content-Length and lets Starlette encode the filename rather than interpolating the job id into a content-disposition header. Two export branches had no test: a job with no run directory, whose manifest is synthesized from the job row and whose files resolve against the output root, and an output: reference copied under the run it names. Both now have one, and the second's run manifest lists the same file twice so it also covers the new de-duplication of output copies. A copy that fails now reports the reference rather than the absolute server path, and the escape test for a manifest path is exact rather than a prefix match on '..'. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
WORKFLOW_GUIDE gets the realized file beside the manifest and, in the agent-authoring section, the get_job_workflow -> save_workflow loop after a long inline run. CLAUDE.md mirrors it in the type-system list and the run-directories gotcha. MCP.md documents the two tools with the server-machine caveat, SERVER.md the three routes, WORKSPACES.md the workflow.json in a run directory and the reserved 'exports' name. dw/server/CLAUDE.md gains a short section on exports.py, matching the other modules already documented there. Each family skill's "Run and judge" gains the same one-line habit. The proposal moves to implemented, and resume.md records that its stage 2 is satisfied by the realized file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Export's asset search path now comes from the job's own asset_dir (falling back to the read-only example libraries, then the selected workspace) rather than the workspace the caller happens to be scoped to, so exporting a job across workspaces still finds its assets. - export_directory refuses cleanly (409) instead of raising a bare TypeError when the server has no workspace root. - realize_workflow also pins a variable-referenced top-level seed into that variable's realized value, so a rerun with no seed argument can't put a null default back over the pinned integer. - Documents the run_start event, fixes an overclaiming sentence about reproducing a run from inside its directory, moves misplaced prose in docs/SERVER.md and CLAUDE.md, repairs a numbered list broken by a bullet in the three plugin skills, updates a UI comment/type for realized job workflows, and notes in the MCP export docstrings that the copy costs disk again and total_bytes reports it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| directory = export_directory(ws.root, job_id) | ||
| except SecurityError: | ||
| raise HTTPException(status_code=404, detail="No export for this job") | ||
| if not os.path.isdir(directory): |
| try: | ||
| with handle: | ||
| with zipfile.ZipFile(handle, "w", zipfile.ZIP_DEFLATED) as archive: | ||
| for current, _dirs, names in os.walk(directory): |
| ) | ||
|
|
||
| target = export_directory(workspace_root, job_id) | ||
| if os.path.exists(target): |
| f"An export of job {job_id} already exists - pass overwrite to " | ||
| f"replace it" | ||
| ) | ||
| shutil.rmtree(target) |
| f"replace it" | ||
| ) | ||
| shutil.rmtree(target) | ||
| os.makedirs(target) |
| """ | ||
| destination = os.path.join(target, relative) | ||
| try: | ||
| os.makedirs(os.path.dirname(destination), exist_ok=True) |
| destination = os.path.join(target, relative) | ||
| try: | ||
| os.makedirs(os.path.dirname(destination), exist_ok=True) | ||
| shutil.copyfile(source, destination) |
| def _write_text(summary, target, name, text): | ||
| path = os.path.join(target, name) | ||
| try: | ||
| with open(path, "w", encoding="utf-8") as file: |
|
|
||
| def _record(summary, path, target): | ||
| try: | ||
| size = os.path.getsize(path) |
| ignored = NAMED_SUBDIRS + (EXPORTS_SUBDIR,) | ||
| try: | ||
| return sorted(entry for entry in os.listdir(path) if entry not in NAMED_SUBDIRS) | ||
| return sorted(entry for entry in os.listdir(path) if entry not in ignored) |
The first drill of the tool unpacked the export into the session scratchpad and doubled the job id in the path. The next hint, the tool description, the MCP doc row and the three skills now say where the archive goes and that it already unpacks into one folder. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
validate_path accepts a path equal to its root, so "." on the zip route resolved to exports/ itself and archived every export in the workspace. The shape check runs before the join. Closes the one real gap among the ten CodeQL path-injection flags on #56; the rest are the validators it does not model. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The model pack from #37 declares the validators as barriers, yet every open path-injection alert postdates it. The proposal is the local reproduction loop that finds out why, and the model fix that follows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every run now leaves a realized workflow beside its manifest, a server job knows which run is its own, and an MCP client can read that workflow and export one job as a git-ready directory and a zip.
Motivation: the 30-second lighthouse short was assembled inline over MCP, ran for most of an hour, and was lost at the last step; its retry regenerated every shot, and the workflow that made the cut existed nowhere a person could read. Proposal:
docs/proposals/job-record-and-export.md; design:docs/superpowers/specs/2026-09-08-job-record-and-export-design.md. Recoverability itself stays withdocs/proposals/resume.md, whose "per-step identity" stage this record satisfies.What changed
dw/realize.py(new):realize_workflowpins every mutable input into a schema-clean copy: arguments folded into variable defaults, the seed the run used,prompt:text inlined,output:.../latest/...rewritten to the concrete run id.asset:,constant:,previous_result:andbuiltin:stay as written; a local sub-workflow path keeps its SHA-256 in the manifest. Realization never fails a run.Workflow.runwritesworkflow.jsonin the run directory before the first step (best effort, like the manifest) and emits arun_startevent withrun_id,identity,run_dir. The manifest'sworkflowblock gainsrealized,promptsandsub_workflows. Flat layout and sub-workflows write nothing, as before.Jobrecordsrun_id/run_dir;jobs.sqlitegains both columns (ALTER when absent, null for older rows);JobManager.realized(job_id)reads the run's file, confined to the job's own output root.GET /api/jobs/{id}/workflowreturnsrealized: true|falseand falls back to the submitted definition for older jobs.POST /api/jobs/{id}/exportgathers one finished job into<workspace>/exports/<job id>/(README.md,workflow.json,manifest.json,job.json,assets/,inputs/,outputs/), copies rather than links, resolves assets against the job's own library, and lists what it could not find.GET /exports/{id}.zipstreams the tree.exportsis a reserved workspace name.get_job_workflow(job_id)andexport_job(job_id, overwrite=False); the latter says the directory is on the server machine and quotestotal_bytes.dw/server/CLAUDE.md; each plugin skill's "Run and judge" tells an agent toget_job_workflowandsave_workflowafter an inline run worth keeping.Follow-ups
docs/proposals/resume.mdstage 3): rehydrate the step cache from a previous run's files, so the lighthouse retry becomes a one-step rerun.Test plan
python -m pytest: 3288 passed, 5 skipped;black --check dw dw_mcp testscleantests/test_realize.py(one per realization rule, schema validation, no input mutation),tests/test_runs.py(file beside manifest, flat layout, best effort),tests/test_server_jobs.py(event, migration,realized(), path escape),tests/test_server_exports.py(tree, historical job, synthesized manifest,output:inputs, dedupe, 404/409, zip parity, job's own asset dir, no-workspace 409),tests/test_mcp_diagnose.py,tests/test_mcp_exports.pyworkflow.jsonbesidemanifest.json;get_job_workflowon that job;export_joband open the zip URL🤖 Generated with Claude Code