Workspaces - #36
Closed
dkackman wants to merge 34 commits into
Closed
Conversation
…puts Content location was bound to the repository: three roots with three different discovery rules, all anchored to the process working directory. That was right for a CLI run from a checkout and stops being right once an agent or a remote client authors - generated media does not belong in a source tree, and an agent's workflows do not belong in the example corpus. dw/workspace.py resolves a single root holding workflows/, prompts/, assets/ and outputs/: --workspace, then DW_WORKSPACE, then the 'workspace' setting, then the working directory when it looks like a workspace, then ~/diffusers-workspace. A checkout satisfies the fourth rule, so every default lands exactly where it did before and nothing moves yet. The per-folder flags still override one folder each. set_workspace pins the root and how it was chosen, so a spawned worker does not read an inferred workspace back as one the user named - get_prompt_dir yields to its older discovery for an inferred workspace and not for an explicit one, which keeps a repo workflow reaching the library it lives beside. Stage one of docs/proposals/workspaces.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A workflow's media paths resolve against the workflow file's own directory, so a workflow that reads anything has to keep that thing beside it - which is why generated media ends up gitignored inside the source tree (workflows/*/assets/, local_inputs/). 'asset:name.ext' is rooted at the asset library instead, the way 'prompt:name' is rooted at the prompt library. It resolves to a path, first and under any argument name, so everything that already loads a path - image/video keys, a from_file, a list of them - loads it unchanged. The library is DW_ASSET_DIR / --asset-dir, else a named workspace's assets/, else ./assets, else the walk up from the workflow file; a reference is confined to it, symlinks out included. Browser uploads move from <output_dir>/uploads to the library's uploads/ and come back as 'asset:uploads/<name>', so input stops being filed among generated output and a workflow saved after an upload still resolves on the next run. A server with no asset library configured keeps the old behavior. Stage two of docs/proposals/workspaces.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Output layout was keyed off a path segment literally named 'workflows', so
the shape of a checkout was the grouping key and a workflow moved out of
that tree silently flattened. It is now keyed off the workflow's own
identity, and each execution gets its own directory:
<output_dir>/<identity>/<run id>/<files> + manifest.json
The run id is a UTC timestamp plus eight hex of the spec that ran, with a
counter when two runs of the same spec start in the same second - one
execution is one directory. A sub-workflow is part of its parent's run: it
writes into the same directory and rolls up into the same manifest.
manifest.json records status, seed, arguments, device, version and each
step's files, relative to the directory so it keeps describing itself when
moved. It is written for a failed run too - the files it wrote are on disk
either way. A CLI run has never been recorded anywhere before this.
The step cache now validates against the output *root* rather than the
per-run directory, which is new every execution and would have defeated it;
an unchanged rerun still writes nothing and reports the earlier run's files
as reused. The gallery groups a workflow's runs under one folder by
stripping the run id, so the folder filter does not grow one entry per run.
--output-layout flat / DW_OUTPUT_LAYOUT / the output_layout setting keep the
previous layout for scripts that glob the output directory.
Stage three of docs/proposals/workspaces.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…kout outputs/ already covers run directories and their manifests - the rule matches at any depth. The asset library is new and was not covered: with the checkout as the workspace, dw.serve creates <repo>/assets/ and browser uploads land in it. Anchored at the root so the SPA's own assets/ folders are untouched. test_serve_main was creating that directory in the working tree on every run, because main() with no --workspace resolves the working directory - give it a workspace of its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/api/server has reported both since the workspace and asset work landed; the page listed only workflows, outputs and prompts, so the root they are folders of was invisible in the UI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workflow directory was both the library and the place saves landed, so with a checkout as the workspace every save from the editor or an MCP client wrote into the example corpus. Reads now span a search path - the workspace's workflows/ first, then any --examples-dir, read-only - while writes only ever reach the front. A name in an earlier root shadows the same name later, so "open an example, change it, save" writes a copy into the user's library and shadows the example instead of overwriting it. Deleting from a read-only root answers 403 with the source it came from rather than a 404 that reads like a missing file. A job now carries the root it is confined to, so a workflow read from an examples directory runs confined to that directory - its sub-workflow steps and relative assets resolve where it lives, not in a writable root it will never be saved to. GET /api/workflows reports the path as `sources` and tags every entry with origin and writable. The UI hides delete and marks a read-only workflow, and submits by name rather than by a path composed from the workflow directory, which only ever named the writable root. MCP passes both through. The packaged dw/workflows/ stay off the path: they are the pieces a 'builtin:' sub-workflow step names, resolved by the engine where that step is read, not workflows anyone browses. Stage four of docs/proposals/workspaces.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Vite emits the SPA's own bundles under /assets/, and the asset library mounted there shadowed them: with a library configured - which dw.serve always configures - every script and stylesheet 404'd, so the web UI loaded and then rendered nothing. Live since the asset library landed. No unit test could see it: they build an app with no UI mounted. The e2e suite caught it as 36 specs failing on locators as basic as the Jobs heading. Added the server-level guard that was missing - a fake SPA bundle and an asset library in one app, both resolving. /inputs is also the symmetric name next to /outputs. The asset: reference a workflow carries is unchanged; only the preview URL moves. The e2e fixture now pins its scratch directory as the workspace too: it copies workflows and prompts precisely so specs never touch the checkout, but the asset library the server creates was landing there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Multi-stage work meant copying files out of the output directory and back in beside the next workflow - the GYRE examples still carry a generated score and a still copied into their assets folder by hand. An 'output:' reference names them where they are: 'output:ltx2/Gyre/latest/still.png'. The name is the path under the output root that run directories already gave every file - workflow identity, run id, file - and 'latest' in the run-id position resolves to the newest run, so a second-stage workflow need not be edited after every run. Runs sort by their id, which starts with a UTC timestamp, so newest needs no mtimes and survives a copy. Like asset:, it resolves to a path before every other convention, so whatever loads paths loads it - image/video keys, a from_file, a list. It resolves against the output root the run was told to write to, activated by Workflow.run, and is confined to it: traversal, absolute paths and symlinks pointing out are refused. Stage five of docs/proposals/workspaces.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An agent could name assets that already existed on the box but had no way to put bytes there, so it could author workflows it could not supply inputs for. dw_mcp/CLAUDE.md recorded POST /api/uploads as deliberately uncovered because "an MCP client has no bytes to push" - true only while uploads landed among generated output. upload_asset reads a file on the machine the MCP server runs on and pushes it, which is the machine an agent authoring against a remote dw.serve is actually on, and returns the asset: reference rather than a path: a path here means nothing there. list_assets reports the library the same way, by reference, so a workflow can be written against what is already present. GET /api/assets is the route behind the listing, empty rather than an error on a server with no library. Uploads now accept audio as well as image and video - the library holds it and workflows read it (an H3 audio reference is built from a .wav), so refusing it left one input kind with no way onto the machine. Stage six, level one, of docs/proposals/workspaces.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Level 2 changed shape once level 1 landed, so it no longer fits as a paragraph in the workspaces proposal. asset: references are rooted at the server's library rather than at the workflow file, and upload_asset can push bytes - between them the base_dir confinement problem that document called level 2's real design work is already gone, and what remains is bookkeeping: keep the files locally, translate at submit time. Records the one server change it needs (content-addressed uploads, so a sync does not re-push a reference video every run), what it deliberately does not do, the alternatives including doing nothing, and the open questions - chiefly how a local and a server workflow of the same name are presented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first draft made a client-side workspace the system of record and submitted runs as anonymous inline definitions, which left work authored through MCP invisible to the web UI - the gallery, editor and prompts pages read the server's directories - and required a rewriting layer (inlining prompts, rewriting asset references) that existed only because the server had nowhere to put a workflow that was not its own. Named workspaces on the server remove both. Ownership is per artifact class, not per workspace: the client is canonical for what a person authors and the server for what the GPU generates, so nothing is canonical twice and there is no merge. A mirrored workspace is read-only on the server, which stage four's origin/writable vocabulary already expresses - save from the UI copies into a server-owned workspace, the same gesture that already works for an example. Records the plumbing: output_dir and workflow_dir already travel with a job, so the work is moving the prompt and asset roots off environment pinning onto per-job values, the way stage five's output root already works. Supersedes the content-addressed upload naming the previous draft proposed - a mirror must preserve names or every reference needs rewriting, so digests detect change rather than name files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…argets Three decisions folded into the proposal. The prompt library stays one, shared by every workspace and server-owned. dw/prompts.py already describes it as shared by reference rather than copied, and 'prompt:scenic' meaning different text per workspace would break that. It also shrinks the plumbing: the prompt root stays process-wide and only the asset root has to become per-job. The layout is the existing workspace directory as the root - its own workflows/assets/outputs stay the default workspace, named workspaces are subdirectories beside them, and prompts/ at the root is the shared library. Nothing on disk moves, and --workspace keeps meaning what it means. The cost is asymmetry between the default workspace and named ones, and four reserved names. The alternatives - sibling directories under a new workspaces root, and a registry pointing anywhere on disk - are recorded with what each would buy; the registry is the way back if per-workspace volumes ever matter. The target is one user, one server, several MCP clients, stated up front so that per-workspace permissions, quotas and tokens stay out until there is someone to need them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One server, one workspace, was the shape until now: everything an MCP client authored landed in whichever workspace the server was started with, in one namespace shared with the web UI and every other client. The workspace root now holds them. Its own workflows/assets/outputs are the 'default' workspace and a named one is a subdirectory beside them, so nothing on disk moves and --workspace keeps meaning what it meant. prompts/ stays at the root, shared by every workspace: a stored prompt is shared by reference, and 'prompt:scenic' resolving to different text per workspace would defeat that. workflows, prompts, assets and outputs are reserved names. Routes take an optional workspace and default to the default one, so every call that predates this behaves exactly as it did. A job carries its own output, asset and workflow roots, so one queued for a workspace still runs in it after the manager has served another - and the worker activates the asset root per job, the one root that could not stay a process-wide environment variable the way the shared prompt library can. Deleting a workspace answers with what it would remove and refuses until acknowledged, refuses the default, and refuses one with jobs still queued. jobs.sqlite gains a workspace column, with rows that predate it backfilled to default. The gallery stays default-only for now; scoping it belongs with the UI's workspace switcher. Stage one of docs/proposals/server-workspaces.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…k one in the UI The static mounts could not follow: StaticFiles binds to one directory at startup, and a workspace can be created afterwards. Mounting the workspace root instead would have exposed workflows/ and prompts/ over an ungated path, and mounting per workspace is impossible - the SPA's catch-all mount at / is registered last, so anything mounted later never matches. So /outputs and /inputs are routes now, resolving per request against the selected workspace, with the same containment check the gallery routes already use. Starlette's FileResponse - what StaticFiles returns anyway - handles Range, so video still seeks; a test pins that. They stay ungated like the mounts they replace, because an <img> tag cannot attach an Authorization header. The gallery, its metadata, thumbnail, download, archive and delete routes all take the workspace, and gallery entries carry a URL that already has it, so a client never composes one. In the UI the selection lives in one place: setApiWorkspace scopes every request and every file URL, so no page threads a workspace through its calls - a page refetches on a switch by reading workspace.current in its load effect. The picker appears on Workflows and Gallery once there is more than one workspace, the Server page creates and deletes them, and the choice is restored from localStorage before the first request goes out. Stage two of docs/proposals/server-workspaces.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Several MCP clients against one server shared one namespace: whatever an agent authored landed in the same workflows and outputs as everyone else's. A session now picks a workspace - --workspace or DW_MCP_WORKSPACE at startup, use_workspace to switch - and DwClient adds the selector to every request's query string, so no handler carries a workspace parameter and the default sends nothing at all. A switch is one visible call in the transcript rather than an argument that can be forgotten on the one call where it mattered, and it is checked against the server so a typo fails there instead of scoping every later call to a workspace that does not exist. list_workspaces, create_workspace and delete_workspace complete the set; delete refuses without acknowledged_cost and reports what it would remove, like the other costly tools. The environment variable is DW_MCP_WORKSPACE, not DW_WORKSPACE: one names a workspace on the server, the other names a directory to the engine. POST /api/jobs and /api/validate now accept the selector as a query parameter as well as in the body, so one mechanism scopes every call a client makes. Stage three of docs/proposals/server-workspaces.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Namespace separation and reuse on the server, not source control: that is handled on the client with the tools already in use. Which makes stages four and five unnecessary rather than merely speculative - where the working copy and the server share a filesystem, git in the workspace directory is a better mirror than this design would be, with real merges instead of a client-wins rule. Kept as a record with the reasoning, marked not planned. Names what actually follows stage three: assets are still siloed per workspace, so an image two projects use is uploaded twice and 'asset:iris.png' means different files in different workspaces - exactly what the shared prompt library exists to avoid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Generated files are named by the run that made them, which is the right place for them and the wrong thing for a later workflow to depend on: 'latest' moves under you and a pinned run id breaks the moment outputs are pruned. Nothing turned "this still is the hero frame for this project" into a name that stays put. POST /api/assets/keep copies one from the workspace's outputs into its assets under a chosen name and answers with the asset: reference. Inside the workspace, so nothing crosses a namespace, and server-side, so an MCP client no longer has to download a render and upload it back to reuse it - a round trip that is silly on one box and absurd against a remote one. A hard link where the filesystem allows one: keeping a frame of a multi-gigabyte render should not cost another copy, and both names refer to the same content. Falls back to a copy across filesystems. An existing name is refused rather than replaced silently. Reaches the UI as the gallery's "Keep as asset" and MCP as keep_output. Completes the generate-keep-reuse loop workspaces exist for: outputs land in the workspace, the ones worth keeping become assets, and later workflows name them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…those are deleted One folder was enough to count as a workspace, so a checkout used as the root listed its own dw/ (dw/workflows holds the builtins) as a workspace - browsable, and deletable through the API with rmtree. A named workspace now needs workflows/, assets/ and outputs/ together, which is what create_workspace makes, and delete_workspace refuses a directory holding anything else (409, naming the entries). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…xplicit-workspace prompt rule Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s completed The id was documented as a UTC timestamp - 'latest' sorts on it - but was stamped in local time, so a DST fall-back could order a newer run before an older one. And a workflow with no steps returned before the run's status was set, so its manifest said failed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…here runs are A failed run, or one whose every step was a cache hit, leaves a directory holding only its manifest; picking the newest directory pointed a second-stage workflow at nothing while the stage before had plainly produced the file. Runs are now tried newest first. And 'latest' was expanded wherever it appeared in a name, so a workflow or file called latest could not be referenced - it now selects a run only in a directory that holds runs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ests out of it tests/test_workflow.py wrote to ./output, which run directories turned into output/w/<run id>/manifest.json on every run - thirty of them got committed. The tests write to a temporary directory now, and /output/ is ignored as a guard. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ost one points at the fix Plus a REPL note that a running worker keeps its libraries until restart when the workspace is switched under it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… says which outputUrl takes the job's workspace over the picker's, the jobs page filters by workspace and chips each row, the workspace listing is fetched once per session rather than per page mount, a workflow's origin comes from its own GET rather than a full listing, and the selection lives in one store that api.ts reads directly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e confine to it Sixteen handlers unpacked the same (name, directories) tuple by hand, and two of them had already drifted back to the process-wide defaults: an inline validate confined base_dir to the default workspace's workflows, and a save validated the candidate there too, so a sub-workflow path valid in the workspace being saved into was refused. Each route now takes ws: Workspace = Depends(selected_workspace); the default is a ConfiguredWorkspace over the server's own directories. A named workspace is checked by its one directory rather than a listing of the root, and /api/workspaces lists the root once. resolve_workflow_reference returns the source too, so confinement is not rediscovered. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… files serve with 304s again
An inline job recorded the default workflow root beside a base_dir in the
named workspace, so the worker refused it; a rerun dropped every root the
original ran against and landed in the default workspace. Both now carry
workflow_dir/output_dir/asset_dir/workspace, jobs report their workspace
and /api/jobs can filter by it, and a gallery file is attributed to a job
in its own workspace rather than any job that wrote the same name.
/outputs and /inputs serve through a per-root StaticFiles so ETag/304 and
Range come back (a bare FileResponse never answers 304), an upload's
preview URL carries the workspace selector, asset entries get a URL, and
GET /api/workflows/{name} reports its origin in headers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…a walk get_prompt_dir and get_asset_dir encoded the same precedence line for line; the four reference/name validators repeated the same three checks; list_assets re-walked what _iter_gallery_files already walks. Each now has one home. workspace_exists had no callers and goes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| except SecurityError as e: | ||
| raise HTTPException(status_code=400, detail=str(e)) | ||
|
|
||
| if os.path.exists(destination) and not request.overwrite: |
| f"to replace it", | ||
| ) | ||
|
|
||
| os.makedirs(os.path.dirname(destination), exist_ok=True) |
| ) | ||
|
|
||
| os.makedirs(os.path.dirname(destination), exist_ok=True) | ||
| if os.path.exists(destination): |
|
|
||
| os.makedirs(os.path.dirname(destination), exist_ok=True) | ||
| if os.path.exists(destination): | ||
| os.remove(destination) |
| marker is enough" and _holds_a_workspace's stricter "all three, or it | ||
| isn't a workspace".""" | ||
| test = all if all_of else any | ||
| return test(os.path.isdir(os.path.join(path, name)) for name in names) |
| def _foreign_entries(path): | ||
| """What a directory holds besides a workspace's own three folders.""" | ||
| try: | ||
| return sorted(entry for entry in os.listdir(path) if entry not in NAMED_SUBDIRS) |
| directory = os.path.join(workspace.root, folder) | ||
| files = 0 | ||
| total = 0 | ||
| for current, _dirs, names in os.walk(directory): |
| for current, _dirs, names in os.walk(directory): | ||
| for entry in names: | ||
| try: | ||
| total += os.path.getsize(os.path.join(current, entry)) |
| foreign = _foreign_entries(target.root) | ||
| if foreign: | ||
| raise NotAWorkspaceError(name, foreign) | ||
| shutil.rmtree(target.root) |
…ames feat: add CodeQL model pack for security barriers and path validation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.