Skip to content

Home

Jump to bottom
Don Kackman edited this page Jan 15, 2025 · 18 revisions

Overview

A workflow JSON must conform to this top-level structure:

  • id (string): A unique identifier for your workflow.
  • variables (optional): An object matching the “arguments” schema, used to store global variables used by the workflow.
  • seed (optional): Default random seed (integer) for the entire workflow.
  • steps (array): An ordered list of steps defining what is done in the workflow.

At a minimum, your JSON must include id and steps.

Example:

{
  "id": "FluxDev",
  "steps": [
    {
      "name": "flux",
      "pipeline": {
        "configuration": {
          "component_type": "FluxPipeline",
          "offload": "sequential"
        },
        "from_pretrained_arguments": {
          "model_name": "black-forest-labs/FLUX.1-dev",
          "torch_dtype": "torch.bfloat16"
        },
        "arguments": {
          "prompt": "a marmot wearing a hat"
        }
      },
      "result": {
        "content_type": "image/jpeg"
      }
    }
  ]
}

Steps

Each step is one of:

  • A step that runs a task
  • A step that defines a pipeline
  • A step that references a pipeline by name (pipeline_reference)
  • A step that references or runs another workflow (workflow)

Common Step Fields

  • name (string): A user-chosen name for the step.
  • seed (optional): Random seed (integer) for the step.
  • result (optional): An object describing how the output of this step is saved or returned.

Running a Task

A task object instructs the workflow engine to run a certain command, optionally passing it arguments or inputs.

  • command (string): A built-in operation like gather_images, resize_center_crop, etc.
  • arguments: Key/value pairs that configure that command.
  • inputs (optional): An array of dynamic or previously fetched data references.

There are a number of built-in tasks for

Example of a step running a task:

{
  "name": "resize",
  "task": {
    "command": "resize_center_crop",
    "arguments": {
      "image": "previous_result:flux",
      "height": 768,
      "width": 768
    }
  }
}

Pipelines

A pipeline step sets up a diffusers pipeline (or possibly a pipeline-like object) with specific components, such as a scheduler, model, and other parameters. You must at least define:

  • configuration
  • from_pretrained_arguments (defines how to load the pipeline from pretrained resources)
  • arguments (arbitrary key/value settings)

It can also declare sub-components:

  • scheduler: Switch out the default scheduler for something else.
  • model, vae, transformer, text_encoder, etc.: Points to separate pipeline components (again potentially loaded from pretrained arguments).
  • controlnet, lora, ip_adapter: Specialized optional fields for advanced usage.
  • seed: If you need a unique seed for this pipeline.

Example of a pipeline step:

{
  "id": "test_job",
  "steps": [
    {
      "name": "apple_image_pipeline",
      "pipeline": {
        "configuration": {
          "component_type": "StableDiffusionPipeline"
        },
        "from_pretrained_arguments": {
          "model_name": "stable-diffusion-v1-5/stable-diffusion-v1-5",
          "torch_dtype": "torch.float16"
        },
        "arguments": {
          "prompt": "an apple falling far form the tree",
          "num_inference_steps": 25
        }
      },
      "result": {
        "content_type": "image/jpeg"
      }
    }
  ]
}

Once defined, this pipeline can be referenced in subsequent steps by name.

Pipeline Reference

If you define a pipeline in one step, you can reference it in another with pipeline_reference. Pipeline reference are specialized cases where a workflow requires the same pipeline instance for subsequent operations. RF Inversion and microsoft.Florence-2 is an examples:

{
  "id": "describe_image",
  "steps": [
    {
      "name": "desscribe_image_processor",
      "pipeline": {
        "configuration": {
          "component_type": "transformers.AutoProcessor",
          "no_generator": true
        },
        "from_pretrained_arguments": {
          "model_name": "microsoft/Florence-2-large",
          "trust_remote_code": true
        },
        "arguments": {
          "text": "<DETAILED_CAPTION>",
          "images": {
             "location": "https://pbs.twimg.com/media/GfFrodVWkAABXxy?format=jpg&name=medium"
          },
          "return_tensors": "pt"
        }
      }
    },
    {
      "name": "describe_image_model",
      "pipeline": {
        "configuration": {
          "component_type": "transformers.AutoModelForCausalLM",
          "no_generator": true,
          "generate": true
        },
        "from_pretrained_arguments": {
          "model_name": "microsoft/Florence-2-large",
          "trust_remote_code": true
        },
        "arguments": {
          "input_ids": "previous_result:desscribe_image_processor.input_ids",
          "pixel_values": "previous_result:desscribe_image_processor.pixel_values",
          "max_new_tokens": 4096,
          "num_beams": 3,
          "do_sample": false
        }
      }
    },
    {
      "name": "decode_image_description",
      "task": {
        "command": "batch_decode_post_process",
        "pipeline_reference": "desscribe_image_processor",
        "arguments": {
          "generated_ids": "previous_result:describe_image_model.generated_ids",
          "task": "<DETAILED_CAPTION>"
        }
      }
    }
  ]
}

Here, reference_name must match the name of the prior pipeline step. The schema also allows you to override arguments (or add new ones) that will be applied for this usage of the pipeline.

Workflow Reference

You can embed or invoke another workflow by referencing a workflow file or built-in workflow:

  • path (string): Points to a .json workflow file or builtin:workflow_name.
  • arguments: Key/value pairs to pass into that sub-workflow.

Example of calling the built-in augment_prompt workflow which uses a local LLM to augment a prompt:

{
  "name": "augment_prompt",
  "workflow": {
    "path": "builtin:augment_prompt.json",
    "arguments": {
      "prompt": "a picture of a happy puppy"
    }
  },
  "result": {
    "content_type": "text/plain"
  }
}

Results

Each step can declare result, which controls how the step’s output is saved or serialized:

  • content_type (string): e.g., "image/png", "video/mp4", or "audio/wav".
  • save (boolean): Whether to save the output to disk (defaults to true).
  • file_base_name (string): The base of the filename used if saving.
  • fps, samplerate: For video or audio outputs.

If the step omits the result, it will not be serialized but is usabled by subsequent steps.

Example:

{
  "id": "FluxLogo",
  "steps": [
    {
      "name": "logo",
      "workflow": {
        "path": "./FluxLora.json",
        "arguments": {
          "prompt": "variable:prompt",
          "lora": "Shakker-Labs/FLUX.1-dev-LoRA-Logo-Design"
        }
      },
      "result": {
        "content_type": "image/jpeg"
      }
    }
  ]
}

The file name will be auto-generated based on the workflow is and steps. The file name can be configured with the file_base_name property of the result.

Passing Results Between Steps

Steps are run in the order they appear in the json. Once run, the output of a step can be referenced in subsequent steps using the previous_result: prefix. This allows you to chain together multiple steps, passing the output of one step as the input to another.

The name of the step is the reference to its result. In theis exmaple logo is the step name and the result of this step can be referenced as previous_result:logo:

{
  "variables": {
    "prompt": "An eco-friendly crypto currency logo"
  },
  "id": "FluxLogo",
  "steps": [
    {
      "name": "logo",
      "workflow": {
        "path": "./FluxLora.json",
        "arguments": {
          "prompt": "variable:prompt",
          "lora": "Shakker-Labs/FLUX.1-dev-LoRA-Logo-Design"
        }
      },
      "result": {
        "content_type": "image/jpeg"
      }
    },
    {
      "name": "remove_background",
      "task": {
        "command": "remove_background",
        "arguments": {
          "image": "previous_result:logo"
        }
      },
      "result": {
        "content_type": "image/png"
      }
    }
  ]
}

Variables

Variables can be declared in a workflow that specify configurable properties of the workflow. These can then be set on the command line or, when using workflow references, passed from parent to child workflows.

Referencing Variables

Once declared for a workflow, variables are references using the variable: prefix. To be well formed json the reference must be a string even if the variable is not. The proper type will be assigned when the variable is de-referenced.

Example of declaring and referencing variables:

{
  "variables": {
    "prompt": "Isometric 3D, a 3D model of a tropical island is displayed on a light blue backdrop. The island features a small body of water, surrounded by gray rocks and green grass. There are palm trees and small bushes scattered throughout the island, adding a pop of color to the scene.",
    "lora": "strangerzonehf/Flux-Isometric-3D-LoRA",
    "num_images_per_prompt": 1,
    "num_inference_steps": 25,
    "guidance_scale": 3.5
  },
  "id": "FluxLora",
  "steps": [
    {
      "name": "image_generation",
      "pipeline": {
        "configuration": {
          "component_type": "FluxPipeline",
          "offload": "sequential"
        },
        "from_pretrained_arguments": {
          "model_name": "black-forest-labs/FLUX.1-dev",
          "torch_dtype": "torch.bfloat16"
        },
        "loras": [
          {
            "model_name": "variable:lora"
          }
        ],
        "arguments": {
          "prompt": "variable:prompt",
          "guidance_scale": "variable:guidance_scale",
          "num_inference_steps": "variable:num_inference_steps",
          "num_images_per_prompt": "variable:num_images_per_prompt",
          "max_sequence_length": 512
        }
      }
    }
  ]
}

Notice that this is the same workflow used by the logo example above. Child workflows allow you to reuse the same step structure but with very different inputs and results.

Putting It All Together

Below is a minimal example showing how the schema might be used in a real workflow:

{
  "variables": {
    "prompt": "A stylish futuristic city",
    "num_images_per_prompt": 1,
    "num_inference_steps": 15,
    "width": 768,
    "height": 768
  },
  "id": "sd35",
  "steps": [
    {
      "name": "main",
      "pipeline": {
        "configuration": {
          "component_type": "StableDiffusion3Pipeline",
          "offload": "sequential"
        },
        "from_pretrained_arguments": {
          "model_name": "stabilityai/stable-diffusion-3.5-large",
          "torch_dtype": "torch.bfloat16"
        },
        "arguments": {
          "prompt": "variable:prompt",
          "num_inference_steps": "variable:num_inference_steps",
          "guidance_scale": 4.5,
          "max_sequence_length": 512,
          "num_images_per_prompt": "variable:num_images_per_prompt",
          "width": "variable:width",
          "height": "variable:height"
        }
      },
      "result": {
        "content_type": "image/png",
        "file_base_name": "my_sci_fi_city"
      }
    }
  ]
}

Further Details

  • Refer to the workflow json schema for more details about the structure and properties of objects.
  • The arguments objects throughout the schema allow freeform key/value data passing (supports strings, numbers, objects, arrays, booleans). These are passed directly to the underlying pipeline, so what is valid or required is defined by the pipeline in question.
  • The schema supports advanced usage like controlnet, LoRA, IP-Adapter, and more. These must appear under the correct pipeline subfields if you need them.
  • Use steps to chain together multiple tasks/pipelines/subworkflows.

Clone this wiki locally