Where should capability-level evaluation evidence attach in a composed workflow?

[yuehua1213]

Agent Framework supports agents, tools, middleware, and graph workflows with checkpointing, human-in-the-loop control, and multiple providers. In a production system, that makes a single framework-level label too broad for deciding whether a reusable capability is reliable.

A failed workflow may come from one tool, one agent, routing logic, provider behavior, or environment dependencies. Recording evaluation evidence at the capability/node level—while preserving the full workflow context—could make failures and version changes easier to compare.

Does the framework have a recommended stable identifier and evidence model for attaching evaluation results to individual agents, tools, or workflow nodes across Python and .NET?

[ECD5A]

On current main, I don't think Agent Framework has one canonical cross-language identity/evidence model for this yet.

The workflow evaluation helpers currently aggregate per-agent results by the local executor ID in both Python and .NET. However, neither Python's EvalItem nor .NET's EvalItem carries a stable capability reference or general evidence metadata.

I would keep two identities separate:

1. Capability identity: application-owned and stable across runs: a namespaced ID/path, capability kind (agent, tool, or workflow_node), and an explicit version or digest covering the implementation inputs that matter (code, prompt, model/configuration, and tool schema).
2. Invocation identity: unique to one execution: trace/span ID, session ID, checkpoint ID, provider response ID, and timestamp.

For example, an evidence record could use:

{
  "capability": {
    "kind": "workflow_node",
    "id": "claims/security/analyze",
    "version": "sha256:..."
  },
  "workflow_version": "sha256:...",
  "evaluation_suite": "security-regression@3",
  "invocation": {
    "trace_id": "...",
    "span_id": "...",
    "session_id": "..."
  },
  "result": { "status": "pass", "scores": {} }
}

I would not use a trace/span/checkpoint ID as the capability ID: those identify an execution or restoration point, not the reusable capability. Python's Workflow.id is also explicitly generated per workflow instance and documented as non-stable. Its graph_signature_hash is useful as a topology version, but it does not version prompts, models, tool implementations, or environment dependencies.

Likewise, I would not treat an unqualified executor ID as globally stable. The nested-workflow collision described in #7763 shows why a scoped path (or separate scope plus local ID) is needed. Tool IDs should be namespaced similarly because the same tool name can appear under multiple agents.

With today's API, I would keep this envelope in the application's evaluation store and associate it with the per-executor sub-result. A small framework-level extension that adds an optional CapabilityRef/metadata to EvalItem and preserves it into item results would make this portable across providers without coupling the core evaluation API to a particular backend.

[Sylvester Kaczmarek (sylvesterkaczmarek)]

I couldn't find a framework-defined cross-runtime evidence schema for this. I'd give each capability an application-owned stable ID, then record the agent/tool/node version, config and workflow run/checkpoint with each evaluation result.