Skip to content

Tie harness version and workflow to one release #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
danielholanda wants to merge 1 commit into from
Closed

Tie harness version and workflow to one release #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 16 additions & 11 deletions .github/workflows/reusable.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ name: reusable
#
# jobs:
# evals:
# uses: amd/skillscope/.github/workflows/reusable.yml@main
# uses: amd/skillscope/.github/workflows/reusable.yml@v0.1.1
# secrets:
# api_key: ${{ secrets.ANTHROPIC_API_KEY }}
# with:
Expand All @@ -17,6 +17,14 @@ name: reusable
# That is a whole caller. Everything else has a default, and the defaults are
# the strict answer: every step runs, and every step can fail the run.
#
# The ref you pin is the harness you get. Every step below references
# `amd/skillscope@v0.1.1`, the release this file ships in, and the action
# installs the harness out of that checkout rather than resolving a version
# from anywhere else -- so `reusable.yml@v0.1.1` grades your skills with
# skillscope v0.1.1, and there is no second setting that can quietly say
# otherwise. The workflow and the harness move as one release; `version` is the
# only way to break the pair, for a caller trying a build before it ships.
#
# discover --> structural --> routing (one runner per skill)
# --> behavioral (one runner per skill)
# --> results
Expand Down Expand Up @@ -101,8 +109,9 @@ on:
default: "1"
version:
description: >-
Harness version to run, overriding the pin in each skill's evals.json.
Blank runs those pins, and this workflow's own ref where there is none.
A harness build to run instead of the one this workflow's release
ships with: a tag, a branch, or a commit. Blank is the answer that
cannot drift from the ref you pinned this workflow at.
type: string
default: ""
api_key_secret:
Expand Down Expand Up @@ -166,7 +175,7 @@ jobs:
# everything else that runs against it.
- name: List the skills
id: found
uses: amd/skillscope@main
uses: amd/skillscope@v0.1.1
with:
command: list-skills
skills: ${{ steps.globs.outputs.globs }}
Expand Down Expand Up @@ -241,7 +250,7 @@ jobs:
uses: actions/checkout@v4

- name: Check every skill's structure
uses: amd/skillscope@main
uses: amd/skillscope@v0.1.1
with:
command: structural
skills: ${{ needs.discover.outputs.globs }}
Expand Down Expand Up @@ -290,7 +299,7 @@ jobs:
fi

- name: Run the routing eval for ${{ matrix.skill }}
uses: amd/skillscope@main
uses: amd/skillscope@v0.1.1
with:
command: routing
# One skill in the room, named, so the score is about this skill alone
Expand All @@ -303,7 +312,6 @@ jobs:
--output routing-report.json
--keep-logs routing-logs
skills: ${{ needs.discover.outputs.globs }}
skill: ${{ matrix.skill }}
version: ${{ inputs.version }}
install-claude: "true"

Expand Down Expand Up @@ -356,16 +364,13 @@ jobs:
fi

- name: Run the behavioral cases for ${{ matrix.skill }}
uses: amd/skillscope@main
uses: amd/skillscope@v0.1.1
with:
command: behavioral
args: >-
--skill '${{ matrix.skill }}'
--output behavioral-report.json
skills: ${{ needs.discover.outputs.globs }}
# This skill's own evals.json may pin the harness version its cases
# were written for, and this is the leg that runs them.
skill: ${{ matrix.skill }}
version: ${{ inputs.version }}
install-claude: "true"

Expand Down
37 changes: 26 additions & 11 deletions .github/workflows/selftest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,7 @@ name: selftest
# and runs on every platform a behavioral leg can be scheduled on, because that
# is where the path handling breaks. The smoke test drives the real launcher
# against a throwaway repo built in the job, which is the only way to find out
# that action.yml, the version resolver, and the CLI still agree on their
# interface.
# that action.yml, the launcher, and the CLI still agree on their interface.

on:
push:
Expand Down Expand Up @@ -92,19 +91,39 @@ jobs:
}
EOF

# `source: .` installs this checkout rather than a published ref, so a
# pull request tests the launcher and the payload it is about to change.
# No `version`: the fixture pins nothing, which exercises the bottom rung
# of the resolution ladder.
# `uses: ./` makes this checkout the action, and the action installs the
# harness from the checkout it is running out of -- so a pull request
# tests the launcher against the payload it is about to change, which is
# the same path a caller takes at a tag. No `version`, which is the input
# that would reach the network for someone else's build instead.
- name: Check the fixture structurally
id: structural
uses: ./
with:
command: structural
# The card and its sections are a repo's own policy, so the fixture
# asks for them the way a repo with a governance card would.
args: --skill-files skill-card.md --skill-sections Description,Owner,License
repo: fixture
source: .

# The version story, end to end: nothing above named a version, so the
# harness that just ran has to be the one in this checkout. A launcher
# that fetched a build from anywhere else would report a git ref here
# instead, which is the failure a caller pinning a tag cannot see for
# themselves.
- name: Check it ran the harness from this checkout
shell: python
env:
REPORTED: ${{ steps.structural.outputs.version }}
run: |
import os
import re

pyproject = open("pyproject.toml", encoding="utf-8").read()
declared = re.search(r'(?m)^version\s*=\s*"([^"]+)"', pyproject).group(1)
reported = os.environ["REPORTED"]
assert reported == declared, f"ran {reported!r}, but this checkout is {declared!r}"
print(f"skillscope {reported}, from this checkout.")

# This repo has no skills, so what is left is its own prose -- which is
# exactly the half a harness for checking references should not be
Expand All @@ -116,7 +135,6 @@ jobs:
with:
command: structural
args: --docs '*.md,docs/**/*.md'
source: .

- name: Select runs for a change
id: select
Expand All @@ -125,7 +143,6 @@ jobs:
command: select
args: --all --routing-room demo-skill
repo: fixture
source: .

- name: Select runs without naming the routing set
id: inferred
Expand All @@ -134,7 +151,6 @@ jobs:
command: select
args: --all
repo: fixture
source: .

- name: Select runs with routing turned off
id: unrouted
Expand All @@ -143,7 +159,6 @@ jobs:
command: select
args: --all --routing-room none
repo: fixture
source: .

- name: Check the plan
shell: python
Expand Down
32 changes: 13 additions & 19 deletions .github/workflows/skill-evals.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ name: skill-evals
#
# jobs:
# skill-evals:
# uses: amd/skillscope/.github/workflows/skill-evals.yml@bootstrap
# uses: amd/skillscope/.github/workflows/skill-evals.yml@v0.1.1
# secrets: inherit
# with:
# skill_globs: skills/*
Expand Down Expand Up @@ -189,8 +189,9 @@ on:
default: false
version:
description: >-
Harness version to run, overriding the pin in each skill's evals.json.
Blank runs the ref the caller referenced this workflow at.
A harness build to run instead of the one this workflow's release
ships with: a tag, a branch, or a commit. Blank is the answer that
cannot drift from the ref you pinned this workflow at.
type: string
default: ""
coordinator_runner:
Expand Down Expand Up @@ -268,7 +269,6 @@ jobs:
scoped: ${{ steps.plan.outputs.scoped }}
scoped_any: ${{ steps.plan.outputs.scoped_any }}
skipped: ${{ steps.plan.outputs.skipped }}
version: ${{ steps.plan.outputs.version }}
steps:
- name: Check out repository
uses: actions/checkout@v4
Expand All @@ -281,7 +281,7 @@ jobs:
# should never start against a broken dataset. External URLs are left to
# the job below; everything here is offline and deterministic.
- name: Check every skill's structure
uses: amd/skillscope@bootstrap
uses: amd/skillscope@v0.1.1
with:
command: structural
args: >-
Expand Down Expand Up @@ -370,7 +370,7 @@ jobs:

- name: Select what to run
id: select
uses: amd/skillscope@bootstrap
uses: amd/skillscope@v0.1.1
with:
command: select
args: ${{ steps.how.outputs.args }}
Expand Down Expand Up @@ -405,7 +405,6 @@ jobs:
# Emitted as the flag itself: every run job passes it through
# verbatim rather than re-deriving the same boolean.
"extended=" + ("--extended" if plan["extended"] else "--no-extended"),
"version=" + plan.get("version", ""),
]
for key in ("default", "scoped", "skipped"):
lines.append(key + "=" + json.dumps(plan[key]))
Expand All @@ -431,7 +430,7 @@ jobs:
uses: actions/checkout@v4

- name: Fetch every external reference
uses: amd/skillscope@bootstrap
uses: amd/skillscope@v0.1.1
with:
command: structural
args: >-
Expand Down Expand Up @@ -497,7 +496,7 @@ jobs:
print("Exported: " + ", ".join(exported))

- name: Run the routing eval
uses: amd/skillscope@bootstrap
uses: amd/skillscope@v0.1.1
with:
command: routing
# By default a wrong routing decision fails this job, the same way a
Expand All @@ -517,7 +516,7 @@ jobs:
--output routing-report.json
--keep-logs routing-logs
skills: ${{ inputs.skill_globs }}
version: ${{ inputs.version || needs.discover.outputs.version }}
version: ${{ inputs.version }}
python-version: ${{ inputs.python_version }}
setup-python: "false"
install-claude: "true"
Expand Down Expand Up @@ -590,14 +589,12 @@ jobs:
print("Exported: " + ", ".join(exported))

- name: Run behavioral cases for ${{ matrix.skill }}
uses: amd/skillscope@bootstrap
uses: amd/skillscope@v0.1.1
with:
command: behavioral
args: --skill '${{ matrix.skill }}' ${{ needs.discover.outputs.extended }}
skill: ${{ matrix.skill }}
skills: ${{ inputs.skill_globs }}
# The skill's own pin, resolved during selection.
version: ${{ inputs.version || matrix.version }}
version: ${{ inputs.version }}
python-version: ${{ inputs.python_version }}
setup-python: "false"
install-claude: "true"
Expand Down Expand Up @@ -665,13 +662,12 @@ jobs:
print("Exported: " + ", ".join(exported))

- name: Run behavioral cases for ${{ matrix.skill }}
uses: amd/skillscope@bootstrap
uses: amd/skillscope@v0.1.1
with:
command: behavioral
args: --skill '${{ matrix.skill }}' ${{ needs.discover.outputs.extended }}
skill: ${{ matrix.skill }}
skills: ${{ inputs.skill_globs }}
version: ${{ inputs.version || matrix.version }}
version: ${{ inputs.version }}
python-version: ${{ inputs.python_version }}
setup-python: "false"
install-claude: "true"
Expand Down Expand Up @@ -701,9 +697,7 @@ jobs:
BEHAVIOR_WANTED: ${{ needs.discover.outputs.default_any }}
SCOPED_WANTED: ${{ needs.discover.outputs.scoped_any }}
SKIPPED: ${{ needs.discover.outputs.skipped }}
VERSION: ${{ needs.discover.outputs.version }}
run: |
echo "harness: ${VERSION:-(the launcher's own ref)}"
echo "discover: $DISCOVER"
echo "routing: $ROUTING (requested: $ROUTING_WANTED)"
echo "behavioral: $BEHAVIOR (requested: $BEHAVIOR_WANTED)"
Expand Down
5 changes: 4 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,16 @@ Simply add an `evals.json` file to your skill and a workflow that points to our
```yaml
jobs:
evals:
uses: amd/skillscope/.github/workflows/reusable.yml@v0.1.0
uses: amd/skillscope/.github/workflows/reusable.yml@v0.1.1
secrets:
api_key: ${{ secrets.ANTHROPIC_API_KEY }}
with:
skills: skills/*
```

The ref is the version: `@v0.1.1` grades your skills with skillscope v0.1.1, and
keeps doing so until you change that one line.

You may also choose to customize or even disable some tests if you prefer it. See [docs/usage.md](docs/usage.md) for details.

## Commands
Expand Down
Loading
Loading