Harbor export - #62
Closed
Frame17 wants to merge 14 commits into
Closed
Conversation
Co-Authored-By: Claude <claude@air.dev>
Move the Harbor task exporter (added in 25f8dde) out of programbench and into a new sibling package src/kotlinai: core conversion, CLI, verifier scripts, and Jinja templates. harbor.py now resolves its own package root and loads templates via PackageLoader("kotlinai", ...); programbench keeps providing shared data-loading/filtering infrastructure. The build backend ships both modules and the programbench CLI registers the harbor sub-app from kotlinai. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add `solve.sh` template to reconstruct source and build gold executables. - Enhance `test.sh` to retry tests serially on xdist crashes. - Extend verifier to include per-branch test failures in reports. - Replace `solve.sh` with branch-agnostic `compile.sh` in solution packages. - Update `README.md` with golden solution details and usage instructions. - Bump `programbench` dependency to 1.2.4.
…onverter. Implement task configuration options.
…omparison CLI - Enable Kotlin and Java toolchains for agent and verifier Dockerfiles. - Add `compare` CLI command for running cross-language experiments. - Enhance `config.yaml` to support language-specific tasks and parallelism. - Update README with language comparison experiment documentation. - Include tests for toolchain integration and language comparison workflows.
- Copy task-level build script (`build.sh`) if not provided in the destination. - Add `scripts/run_oracle_verification.sh` for streamlined Oracle agent execution.
- Introduce `harbor export-config` CLI command for exporting tasks from a Harbor config. - Update shell scripts (`run_agent.sh`, `run_oracle_verification.sh`) to support skipping config exports using the `HARBOR_SKIP_EXPORT` environment variable. - Extend test coverage to include Harbor config export functionality.
…ison experiments
- Raise `n_concurrent_trials` and `parallelism` from 2 to 8 for increased task export concurrency. - Implement `on_error` handler to skip and log failed task conversions without halting the entire batch. - Extend failure reporting to include skipped instances in conversion summaries. - Add new task repositories and task-level `build.sh` script for the `rust-lang__mdbook` task. - Introduce fallback mechanism for task-level build scripts when missing in branches.
…olicies - Introduce `--agent-user` option in `harbor export` to enable Oracle-verifiable tasks requiring root access. - Refine `Dockerfile` to drop unnecessary `NOPASSWD` sudo access for enhanced security. - Update `task.toml` to hand off the full workspace without exclusions, avoiding symlink issues. - Add paired-difference summaries for language comparisons with descriptive 95% confidence intervals. - Pin agent CLI version for experiment consistency, avoiding run-to-run confounds. - Expand agent image with inspection tools (`file`, `xxd`) necessary for trials.
… symlink protection - Move reference binary to a root-only directory (`/opt/programbench/reference`) and replace it with a symlink to prevent unauthorized modifications. - Harden `test.sh` by dropping symlinks that escape or dangle outside the workspace. - Ensure verifier image wipes relocated reference binaries for cleanroom guarantees. - Expand testing to cover relocation, verifier cleanup, and symlink drop logic.
…periment robustness - Extend `AGENT_TIMEOUT_SEC` to 3 hours for long-running tasks, with per-run spend capped at $30 to maintain budget control. - Define a default retry policy (`max_retries=2`) to handle transient API errors while excluding genuine timeouts. - Update `run_agent.sh` with Anthropic API key and base URL support. - Ensure user-supplied configurations override defaults for retry and spend cap. - Expand unit tests to validate new retry logic, cost cap enforcement, and timeout adjustments.
…owed hosts - Require Gradle project structure for Kotlin/Java tasks with offline-compatible dependencies. - Extend default agent allowed hosts to include Maven, Gradle, Kotlin, and related build-tool documentation. - Adjust default retry policy: set `n_attempts=3` and `max_retries=2`. - Update templates and Dockerfile to reflect Gradle integration. - Introduce new dependencies (`ipykernel`, `matplotlib`, `pandas`) in `pyproject.toml`. - Expand test coverage for Gradle enforcement, allowed hosts, and policy adjustments.
|
Hi @Frame17! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.