Skip to content

fix: deterministic path resolution in scoring#335

Draft
n0w0f wants to merge 2 commits into
devfrom
fs_fix
Draft

fix: deterministic path resolution in scoring#335
n0w0f wants to merge 2 commits into
devfrom
fs_fix

Conversation

@n0w0f

@n0w0f n0w0f commented Apr 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Remove heuristic path resolution (smart_resolve_path, rglob, regex parsing) from the scoring hot path — replaced with deterministic resolution: relative paths are prepended with the trial workspace directory, absolute paths and JSON data pass through unchanged.
  • WriteFileTool and CopyFileTool now return absolute paths so agents see the real file location, closing the FSManager abstraction gap.
  • FSManager scoped to per-trial workspace instead of the global base_work_dir, so relative paths from tools resolve correctly within each trial.
  • Dead resolve_path functions removed from catalyst and ML score modules.
  • ML tools fixed to consistently resolve paths via resolve_working_dir_path (consolidate, select, filter, prepare, train tools).

Files changed

File Change
src/corral/utils/task_group.py Deterministic score(), FSManager scoped to trial workspace
src/corral/utils/io_tools.py WriteFileTool/CopyFileTool return absolute paths
tasks/catalyst/src/catalyst/score.py Remove smart_resolve_path, dead resolve_path
tasks/ml/src/ml/score.py Remove dead resolve_path
tasks/ml/src/ml/tools.py Simplify + fix path resolution in tools
tasks/resistor_network/src/resistor_network/score.py Remove smart_resolve_path
tasks/catalyst/tests/test_environment_integration.py Updated for new path behavior
tasks/ml/tests/test_unit.py Remove deleted resolve_path test

Test plan

  • cd tasks/catalyst && CORRAL_WORK_DIR=$(mktemp -d) uv run python -m pytest tests -v --rootdir=.
  • cd tasks/ml && CORRAL_WORK_DIR=$(mktemp -d) uv run python -m pytest tests -v --rootdir=.
  • cd tasks/resistor_network && CORRAL_WORK_DIR=$(mktemp -d) uv run python -m pytest tests -v --rootdir=.
  • Run catalyst + ML benchmarks end-to-end (ToolCalling and ReAct)

n0w0f added 2 commits March 31, 2026 16:17
… TaskGroupEnvironment

- Add WorkspaceRegistry (SQLite-backed) for tracking workspaces and files across trials
- UUID-based workspace directory naming for uniqueness and persistence
- Global FSManager scope (base_work_dir) enabling cross-trial file access for interventions
- Registry-backed path resolution replacing fragile rglob search
- File tracking on FSManager writes via registry
- Consolidate duplicate TaskGroupEnvironment classes: ML and samplemath now use core class
- Core TaskGroupEnvironment supports hidden_args_keys, fsmanager_app, extra_file_tools, extra_fsmanager_tool_classes
- corral_md subclasses core, overriding only prompt and score
- Add workspace_id to schema, server, router, and report plumbing
- Add CLAUDE.md for repository guidance
…in scoring

Remove smart_resolve_path/rglob from the scoring hot path. Tools now
return absolute paths (WriteFileTool, CopyFileTool) and score() resolves
relative paths deterministically against the trial workspace directory.
Eliminates fragile regex parsing, registry lookup, and filesystem search.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, we are unable to review this pull request

The GitHub API does not allow us to fetch diffs exceeding 20000 lines

@coderabbitai

coderabbitai Bot commented Apr 2, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: d3ac9451-55df-4428-94ed-175878d6404f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

Comment @coderabbitai help to get the list of available commands and usage tips.

@MrtinoRG

Copy link
Copy Markdown
Collaborator

Is this not needed for the webpage as well? Or for what is this PR?

@MrtinoRG

Copy link
Copy Markdown
Collaborator

@n0w0f this should be merged before #345, as I understand that it is needed for the correct behavior of the landing page, right?

@n0w0f
n0w0f changed the base branch from main to dev April 19, 2026 16:31
@n0w0f

n0w0f commented Apr 19, 2026

Copy link
Copy Markdown
Collaborator Author

@n0w0f this should be merged before #345, as I understand that it is needed for the correct behavior of the landing page, right?

This is not needed for the landing page, this is an enhancement on utils for the environments that need a workspace to write and read files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants