Skip to content

Support multiple initial programs #126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Support multiple initial programs #126

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
352ee33
update controller
0x0f0f0f Jul 7, 2025
286a70a
change cli
0x0f0f0f Jul 7, 2025
fb5e654
update tests and controller
0x0f0f0f Jul 7, 2025
529a06e
update cli
0x0f0f0f Jul 7, 2025
176849f
Merge remote-tracking branch 'upstream/main' into ale/multiple-initia…
0x0f0f0f Jul 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ An open-source implementation of the AlphaEvolve system described in the Google
OpenEvolve is an evolutionary coding agent that uses Large Language Models to optimize code through an iterative process. It orchestrates a pipeline of LLM-based code generation, evaluation, and selection to continuously improve programs for a variety of tasks.

Key features:

- Evolution of entire code files, not just single functions
- Support for multiple programming languages
- Supports OpenAI-compatible APIs for any LLM
Expand All @@ -34,6 +35,7 @@ The controller orchestrates interactions between these components in an asynchro
### Installation

To install natively, use:

```bash
git clone https://github.com/codelion/openevolve.git
cd openevolve
Expand All @@ -51,7 +53,7 @@ from openevolve import OpenEvolve

# Initialize the system
evolve = OpenEvolve(
initial_program_path="path/to/initial_program.py",
initial_programs_paths=["path/to/initial_program.py"],
evaluation_file="path/to/evaluator.py",
config_path="path/to/config.yaml"
)
Expand Down Expand Up @@ -83,6 +85,7 @@ python openevolve-run.py path/to/initial_program.py path/to/evaluator.py \
```

When resuming from a checkpoint:

- The system loads all previously evolved programs and their metrics
- Checkpoint numbering continues from where it left off (e.g., if loaded from checkpoint_50, the next checkpoint will be checkpoint_60)
- All evolution state is preserved (best programs, feature maps, archives, etc.)
Expand Down Expand Up @@ -145,6 +148,7 @@ python scripts/visualizer.py --path examples/function_minimization/openevolve_ou
```

In the visualization UI, you can

- see the branching of your program evolution in a network visualization, with node radius chosen by the program fitness (= the currently selected metric),
- see the parent-child relationship of nodes and click through them in the sidebar (use the yellow locator icon in the sidebar to center the node in the graph),
- select the metric of interest (with the available metric choices depending on your data set),
Expand All @@ -157,6 +161,7 @@ In the visualization UI, you can
### Docker

You can also install and execute via Docker:

```bash
docker build -t openevolve .
docker run --rm -v $(pwd):/app --network="host" openevolve examples/function_minimization/initial_program.py examples/function_minimization/evaluator.py --config examples/function_minimization/config.yaml --iterations 1000
Expand All @@ -179,6 +184,7 @@ database:
```

Sample configuration files are available in the `configs/` directory:

- `default_config.yaml`: Comprehensive configuration with all available options

See the [Configuration Guide](configs/default_config.yaml) for a full list of options.
Expand All @@ -205,18 +211,23 @@ return EvaluationResult(
```

The next generation prompt will include:

```markdown
## Last Execution Output

### Stderr

SyntaxError: invalid syntax (line 15)

### Traceback

...
```

## Example: LLM Feedback

An example for an LLM artifact side channel is part of the default evaluation template, which ends with

```markdown
Return your evaluation as a JSON object with the following format:
{{
Expand All @@ -226,6 +237,7 @@ Return your evaluation as a JSON object with the following format:
"reasoning": "[brief explanation of scores]"
}}
```

The non-float values, in this case the "reasoning" key of the json response that the evaluator LLM generates, will be available within the next generation prompt.

### Configuration
Expand All @@ -239,7 +251,7 @@ evaluator:

prompt:
include_artifacts: true
max_artifact_bytes: 4096 # 4KB limit in prompts
max_artifact_bytes: 4096 # 4KB limit in prompts
artifact_security_filter: true
```

Expand All @@ -266,6 +278,7 @@ A comprehensive example demonstrating OpenEvolve's application to symbolic regre
[Explore the Symbolic Regression Example](examples/symbolic_regression/)

Key features:

- Automatic generation of initial programs from benchmark tasks
- Evolution from simple linear models to complex mathematical expressions
- Evaluation on physics, chemistry, biology, and material science datasets
Expand Down
18 changes: 12 additions & 6 deletions configs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,23 +5,29 @@ This directory contains configuration files for OpenEvolve with examples for dif
## Configuration Files

### `default_config.yaml`

The main configuration file containing all available options with sensible defaults. This file includes:

- Complete documentation for all configuration parameters
- Default values for all settings
- **Island-based evolution parameters** for proper evolutionary diversity

Use this file as a template for your own configurations.

### `island_config_example.yaml`

A practical example configuration demonstrating proper island-based evolution setup. Shows:

- Recommended island settings for most use cases
- Balanced migration parameters
- Complete working configuration

### `island_examples.yaml`

Multiple example configurations for different scenarios:

- **Maximum Diversity**: Many islands, frequent migration
- **Focused Exploration**: Few islands, rare migration
- **Focused Exploration**: Few islands, rare migration
- **Balanced Approach**: Default recommended settings
- **Quick Exploration**: Small-scale rapid testing
- **Large-Scale Evolution**: Complex optimization runs
Expand All @@ -34,9 +40,9 @@ The key new parameters for proper evolutionary diversity are:

```yaml
database:
num_islands: 5 # Number of separate populations
migration_interval: 50 # Migrate every N generations
migration_rate: 0.1 # Fraction of top programs to migrate
num_islands: 5 # Number of separate populations
migration_interval: 50 # Migrate every N generations
migration_rate: 0.1 # Fraction of top programs to migrate
```

### Parameter Guidelines
Expand Down Expand Up @@ -66,8 +72,8 @@ Then use with OpenEvolve:
```python
from openevolve import OpenEvolve
evolve = OpenEvolve(
initial_program_path="program.py",
evaluation_file="evaluator.py",
initial_program_paths=["program.py"],
evaluation_file="evaluator.py",
config_path="my_config.yaml"
)
```
47 changes: 41 additions & 6 deletions openevolve/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,15 @@ def parse_args() -> argparse.Namespace:
"""Parse command-line arguments"""
parser = argparse.ArgumentParser(description="OpenEvolve - Evolutionary coding agent")

parser.add_argument("initial_program", help="Path to the initial program file")

parser.add_argument(
"evaluation_file", help="Path to the evaluation file containing an 'evaluate' function"
)
parser.add_argument(
"initial_program",
nargs="?",
help="Path to the initial program file",
default=None,
)

Copy link
Author

@0x0f0f0f 0x0f0f0f Jul 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should discuss the positioning of arguments

The MR currently gives the openevolve-run.py CLI this usage:

usage: openevolve-run.py [-h] [--config CONFIG] [--output OUTPUT] [--iterations ITERATIONS] [--target-score TARGET_SCORE]
                         [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [--checkpoint CHECKPOINT] [--api-base API_BASE] [--primary-model PRIMARY_MODEL]
                         [--secondary-model SECONDARY_MODEL] [--initial-programs-dir INITIAL_PROGRAMS_DIR]
                         evaluation_file [initial_program]

Before, initial_program was the first positional argument. I've swapped them, and also added [--initial-programs-dir INITIAL_PROGRAMS_DIR].

It would probably best, if instead of passing the directory, the first argument stayed the evaluator, and then we require one or more initial programs as positional arguments, so users can do something like python /path/to/openevolve-run.py evaluator.py initial_program1.py initial_program2.py other_programs_dir/*.py

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in latest version

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about we keep the existing ordered args for backwards compatibility, in addition we can add a --initial-programs argument that can take either a directory or a list of paths.

Copy link
Author

@0x0f0f0f 0x0f0f0f Jul 8, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was what I attempted in the previous commit, and the logic for args parsing was not looking very good. With the current state, we can definitely do python openevolve-run.py evaluator.py prog1.py prog2.py ./a/b/c/*.py and all other bash goodies. Having positional arguments be initial_program.py evaluator.py and then --initial_programs dir/*.py then causes some issues. One has to always provide one initial program, and cannot omit the first positional argument, and the behavior/ux of defining both initial_program argument and --initial_programs together is not really clear.

I would go for the breaking change if possible! (It also keeps things very simple on OpenEvolve side, no directory listing, etc...)

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have an example where multiple initial programs would be required? If the idea is to evolve like an entire codebase at once, then just having a folder path instead of file path as first argument should be sufficient. We can then use the config.yaml to control what file types or other things are in scope of evolution v.s. out of scope similar to what was suggested in #111

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my use case, then I just want to pick a single output program from many starting programs.

The issue with having the folder path as the first argument (which is what I attempted at the beginning of this branch) is that it requires openevolve cli to perform a lot of extra logic: directory listing, extension parsing, etc...

Having a + vararg allowed me to remove all of this extra logic in latest commits. If the user wants to have many initial programs which are initialized per-island, then the logic of selecting these programs from a directory then can be done via normal bash globbing: openevolve-run evaluator.py file1 file2 dir1/foo/{a.py,b.py}, dir2/**.py.

If we go for something like openevolve.py initial_program.py evaluator.py --initial-programs=dir/ then we enter problems:

  1. All the extra directory listing logic is required in openevolve: (Ignore readmes, check file extensions, etc...)
  2. Users then have to move their initial programs to a directory with no other files.
  3. Being the first positional argument, initial_program.py is not optional, so initial_program.py and --initial-programs=dir/ would not be interchangeable.

So I think this breaking change is worth it, to not reinvent the wheel :)

For evolving an entire codebase, which is a different goal from mine, we could have a openevolve-run.py --codebase mode later. The logic would be that each starting program will need an output program, so it's quite different.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it is about just assigning different initial programs to different islands, we can do it via the config.yaml We can have one initial program.py as we do and in teh config.yaml with the island configs we can also add path to other initial programs which can be initialized to the islands. Will need to ensure that the config has sufficient islands so it may be better to define them at the same place in the config itself.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will need to ensure that the config has sufficient islands

This is done in this PR. After initializing config, if number of islands is < number of programs then it throws. Very simple check

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's done when loading the programs

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also thought about that, but it might also introduce additional logic. I don't really like the idea, because having inputs defined in a config feels like violating the UNIX philosophy

              | Config           
              v                  
        +----------------+       
Input(s)|                | Output
 ------>|   Program      +------>
        |                |       
        +----------------+

parser.add_argument("--config", "-c", help="Path to configuration file (YAML)", default=None)

Expand Down Expand Up @@ -57,6 +61,8 @@ def parse_args() -> argparse.Namespace:

parser.add_argument("--secondary-model", help="Secondary LLM model name", default=None)

parser.add_argument("--initial-programs-dir", help="Path to initial programs directory", default=None)

return parser.parse_args()


Expand All @@ -69,9 +75,38 @@ async def main_async() -> int:
"""
args = parse_args()

# Check if files exist
if not os.path.exists(args.initial_program):
print(f"Error: Initial program file '{args.initial_program}' not found")
# Check if files exist.
# If args.initial_program is present, it should be a file.
# If args.initial_programs_dir is present, it should be a directory, and args.initial_program should be `None`.
if args.initial_programs_dir:
if args.initial_program:
print("Error: Cannot specify both initial-program and --initial-programs-dir")
return 1
if not os.path.isdir(args.initial_programs_dir):
print(f"Error: Initial programs path '{args.initial_programs_dir}' is not a directory")
return 1
elif args.initial_program:
if args.initial_programs_dir:
print("Error: Cannot specify both --initial-programs-dir and initial_program")
return 1
if not os.path.isfile(args.initial_program):
print(f"Error: Initial program file '{args.initial_program}' is not a file")
return 1

# Populate the initial_programs_paths vector.
initial_programs_paths = []
if args.initial_programs_dir:
initial_programs_paths = [
os.path.join(args.initial_programs_dir, f)
for f in os.listdir(args.initial_programs_dir)
if f.endswith(".py")
]
elif args.initial_program:
initial_programs_paths = [args.initial_program]

# Check that initial_programs_paths is not empty
if not initial_programs_paths:
print("Error: No initial programs found. Please provide a valid initial program or directory.")
return 1

if not os.path.exists(args.evaluation_file):
Expand Down Expand Up @@ -100,7 +135,7 @@ async def main_async() -> int:
# Initialize OpenEvolve
try:
openevolve = OpenEvolve(
initial_program_path=args.initial_program,
initial_programs_paths=initial_programs_paths,
evaluation_file=args.evaluation_file,
config=config,
config_path=args.config if config is None else None,
Expand Down
83 changes: 54 additions & 29 deletions openevolve/controller.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ class OpenEvolve:

def __init__(
self,
initial_program_path: str,
initial_programs_paths: List[str],
evaluation_file: str,
config_path: Optional[str] = None,
config: Optional[Config] = None,
Expand All @@ -86,9 +86,15 @@ def __init__(
# Load from file or use defaults
self.config = load_config(config_path)

# Set up output directory
# Assert that initial_programs_paths is a list, and not empty
if not initial_programs_paths:
raise ValueError("initial_programs_paths must be a non-empty list of file paths")

# Set up output directory.
# If output_dir is specified, use it
# Otherwise, if initial_programs_paths has a single path, use the directory of the initial program.
self.output_dir = output_dir or os.path.join(
os.path.dirname(initial_program_path), "openevolve_output"
os.path.dirname(initial_programs_paths[0]), "openevolve_output"
)
os.makedirs(self.output_dir, exist_ok=True)

Expand Down Expand Up @@ -122,20 +128,31 @@ def __init__(
logger.debug(f"Generated LLM seed: {llm_seed}")

# Load initial program
self.initial_program_path = initial_program_path
self.initial_program_code = self._load_initial_program()
self.initial_programs_paths = initial_programs_paths
self.initial_programs_code = self._load_initial_programs()

# Assume all initial programs are in the same language
if not self.config.language:
self.config.language = extract_code_language(self.initial_program_code)
self.config.language = extract_code_language(self.initial_programs_code[0])

# Extract file extension from initial program
self.file_extension = os.path.splitext(initial_program_path)[1]
self.file_extension = os.path.splitext(initial_programs_paths[0])[1]
if not self.file_extension:
# Default to .py if no extension found
self.file_extension = ".py"
else:
# Make sure it starts with a dot
if not self.file_extension.startswith("."):
self.file_extension = f".{self.file_extension}"

# Check that all files have the same extension
for path in initial_programs_paths[1:]:
ext = os.path.splitext(path)[1]
if ext != self.file_extension:
raise ValueError(
f"All initial program files must have the same extension. "
f"Expected {self.file_extension}, but got {ext} for {path}"
)

# Initialize components
self.llm_ensemble = LLMEnsemble(self.config.llm.models)
Expand All @@ -160,7 +177,7 @@ def __init__(
)
self.evaluation_file = evaluation_file

logger.info(f"Initialized OpenEvolve with {initial_program_path}")
logger.info(f"Initialized OpenEvolve with {initial_programs_paths}")

# Initialize improved parallel processing components
self.parallel_controller = None
Expand Down Expand Up @@ -189,10 +206,13 @@ def _setup_logging(self) -> None:

logger.info(f"Logging to {log_file}")

def _load_initial_program(self) -> str:
"""Load the initial program from file"""
with open(self.initial_program_path, "r") as f:
return f.read()
def _load_initial_programs(self) -> str:
"""Load the initial programs from file"""
programs = []
for path in self.initial_programs_paths:
with open(path, "r") as f:
programs.append(f.read())
return programs

async def run(
self,
Expand Down Expand Up @@ -226,29 +246,34 @@ async def run(
should_add_initial = (
start_iteration == 0
and len(self.database.programs) == 0
and not any(
p.code == self.initial_program_code for p in self.database.programs.values()
)
)

if should_add_initial:
logger.info("Adding initial program to database")
initial_program_id = str(uuid.uuid4())
logger.info("Adding initial programs to database")

# Evaluate the initial program
initial_metrics = await self.evaluator.evaluate_program(
self.initial_program_code, initial_program_id
)
if len(self.initial_programs_code) > len(self.database.islands):
raise ValueError(
"Number of initial programs exceeds number of islands."
)

initial_program = Program(
id=initial_program_id,
code=self.initial_program_code,
language=self.config.language,
metrics=initial_metrics,
iteration_found=start_iteration,
)
for i, code in enumerate(self.initial_programs_code):
initial_program_id = str(uuid.uuid4())

# Evaluate the initial program
initial_metrics = await self.evaluator.evaluate_program(
code, initial_program_id
)

initial_program = Program(
id=initial_program_id,
code=code,
language=self.config.language,
metrics=initial_metrics,
iteration_found=start_iteration,
)

self.database.add(initial_program)
# TODO. Should the island be incremented and reset here?
self.database.add(initial_program, 0, i)
else:
logger.info(
f"Skipping initial program addition (resuming from iteration {start_iteration} "
Expand Down
Loading