GitHub - allenai/codescientist: CodeScientist: An automated scientific discovery system for code-based experiments
8. Benchmark
Included in this repository is a "benchmark" of 50 ideas (paired with plans), that were used to study CodeScientist. These ideas were hand-picked to be relatively diverse, interesting, and likely to be implementable by CodeScientist. They can provide a starting point if you'd like to (for example) (a) study the variance in different aspects of CodeScientist, (b) re-run the same expeirments as in the paper, or (c) run these same ideas through your own automated discovery system to s...
Read more at github.com