<\/span><\/h2>\n\n\n\nThomas Helmuth and Lee Spector<\/p>\n\n\n\n
In 2012, researchers in the genetic programming community realized that the field was being held back by reliance on toy benchmark problems [1]. By 2015, some work had been done to improve benchmarking in specialized areas such as symbolic regression [2]. However, no standardized benchmark problems had yet been developed for general program synthesis, the task of automatically generating programs that are similar in many respects to the kinds of programs that humans usually write.<\/p>\n\n\n\n
At GECCO 2015, we published the paper General program synthesis benchmark suite<\/em>, which includes 29 problems taken from introductory programming textbooks [3]. This paper was awarded the ACM SIGEVO 10-years Impact Award at GECCO 2025. Here we briefly reflect on the paper, its impact, and our thoughts about benchmarking program synthesis systems moving forward.<\/p>\n\n\n\nThe benchmark problems provided in our 2015 paper require solution programs to manipulate eight different data types and use large, general-purpose instruction sets that include conditional execution and looping or recursion. Each problem is defined as a supervised learning problem, with specified inputs and corresponding outputs. As an example, the Syllables problem asks for a program that, given a string, returns the number of vowels in the string.<\/p>\n\n\n\n
Our suite of problems, now referred to as PSB1, was accompanied by recommendations for the design of experiments that could use the benchmarks to fairly compare alternative program synthesis methodologies. The paper also included an example usage, experimentally comparing three different parent selection methods (tournament selection, implicit fitness sharing, and lexicase selection) using the PushGP genetic programming system. The results from this experiment are below.<\/p>\n\n\n\nSince 2015, over 200 studies have been published that cite the original PSB1 paper, many of which use the problems to benchmark genetic programming and other program synthesis systems. Sobania, Schweim, and Rothlauf surveyed the use of evolutionary algorithms for program synthesis by looking at all studies that made use of PSB1 [4]. They found that stack-based genetic programming (primarily PushGP), grammar-guided genetic programming, and linear genetic programming were the main types of genetic programming benchmarked with PSB1, and that papers using PSB1 had been published in a variety of conferences and journals. Below is a list of genetic programming systems and techniques that have been benchmarked using PSB1:<\/p>\n\n\n\n
\nlexicase parent selection (and many variants)<\/li>\n\n\n\n grammar design patterns for grammatical evolution<\/li>\n\n\n\n domain knowledge integration in grammatical evolution<\/li>\n\n\n\n novelty selection<\/li>\n\n\n\n generalization of genetic programming solutions<\/li>\n\n\n\n higher-order typed genetic programming<\/li>\n\n\n\n phylogeny-informed fitness estimation<\/li>\n\n\n\n program synthesis with variational autoencoders<\/li>\n\n\n\n code similarity and modularity metrics<\/li>\n<\/ul>\n\n\n\nIn addition, a handful of non-genetic programming program synthesis techniques have used problems from PSB1.<\/p>\n\n\n\n
So, what is the state of the field for program synthesis using genetic programming? We see improvement from 2015, when 22 out of 29 of the PSB1 problems were solved at least once out of 100 runs using PushGP, to today, when 26 of the problems have been [4]. Additionally, PushGP and grammar-guided genetic programming have dramatically improved their frequency of success on many of the PSB1 problems. To avoid over-specialization on this one set of problems, a second program synthesis benchmark suite, PSB2, was published in 2022 [5]. Genetic programming systems have solved some, but not all, of these new problems, and still struggle when given tasks that require larger, more complex programs to solve.<\/p>\n\n\n\n
In the present, when large language models (LLMs) are dominating the conversation of program synthesis, how should we think about genetic programming for program synthesis, and of benchmarking program synthesis? Both in 2015 and today, we see two primary roles for genetic programming in program synthesis: to automate programming tasks based on examples, and to write programs that solve unsolved problems. <\/p>\n\n\n\n
With these goals in mind, are there places where genetic programming may shine compared to LLM program generation? To begin, some solutions to unsolved problems may look so different from anything humans have ever written that LLMs are unlikely to be able to produce them. Additionally, genetic programming usually behaves entirely as a supervised learning system: it is only given examples of the desired behavior, instead of an English-language description of the system. It may therefore make sense to use genetic programming for problems in which it is more natural to give specifications based on inputs and outputs, rather than descriptions of the desired programs. We may also find opportunities to use LLMs and GP together, in ways that produce results that are superior to those of either approach when used individually.<\/p>\n\n\n\n
While genetic programming has been successful in solving the introductory programming problems of PSB1, current systems have trouble scaling up to larger and more complex problems and programs. If we want genetic programming to build larger pieces of software, we will have to improve our methods for automatically generating the modular, hierarchical program structures that humans use to build software. We will also need benchmark problems that assess the ability of our systems to solve novel problems. <\/p>\n\n\n\n
While PSB1 has provided useful benchmarking targets for program synthesis systems over the last decade or so, the research community will require updated benchmark problems and benchmarking practices for solving problems that are larger and more novel. These considerations should guide the future creation of program synthesis benchmark problems.<\/p>\n\n\n\n
References<\/h3>\n\n\n\n\n
\n
\n
[1] James McDermott, David R. White, Sean Luke, Luca Manzoni, Mauro Castelli, Leonardo Vanneschi, Wojciech Jaskowski, Krzysztof Krawiec, Robin Harper, Kenneth De Jong, and Una-May O’Reilly. 2012. Genetic Programming Needs Better Benchmarks. In Proceedings of the 14th annual conference on Genetic and evolutionary computation (GECCO ’12). ACM.<\/p>\n\n\n\n
[2] Miguel Nicolau, Alexandros Agapitos, Michael O’Neill and Anthony Brabazon. 2015. Guidelines for defining benchmark problems in Genetic Programming. In 2015 IEEE Congress on Evolutionary Computation (CEC). IEEE.<\/p>\n<\/div>\n<\/div>\n\n\n\n
[3] Thomas Helmuth and Lee Spector. 2015. General Program Synthesis Benchmark Suite. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation (GECCO ’15). ACM.<\/p>\n\n\n\n
[4] Dominik Sobania, Dirk Schweim and Franz Rothlauf. 2023.A Comprehensive Survey on Program Synthesis With Evolutionary Algorithms. In IEEE Transactions on Evolutionary Computation. IEEE.<\/p>\n\n\n\n
[5] Thomas Helmuth and Peter Kelly. 2021. PSB2: The Second Program Synthesis Benchmark Suite. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO ’21). ACM.<\/p>\n\n\n\n
\n<\/div>\n<\/div>\n\n\n\n
About the Authors<\/h3>\n\n\n\n\n
Thomas Helmuth<\/strong> is the Sacerdote Family Distinguished Associate Professor of Computer Science at Hamilton College in New York, USA. He received his Ph.D. from University of Massachusetts Amherst. His research focuses on algorithmic improvements to genetic programming for general program synthesis. Specific areas of study include genome\/program representation, parent selection, and benchmarking program synthesis.<\/p>\n<\/div><\/div>\n\n\n\n\n
Lee Spector<\/strong> is the Class of 1993 Professor of Computer Science and the Chair of the Department of Computer Science at Amherst College, and an Adjunct Professor and member of the graduate faculty in the College of Information and Computer Sciences at the University of Massachusetts, Amherst. He works on a variety of topics at the intersections of computer science with cognitive science, philosophy, physics, evolutionary biology, and the arts. He formerly served as the Editor in Chief of Genetic Programming and Evolvable Machines and as a member of the ACM SIGEVO Executive Board<\/p>\n<\/div><\/div>\n\n\n\n