The Verification Playground

Verification Playground focuses on advanced functional verification, an aspect of software testing that lies beyond the capabilities of fully automatic verification. Instead, it relies on human expertise to suitably encode programs and specifications into meaningful invariants. This approach requires an understanding of program design and programming, as well as knowledge about how to write partial specifications and implement invariants efficiently.

The Verification Playground consists of a series of competitions that focus on challenging participants' abilities to verify detailed behavioral properties--aspects of program design that lie outside the realm of most verification tools and are thus difficult to automatically test. The competitions also emphasize the human aspect of the verification process, which in many cases can be overlooked in the rush to produce solutions.

A few months before the competition, we sent out a "Call for Problems" asking for suggestions of challenges that could be used during the competition. Two people submitted proposals by the recommended deadline, and one more problem proposal arrived later, close to the competition date.

We analyzed the proposals and arranged interviews with teams in order to gain insight into their preparations. We asked each team to explain their solution and point out areas where they needed more help. In addition, we compiled an extensive list of possible 검증놀이터 topics that could appear in the competition.

As a result of this analysis, we decided that the competition challenges were largely similar in nature and focused on common aspects of program design. Nevertheless, the challenges were different enough that teams had to adapt their preparations and solve them in a unique way.

In addition, the competition challenges were selected to be suited to the skills and experience of a broad range of teams. This ensured that no team would be disadvantaged by the fact that they were using a specialized tool for their implementation.

Moreover, the competition challenges were selected to target various domains and properties, so that no tool was exclusively advantaged or disadvantageed by its implementation of these features.

The competition challenges were based on a series of specifications provided by the organizers, with a few minor modifications adapted to the specific needs and challenges of the participating teams. This approach enabled us to judge the solutions in a more objective manner than we could have done had the tasks been formulated independently of the tools involved.

We had a small number of participants in the competition, so we were able to give each team a thorough interview during which they explained their approach to the challenges and pointed out areas where they needed more help. This gave us a better idea of how to improve the future challenges, as well as what problems and features we should avoid addressing in VerifyThis's future editions.

Despite their relatively limited time, most of the teams were able to submit solutions that both were correct and complete. For example, Team openjml completed challenge 2's part A while also completing the task traversal of part B--despite the fact that it had not been explicitly marked as optional. Similarly, Team vercors managed to work on both part A and part B, even though it was only a minor hint that part B was simpler than part A.