Detail

MENTOR: Automated Feedback for Introductory Programming Exercises

The increasing demand for programming education has given rise to all kinds of online evaluations such as Massive Open Online Courses (MOOCs) focused on introductory programming assignments (IPAs). As a consequence of a large number of enrolled students, one of the main challenges in these courses is to provide valuable and personalised feedback to students. In my talk, I will present MENTOR, a semantic automated program repair (APR) framework designed to provide Automated Feedback for Introductory Programming Exercises. MENTOR addresses this challenge by generating possible repairs for faulty student programs, enabling semantic repairs validated through execution on a test suite and by highlighting these faulty statements to the students. Hence, in the context of this work, we provide scientific contributions in several areas, such as program clustering and analysis, automated fault localisation and program repair. MENTOR advances the state of the art in the referred areas and provides an innovative practical framework to be deployed in educational environments.

Unlike symbolic repair tools like Clara and Verifix, which require correct implementations with identical control flow graphs (CFGs), MENTOR’s Large Language Model (LLM)-based approach enables flexible repairs without strict structural alignment. MENTOR clusters successful submissions regardless of CFGs and employs a Graph Neural Network (GNN)-based variable alignment module for enhanced accuracy. MENTOR’s fault localisation module, CFaults, leverages Maximum Satisfiability (MaxSAT) techniques to pinpoint buggy code segments precisely. MENTOR’s program fixer integrates Formal Methods (FM) and LLMs through a Counterexample Guided Inductive Synthesis (CEGIS) loop, iteratively refining repairs. Furthermore, in my talk I will also present GitSEED, which is a language-agnostic automated assessment tool that enhances student learning by providing personalised feedback on code submissions and successfully integrates CFaults for effective fault detection on student code. Experimental results on C-Pack-IPAs demonstrate that MENTOR significantly improves repair success rates, achieving 64.4%, compared to just 6.3% for Verifix and 34.6% for Clara.