PhD Oral Exam - Cheng Cheng, Software Engineering

When studying for a doctoral degree (PhD), candidates submit a thesis that provides a critical review of the current state of knowledge of the thesis subject as well as the student’s own contributions to the subject. The distinguishing criterion of doctoral graduate research is a significant and original contribution to knowledge.

Once accepted, the candidate presents the thesis orally. This oral exam is open to the public.

Abstract

Large language models (LLMs) increasingly support code generation at different granularities, from individual functions to complete software projects. However, generated code may contain security weaknesses even when it executes successfully or satisfies functional requirements. Existing research investigates different stages of vulnerability management and provides benchmarks and evaluation methods for studying the security of LLM-generated code. Nevertheless, three limitations remain. First, benchmark coverage across code-generation granularities is limited. Second, methodological support for systematic security-centered evaluation remains insufficient. Third, securing LLM-generated code through integrated vulnerability management remains insufficiently investigated. This thesis addresses these limitations through three complementary studies. The first study establishes security-oriented benchmarking settings across function-level, class-level, and file-level generation to characterize weaknesses in LLM-generated code. The second study develops CFCEval, a security-centered evaluation framework for assessing generated code and fixes through consistent procedures and multiple dimensions. These include programming language quality, fixing capability, post-transformation fixing capability, and element-level relevance to security-relevant reference code. The third study investigates project-level vulnerability management through a Detect–Confirm–Repair–Verify (DCRV) workflow. Using LLMGProjects, a benchmark comprising 48 implementations of 16 project specifications generated by three LLMs, it examines detection reliability, the effects of verified repair guidance, and the marginal benefit of iterative repair. The project-level findings show that more reported vulnerability candidates do not necessarily imply higher detection precision, and richer descriptive guidance does not consistently improve repair outcomes. Guidance that includes test information produces the strongest observed secure-and-correct outcomes. Additional repair attempts provide limited gains and can introduce regressions, highlighting the importance of independent vulnerability confirmation and joint functional and security verification.

Together, these studies advance the security assessment and improvement of LLM-generated code by establishing benchmarks across code-generation granularities, developing a systematic framework for evaluating generated code and fixes, and empirically examining the effectiveness and limitations of integrated vulnerability management for securing generated projects.