PhD Oral Exam - Alaa Nfissi, Information and Systems Engineering

When studying for a doctoral degree (PhD), candidates submit a thesis that provides a critical review of the current state of knowledge of the thesis subject as well as the student’s own contributions to the subject. The distinguishing criterion of doctoral graduate research is a significant and original contribution to knowledge.

Once accepted, the candidate presents the thesis orally. This oral exam is open to the public.

Abstract

Emotion is expressed through patterns of speech, language, and visual behavior, but these observations also carry variation unrelated to an affective decision. A learning system must expose useful distinctions while deciding what to retain, suppress, or combine. This dissertation investigates how representations can adapt to that task without abandoning the structures that organize information within and across observations. The investigation begins with acoustic feature spaces refined through supervised feedback and model explanation. It then moves representation formation into networks that learn local waveform responses and their temporal aggregation. Multiresolution analysis organizes this learning within a hierarchy of scales, with adaptable filters and coefficient processing. Learnable frequency, cycle, and order parameters offer a further way to shape time–frequency resolution. Together, these methods locate flexibility in identifiable representation decisions rather than only in a final classifier. A reconstructive branch asks what changes when the output must be a waveform rather than an emotion label. Learned decomposition, coefficient attenuation, and coupled synthesis are developed under an objective that balances reconstruction and sparsity. This branch tests the information-preserving responsibilities of a structured representation. The argument extends beyond speech through modality-specific encoders and cross-attention, which learn how acoustic, linguistic, and visual observations inform one another. Experiments on benchmark corpora and annotated helpline speech examine these choices through task performance, component comparisons, and resource measurements. Their combined contribution is an account of adaptive representation learning in which the location of adaptation, the retained organization, and the required output are considered together. The dissertation develops this account across classification, reconstruction, and multimodal interaction, providing concrete methods and experimentally grounded questions for the continued study of affective information.