Publications

Large Language Models (LLMs) are widely used for code generation, repair, and explanation, yet they still make mistakes when instructions are vague or when their initial assumptions are wrong. Current prompting methods help models show their reasoning but do not encourage them to check whether that reasoning is sound. This confirmatory study introduces MIND, a Socratic-style prompting framework that guides models through self-questions aimed at clarifying the task, identifying missing information, and checking intermediate conclusions. We will evaluate the framework across four benchmarks covering code generation, repair, explanation, and efficiency, comparing it with methods such as Chain-of-Thought, Self-Ask, Self-Refine, and ReAct. We will also run an ablation study to examine which stages of the questioning process contribute most to performance.

As Large Language Models (LLMs) are increasingly adopted in software engineering, recently in the form of conversational assistants, ensuring these technologies align with developers' needs is essential. The limitations of traditional human-centered methods for evaluating LLM-based tools at scale raise the need for automatic evaluation. In this paper, we advocate combining insights from human-computer interaction (HCI) and artificial intelligence (AI) research to enable human-centered automatic evaluation of LLM-based conversational SE assistants. We identify requirements for such evaluation and challenges down the road, working towards a framework that ensures these assistants are designed and deployed in line with user needs.

A growing number of tools have used Large Language Models (LLMs) to support developers' code understanding. However, developers still face several barriers to using such tools, including challenges in describing their intent in natural language, interpreting the tool outcome, and refining an effective prompt to obtain useful information. In this study, we designed an LLM-based conversational assistant that provides a personalized interaction based on inferred user mental state (e.g., background knowledge and experience). We evaluate the approach in a within-subject study with fourteen novices to capture their perceptions and preferences. Our results provide insights for researchers and tool builders who want to create or improve LLM-based conversational assistants to support novices in code understanding.