Adam Torek recently traveled to Washington D.C. to attend the Frontiers in Education Conference, “A major international conference focusing on educational innovations and research in engineering and computing education leading the world to develop[…]new research insights and educational approaches.”
However, Adam didn’t just attend the conference, he also presented his research, titled “A Systematic Evaluation of Code-generating Chatbots for Use in Undergraduate Computer Science Education,” which was published by IEEE. Adam collaborated on the project with fellow Computer Science students Elijah Sorensen and Natalie Hahle, and together they wrote and submitted the research paper.
Since late 2022, large language models (LLMs) like ChatGPT and Gemini have become widely used across all areas of science, industry, and education, particularly for writing tasks. These LLMs can generate responses to written prompts, whether that be answering a question, summarizing a long document into a brief abstract, or even producing essays. In addition to writing, models can also write computer programs based on prompts, which can be used in developing software.
Adam, Elijah, and Natalie conducted a systematic evaluation of several LLMs using assignments from Computer Science courses. They then created a programming assistant that utilized an LLM for a popular programming environment, VSCode. Finally, they evaluated how both beginner and advanced programmers interacted with this assistant.
Their work highlights the importance of systematically evaluating LLMs on commonly used assignments to assess their effectiveness. This helps instructors understand what adjustments may be necessary for their assignments, whether that is to prevent these assignments from being easily solved by an LLM or to design tasks so students can intentionally incorporate LLMs for learning purposes. Additionally, it provides the department valuable insight on where the LLMs excel (such as basic programming) and where they struggle (like systems programming and theory of computation).
Adam represented Elijah, Natalie, and Dr. Kennington, the faculty member overseeing the course in which these projects were completed and who assisted with the publication process. Dr. Kennington’s research focuses on Natural Language Processing and Spoken Dialogue Systems, and he has been teaching about LLMs in his course since 2020.
Adam completed his Bachelor’s degree in Computer Science in Spring 2024. Prior to finishing his degree, he enrolled in the Computer Science accelerated master’s program, allowing him to take graduate-level courses that counted towards his master’s degree as well as his bachelor’s degree. He transitioned into a full-time master’s student in Fall 2024, in continuing his degree.
Currently, Adam is expanding his research by working on a project aimed at enhancing language models with multi-source information, integrating not just text but also images and real-time interactive data.