Unsupervised concept extraction in an introductory programming course

Abstract

Manually determining concepts present in a group of questions is a challenging and time-consuming process. However, the process is an essential step while modeling a virtual learning environment since a mapping between concepts and questions using mastery level assessment and recommendation engines are required. This thesis provides tools to assist in closing the learning feedback loop focused on concept extraction. We investigated unsupervised semantic models (known as topic modeling techniques) to assist computer science teachers in this task and propose a method to transform Computer Science 1 teacher-provided code solutions into representative text documents, including the code structure information. First, we projected, implemented, and deployed a learning environment based on a teaching methodology to collect professors and student data. Then, we extracted the underlying relationship between questions and validated the results using an external dataset by applying non-negative matrix factorization and latent Dirichlet allocation techniques. We considered the interpretability of the learned concepts using 14 university professors’ data, and the results confirmed six semantically coherent clusters, achieving 0.75 in the normalized pointwise mutual information metric. The metric correlates with human ratings, making the proposed method useful and providing semantics for large amounts of unannotated code. Finally, we compare this method with methods focused on the students’ knowledge to extract the questions’ latent semantic relationship through the students’ perspective. As we could not find a significant relationship between concepts found using the student-focused methods and the ones provided by the professors, we proposed a new visualization to track student performance and pinpoint students’ difficulties and achievements.