Detecting duplicate multiple choice questions (MCQs) in the large question bank

Abstract

A question bank is a database of questions in a variety of formats, used as a central repository for building tests. Question banks are the core tool for innovative testing and assessment of learners' learning outcomes. It can be built and added over time, from many sources, and many people. So creating new questions may result in duplicate questions and deciding whether to include that question in the real database will take time to manually search across data files. This study proposes an end-to-end machine learning architecture to combine the information from the text data and the data from the image in question through optical character recognition into an encoded vector. From there, the system can query to find similar questions based on similarity ranking. The machine learning model was evaluated on a part of the question bank of FPT University, a university of information technology in Vietnam. The obtained F1 score of 0.95 proves that the model can be used for intelligently managing the question bank of the FPT education system as well as of other educational institutions