1.   Introduction

Education from, traditional pedagogy to modern pedagogy a matter of transition has been done over the year from pedagogy to other pedagogical paradigm a matter of transition has been from basic pedagogy to modern pedagogy like some traditional but digitalized, interactive more personalized learning process have been done. Nevertheless, there is a major drawback in existing training and testing platforms, which is the limited nature of their question databanks when it comes to up-to-dateness, diversity and pedagogical relevance. Such shortcomings are obstacles to the efforts of students to engage in deep and nuanced learning as well as places a heavy demand on instructors, who are ever expected to develop fresh, high-quality assessment content [1]. The state-of-the-art studies are mainly based on heterogeneous static and fixed set of questions that are far from being able to cater for dynamic and personalized learning of students.

The aim of this proposal is to develop an emerging domain for Question generation with the Generative artificial intelligence (AI)-based system which can able to generate rich semantic and pedagogical sound question types from a variety of diverse heterogeneous sources such as text, audio, video and heterogeneous document types like PDF, PPTX and DOCX [2]. Such an advanced system is intended to alleviate the mentioned disadvantages of periodic platforms in a strong degree, to the level that the question generation burden posed on teachers is highly relieved and to feed the provision of up to date content on the current learning trajectory of students. To this end, our system selectively employs the power of pre-trained large language models (LLMs: Large Language Models - artificial intelligence models containing billions of parameters with natural language understanding and generation capabilities), a backbone of generative AI, to generate questions across a continuum of difficulty and types, for example, multiple-choice, true/false, fill-in-the-blank, open-ended, matching, and short-answer text forms [3]. In addition, the Human-in-the-Loop (HITL) mechanisms allow to iteratively improve the linguistic consistency as well as the pedagogical relevance of the generated questions in context.

Our study begins with a structural examination of inadequacies in current educational systems, highlighting the need for AIbased automatic question generation [4]. But more than this static quality and intrinsic limitations, the never-ending requirement for teachers to develop new content is recognized as a key element that reduces the efficiency of education. To cope with these, we present a dynamic question generation pipeline which automatically aggregates and composes information from heterogeneous documents. This approach empowers and personalizes students' learning experiences and frees educators from the never-ending chore of generating new questions.

Contributions The core contributions of this work include the design of a complete generative AI architecture that combines state-of-the-art knowledge extraction from heterogeneous data, advanced summarization of text, and a single framework for data processing. A comprehensive comparative study on the performance of different large language models is performed and human-quality outputs are evaluated in a rigorous manner based on question meaningfulness, question format, difficulty level, answer correctness, question clarity, and overall quality [5]. Experimental results clearly show that the questions produced by our system are technically sound questions and pedagogical suitable for students' needs. The empirical results of the generated system demonstrate that our system is able to generate high-quality, up-to-date questions of widely varied question types and difficulty levels. Importantly, by integrating human feedback mechanisms, the grammatical accuracy, contextual relevance and pedagogical validity of the automatically generated questions can be greatly improved. These results provide strong evidence that question generation from generative AI systems can be a powerful tool for transforming how assessment and evaluation take place in educational settings. In summary, this study sets an important groundwork for future investigation by demonstrating the practical feasibility and tremendous power of automated question generation with question generation-based generative AI in educational technology. The rigorous process and the attractive outcomes were able to 45 overcome not only the shortcomings of other solutions but also to open the door to new proposals which aim at increasing the quality and efficiency of the educational process.

This study sets new standards for educational technology by showing that automatic question generation gives rise to pedagogical validity which is on par with expert-authored content when combined appropriately with human expertise and multimodal data processing. The systematic comparison of model performance is indispensable to guide educational technology transfer, and the HITL integration approach can serve as a replicable blueprint to ensure educational quality of AI-supported content production. These are the implementations that allow to automatically generate educational content on teacher-specified topics. Beyond current limitations and serves as a canvas for exploring future refinements in automatic generation of educational content.

The main technical novelty of this work is to adopt the full modal approach. As opposed to the traditional methods that are largely text-centric question generation [15,20,32], this work is the first to build an end-to-end system that collectively involves…(I) audio, video, document, (II) advanced speech-to-text conversion with faster-whisper technology (it is an improved version of OpenAI’s Whisper model for speech-to-text conversion), sophisticated document parsing using PyMuPDF (a Python library that could extract the textual content from pdf files), python-docx (a Python library used for manipulating Word files), python-pptx (a Python library that helps in the manipulation of PowerPoint files) and (III) intelligent summarization methods. This resolves the major limitation on the existing works that the heterogeneity of the educational resources are still under-exploited for automatic generation of assessment, leaving a big gap between the sophisticated multimodal educational contents and the text-only automated assessment systems.

Additionally, this paper presents the first comprehensive end-to-end HITL pipeline tailored to educational question generation. While some work acknowledges partial expert validation [7] or teacher feedback [12], our iterative feedback process with automatic revision mechanisms stands as a step change from static question generation to dynamic, expert-driven content optimization. This ‘bridges’ the two key quality and pedagogical gap that have identified in the literature [27,28], where automated systems may generate technically correct but pedagogically incorrect material since there is no systematic expert validation at the time of material generation.

The work also introduces adaptive prompt engineering strategies that adapt based on content difficulty, model performance, and educational goals. Our system differs with the previous researches in which static type prompt approaches are used [29, 30] instead our system is using smart selection of prompt and it is designed a prompt generation as a function of input pattern to generate the question according to the given level of difficulty and educational context. This is a remarkable step forward for the application of generative AI to education over one-size-fits-all prompting strategies of yester-years by enabling context-aware, adaptive content generation which is contingent on specific content traits and target learning objectives. In addition, we provide the largest-scale comparison cross-LLMs in educational question generation literature to date, closing the gap on seven state-of-the-art LLMs on six types of questions using well-defined pedagogical measures. Existing work has often examined single models or narrow comparisons [31], and our framework fills a significant gap in model selection recommendations for educational applications by presenting data on a model's performance across a variety of educational contexts and types of questions. This systemization approach facilitates evidence-based decision making for the adoption of educational technology.

The work also presents the initial effort to systematically generate question that align with grade level (5th - 8th) with teacher validated difficulty level. This meets the pressing need for intelligent content calibration for a particular set of developmental stage and cognitive ability of their target student populations, since the age-appropriate for automatically generated tests has already been a massive gap to be studied, as already highlighted in literature [26]. Existing automatic question generation systems have several key limitations that are directly targeted by this work such as small modality support as the majority only process text data [9, 10, 25]; narrow question type coverage as they are focused on one or two of the possible question formats [17, 33]; lack of systematic expert feedback provisions in most systems [21, 24]; inadequate educational domain fine-tuning available in general-purpose models [22, 23]; and lack of extensive evaluation frameworks with little pedagogical assessment criteria available [32].

2.   Related Works

In this section, we survey work in the domains of automatic question generation, HITL in education and Heterogeneous data processing over heterogeneous sources.

2.1. Automatic Question Generation

Automatic question generation has become more and more important for dynamically updating educational resources and enhancing assessment service. The last few years, transformer-based models (attention mechanism-based deep learning architectures that show high performance especially in language processing tasks), deep learning and hybrid approaches have been focused on developing new questions from the text, audio, video, and document content. Bhowmick et al. [6] introduced a complete strategy to automated question generation from educational text, addressing context-loss, question-quality, diversity, and linguistic inconsistency issues. They combined deep learning–based NLP (Natural Language Processing: the technology for computers to understand and process human language) with handcrafted rules and patterns for educational content, and used attention mechanism and context analysis methods to recognize the main concepts and relationships of the text. In particular, the rule-based system was successful in producing accurate and meaningful question templates that cover educational terminology and conceptual structures. Experiments on wide-spectrum datasets (textbooks, online learning data) demonstrated that with higher accuracy, better context alignment, more question diversity over previous methods. However, issues included semantic ambiguity, requiring full context, and adjust- ing to educational domains. These were resolved through error examination and the model was optimized iteratively. At a high level, the authors provided a novel manner to improve automated question generation for education, increase student attention and awareness, and to make instructional materials more effective.

Recent innovations in educational question generation using large language model applications have shown promise in higher-education settings. Ling and Afzaal [56] studied the use of pre-trained large language models for the automatic generation of question-answer pairs in higher education scenarios. They conducted an extensive study on performance of transformer-based models when generating contextually relevant and pedagogically appropriate questions based on academic documents. The authors have shown that LLMs could make a significant relief for teachers in their effort to maintain high-quality education material for students. The experimental results indicated that there were significant improvements in question diversity and Semantic coherence comparing to traditional rule-based approaches, while the issues of domain expansion and the calibration of the degree of difficulty, are also worth to solve in different levels.

Recent advances in deep learning methods have also helped improve automatic question generation. Abdel-Galil et al. [57] proposed an end-to-end question generation model with CNN, RNN and reinforcement learning for high-quality and relevant educational content generation. Their model, leveraging state-of-the-art neural network architecture, was able to capture textual context and produce questions that target teaching goals. The study showed the superiority of attention mechanisms combined with contextual encoding for question clarity and educative value. Nevertheless, the authors reported current difficulties to keep up with semantic coherence in heterogeneous educational domains, and that further model refining was needed from pedagogical feedback.

Recent developments in deep learning approaches have also contributed to advancing automatic question generation capabilities. Abdel-Galil et al. [57] presented an automatic question generation model based on deep learning approaches, focusing on improving the quality and relevance of generated educational content. Their model incorporated advanced neural network architectures to better understand textual context and generate questions that align with learning objectives. The research demonstrated improvements in question clarity and educational value through the integration of attention mechanisms and contextual encoding techniques. However, the authors noted ongoing challenges in maintaining semantic consistency across diverse educational domains and the need for continuous model refinement based on pedagogical feedback.

Kurdi et al. [7] presented a survey of automatic question generation for educational purposes. They looked at rule-based, template-based, and deep learning – based methods and how they address context alignment, question quality and difficulty level determination and alignment to educational objectives. They carried out a multi-dataset evaluation including several educational sources, highlighting (dis)advantages in terms of different metrics. The addresses prioritized issues that keep coming up, such as a lack of question variety and loss of context, suggesting future research to address these concerns. Kusuma et al. [8] proposed an ontology-based approach for question generation with various levels of difficulty. They recognize key concepts in education texts through ontologies and apply specialized query templates for generating questions. This hybrid approach combines ontological structures with deep learning models, and has been experimentally evaluated in different educational contents. Difficulties were encountered in creating the ontology and in the right template choice, these difficulties were alleviated by parameter adjustment and data cleaning. The results showed that the system is able to generate questions that are faithful to the textual content while adjusting to varying student skill levels.

Srivastava et al. [9] introduced an end-to-end QA system “Questionator,” which relies on deep learning to generate questions automatically. Their pipeline has the text preprocessing phase, the neural network architecture (focus on attention mechanisms), and the rule-based components to keep the context and diverse questions. The educational materials that were collected came from different sources and the results of some in-lab testing showed better efficiency and precision than the conventional methods. Other remaining challenges including linguistic uncertainty and lack of retention of full context were tackled with a process of refinement and further rule-based logic. Shi et al. [10] proposed an architecture for an automatic question generating system, including text preprocessing, feature extraction and model training on NLP platforms. Their technique is designed to harvest useful information from educational texts and use it to generate intelligible questions. Based on deep learning and language modeling, the system achieved good performance in context alignment and the selection of correct answers. Challenges related to linguistic irregularities and data quality were addressed by fine-tuning the model and additional preprocessing.

Madhav et al. [11] Leveraged LangChain for generating questions through PDFs in automation fashion. In their paper, the author describes ‘how we can use a PDFs’ text extraction to concatenate with the NLP model, and then converting those into questions’. The method exploits LangChain, language modeling and text extraction algorithms. Experiments conducted on multiple document types verified its efficiency, data quality and extraction errors were considered by improving the preprocessing and tuning it. Morón et al. [12] proposed a system that can generate automatic ques- tions for elementary level of English learning. They constructed simple question templates based on basic grammar and vocabulary with the help of NLP approaches. Questions were filtered by rule-based filtering and level detection so that they were consistent with the proficiency of the learner. The datasets included beginner-level texts, and the experimental results suggest the tool can help learners learn the language. Problems of language simplicity and semantic coherence were mitigated through further customization of language-model settings.

Tsai et al. [13] presented an autoquestion generation technique for repeat testing to enhance student learning. They utilized deep learning models and NLP algorithms for contextual coherence preservation and to generate diverse and accurate questions from educational documents. In this study, multiple sources of data were examined, and a statistically significant effect was obtained on student achievement. Difficulties such as language ambiguity and context matching were addressed with not only model optimization, but also additional strategies. Varalakshmi and Bugath [14] developed a customized testing system for interviews, by generating the dynamically question and answer pair for the candidate resume. It scrutinizes resume records with natural language processing (NLP) and designs questions based on the candidate’s strengths. The system is based on language processing algorithms and AI-powered analytics and enhanced interview preparation based on experimental evaluations. Issues of data size and precise classification of resume were addressed using sophisticated pre-processing and classification.

Rathi et al. [15] offered a natural language processing (NLP) technique for automatic question-generating from text-like information, which involves text pre-processing, keyword extraction, summarization, and question construction. Their models use language modeling and machine learning algorithms to maintain context and generate diverse questions. Experimental results demonstrated the effectiveness of our System on aligning contexts, correctness and diversity of questions. Linguistic ambiguities and partial context references were refined thereafter by subsequent optimisation and filtering. Paramasivam and Nirmal [16] used an entailment centred question generation system to enhance reading comprehension by determining important concepts and relationships in passages. They used deep learning and language modeling to generate questions that elicited deeper-level processing. Experimental results indicated students had a better understanding of concepts. Limitations, e.g., partial coverage of inference relations and linguistic irregularities, were remedied by optimizing the model and extending preprocessing.

Pochiraju et al. [17] proposed a two-steps process which first has a headline generation stage and then produces MCQ based on the headline. The summary is extracted by XLNet-based model, and distractors are generated by specialized algorithm. The system perfor med well on context alignment, question quality, and diversity, although infor mation loss in the summarization and the difficulty of generating distractorseture additional optimization. Ch and Sah [18] worked in the field of automatically generating multiple-choice questions from school-level textbooks. They retrieve main ideas and content, create question patterns and then generate distractors based on NLP and rule-based techniques. Experiments showed the system can generate correct, readable questions relevant to school level educational materials. Issues related to linguistic simplicity and coherence of content, as well as distractor generation were addressed using additional rule sets and model’s refinements.

Ch and Sah [19] provided a comprehensive survey of automatic MCQ generation techniques, including rule-based, template-based, and machine learning-based methods. Their conversation emphasized issues such as the retention of context, linguistic variety, distractor generation, and question difficulty. They cross-examined various datasets and measures which clarify where each method performs well and how future research could be improved. Kumar et al. [20] presented an encoder-decoder model with attention mechanisms for automatic question-answer pair generation. Attention is payed in order to preserve the context information and align the meaning of the answers to the question. On educational datasets, they experimentally demonstrated high accuracy, though a few isolated cases of context misalignments and attention oversights indicated a need for more careful model tuning and data cleaning or preprocessing.

Islam et al. [21] studied AI- empowered collaborative learning platforms and found deficiencies in the personalization, feedback and interaction types provided. By using AI techniques (for example, recommendation systems, clustering) to enhance adaptivity, they experimented with real-world learning datasets and found an increase in student engagement and collaborative learning. Technical challenges such as scalability and integration with legacy systems were solved through state-of-the-art data processing technologies. Narayana et al. [22] concentrated on developing automatic question papers, which made use of NLP methods to pose questions, such as balanced question sets, consistency, and different difficulty levels. Their method is based on a combination of text analysis, template-based question generation, and semantics inference. Experiments on past exam papers and textbooks show story and question diversity and adequate difficulty calibration, and iterative improvements are found to address linguistic inconsistencies. Fung et al. [23] tried to enhance the English reading understanding by an auto question-generation system. They addressed the challenges of maintaining context, creating a variety of question types (knowledge-level, inferential, etc.), and generating distractors. The pipeline synchronizes syntactic parsing, semantic role labeling, and rule-based logic and combines them with machine learning to extract essential information from reading passages. The results showed positive benefits for student comprehension and the challenge of maintaining language-level coherence of the style despite tuning the model. Susanti et al. [24] combined automatic question generation with computerized (adaptive) test generation; questions can be adapted, in response to student's performance. Their approach integrates NLP and statistical methodologies (e.g., item response theory) to adapt in real time assessments. Experiments on standard test datasets demonstrated the effectiveness of test efficiency and question validity, as well as the problems of question quality and consistency were solved, and the model kept optimized for these problems. Thotad et al. [25] constructed an NLP-based AQG system, and they observed that linguistic ambiguity, semantic correspondence, and difficulty-level selection were serious barrier to overcome. Their pipeline blends rule-based techniques with statistical language models in order to retrieve and ›lter questions from input text. They obtained encouraging accuracy and consistency on domain-specific data, and misaligned context was answered by successive refinements. Steuer et al. [26] aimed at enhancing the reading comprehension of students with the first language other than English with an automated system for the generation of questions. Conventional materials typically lack in appropriately aligned content and difficulty levels. By integrating NLP techniques into student data analysis, the authors proposed adaptive algorithms, which aimed to generate questions. implemented an instructional case study in reading comprehension, which found significant improvements, although linguistic simplicity and contextual fit were achieved through adaptive algorithms and by means of rigorous quality controls. Das et al. [27] presented a comprehensive review on automatic question generation and answer assessment, covering approaches from rule-based methods to deep learning-based techniques. They debated shared datasets, evaluation metrics and problems, such as losing the context and linguistic confusion. Through the comparison of these techniques, the authors stressed on sophisticated more advanced language models and hybrid systems to make this approach work effectively in the real educational environments.

Mulla and Gharpur [28] provide a good overview of approaches, data sets, evaluation metrics and application in the area of automatic question generation. They have then compared how both of these techniques address context alignment, scalability and semantic consistency pros and cons. The review exposed areas for improvements and provided, based on a literature overview, strategic directions for addressing structural differences with the potential to consolidate different methodologies. Wang et al. [29] studied the effect of prompt patterns for generative AI in automatic question generation. They systematically verified different prompt designs and how it impacted on coherence level, alignment to context, question diversity. They learned the best setting of prompts using data from textbooks. The authors also discussed technical difficulties such as prompt sensibility and model bias, and provided recommendations for maximizing the efficiency of the generative models. Lee et al. [30] investigated how few-shot (minimal example) prompt engineering with ChatGPT can be used to generate natural English language questions. Their hand-crafted prompt templates target coherence, contextuality, and grammaticality, given limited training instances. Empirical evaluation on educational datasets showed that few-shot learning reduces the amount of data required to perform training while increasing the quality of questions.

A comparative study on automatic question generation techniques which includes rule-based, template-based and deep learning-based methods is presented in Sewunetie and Ková [31]. Each approach was evaluated based on quality, diversity and accuracy of posed questions. These results, gathered using standard datasets and established metrics, indicated that, while deep learning enables flexibility, it tends to have difficulty with context and consistency. The authors noted the potential benefits of hybrid approaches which combine semantic analysis and rule-based reasoning. Panchal et al. [32] presented a system which automatically generates questions and evaluates the quality of generated questions. They system is built on NLP and machine learning models, it uses academic texts to generate questions and then it analyzes clarity, relevance, and difficulty level. Experiments on educational domain datasets proved that the model successfully prunes off low-quality questions yet keeps those with higher Contextual alignment. Zou et al. [33] introduced a true/false question generating mechanism for education systems to deal with ambiguity and inconsistency which can arise while converting mined RDF triples into true/false formats. Using NLP and machine learning, they extracted true statements from educational texts and converted them to true/false questions through controlled generation. Experiments revealed effective question generation, ambiguity was resolved by means of filtering and structured checks.

2.2. Human-in-the-loop in Education

In the recent years, human-in-the-loop (HITL) feedback mechanisms have demonstrated significant impact in improving the technical and pedagogical quality of automatically generated questions in educational settings. Recent studies have shown that incorporating expert or teacher knowledge to model predictions can significantly improve language fidelity and the match with learning content. Chen et al. [34] proposed a generative AI-powered teaching material system that imbeds human inputs to tackle content coherence, pedagogical fitness and linguistic accuracy. Through the use of expert feedback, the system had increasingly better alignment with the curricular objectives. The model’s effect was confirmed with data from established teaching materials but domain specificity and plurisemiotic ambiguity were handled with iterative refinements.

The importance of human monitoring for AI supported educational systems is well documented in the literature. Ninaus and Sailer [58] presented an extensive study concerning humans in artificial intelligence for education focusing on `closing the loop' between automated systems and human expertise. Their study indicated that although the use of AI systems may help to serve content-generation and assessment functions, human intervention is still necessary for pedagogical appropriateness and educational soundness. The authors claim that successful AI-education systems should balance the efficiency of automation with human supervision, especially in situations where learning outcomes are directly tied to student learning. Their findings showed that human-in-the-loop strategies result in increased quality of content along with greater teacher confidence in AI based educational tools, and therefore greater acceptance and usability in classroom settings.

The theoretical basis of the HITL in ET has been already investigated in in-depth surveys and empirical studies. Wu et al. [39] presented a broad survey of Human-In-The-Lopp methods in machine learning, highlighting the trade-off between automation efficiency and human oversight quality. They found that applications in education especially benefit from a HITL approach, given the intricacy of pedagogic goals and the necessity of expertise in content validation. The study emphasized that while pure automatic methods can achieve technical adequacy, educational validity involves advanced human intervention strategies which trade off efficiency against pedagogical adequacy.

Mosqueira-Rey et al. [41] presented an extensive state of the art review of human-in-the-loop machine learning, providing useful considerations on the adoption in different domains and effectiveness pattern. Their study showed that educational applications are amongst the most challenging settings for HITL adoption, due to the complexity of level objectives, different student populations and the necessity for instantaneous adaptability. The authors derived success factors such as feedback loops, domain-expert involvement and structured procedures which are especially interesting for educational question generation systems. Utilizations of HITL in educational domains have shown significant enhancements in content quality and pedagogical consistency. Another study conducted by Bhutoria [40] investigated AI based personalized learning systems across countries indicating that human-in-the-loop models for AI have a higher rate of improving outcomes in education than completely autonomous algorithms. The authors stressed that HITL integration facilitates the ability to adapt to various learning styles, cultural contexts, and individual students, while respecting pedagogical standards and expert validity by including expert supervision.

The relevance of HITL methods in the educational scenarios has been tested based on a range of domain applications that underpin the need of human expertise with a view to maintain educational valid and pedagogically appropriate. Zamprogno et al. [35] introduced dynamic human-in-the-loop decision heuristics in assertion generation, for software engineering, and observed how automation was insufficient in terms of accuracy and situational context. Though the domain is different from educational question generation, the concept of domain expert feedback does apply to educational settings where domain expert remains to help improve the model outputs. They showed that sequential round of revisions according to human feedback promoted correctness in aggregate, from which we can derive principles for educational content production.

Newest developments in HITL approach concentrated on methodological integration for maximizing educational benefit and for keeping the use of effort effective. Kumar et al. [38] provided a comprehensive overview of the state-of-the-art for HITL learning, including applications, challenges, and the future, with an emphasis on educational technology applications. They found education question generation as an appropriate domain for integrating HITL as there is a demonstrable effect of expert feedback on content quality and there are educators skilled in the relevant domain. The search revealed several key factors of implementation, such as forms of feedback, conditions of expert trainings and process of iterative revision, which appeared closely related to the educational effect.

Memarian and Dolec [37] investigated the use of HITL AI in educational domains using their entity-relationship analysis to present a framework to comprehend human experts and an AI system’s contacts into educational data. The authors showed that successful realization of HITL highly depends on the types of expertise, feedback and integration while preserving technological limitations and educational principles. The study demonstrated that productive ED HITL systems maintain a pragma of separation between automated efficiency and human expertise but this should always ensure that pedagogy should dominate the fruition of system design and implementation decisions.

2.3. Heterogeneous Data Processing for NLP Tasks

Heterogeneous NLP processing process structures different data modalities such as text, audio, video, and image, in order to build more context-aware representations. It holds special promise in the education context where multimodal resources such as diagrams and speech are used to support learning. Yet a few issues such as modality mismatch and data synchronization need to be resolved to achieve practical application.

Recent developments in multimeasurement analysis have tackled major challenges in incomplete and heterogeneous data sources. Zhan et al. [59] conducted a systematic literature review on incomplete multimodal learning that looks at methods and challenge in dealing with data in which not all modalities of observation are always available. Their extensive study indicated that robust multimodal processing architectures, capable of dealing with partial or corrupted data streams are particularly relevant in the field of educational applications. The authors have demonstrated subchallenge-specific approaches, including cross-modal imputation, adaptive fusion strategy and robust feature extraction, which enable data completeness-insensitive performance of the systems. Their results underscored the need for designing fault-tolerant multimedia systems for educational use, due to the varied quality and availability of data across learning situations.

The merging of various data types in order to achieve context aware learning analytics has been well studied in recent research. Gao et al. [60] performed an excellent review of deep learning-based multimodal data fusion with a focus on neural network architectures that can integrate information from different sources in an efficient way. Their work emphasized the power of attention-based fusion and cross-modal learning for producing better and more expressive representations for educational content analysis purposes. The results from the study indicate that successful multimodal fusion can dramatically increase the quality of automatically generated learning content, by enabling richer context understanding as well as improving the semantic alignment (across disparate types of data).

Multimodal data preparation for NLP use cases has already been applied in a number of educational contexts. Novak et al. [61] researched methods to represent multimodal data to apply to NLP task as educational content analysis and generation. Their work dealt with important preprocessing issues such as data alignment, modality synchronization, and feature selection for educational video content. The authors illustrated that finegrain multimodal data processing has a huge influence on the quality of NLP down-stream task, especially for educational QG task in which context and semantics precision were crucial to pedagogical work.

For instance, Chen et al. [42] designed MURAG for the retrieval-based tasks around generative language modeling to process multi-modal data well. Their paper pointed out some critical challenges including how to maintain the integrity of the context after aligning various modalities. Similarly, Singh et al. [43] addressed multi-modal question answering by proposing MIMOQA, a transformer encoder-decoder model for multi-modal question answering and handling both input and output in multi-modalities, with experiments showing increased contextual understanding without drastic downsides in complexity. Also, from an educational point of view, Mangaroska et al. [44] verifies how multimodal data (text, images, sound) can influence the experience of learning, highlighting the issues and the challenges of preprocessing and ethics. Their study of the views of CS students has given some understanding of how to make educational technologies more effective. Similarly, Crescenzi-Lann [45] carried out a systematic literature review on multimodal learning analytics in early childhood education focusing on the combination of video, audio, and tactile data to enhance learning outcomes, as well as ethical and privacy issues.

Another related study is that of Cukurova et al. [46] explored how multimodal learning analytics can facilitate student interactions through data futurization and information extraction, even though numerous issues exist concerning data interpretation and management. Adversarial robustness in multimodal NLP Studies related to adversarial robustness in multi-modal NLP are introduced by Yigit and Amasyal [47] which investigate adversarial example generation for question answering systems not just on text-based attacks but also with multimodal systems, assessing the susceptibility of the latter. A practical application of multi-modal data in question generation was introduced by Akbar et al. [48], who designed a Multimodal Adaptation Gate (MAG) integrated with BERT-based models to enhance the synchronization of visual and textual data, thereby improving question quality. Similarly, Wu et al. [49] investigated Synthetic Multimodal Question Generation, leveraging generative models to create synthetic questions from text and visual data, with promising results for educational and evaluative purposes.

Table 1. Comparison of question generation approaches in education

Table 1 summarizes various studies on automatic question generation in the field of education. Some studies focus only on multiple choice (MC) or fill-in-the-blank (Cloze) questions, while others may also produce open-ended or summarizing questions. In addition, many studies do not use expert or teacher feedback, even partially, while in some studies notes such as “Yes (Teacher feedback)” or “Partial” indicate that human expertise is involved. In our study, a full HITL cycle is in place and expert feedback is systematically integrated into the model. Some studies focus on the K-12 level (primary and secondary education), while others work on more specific areas such as university level or foreign language education. The datasets used include textbooks, ontologies, PDF documents, exam papers, and even CV information (recruitment processes). This diversity shows that automatic question-generation approaches can be applied in many different fields and can be used at various stages of the educational process. We support 6 different question types (multiple choice, true/false, fill-in-the-blank, open-ended, matching, short answer) and fully adopted the HITL cycle. It is also emphasized that the educational level we target is between 5th and 8th grades, and that our dataset is a special collection of multiple sources (text, audio, video, PDF, etc.). Thus, it is understood that our study offers a broader scope compared to other approaches in the literature in terms of both question diversity and the use of HITL. Integrated implementation of the HITL mechanism offers an innovative and comprehensive contribution to the process of automatic question generation in education. In this way, both technical (model architecture, data processing) and pedagogical (difficulty level, learning objectives) aspects are addressed together.

From a technical viewpoint, this work is drastically different from previous automatic question generation systems due to a number of crucial technical breakthroughs that overcome some key limitations in prior literature. Unlike Bhowmick et al. [6], whose method is still limited to text-only processing with rule-based templates, we equip our system with a sophisticated multimodal architecture handling in a unified processing pipeline video (faster-whisper), audio as well as various document formats (PyMuPDF, python-docx, python-pptx). In contrast to Bhowmick’s system that uses canned linguistic patterns and manually created rules, our system uses prompt engineering that is responsive to content complexity and to the educational context, pointing to a shift from static template-based generation to responsive context-aware content generation.

In contrast to Kurdi et al. [7] that considers partial expert validation as an optional add-on, our system designs HITL as a principal component of the system with iterative feedback loops rather than post-hoc validation. Existing systems (e.g. Srivastava et al. [9] and Shi et al. [10] realize one-way generation pipelines with limited refactoring capacity, our HITL fusion directly includes bidirectional refining mechanisms that automatically encode expert feedback into exactly feedback-inspired prompt adjustments and difficulty re-adjustment as well as content restructuring. This gradually improving system structure grows as you go; this contrasts the static system that requests any retraining/control logic updates to be done manually.

This technical innovation also applies to our full LLM comparison on seven SOTA models and six question types are systematically compared, offering insights for adoption of educational technology based on evidence. Whereas works such as Madhav et al. [11] concentrate on single-model implementations, with restricted scope of evaluation; our comparison identifies model-specific strengths and suitability patterns in education, which also help determine the optimal choice for deployment. Unlike Wang et al. [29] and Lee et al. [30]) whose prompt construction does not change and is domain independent, our framework utilizes adaptive prompt editing procedures, which adapt to content features, student grades, and pedagogical scenarios.

Our HITL system also provides a clear advancement over prior feedback schemes in that our HITL mechanism features a dynamic, real time integration structure. Distinct from Chen et al. [34] in which system-level expert validation is implemented, our architecture comprises automated revision modules that repeatedly integrate educator feedback into system improvements. The fluid loop (described in more detail in Section 4) consists of real time quality screening (i.e., generated questions are automatically filtered before review by the expert), structured feedback collection with domain-specific rubrics, prompt adaptation (any feedback gathered is used to adjust generation parameters), and performance tracking (changes in agreement with the expert across feedback cycles) imachinery.

This is fundamentally different from traditional post-generation validation strategy as in systems such as Morón et al. [12] or Susanti et al. [24], where human feedback is used only for quality checking, but not for improving the system. Our closed-loop learning system makes every expert interaction drive better future performance, so the knowledge and question quality only get better with time. The real-time nature of our feedback incorporation ensures the resolution of general pedagogical concerns in the program generation process and not as adhoc after-the-fact fixes as per the comment above. Unlike state-of-the-art which treats difficult level assignment as a post-processing, our HITL incorporation impacts difficulty calibration by learning through expert guided parameter modification and generates more pedagogically-appropriate content. The feedback-driven prompt evolution is a new way to keep educational relevance while avoiding retraining the entire system, thus making possible such improvement in a sustainable way that can follow the changes in educational standards and teaching styles.

3.    Materials and Methods

In this section, we detail the process of automatically generating questions from heterogeneous data sources. Firstly, we discuss how the data sources are processed, then how questions are generated using LLM, and finally how the HITL integration is realized. The approach is comprehensive in terms of methods and control of quality.

3.1. Data Sources and Preprocessing

Educational materials often encompass a wide variety of resources, making the success of an automated question-generation system critically dependent on the diversity and quality of data sources. In this research, we handle varying types of data formats (video, audio, PDF, Word, PowerPoint and plain text) using diverse technologies. We then combined the extracted full text into one consolidated source. It painstakingly preserves the characteristics of each type while still obtaining a unified representation suitable for the system's processing capabilities.

Figure 1. Proposed methodology for question generation and evaluation

Video and Audio Sources. Video and audio files represent a substantial portion of educational content; thus, high accuracy and speed are paramount in converting these resources to text. To meet these requirements, we employ an optimized version of OpenAI’s Whisper model, known as "faster-whisper". Whisper offers robust performance in speech-to-text conversion, even under varying accents, intonations, and background noise. The faster-whisper variant ensures rapid data processing, suitable for real-time applications. We use this technology because most educational materials are delivered in video or audio form, and the text extracted from these formats must be sufficiently accurate and complete to serve as a foundation for question-generation [50].

PDF Documents. For text extraction from PDF files, we use the PyMuPDF (fitz) library. PyMuPDF can parse text while preserving much of the page layout and formatting. However, in PDF documents containing extensive graphics, tables, or images, the text extraction process can exhibit limitations. Consequently, our approach focuses primarily on text-intensive PDFs, applying optimized techniques to minimize data loss.

Word and PowerPoint Documents. We utilize the python-docx library for Word files and python-pptx for PowerPoint presentations, ensuring the structural elements (headings, paragraphs, tables, notes) remain intact. These documents frequently contain crucial conceptual information, making the choice of these tools critical for preserving text integrity and minimizing data loss. Since educational content in the form of lecture notes and slides often carries significant conceptual detail, these tools were essential for maintaining a high level of fidelity in the extracted text.

Plain Text Files. Plain text files require no additional processing and can be directly integrated into the system. Although these files often contain short definitions or summaries, they are less structured than other formats, allowing for swift and efficient processing.

After collecting text from all these sources, we combine them into a single dataset while preserving topic coherence. If the merged text exceeds the token limit of our AI models, we employ summarization techniques to reduce length. For this, we use the Sumy library (a Python-based text summarization library), which supports multiple algorithms (e.g., Luhn (a frequency-based summarization algorithm), TextRank (a graph-based summarization algorithm)). By leveraging each algorithm's advantages, we determine the most suitable summarization strategy. This approach is particularly valuable for lengthy or complex content, allowing us to create a more compact, yet information-rich, format. Sumy's flexibility, offering multiple summarization methods in a single framework, ensures that crucial information from educational materials is preserved in a concise form.

The integration of these technologies forms the cornerstone of our system’s overall architecture. We prioritize rapid and accurate speech-to-text conversion for video and audio content; for structured data sources like PDF, Word, and PowerPoint, we carefully handle text extraction to preserve layout and minimize data loss. Each dataset underwent preprocessing steps (cleaning, linguistic corrections, removing irrelevant content) tailored to its respective educational domain. Consequently, the final dataset emerges as a high-quality, contextually rich resource for automated question generation. This methodological approach not only enhances the technical efficiency of our system but also supports more dynamic and up-to-date assessment processes in education. By carefully selecting and optimizing these technologies, we ensure direct processing of educational materials while minimizing information loss, thereby providing a robust and coherent textual foundation for question generation.

3.2. LLM-based Question Generation

Our system generates various types of questions using multiple LLMs. This subsection describes the model architecture, utilization processes, and the question types.

3.2.1. Model Architecture and Utilization Process

In this study, both API-accessible and locally deployed LLMs are used to automate question generation. The underlying model architecture primarily involves processing text extracted from different data types (text, audio, video, documents) such that, if the combined text does not exceed the chosen model’s token limit, it is fed directly to the model; otherwise, it is summarized before processing. Each model analyzes the input text and generates questions in a format suitable for the targeted question type (e.g., multiple-choice, true/false) and difficulty level.

Among the API-based (Application Programming Interface: an interface that enables data exchange between applications) models, we include GOOGLE (gemini-1.5-flash) and NVIDIA (nvidia/nemotron-4-340b-instruct), accessed via API keys provided by their respective cloud platforms, namely Google AI Studio [51] and the Nvidia NIM platform [53]. Locally deployed models consist of GOOGLE (gemma2) [27b], MISTRAL (mistral-small) [22b], META (llama3.1) [70b], ALIBABA (qwen2.5) [32b], and Cohere For AI (aya-expanse) [32b]. Each model has a distinct architecture (e.g., transformer-based, RNN-based (Recurrent Neural Network: a recurring neural network architecture designed for sequential data processing)), exhibiting variations in data processing capacity and token limits.

The system is designed to manage outputs from each model by dynamically activating summarization and data merging procedures, depending on the model’s token capacity or context window. As a result, even if heterogeneous sources are merged into a single text, the system can adapt it to fit the model’s constraints, thereby achieving high-accuracy, linguistically coherent sets of questions. This adaptability ensures that each model receives data in an optimal format, maintaining both performance and fidelity of the generated questions.

3.2.2. Generated Question Types

Our system produces six different types of questions: multiple-choice, true/false, fill-in-the-blank, open-ended, matching, and short answer. Each type of question targets distinct cognitive skills in learners. For instance, multiple-choice questions emphasize information verification and memory recall, whereas open-ended questions measure critical thinking and analytical abilities. Matching questions help visualize and reinforce conceptual relationships. Every generated question is categorized as "easy", "medium", or "hard" depending on the text’s linguistic complexity, the depth of concepts, and the question’s structure. These levels of difficulty correspond with the learner's level of cognition, including acquisition of basic knowledge related to a topic, as well as the more complex and analytical.

For evaluating question quality, we use seven different LLMs. We use each of the model in the pre-trained version in the same dataset for question generation. We compare the generated responses by these models in view of semantic consistency, linguistic fluency, question structure, and relevance to the educational objectives aimed to and investigate which model is the best performer. Consequently, we compare their outputs over a variety of tasks to see which model generates the most accurate, understandable, and pedagogically appropriate questions. Question Types and Level of Difficulty:ality of content and the performance of models. This approach enables us to optimize the question bank for automated generation of educational content generation, as well as A/B comparisons of model performance in terms of stability. We want to guarantee the transparency and reproducibility of our automated question generation process so we create detailed prompts for each type of question that we generate. The prompts ask the model to produce questions using only the context by satisfying certain rules and of certain difficulties. In doing so, we ensure that the model’s outputs remain well-calibrated and consistent which improves the quality of the question bank for educational materials and facilitates the downstream evaluation. Here is the sample prompt for question generation.

This prompt instructs the model to produce three Turkish-language questions strictly from the provided data, disallowing any personal interpretation or external knowledge. Additionally, all questions must be classified into one of three predefined difficulty categories-easy, medium, or difficult. The question text must be a complete sentence in question format, and the output should be in JSON, facilitating direct parsing by automated systems. Such formatting aims to standardize and stabilize outputs, thereby improving the quality and assessment of question banks used in educational content. The prompt for multiple-choice questions is as follows.

Designed for multiple-choice questions, this prompt compels the model to produce four-choice items, specifying the correct answer in each question. The JSON output uniformized the ques- tion data, and allowed for easy machine processing, and uniform formats of subsequent evaluations. For true/false questions, the query is.

This particular word prompt works well with true, false questions, a brief, easy-to-evaluate test of fundamental information. In addition, JSON formatted outputs from the tools allow for easy analysis automatically and can be easily in automated testing systems. Fill in the blank questions are presented with with the following prompt.

Fill-in-the-blank questions require students to enter missing information. This architecture guides the model to look for key blanks and generate right answers in the same. It is also allowed to have multiple blanks, which enables detailed assessment of knowledge. The open-ended question prompt is as follows.

Open-ended questions are intended to test higher-order thinking skills, and students are expected to answer in more detailed and reflective ways. The limitation of only one correct answer per question makes the system straightforward while allowing educators to evaluate depth of reasoning. Matching questions should follow this format.

Problems involving matching test the ability to establish appropriate relations between elements in two sets. This prompt ensures that the model generates pairs separated by semicolons, so that the output is regular & easy to evaluate programmatically. The short answer writing assignment is the following.

Short-answer questions measure basic knowledge with succinct responses. This prompt directs the model to produce a single correct answer for each question, ensuring straightforward evaluation of correctness and comprehension. The prompt for difficulty level is as follows.

Here, the chosen difficulty level prompt determines the complexity of the questions generated by the system. Each level is associated with a set of definitions, ensuring that questions align with the learner’s cognitive stage and measure the intended depth of knowledge. All these prompt structures are employed to maintain consistency, clarity, and pedagogical suitability in the automatically generated educational materials. By standardizing the input and output formats, we aim to facilitate objective evaluation of question quality and better support learners’ overall learning processes. This methodological scenario allows us to obtain excellent results in the quality and performance of our system in automatic question generation for educational settings.

3.3. HITL Integration

Recent works highlight the additional benefits of incorporating HITL factors into the AI systems as they can not only reduce the error rates but also lead to more global optimization of the outputs [35]. Within the context of education, it has been shown that the systematic inclusion of expertise and teacher feedback into model output can improve the value of machine generated content from a pedagogical perspective, contributing to higher student achievement and better learning experience [36]. This process includes periodically showcasing the intermediate model’s outputs to human reviewers for rating on factors such as grammaticality and consistency and alignment with instructional objectives, and has completed unfolded and realized the value of both technical and education quality in the final question bank [37].

3.3.1. Feedback Mechanisms

In HITL systems, the choice and incorporation of feedback mechanisms are a key factor for advancing model predictions [41]. The burned-in feedback from expert teachers or domain specialists on the automatically produced questions in terms of correctness of language, level of difficulty and relevance to the context enables teachers to make refinements to lessons yielded by the model. This kind of feedback allows for systematic error detection and parameter (or prompt) adjustment [53]. As a result, a system that is constantly fed by human intervention attains increasingly better accuracy and pedagogical alignment over time [54]. This feedback loop addresses not only linguistic or semantic shortcomings but also guides the creation of content aligned with educational objectives, a balanced distribution of questions at varying difficulty levels, and the development of items that assess specific conceptual skills [55]. Such guidance constitutes one of the most critical phases for improving the quality of learning materials and enhancing students’ conceptual gains.

In our system, we reserve a place for human intervention at each stage of automated question generation. Educators and domain experts review the generated questions for content completeness, linguistic consistency, difficulty level suitability, and pedagogical appropriateness. Feedback is collected according to detailed control criteria; for instance, linguistic errors, contextual mismatches, or gaps in meeting pedagogical goals are detected by the system and, based on these data, are forwarded to automatic revision modules. This architecture way the constant enhancement of model output and reduces error rates. Educator feedback is incorporated into the system’s revision phase in an iterative feedback loop, improving both linguistic and content quality.

3.3.2. HITL System Workflow

The HITL integration process is built on a closed-loop multi-stage model. First, the automatically created question sets are sieved on the basis of predefined quality measures, groups of questions being eliminated in the first stage if they are faulty or incomplete [35]. Whatˈs more, in the expertise feedback process, the model is retrained or prompt updated at the linguistic and content levels by automatic revision modules, which further promotes better quality of the second-stage outputs [36]. This iterative process demands close interaction between human feedback and model automatic update features, allowing fast error spotting and correction to ensure that each new question set produces improved results. [ 37]

In carrying out that process expert review may include not only review of linguistic and content consistency as traditional trait model-based design principles suggest, but also consideration of who the target student audience is, how well it aligns with curricula, and the level of cognitive skills being assessed [41]. Therefore, the model output are not only technically correct but also advanced to a test item bank with high pedagogical utility and ready for application in educational measurement and evaluation [53]. These phases demonstrate how HITL not only improves the performance of the model, but also paves the way for a flexible and adaptive structure where human expertise and AI evolve together [54].

Figure 2. The employed HITL system workflow in question generation process.

As seen in Figure 2, proposed system’s HITL approach not only enhances the reliability and effectiveness of automated question generation in education but also reinforces the model’s adaptability. By integrating user feedback with AI outputs, the system systematically reduces errors and benefits from continuous improvement mechanisms. Consequently, HITL integration raises both the technical and pedagogical quality of automated question generation in education, maximizing the usability of the final question bank. This approach is further substantiated by the experimental results presented in subsequent sections of this paper, clearly demonstrating how our system achieves innovative and effective outcomes in educational measurement and assessment.

The automated question generation process begins with obtaining an initial set of questions produced by large language models. At this stage, the system generates an automatic question set based on broad data sources (e.g., video, audio, documents, text). However due to the fact these early outputs are entirely unfiltered they may need to be improved in terms of the grammatical correctness, serving or line structure and broad content overlap. The resulting questions are automatically screened according to pre-defined quality criteria. In this process, the linguistic errors, format discrepancies and context defeasibility are also verified for the model answers and any of the items not meeting the above-mentioned conditions are thrown away. As a result, only the questions that have met minimum standards of quality are passed on to an expert. This prescreening is important for obtaining final results with high accuracy and robustness.

All submitted resources are independently reviewed by teachers and experts with the emphasis placed on quality. They discuss each question carefully, addressing concerns of linguistic accuracy, context dependency, and pedagogical benefit. This type of review assesses overall completeness of content, alignment of content to grade standards and benchmarks, and effectiveness of the materials for the target student population. Expert commentary points out capricious errors, omissions, and oddities in the question bank. The input feedback is processed and incorporated into automatic revision modules. They iteratively edit the question sets generated by the model, correcting grammar, context alignment, and how well questions meet pedagogical goals. The rephrased questions are sent back to experts to re-evaluate, whether the modifications follow the required quality levels. The manual and automatic question generation processes are mutually adapting, and this (partially automated) process improves systems reliability overtime.

At last, the modified and approved questions are put into the system as the final, ready to be used question bank. This validation step guarantees the fulfillment of all quality criteria (relevance of the question, correctness of the format, adequation of difficulty, correctness of the answer, clarity, and global quality). The result is that technically as well as pedagogically excellent results are obtained in auto- mated query generation. Through the inclusion of HITL facilities, the technical and pedagogical quality of automatically developed questions will be optimized resulting in improved student learning. This methodological method allows for an ongoing improvement of the system and the question bank for learning media quality assurance.

4.    Experimental Results

In this section, we report on the experimental results of our automated question generation system and discuss the experimental setup, datasets, evaluation metrics, and the comparison with different LLMs as well as integration of human feedback.

4.1. Experimental Setup

Experiments are conducted to show the practicability of our system in text extraction, summarization, automatic question generation, and HITL feedback from arbitrary sources. To this end, seven different LLMs (API-based as well as locally deployed) are compared on the same datasets. Locally installed models were run on a machine with an NVIDIA GeForce RTX 4090 GPU, 64 GB of RAM, and an Intel Core i9-14900KF CPU, using Python 3.10. Data processing, model querying, and experiment result recording is done under this setup. A React JS (JavaScript-based user interface development library) interface was developed to visualize at runtime (i.e. live) the generated questions and received feedback, and its data were saved in Firebase Realtime Database (Google's real-time cloud database service) based on questions and sources.

The API-based model also includes GOOGLE (gemini-1. 5-flash) and NVIDIA (nvidia/nemotron-4-340b-instruct), respectively, available via API keys provided from Google AI Studio and the Nvidia NIM platform. The two models run in the cloud, and may have different performance characteristics because of network delay and potential dynamic of the workload sharing the cloud resources. The locally deployed models include GOOGLE (gemma2) [27b], MISTRAL (mistral-small) [22b], META (llama3.1) [70b], ALIBABA (qwen2.5) [32b], and Cohere For AI (aya-expanse) [32b]. Each model features unique architectural properties concerning data processing capacity, token limits, and context analysis.

The datasets are comprised of a variety of sources-video, audio, PDF, PPTX, DOCX-for different grade levels and subject areas. Text extracted from these sources is merged to form inputs suitable for each model. When the combined text exceeded a model’s token limit, summarization was applied to reduce its length. By respecting each model’s constraints and minimizing data loss, experiments are systematically conducted to support six question types (multiple choice, true/false, fill-in-the-blank, open-ended, matching, short answer) and three difficulty levels (easy, medium, difficult). Each model is run under the same parameter set (e.g., number of questions, desired difficulty level), and the results are recorded by model and question type. This approach ensured a fair comparison across models, clarifying how each performed in various scenarios.

The automatically generated questions are assessed by domain experts and educators on criteria such as grammatical correctness, contextual consistency, difficulty level, and pedagogical suitability. Feedback is then fed into the system’s automatic revision modules, and used during subsequent refinement stages to continually improve the question bank. All experiments take place under fixed hardware and software configurations, objectively measuring factors such as query response speed, memory usage, and output quality. Results are reported through tables and graphical methods, providing a reproducible and controllable framework from both technical and pedagogical standpoints.

4.2. Description of Datasets used for Evaluation

Evaluation is conducted using specialized datasets prepared for different grade levels (5th, 6th, 7th, and 8th) and subjects (Turkish, social studies, science). The provided datasets are pretreated to order contents on topics into natural language, remove unneeded information, and correct errors in the grammar.

Turkish subject dataset:  The Turkish dataset consists of passages of text and solutions related to the category of Grammar. They include plenty of information about these three topics – sentence materials, samples of literary texts or language rules.

Dataset of social studies subject: The Turkish dataset consists of passages of text and solutions related to the category of Grammar. They include plenty of information about these three topics – sentence materials, samples of literary texts or language rules.

Science subject dataset: Constructed from items with experimental data, theoretical constructs, and scientific explanations, this dataset measures student analytical reasoning and problem-solving abilities.

Levels dataset: Both datasets are divided to different difficulty levels per grade. In this way, questions will be attuned to the appropriate student ability, addressing factual and conceptual queries.

Table 2. Covered 60 topics and their grade level, subject and data sources

Table 2 delineates 60 topics of three different subjects (i.e., Turkish, Social Studies, Science) from 5th, 6th, 7th and 8th grades. It provides sources of various content types (video, audio, PDF, PPTX, DOCX, text) for each topic. This organizationigan-tion emphasizes the need for multimodalessential that instructional materials are designed in a way that engage students and makes learningaving accessible and engaginglearner provide students with an active and interactive learnlearning experience where there are multiple faculty-student andstudent-student interactions. For instance, regarding 5th-grade Turkish, subjects like "Word Meaning", "Sentence Meaning", and "Passage Meaning" are offered in video, audio and document (PDF, PPTX, DOCX) formats. This allows students to study the same content through different media, supporting visual, auditory, and text-based perception of the content. Thus, students can study the same material using various media (books, lecture notes, audio etc.) and support their learning through the visual, aural and textural modes of learning.

Similarly, for social studies and science applications, the heterogeneity of types of information suggests that there may be value in multi-source integration, in which more comprehensive content is constructed from information in different formats. For example, in the 6th-grade Social Studies course the subject is "Social-Cultural Life in Early Turkish States," and it has videos, audio, PDF, PPTX documents that provide more detailed information about the chronological events, and the conceptual relations etc. Table 3 also displays that for each topic as many as six kinds of resources can be applied, whilst not for every lesson there are video, sound and written resources. Accordingly, the fusion of heterogeneous data boosts and characterizes the information. A grand total number of 238 resources are used to describe topics as will clearly show how sizeable and spacey are these objects. In sum, Table 3 highlights the importance of integrating multi-dimensional data sources for educational question generation and evaluation. Employing diverse data types enables the creation of more comprehensive and context-rich question banks, aligning with varied learning styles and cognitive requirements. This approach ultimately enhances the effectiveness of measurement and evaluation processes in education.

4.3. Evaluation Metrics

In this study, we adopt a multidimensional evaluation approach to objectively measure both the technical and pedagogical qualities of automatically generated questions. Although metrics such as BLEU, ROUGE, METEOR, or BERTScore are frequently used in the literature for assessing textual similarity and semantic closeness in automatic question generation, these metrics typically focus only on linguistic or semantic accuracy. However, questions intended for use in educational environments must also address pedagogical aspects such as difficulty level, alignment with the student’s proficiency, and integration with instructional objectives. Consequently, we devised a comprehensive set of evaluation criteria that extends beyond textual similarity measures.

Alternative evaluation methods include approaches that concentrate solely on linguistic correctness or user experience-via surveys and expert opinions. In this study, however, our proposed multidimensional system surpasses the one-dimensional limitations of those methods by incorporating HITL feedback mechanisms, thereby facilitating iterative improvement at every stage of question generation. As a result, we employ a strategy that concurrently accounts for difficulty level, format integrity, and pedagogical alignment, rather than relying exclusively on text similarity or semantic proximity. Our system evaluates the quality of automatically generated questions according to six criteria:

Relevance and Format Appropriateness: This metric examines how accurately and consistently the generated questions reflect information from the source text, while also checking whether they conform to the expected question format.

Question Type Appropriateness: Here, we assess how well each question aligns with the designated question types (multiple choice, true/false, fill-in-the-blank, open-ended, matching, short answer).

Difficulty Level Appropriateness: We compare for a question if it fits the pre-defined ‘easy’, ‘moderate’ or ‘hard’ buckets. The difficulty level will be rated by the teachers, i.e., experienced teachers will rate whether the item fits the criterion based on two aspects: 1) grade level (5th–8th) in primary and middle school students; and 2) the degree of cognitive complexity. Teachers evaluate questions according to a range of pedagogical dimensions such as conceptual demands, language complexity, cognitive load, and appropriateness to the developmental stage of the given student cohort. This expert human assessment methodology has also been adopted in the context of other automated assessment tasks; including workload estimation for human reading; and is thus considered to ensure that difficulty ratings are based on real educational measurements rather than surrogate ones such as linguistic complexity scores or reading ease indices. The teacher evaluation workflow consists of domain experts to review each question with reference to official educational frameworks and grade level expectations resulting with rating on a scale from 1-5, where 1 means not at all on a grade level and 5 means perfect question difficulty for given grade level.

Correctness and Clarity of Answers: The correctness, consistency, and clarity of the answers are critical, particularly for ensuring that students accurately grasp the underlying concepts.

Clarity and Conciseness of the Question: This criterion evaluates grammatical accuracy, clarity of expression, and overall comprehensibility of the question, all of which directly influence its educational utility.

Overall Quality of the Question: By synthesizing all the aforementioned measures, this metric reflects the question’s pedagogical value and its suitability for its intended purpose.

Each question is rated on a scale from 1 to 5 according to these criteria. We then aggregate these scores by model and question type, presenting them in detailed tables. This comprehensive assessment aims to objectively gauge the system’s performance in both technical and pedagogical dimensions.

Figure 3. Demonstration of user evaluation of retrieved questions

Figure3 illustrates how three questions generated in a single pass are evaluated. On the left, three automatically generated questions are displayed, each linked to a specific lesson and topic (e.g., ``5th Grade Social Studies – Unity and Solidarity in Society"). When the prompt is executed, the system produces a set of three questions that the user (e.g., a teacher or a subject matter expert) can evaluate collectively. Each question’s body, answer choices, and correct solution are listed separately for easy inspection.

On the right side, a multidimensional evaluation system is provided for these three questions. Criteria such as ``Are the questions relevant and properly formatted?", ``Is the question type appropriate?", and ``Does the question’s difficulty level match the specified category?" aim to assess both the technical and pedagogical quality of each question. A star-based rating system enables the user to quickly and intuitively offer feedback. The ``Evaluation Details" section below it summarizes explanatory scores for each criterion and includes general notes. This method compiles data on how to enhance both the entire automatically generated question set and each individual question. Hence, the user interface allows for simultaneous examination of the three questions generated by a single prompt, along with rating them and gathering feedback, which the system uses to continuously refine its question bank. This strategy ensures that both technical (grammar, format, correctness) and pedagogical (difficulty level, alignment with learning goals) quality checks are performed in parallel in the realm of automated question generation for education.

4.4. Performance Comparison

The experimental evaluation phase aims to objectively demonstrate the technical and pedagogical performance of our automated question generation system based on multidimensional criteria. Each question is given a score between 1 and 5 based on six main criteria: alignment with the source text, question type appropriateness, difficulty level accuracy, accuracy and clarity of answers, linguistic clarity of questions, and overall question quality. This comprehensive evaluation reveals how effective the output of the system is not only in terms of technical accuracy, but also in an educational context. Thus, the outputs of different large language models were meticulously analyzed with data collected both for all question sets and for each question type separately, and differences in performance and areas for improvement were identified on a model-by-model basis.

4.4.1. Comparison of Question Generation Times According to Models

Table 3 shows the fastest and slowest generation times for each model, as well as the average time calculated on a given subset. These data clearly show the impact of different model architectures and access methods (API-based or local) on throughput. In particular, the slowest time values are higher for API-based models due to network latency or the load of shared cloud resources, whereas the average production time varies for local models depending on hardware capacity. This comparison emphasizes that for time-sensitive applications such as automatic question generation in education, model selection should consider not only linguistic accuracy or content quality but also processing speed.

Table 3. Comparison of question generation times (Total 22680 questions)

Table 3 compares the question generation times of different LLMs over a total of 22,680 questions, and shows the fastest (minimum), slowest (maximum), and average time values calculated for each model based on a subset (3,240 questions). Among the models accessed through the API, GOOGLE (gemini-1.5-flash) has a very short fastest generation time of 1.71 seconds, but the slowest time of 249.32 seconds shows that it can experience high latency at times. Likewise, NVIDIA (nvidia/nemotron-4- 340b-instruct) shows a relatively consistent but relatively slow profile, with a fastest throughput of 14.91 seconds, a slowest time of 159.15 seconds, and an average of 33.29 seconds.

Looking at the locally running models, GOOGLE (gemma2) [27b] and MISTRAL (mistral-small) [22b] offer moderate performance with average throughput times of 17.50 and 15.16 seconds respectively, with no significant difference between the fastest and slowest values, indicating that these models provide a more stable computation time with respect to query size. On the other hand, META (llama3.1) [70b] stands out as the model with the highest latency range in the table, with the slowest production value of 697.56 seconds, despite recording the fastest time of 116.91 seconds; its average of 231.26 seconds indicates that the model incurs high computational cost due to its large parameter set. The ALIBABA (qwen2.5) [32b] and Cohere For AI (aya-expanse) [32b] models offer moderate speed with average times of 20.71 and 25.58 seconds, while the slowest generation time of 1136.85 seconds, especially in Cohere For AI (aya-expanse) [32b], indicates that very long processing times can occur in certain situations.

Overall, Table 3 highlights that in time-critical applications, such as automated question generation in education, model selection is important not only for linguistic accuracy or output quality, but also for the stability and consistency of production speed. For API-based models, network latency and the dynamic load of cloud resources can lead to high maximum time values, while for local models, hardware configuration and model size are the main performance determinants.

The generation time findings highlight some key infrastructural issues that have bearing on the pragmatic use of AQG systems. API-based models have completely different performance characteristics to a local model, with a range of key factors that contribute to the variance in response times. The API determines network latency, the largest factor contributing to variation in performance of API models, which is influenced by geographical distance, connection quality, and routing efficiency between the client and server. The extreme contrast between GOOGLE (gemini-1. 5-flash) fastest and slowest times are 1.71 seconds and 249.32 seconds, respectively, demonstrating that between network time can differ in time by more than 100x.

Cloud resource interactions arise as another important element influencing API-based model performance. At high load times, shared hardware resources get overwhelming requests that cause throttlings and queuing times leading to degradation of response time. The NVIDIA (nvidia/nemotron-4-340b-instruct) slow, yet consistent (14.91-159.15 seconds) response times indicates complex load balancing processes that ensure service stability yet retain resource allocations under increasing numbers of users.

API rate limit and auth overhead add further layers of latency that don't exist on premises. Every API request has to pass through an authentication check, rate limit check and request routing step that introduces constant baseline latencies. In addition, API service providers can apply dynamic throttling depending on requests patterns, account levels, or infrastructure load, and this can introduce unpredictable performance fluctuations that greatly affect the quality of real-time educational applications.

The overhead of data transmission is especially noticeable in educational content processing because the input texts can be large. The requirement to send educational stream and get compacted questions, and to deal with possible retry cycles when requests fail imposes further level of latency proportional to the complexity of the content. This may account for part of the large variations seen in API-based models with different educational content types.

Local models tell a different story in terms of stability properties. GOOGLE (gemma2) [27b], and MISTRAL (mistral-small) [22b] display smaller performance spread (12.20-39.31s, and 6.20-33.67s). 70s, respectively), suggesting that local hardware performance offers more predictable compute environments. Meanwhile, local deployment itself also brings certain limitations, such as limited local memory, thermal throttling, and the competition with other concurrent progresses, which may influence the robustness of performance.

META (llama3. 1) [70b]'s large inference time range (116.91-697.56 seconds) when deployed with local computing, and the effects of the complexity of model architecture from the perspective of hardware. The model with 70 billion parameters requires significant amount of memory and computational power, which may overload the local resources and then cause performance drops when handling sophisticated educational contents and when the system cannot be fully dedicated because of other concurrent tasks.

These are sobering performance attributes that have important implications for strategies to work the new educational technologies. This means that the real-time classroom applications where the immediate question generation is preferred, will support slightly less expensive models that can be submitted locally and with acceptable quality degradation for stability in performance. In some curriculum development batch processing situations, high-quality API-based models can be used with varying response times as a result of asynchronous processing to address these latency challenges.

The cost-performance balance between API-based and local deployment modalities are more than costs based financial implications, but also about the management overhead, disparities in deployment scale, and technical skillsets. It is important for educational institutions to consider their use cases, technical infrastructure, and performance requirements in choosing the right model deployment strategies for AQG systems.

4.4.2. HITL Assessment by Models and Question Types

The quality of the auto-generated questions is determined as much by the HITL feedback process as by the linguistic capacity of the model. Examples of generated questions are given in Table 12 in the appendix section. For various question types (multiple-choice, true-false, fill-in-the-blank, open-question, matching, short-answer), when expert feedback and editing modules are activated, the grammatical errors of the questions are significantly reduced and their pedagogical relevance is increased. This effect is particularly evident in the criteria of difficulty level matching and accuracy of answers. Since different models benefit from the HITL cycle to different extents, we observe that some models improved quickly with expert feedback, while others showed more limited gains. Thus, the extent to which each model can flex both technically and educationally with HITL integration and adapt to revisions plays a critical role in shaping the final quality of the question bank.

Table 4. Performance evaluation of LLMs for multiple choice questions

As seen in Table 4, the scores obtained for the multiple-choice questions show that all models performed well overall. In the criteria of “Meaningfulness and Appropriateness of Questions to Question Format” and “Appropriateness of Question Type”, the models reached values in the range of 4.88-4.94, while the accuracy and clarity of the answers were in the range of 4.77-4.94. This shows that the language models used in the generation of multiple-choice questions accurately and consistently reflect the information in the source text. In particular, GOOGLE (gemma2) [27b] and META (llama3.1) [70b] models stood out in terms of both question type conformity and answer accuracy; they also performed well in the 'difficulty level appropriateness' criterion.

The generated multiple choice questions give us the following insights into the model architectures and their potential to be used in educational content creation. GOOGLE (gemma2) [27b] generates highest overall quality score of 4.38, indicating that smaller models fine-tune achieve better task performance than larger counterparts. We believe this model's strong performance can be attributed to the balanced parameter size (27B) that offers enough complexity to interpret educational context and yet still computationally efficient without overfitting.

MISTRAL (mistral-small) [22b] revealed some worrying weaknesses in the generation of multiple-choice questions, particularly in difficulty level (4.00) and clarity (4.35). This disappointing performance can be attributed to a number of architectural and training-related issues. It seems that the training corpus of the model is not rich in educational texts and therefore the model shows deficiencies in pedagogical understanding, as in using age-appropriate language. Secondly, the coherence of the question stems of the questions of a test with the answer options are hard to achieve, that is, the attention mechanism for educational content is not used effectively. MISTRAL is also perceptually more sensitive to prompt function variations, and hence demands finer-grained instruction engineering for stable responses.

The good performance of most models under relevancy and format criteria (4.76-4.92) demonstrates that the automatic creation of multiple-choice questions can potentially be used to assist in curriculum-specific assessment. But the level calibration difficulty from 3.99 to 4.93 has laid bare the need for educational domain knowledge in AI training. Teachers planning to use smaller models such as MISTRAL could be encouraged to adjust the level of difficulty.

Table 5. Performance evaluation of LLMs for true/false questions

In the true/false questions, as seen in Table 5 the models generally scored high, with values between 4.81 and 4.97 in the “Appropriateness of Question Type” and “Appropriateness of Question Meaningfulness and Format” criteria. In this question type, the ability of the models to make sense of the context is evident, as the language is expected to be succinct and concise. The META (llama3.1) [70b] and ALIBABA (qwen2.5) [32b] models slightly outperformed their competitors in the difficulty level matching and answer accuracy criteria, while the overall accuracy and clarity of all models were high.

True/false items are the simplest linguistically, but they show that all the logic described are worth considerations. META (llama3. 1) [70b] had an overall quality of 4.71, which implies that larger parameter counts are likely to benefit binary decision tasks due to better contextual reasoning. The 70B parameters in the model provide a more fine-grained sense of factual versus opinionated statements, essential for generating educationally defensible true/false items.

ALIBABA (qwen2. 5) [32b] with the best overall quality of 4.78, hence indicating that educational content optimization is attained despite a moderate size of the parameters. I would think that part of the reason for this success could be due to the multilingual training method of such models, which would help the model understand many different educational concepts and many different school contexts.

The consistent MISTRAL underperformance is reinforced in the true/false generation task, where the lowest relevance score is for MISTRAL (4.67) and overall quality doesn't reach 4.32. This is indicating some underlying restrictions of the model when it comes to learn the representation of educational material. The problems seem to arise from representation of factual knowledge, with the model failing to differentiate between provable and interpretive sentences. The model also suffers from context boundary issues which make it hard to hold coherent logical chains within the educational parts and calibration issues where the level of difficulty of the generated content doesn’t fit with the grade levels disparity between the two sources of knowledge.

Across models, the high aggregate quality (overall quality = 4.30–4.78) indicates that the technology for true/false generation is mature for classroom usage. But instructors need to know that questions, although technically sound, might not completely measure higher order problem solving ability. The strong performance shows that this format is well suited for efficient content review and basic comprehension assessment in education.

Table 6. Performance evaluation of LLMs for fill-in-the-blank questions

As seen in Table 6, there are some significant differences between the models. While the average scores of the models ranged between 4.61 and 4.83 in the criterion of “Meaningfulness of Questions and Appropriateness of Question Format”, the scores of the models in the criterion of “Appropriateness of Difficulty Level” ranged between 3.95 and 4.24. This shows that it is difficult to correctly identify critical pieces of information in the text and to apply appropriate gap placement strategies in the generation of fill-in-the-blank questions. In particular, the MISTRAL (mistral-small) [22b] model scored lower than the other models in this question type, suggesting that fill-in-the-blank questions need further improvement in terms of linguistic structure and contextual integrity.

Fill-in-the-blank questions are the hardest type of question to automatically design, as they require a deep understanding of hierarchy of concepts and pedagogical gap selection. The test results provide important information about strengths and weaknesses of the models. All met models had trouble with difficulty level appropriateness for identifying the (5–8)53, providing evidence that recognizing appropriate gaps is a complex skill requiring deep pedagogical knowledge not found in the current LLMs. This is particularly problematic because a model must be able to separate key concepts and supporting details, in that, a model can tell which removal of information is constructive for learning rather than cases of trivial recalls. However, gaps should be placed in a manner than retained grammatical structure but leave enough context to provide clues, and questions must be difficult enough to test comprehension but still solvable with the rest of the context.

We also see MISTRAL experinecing a drop in quality, where it's been fine-tuned for co-reference resolution came at the co-presense of co-reference fill-in-the-blank generation quality, specifically for relevance (4.62), question type appropriatness (4.74), and overall quality (4.17). This lack lustre performance exposes some fundamental flaws in the model's architecture, such as its lack of semantic parsing, where the model cannot achieve the separation of semantic boundaries and thus cannot correctly place a gap in a sentence, ruining its meaning. The model suffers from educational context blindness, i.e., it does not know which concepts are central as opposed to peripheral in educational contexts, as well as syntactic rigidity, which makes it hard to modify sentence structures by removing information while still making them grammatical.

Cohere For AI (aya-expanse) [32b] obtained the best overall quality score of 4.65 which indicated effective gap filling strategies. This success might be due to the multilingually trained model, which helps in understanding the flexibility of syntax and in preserving meaning. GOOGLE (gemini-1. 5-flash) also did well at 4.55, probably owing to its newtigma-1 (d size) and tuning for education task. Considering the universal challenge in this format, human review should be preferred for fill-in-the-blank questions over others, the generated fill-in-the-blank questions can play as entry point with heavy pedagogical tweaking, reviewing AI-generated content on concept-critical missing point over detail-specific blank, and a mixture approach by having AI to hunt for potential missing points for human expert' further selection.

Table 7. Performance evaluation of LLMs for open-ended questions

When the open-ended questions are evaluated in a format where the answers should be specific and detailed, the overall performance of the models is quite high as seen in Table 7. In the “Accuracy and Clarity of Answers” criterion, META (llama3.1) [70b] and ALIBABA (qwen2.5) [32b] models performed particularly well, scoring between 4.86-4.92. However, the MISTRAL (mistral-small) [22b] model showed lower values (e.g. 3.65) in the “Difficulty Level Appropriateness” criterion, suggesting that sometimes conceptual depth is not sufficiently captured in the generation of open-ended questions.

Open-ended questions require the most pedagogical sophistication as models must know learning objectives, cognitive taxonomies, and assessment. The performance fluctuation also provides some important insights on what the model can do for higher-order thinking assessment. META (llama3. 1) [70b] showed that with a 4.53 overall quality:“The performance of large-scale models is superior in these complex reasoning tasks”. The model's effectiveness is due to improved cognition mapping with a clearer perception of Bloom's taxonomy levels and question complexity required for different grade levels, contextual deepness in recognizing the themes and concepts that needs analytical instead of factual response, and meaningful rigorous discourse toward the relationship among core concepts.

MISTRAL's very low difficulty level score of 3.65 is evidence of automated pedagogical content generation crisis. This failure are symptoms of lack of understanding of educational scaffolding, as the model is unable to differentiate between superficial and deep learning questions. Cognitive level confusion A model that is unable to calibrate questions at the appropriate grade-level thinking skills Assessment purpose misalignment A model that fails to assess the targeted learning outcomes that could misreport mastery levels to students and teachers both.

The high performances of best models such as META and ALIBABA demonstrate that open-ended question generation is of great help for valid assessment if designed well. However, the large differences in the performance provide evidence that model choice is important to educational validity. Poorly performing models such as MISTRAL may cause active harm to learning in terms of introducing surface learning propagated by questions that encourage memorization over understanding, misrepresenting student capability through assessments that do not reflect the level of understanding, negative teacher confidence driven by inconsistent quality, which degrades trust towards AI-supported education. For open-response, teachers need to ensure expert review of all content generated, that cognitive level benchmarks are set before content is automatically generated, that performance data determines which models may be used for a given educational purpose, and human oversight for alignment with learning goals and standards is kept in place.

Table 8. Performance evaluation of LLMs for matching questions

Matching questions generally receive lower scores than other question types as seen in Table 8. The models have values between 4.49 and 4.85 in the criterion of “Meaningfulness of Questions and Compliance with Question Format” and between 3.44 and 3.90 in “Overall Question Quality”. This suggests that matching questions pose more complex challenges in terms of correctly establishing the relationship between items and generating compatible answer pairs. In particular, the scores of the MISTRAL (mistral-small) [22b] model in this question type are significantly lower than the other models.

Matching questions are the most structurally complex, and they demand nuanced knowledge of related concepts, semantic subcategories, and pedagogical arrangement. The uniformly low ratings (3.44-4.41 average quality) demonstrate inherent difficulties in automating the relation mapping for educational resources. The bad performance of all models demonstrates the difficulty in aligning the question generation, and in particular the relationship detection where the models need to understand multiple types of relationship (e.g. cause-effect, part-whole and category-example) in educational content. Also, with respect to cognitive load management, the suitable quantity of items is created in an attempt to provide cognitive overload without providing too much challenge for the learner, and with distracter balance, wrong pairings are made plausible but not confusable with the right ones.

MISTRAL received the lowest scores on all criteria, with catastrophe difficulty 2.82 and overall quality 3.62. This is an utter breakage of capacity to generate educational content. An analysis of the breakdowns reveals problems in reasoning about relationships indicating underlying inability to comprehend conceptual relationships present in the educational content, problems with clustering concepts based on semantics revealing lack of appropriate grouping of related concepts and leading to nonsensical matching pairs, and lack of educational logic checking to ensure that questions are composed in accordance to some very basic pedagogical principles of assessment design.

Cohere For AI (aya-expanse) [32b] was the strongest at 4.41 overall quality, indicating that multilingual training supports varying relationship types. Perhaps the successful learning of the model can be attributed in part to its exposure to diverse types of conceptual scenarios in different languages and cultures. The sub-par performance on match questions reflects a substantial disassociation with the existing level of AI-based education. This approach relies on human expert scrutiny with all generated matching questions needing to be thoroughly pedagogically evaluated, (ii) pre-specification of relationships by teachers stipulating relationship types that should guide generation, and (iii) over the course of multiple generation cycles, expert feedback needs to steer the course of generation toward educational soundness. In the face of these difficulties, instructors using AI generation should:  use exercise pairing with great care,  provide as much context of desired relation type and complexity level to AI as possible,  use AI as an inspiration for rather than replacement of existing pool of matching questions and  invest resources in teacher training for quality assessment and improvement of AI generated matching questions.

Table 9. Performance evaluation of LLMs for short-answer

In the evaluation of short-answer questions, all models perform well as in Table 9. Scores between 4.85 and 4.97 are obtained for the “Accuracy and Clarity of Answers” criterion and between 4.51 and 4.76 for the “Clarity and Understandability of Questions” criterion. This shows that the models performed adequately in generating correct and clear answers in the generation of short-answer questions. In general, the performance differences between the models for short-answer questions are quite small. Short-answer question (SAQ) has a good strategic level in educational measurement being the link between simple recall formats and complex analytical task. These are the type of questions where the students need to be able to write short, but substantive answers to show they understand, but not give a ton of on and on and on elaboration. The performance results provide important insights into the capabilities of models for this currently educational but educationally valuable data format, demonstrating quite large variation in effectiveness across architecture variations.

Most recently nvidia/nemotron-4-3 40b-instruct achieved an overall quality score of 4.76, out-performing the other systems across several evaluation dimensions, with perfect scores for question type appropriateness (4.97) and correctness (4.97). This impressive performance seems to be related to the fact that the model can decompose the response scope to an optimal extent, generating questions on which relatively short answers are appropriate, while providing clear expectation setting of the desired answer length and depth. The success of the model is also indicative of its content precision–it can process the emphasis content that can be meaningfully judged in short utterances.

ALIBABA (qwen2. 5) [32b] fared well overall with a quality score of 4.64 and had especially good clarity metrics (4.93). The success of this model demonstrates the impressive ability to formulate questions from linguistic clarity to unambiguity and ease of interpretation. Modeling performance reveals the successful development of scope definition capabilities with clear distinctions made as to expected length and response content, balanced with cultural flexibility across educational contexts.

MISTRAL (mistral-small) [22b], on the other hand, still exhibited worrying gaps in short-answer generation, obtaining an overall quality of 4.22 with especially poor scores in difficulty level appropriateness (3.81). This substandard performance reflects systemic problems with the design of assessments in the form of the inability to design questions that are at appropriate levels for the students’ development. The model suffers from response scope confusion, which results in asking questions that are either too shallow, or too detailed for the short answer format, and thus has inherent misfit with educational goals and outcomes.

In conclusion, the general results show that SAQG is based on has reached a level of maturity suitable for being deployed in education under the guidance of quality control. These are very good questions for cognitively complex demand that do not require too much or too little cognitive demand, which can be useful in providing fair and purposeful assessments while the grading load remains reasonable, and hence give valuable information of students’ understanding. The format is adaptable to formative and summative assessment contexts, but model selection has a considerable impact on educational quality measures to be achieved.

Table 10. Overall evaluation of all questions (total 21798 questions)

Table 10 shows that the models are evaluated between 4.80 and 4.86 in the “Meaningfulness and Format Appropriateness of Questions” criterion, between 4.88 and 4.93 in the “Type Appropriateness of Questions” criterion, between 3.71 to 4.13 in “Appropriateness of Difficulty Level” criterion, 4.81 to 4.88 in “Accuracy and Clarity of Answers” criterion, 4.50 to 4.60 in “Clarity of Questions” criterion and 4.14 to 4.37 in “Overall Question Quality” criterion. These findings indicate that our automated question generation system generally produces high quality questions. Although there are small differences between the models, META (llama3.1) [70b] and Cohere For Al (aya-expanse) [32b] show slightly superior performance, especially in terms of accuracy, clarity and overall question quality. On the other hand, the MISTRAL (mistral-small) [22b] model has lower overall quality scores, especially in terms of difficulty level matching and some question types. This suggests that each model may perform differently on different question types and that each needs specific areas for improvement.

Performance in 21,798 questions provide important information for educational technology deployment and systemics in automated question generation performances. The rankings of the models indicate that there are well-defined tiers of models appropriate for educational use. GOOGLE (gemma2) [27b] bears the best overall flexibility of between performance and efficiency for deployment in the classroom of 4.57 among all architectures, while GOOGLE (gemini-1. 5-flash) 4.50 overall quality is capable of providing excellent performance with fastest generation times and can be applied to real-time applications. META (llama3. 1) [70b] works well for sophisticated reasoning tasks but has a higher computational cost with an average quality of 4.37.

Average educational readiness is indicated by Cohere For AI (aya-expanse) [32b] (4.35 OV-alignense show good and us, xuu) with 4.35 overall quality (monolingual capabilities as well strong abilities for multi-lingual) interesting in various educational backgrounds, and ALIBABA (qwen2. 5) [ 32b] 4.32 overall quality and fair performance and cultural adaptability. Models in need for heavy human curation are NVIDIA (nvidia/nemotron-4-340b-instruct) with 4.23 overall quality and a reasonable performance quality with a large parameter count, as well as MISTRAL (mistral-small) [22b] (4.14 overall quality) showing a critical issue for educational purposes.

The continuously low performance of the MISTRAL classifier, not surpassing the 5th place in 5 out of 6 categories, indicates a model architecture that is fundamentally mismatched to the educational needs. The limitations cut across several key dimensions, with difficulty calibration catastrophe (scores of 3.71/5.0), where the model’s shortcoming in judging difficulty presents significant learning hazards, as it may overestimate or underestimate cognitive demands as being too easy, which can lead to surface learning, or too difficult, in causing student frustration or withdrawal. The overarching issues are the grade-based mismatch between content and the levels of students who are targeted for taking this curricular approach (below their actual developmental capacity) and the validity threats to the assessments through unreliable measurements of students' comprehension and achievement.

Training of model seemed to lack educational domain knowledge (e.g., pedagogical principle violations in which there are no violations of established educational best practices, learning objective misalignment in which little relation exists between the content on which the question is based and intended curricular objectives, student-centered design ignorance, in which linguistic complexity is prioritized over educational value). MISTRAL performs extremely variably on question types, suggesting that teachers cannot expect consistently useful outputs, adds another layer of review burden, requiring that human oversight on par with other models, is risky to implement and could lead to negative effects on student learning and teacher practice.

The comprehensive evaluation shows that although automatic question generation is technically feasible, educational validity relies on careful choice of modelling, systematic quality assurance, and continuous human expert oversight. Such large gaps in performance between the models highlight just how critical evidence-based deployment is in the adoption of education technology. Parameter optimization seemed more important than scale — GOOGLE (gemma2) with only 27B parameters did well considering the scale of the web, indicating that ergonomics mattered to an even greater extent than raw model size. Training data quality reveals that models with obvious educational content in the training data perform much better than those without similar exposure while architectural suitability suggests that some architectures could be fundamentally bad at generating educational content despite fine-tuning.

We found that the HITL integration worked with different effectiveness levels, providing interesting insights on the nature of AI-generated educational content and the concrete aspects where human experience is more effective. The analysis of the improvement patterns revealed that HITL mechanisms had the greatest positive influence on structurally complex question types.  Fill-in-the-blank questions saw the most substantial relative improvement via human feedback: success rates surged for all models. This significant improvement illustrates the vital role that pedagogical judgment of human provides to recognize the relevant information gap and make sure that blanks aim at content critically relevant to the concept, not peripheral information.

Matching questions were also substantially improved by HITL intervention, but were still overall the most difficult question type encountered. Human expertise was crucial to validating conceptual relationships and maintaining that matching word pairs represented meaningful learning ties and not simply literal associations. The expert comments focused on relationship clarity and management of cognitive load in particular, and they were key factors that contributed to evolving fossils of an uncomprehensible matching question into a learning-oriented assessing tool.

For open-ended questions, Human-in-the-Loop process indicated modest, yet consistent gains with focus on the difficulty calibration and alignment with LOs. Expert support was key to the fact that questions would place demands on appropriate cognitive levels and have clear expectations for answers from the students. The advances were most pronounced for the systems for which educational context understanding was a weak point initially.

By contrast, multiple-choice and true/false questions showed the smallest relative benefit from HITL mechanisms, meaning that these types are more easily automatically generated with little human effort. But even with these seemingly straightforward formats the benefits of expert review for difficulty level suitability and choice quality were apparent, so human oversight is still a positive feature for all but the simplest question types.

Short answer questions were of medium HITL effectiveness where human feedback mostly dealt with scope definition and response expectation clarity. The only moderate levels of improvement indicate that these questions can be produced with reasonable automated quality, but that expert touch-up is required to really refine them and make them more useful to students.

The differential effect of HITL on question types has important implications for the design of educational technology. For resource allocation, focus on human expert review for more structurally complex formats such as fill-in-the-blank and matching questions and allow more automated processing of multiple-choice and true/false items with intermittent expert validation. Such two-level quality assurance can enhance the learning effect and the operational efficiency of AQG systems.

To compare the performance of large language models, we use statistical tests like the Friedman test and Nemenyi test. The Friedman test is used to detect differences in the performance of multiple models across evaluation metrics. The Nemenyi post-hoc test is used after the Friedman test to identify which models are significantly different. For these tests, we focus on the accuracy values of seven language models GOOGLE (gemini-1.5-flash)", GOOGLE (gemma2) [27b]", MISTRAL (mistral-small) [22b]", META (llama3.1) [70b]", ALIBABA (qwen2.5) [32b]", Cohere For AI (aya-expanse) [32b]", and NVIDIA (nvidia/nemotron-4-340b-instruct)". After Nemenyi post-hoc test (threshold q_alpha = 5.9 from the studentized range table), significant differences between models are changing. Nemenyi post-hoc test for models is depicted in Figure 4 “aya-expanse” has the best evaluation value for overall metrics.

Figure 4. Proposed methodology for question generation and evaluation

4.5. Impact of Human Feedback

The contributions of HITL integration to the automated question generation process have been examined in detail using experimental data and expert evaluations. The feedback provided by expert instructors not only addressed linguistic errors and reduced ambiguities but also significantly enhanced question quality, content richness, and pedagogical alignment. This procedure contributed to the reconfiguration of model inputs (prompts), more precise calibration of difficulty levels, and increased effectiveness of questions for the target student audience. In terms of language clarity and content relevance, expert evaluations result in a significant reduction in ambiguities and phrasing mistakes in the system’s deterministic questions. It comes with an important gain regarding clarity and conciseness of question criterion where questions recognized as ambiguous or unnecessarily hard get recast based on expert input and get clearer formulations. These changes reduce the challenge for the students but also improve clarity and continuity for educators who evaluate the question bank.

In terms of pedagogical appropriateness, the expert feedback can support the fit between the content of the question and student proficiency levels and instructional goals. For example, quality of difficulty level (the Difficult questions being either too hard or too easy) are corrected to a point where the “Difficulty Level Appropriateness” and “Overall Quality of Question” metrics encountered a noticeable rise. In this sense, questions with little conceptual depth are extended and adapted to low or high performing students as per the target student population. Furthermore, expert feedback has a strong impact in the prompt-engineering domain. Recommendations for modifying content and form of the model input (prompts) by experts lead to more recognizable improvements, especially in composing conceptual and analytical aspects-question. These refinements allow for a better application of text summarization methods, they ensure that the context is preserved in the paraphrased question bodies, and the refactored answer choices preserve the tradeoff between correct and distractor answers. And finally, these enhancements are already good for overall pays. Experimental results show that mean ratings for each score increase significantly using expert feedback incorporated into the system. This is an increase which suggests that the human intervention (HITL) in the automated question generation not only improves the linguistic aspect of its outputs but also has an important role to play in how the question banks satisfy pedagogical objectives, and remain balanced in their level of difficulty and supportive of students’ learning of the subject.

Table 11. Number of successfully produced questions before and after HITL

Table 11 presents the total number of generated questions for each model, along with the number deemed “successful” before and after the HITL mechanism is applied. Here, “successful” refers to questions meeting criteria such as grammatical correctness, content consistency, alignment with the intended difficulty level, and pedagogical suitability. The table compares the performance of locally running models (e.g., GOOGLE (gemma2) [27b], MISTRAL (mistral-small) [22b]) with models accessed via API (e.g., GOOGLE (germini-1.5-flash), NVIDIA (nvidia/nemotron-4-340b-instruct)) both at the outset and after expert feedback and revision phases.

The contrast between the pre- and post-HITL results is evident from Table 11., the space group and the cell parameters of which are not given, with the ratio GOOGLE (germini-1. 5-flash) demonstrates an increase from 2838 successful questions to 3212 after the HITL while it is a sign of a successful model learning from expert feedback for the minor error fixing and difficulty calibration. Similarly, MISTRAL (mistral-small) [22b] shows a large leap from 1678 to 2969 correct questions, indicating that the expert feedback mechanism is particularly effective in detecting its grammatical and content coverage flaws. Meanwhile, NVIDIA (nvidia/nemotron-4-340b-instruct) is kicked off with 2799+ and rises to 3065, which indicates: while the model is more stable, it still showed some potential benefit from the expert who asked them. The total number of questions with a success across all models also increased from 19528 to 21798. This improvement—about 10 points of gain—indicates that expert instructor feedback and revision cycles contributed to a large extent to the system’s overall performance. These figures suggest that HITL is used not only for ensuring linguistic/semantic fidelity, but also content fidelity, appropriate level of difficulty, and ability to align to student learning activity. These results indicate that intelligent feedback in automatic question generation gives significant technological benefit as well as pedagogical benefit.

5.    Conclusion and Future Work

This paper presents a comprehensive automatic question generation (AQG) system that leverages heterogeneous data sources to produce pedagogically relevant questions tailored to diverse training materials. By integrating data from text, audio, video, and documents, the system employs advanced large language models (LLMs) to generate questions across various types (e.g., factual, inferential, analytical) and difficulty levels. The incorporation of Human-in-the-Loop (HITL) mechanisms ensures continuous improvement in the quality and educational appropriateness of the generated questions, addressing both technical efficiency and pedagogical validity. Experimental results highlight the system's ability to produce high-quality questions, with comparative analyses of different LLMs revealing their strengths and areas for improvement in handling specific question types and data modalities. These findings underscore the potential of the proposed system to enhance educational tools and support adaptive learning environments.

5.1. Practical Implications for Education

“With the AQG system, schools can experience a dramatic innovation in teaching and learning.” By automating the process of generating rigorous, standard-aligned questions, the system frees teachers to spend more time understanding and addressing the instructional implications of assessment results. Teachers are now freed up from this, to educate, engage students and offer individual feedback. For instance, the system may create sets of questions for a quiz, homework or formative testing, to facilitate preparation across different subjects and students' levels. In low-resource settings, e.g. resource-starved schools, such automation could help in tackling the problem of teacher burn out and enhancing instructional efficiency.

In addition, the platform’s capability to create custom questions that correlate to students’ skill levels and learning goals enables differentiation, resulting in better academic results. By producing questions that concern the specific skills, the platform supports adaptive learning that is tailored to the learner. For example, questions based on multimedia/visual content (e.g., short videos or podcasts) may help click students in while boosting comprehension and critical thinking in the history and sciences units. Initial evaluations indicate that students with questions generated by the system are more engaged and informed, especially in self-paced learning environments such as online courses. The flexbility of the system for dealing with heterogeneous data, also makes it work across several educational contexts ranging from K-12 classrooms, higher education or professional training and enables for easy deployment at scale over learning managent systems (LMS) or massive open online courses (MOOC) to democratize access advanced educational tools, especially in underserved areas.

5.2. Limitations and Challenges

Despite the good results obtained in this work, there are a number of limitations and issues that we still need to tackle in order to improve the performance and scalability of the system.

5.2.1. Information Loss in Heterogeneous Data Integration

An interesting problem that arises in this context is the combination of diverse types of data, such as text, audio and video. In the process of converting any of these non-textual modalities (eg., transcribing an audio recording to text, or reducing the content of a video to a series of scenes) into a common format to enable processing, information may be lost. For example, subtleties such as tone, emphasis or visual context will not complete be translated, leading to confusions or wrong questions produced. Such inconsistencies may negatively affect the system's capability of delivering contextually correct and pedagogically adequate outputs, especially in dealing with complicated media content.

5.2.2. Token Capacity and Context Retention

Another limitation of LLMs are their token capacity restrictions, especially with long texts or big datasets. The input sequences are long and oftentimes exceed the models' context windows, rendering summarization or truncation methods crucial. Such procedures, however, can suffer from the loss of important surrounding context, which in turn can lead to a lower quality of generated questions. For instance, a summary of a long historical text would likely cut out subtler cause-and-effect relationships, resulting in questions that do not reflect the full extent of the material.

5.2.3. Inconsistent Question Difficulty Classification

There is also the automatic classification of the levels of difficulty. The system sometimes misplaces questions at wrong difficulty levels because of incoherent or subjective measurements. For example, a question designed for advanced learners could be categorized as intermediate for a model making incorrect assumptions about linguistic complexity or need for domain-specific knowledge. This kind of inconsistency hampers a system’s capability to match questions to users’ competence levels and can diminish effectiveness in adaptive-learning scenarios.

5.2.4. Scalability of HITL Mechanisms

The scalability issue is the most significant obstacle for adopting HITL-assisted question generation systems in general educational technology. Although our experiments show substantial quality gains with human expert feedback, the approach will be infeasible for large-scale institutional adoption with a high demand for human expertise. The manual review process is very efficient for small databases but poses some bottlenecks for real-time processing and large-scale applications where we may have thousands of questions on a daily basis for validation across educational domains and grade levels.

The basic scalability tension results from two conflicting requirements for quality assurance and operational efficiency. The present HITL implementation relies on expert annotators to provide detailed feedback on question relevance, pedagogic appropriateness and difficulty calibration, which takes around 2-3 minutes per question for an extensive review. This time commitment not only is impractical at an institutional level serving thousands of students across the breadth of a curriculum, but also is economically infeasible for organization-wide mandates. The problem is exacerbated for educational systems with limited resources and may not have access to trained or committed expert reviewers.

To overcome these scalability issues, we suggest six aggressive tactics that will preserve quality while still enabling deployment at scale. The first method consists of the creation of crowdsourcing networks of distributed expertise of teachers taking advantage of the wisdom of the practicing educators through well-defined reviewing grids. It operates not by the authority of expert panels centralized across the national landscape, but by the knowledge of classroom teachers who literally do understand what their students and curriculua must have across discipline.

Distributed question review responsibilities are placed across geographically dispersed teacher networks in teacher networks (crowdsourcing) – revolutionary white-label, subject-specialized and grade-level focused quality assurance volunteerism. This method includes enlisting teachers in the field to participate in professional development programs and to receive continuing education units for participating in systematic review of items. Teachers would be grouped according to specific subjects (e.g., mathematics, science, language arts, this particular discipline), and each of these groups could be assigned the responsibility for reviewing questions in a particular content area. The application would leverage micro-credentialing models, in which teachers receive PD points for contributing to quality review, thus creating sustainable incentive structures around existing professional growth requirements.

The crowdsourcing system would have an automated quality control system, which would assure some level of consistency among crowdsourced reviewers. Every question would be subjected to multiple independent reviews by multiple educators in the same sub-specialty, and an algorithmic consensus system would identify items for which there were disparities that would be resolved through expert adjudication. Machine learning models could analyze review patterns, and seek to find highly reliable reviewers whose feedback can be weighted more heavily in final quality assessments, as well as flag potential bias or inconsistency in review patterns that could need additional training or review.

The second strategy to consider consists in the introduction of cutting review in tiers, which, through multi-level quality control schemes, can maximise resource use. This multi-step framework starts with automatic pre-screening, based on state-of-the-art natural language processing models tailored to validate educational content, which are filtering out questions that meet particular requirements in terms of grammatical soundness, semantic plausibility, and relevance to our formats. Questions that survive automated screening are then moved into peer review stages, where expert teachers offer targeted feedback on pedagogical alignment and difficulty settings. Complicated/ambiguous issues which necessitate the proficiency of a master teacher or subject matter expert above would be referred for final decision.

Such tiered systems work to ensure that human expert time is efficiently reserved for those questions that take nuanced pedagogical judgment, and the automation is used for routine quality checks. The automated pre-screening can disqualify about 70-80% of clearly problematic questions (e.g., having grammatical errors, factual errors, content not quite appropriate in difficulty level to target grade ranges). This level of filtering does a significant amount of noise reduction for the human reviews, while keeping a stringent quality on questions advancing through our system. The peer review level deals with questions that need to be evaluated in terms of pedagogical context, as well as whether they are in line with the learning objectives and have an appropriate cognitive level for students of the corresponding age. Questions that are solely a matter of pedagogical complexity or reviewer subjectivity will only be elevated to expert review levels in cases where disagreement among reviewers is evident.

To this end, the third approach is to implement micro-task platforms tailored for the educational question evaluation context, and to split composite review processes into short 5-10 minute review tasks that can be completed in teachers’ spare time. These micro-reviews fit within the busy schedules of practicing teachers but still feature robust quality review through targeted rubric criteria. Teachers join during prep periods, after-school, or professional development days, meaning that their core responsibilities are not interfered with while they help to improve the overall system quality.

Micro-task systems implement game features to stimulate engagement and motivation such as point mechanics, achievement badges, types of users and leaderboards for high quality judge contributions. Tutors are awarded for providing useful and relevant feedback and additional support is given to the top contributors. Overall, the system is developed to enable optimized user experiences such that interfaces for the review chain are intuitive, responsive, and accessible on multiple form factors such as tablets and smart-phones for participation in reviews from different locations and contexts.

The final proposed strategic solution is high-quality, machine learning-based automated pre-screening systems retrained with machine learning models on educational content reviewing algorithms. These screening tools, powered by advanced machine learning techniques, are built based on a set of multidimensional question-passing criteria, namely, question-level linguistic complexity suitable to the grade level, question-standard alignment, answer information verification (i.e., a knowledge base check), and pedagogical quality from assessment design principles. Sophisticated natural language processing models can detect questions that need human review and automatically approve questions that satisfy some predefined quality threshold.

Automatic pre-screening systems can also learn over time from feedback of human reviewers, fine-tuning nudging heuristics by forming patterns in human expert correction and modification. Machine learning algorithms learn from accepted versus rejected questions to identify subtle signals of quality and pertinence that might not be evident in a rule-based filter system. This adaptive learning means screening accuracy would get better over time, lowering the load on humans while maintaining tough quality standards for educational material.

The fifth model sets up regional centres of expertise which act as expert centres for the assessment of educational problems (not necessarily in the same district or geographical area). These centers pool expertise in reviewers and technology resources in support of scalable quality assurance services to surrounding academia. Regional centers can focus concentrations of expertise in certain areas of interest or in certain modes of education, and work broadly with institutional clients in a consortium-like resource pooling model where costs get spread across multiple educational systems.

Regional expertise centers can also achieve economies of scale through the construction of nodes across site costs by aggregating costly expert resources and atomic infrastructures in some places and services to the distributed education institutions. Such centers may be more nimbly able to retain specialized knowledge in new educational areas, adjust to rapidly changing curriculum standards, and offer professional development opportunities to teacher reviewers. The shared model of resources enables smaller districts or entities to have access to high-level review services they would otherwise not be able to sustain in-house.

The sixth and final strategic approach implements asynchronous review workflows that leverage global time zone differences ensuring 24-hour operation of the system in question generation requests. This distributed temporal model synchronizes the review of items across international educator networks, allowing question evaluation on a 24-hour basis, and at suitable times based on local working hours and cultural preferences in various time zones. This allows teachers located in different time zones to perform the review as part of their normal working day, and ensuring system responsiveness without the need to ask reviewers to work overtime or outside of their established schedules.

Asynchronous workflows include advanced project management system with the ability to track question progression through review routing, automatic routing of material to available reviewers by way of expertise, time zone, and transparent progress reporting to sys  Odd number system administrators and en d users alike. This also facilitates immediate for fast response to urgent question generation requirements and ensures that grading work is distributed evenly/seamlessly throughout global reviewer networks, minimizing bottlenecks that could result if review work were concentrated in specific geographical areas or time zones.

Implementing such scalability solutions will have to be carefully coordinated with work already underway in the domain of educational technology infrastructure, such as learning management systems, curriculum databases, and professional development platforms. The integration methods need to satisfy the needs of data privacy, the educators’ compensation model and quality assurance procedures to ensure trust and reliability of the generation of automated educational content. Pilots should be started at voluntary teacher networks and should be developed step by step guided by the effectiveness, and user satisfaction, and make sure that improvements in scalability increase the effectivity of educational value of generated questions rather than decreasing it.

5.2.5. Computational Resource Demands

Processing heterogenous data and more advanced LLMs can be computationally expensive, especially in real-time settings. Heavy computation is expensive for easily-accessible to institutions with little infrastructure, which may serve as a barrier for its wide acceptance. Balancing resources and usage for performance without degradation is still a challenging aspect.

5.3. Future Research Works

The limitations highlighted here suggest a number of exciting areas for further investigation, which are expected to contribute to strengthening the robustness and scalability of the system and its pedagogical value.

5.3.1. Advanced Data Integration Techniques

To overcome data loss when integrating heterogeneous data, possible future work should lie in improving the data-processing algorithms. This could potentially be improved by utilising techniques such as multimodal embeddings, to maintain the textual nuance and context across text, audio, and video. For instance, one might integrate text transformer-based models with CNNs for video frame analysis in order to more effectively capture visual and auditory signals so that the generated questions represent the entire context in the source text. Moreover, the investigation of neural network architectures that focus on cross-modal alignment (such as CLIP-based models) might help to alleviate ambiguities in modality conversion.

5.3.2. Improved Question Difficulty Metrics

To further improve the adequacy of question difficult classification, new AI- based metrics need to be created to better evaluate the difficulty of question. Such metrics would reflect semantic density, syntactic complexity, domain knowledge needs, etc. For example, NLP tools such as dependency parsing or entailment could be employed to measure the cognitive difficulty of questions. Second, if the machine learning models are trained with labeled question difficulty data, they could yield more consistent results and be aligned better learner's levels in various educational scenarios.

5.3.3. Enhanced Multimodal Processing

It is proposed as future work that we can emphasize further on the improvement of advanced multimedia integration methods to handle both the video, audio and visual descriptions more efficiently. Architectures such as vision-language transformers or audio-visual fusion models might further enhance the model’s capacity to abstract and synthesize contextual cues across multiple modalities. FORM EXAMPLE, if the model is capable of incorporating temporal analysis for video data, it might generate questions that relate to the sequence of events or use dynamic visual content. These improvements could make the pedagogical value of questions better, especially for subjects in need of multimedia understanding, like science or history.

5.3.4. Domain-Specific LLMs for Education

The design of the targeted LLMs dedicated to the educational domain is a great opportunity for the question generation. If a model can be further trained from fine-tuned to be education-tailored on education-related data sources (e.g., textbooks, lecture transcripts, or pedagogical guides), the generated ques- tions may be more accurate and relevant. Similarly, domain specific models (possibly combined with knowledge graphs) could get a better grounding on what concepts and relationships exists in a particular subject to be able to generate question related to what would be taught in a curriculum. Comparisons with other approaches and different learning settings and multiple datasets should be performed to further validate the proposed models in other educational scenarios.

5.3.5. Scalable HITL Frameworks

In order to overcome the scalability issues of HITL integration, automatic feedback procedures that require less human intervention could be investigated in future work. For instance, active learning methods can help identify the most uncertain or ambiguous questions for human review, making efficient use of resources. Furthermore, crowd sourcing platforms could be employed to spread feedback requests over a wider number of educators, enabling a more scalable yet still quality-focused feedback process. Investigation of semi-supervised learning methods could reduce the reliance on significant human feedback, making the system capable of dealing with large scale perfectly.

5.3.6. Optimization for Computational Efficiency

In order to reduce the computational burden of the system, model compression strategies like quantization and pruning could be explored in future without any performance tradeoff. Furthermore, by investigating edge computing solutions, the system could run on less performance demanding hardware, and be accessible to a broader audience of users. Methods such as federated learning could also be used to distribute the computation to be performed over different devices to improve the scalability of real time applications.

5.3.7. Evaluation and Benchmarking

The product should be well benchmarked with existing AQG systems to be a success in the long run. This entails assessing the quality, diversity, and pedagogical efficacy of questions in different domains and among different groups of learners. Common benchmarks and evaluation measures -e.g., BLEU for query fluency or human-rated relevance scores- could be also defined in order to build a solid baseline system. Longitudinal evaluations to study the effects of the system on learning outcomes would provide additional evidence of the educational purpose of the system.

5.4. Broader Implications

The innovations of this proposal have the potential to revolutionize educational technology toward more adaptive, personalized, and scalable learning. By overcoming the aforementioned limitations, the AQG system could be applied in the context of larger educational platforms, including learning management systems (LMS) or massive open online courses (MOOCs). In addition, the ability of the system to work with heterogeneous data could enable applications beyond education like automatic content analysis of journalism or generation of training materials in corporate environments. "However, ongoing meaningful collaboration with AI researchers, educators and domain experts is required to capitalize on those opportunities and to guarantee that the system addresses real world needs.

6.    Declarations

Acknowledgement

Not applicable

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Authors` Contributions

All authors have participated in drafting the manuscript. All authors contributed equally to the manuscript and read and approved the final version of the manuscript.

Conflict of Interest

The authors certify that there is no conflict of interest with any financial organization regarding the material discussed in the manuscript.

Data Availability

This review is entirely dependent on previously published literature and does not incorporate any novel datasets. All data that support the findings of this study are available within the cited articles and publicly accessible databases.

Ethics Approval

No need for ethical committee approval.

Declaration of AI Usage

AI-assisted tools were employed in this study for minor tasks such as grammar correction, language refinement, and proofreading. These tools were used transparently and in a manner that does not compromise the authors’ intellectual contribution. The authors affirm that all substantive content reflects original thought and upholds academic integrity.

8.    APPENDIX

Table A. Examples of AI-generated questions