Not specified
The multimedia tools tested were reported in studies from various countries, including Nigeria ( Akinoso, 2018 ), Saudi Arabia ( Aloraini, 2012 ), England ( Bánsági and Rodgers, 2018 ), Ireland ( Davies and Cormican, 2013 ), Australia and Canada ( Eady and Lockyer, 2013 ), Taiwan ( Huang et al., 2017 ), Turkey ( Ilhan and Oruc, 2016 ) Czech republic ( Karel and Tomas, 2015 ), Malaysia ( Maaruf and Siraj, 2013 ), Serbia ( Milovanovic et al., 2013 ), Pakistan ( Shah and Khan, 2015 ) and China ( Wu and Chen, 2018 ).
Various age groups were targeted by the multimedia tool tests. A considerable proportion involved university students with ages starting from 16 or 18 years as specified in the articles ( Bánsági and Rodgers, 2018 ; Huang et al., 2017 ); Hwang et al., 2007 ; Jian-hua & Hong, 2012 ; Kapi et al., 2017 ; Karel and Tomas, 2015 ). Another group targeted were secondary school students ( Akinoso, 2018 ; Maaruf and Siraj, 2013 ) including vocational school students ( Wu and Chen, 2018 ). Shah and Khan (2015) reported testing their multimedia tool on children below the age of 15 years.
The articles involving evaluation were examined to identify the methodologies used for the evaluation, the target groups and sample of the evaluation and the evaluation outcome. The limitations of the evaluation were also identified and whether or not the study outcome could be generalized. Thirteen articles were found and the results are presented in Table 5 .
Summary of Evaluation methods of multimedia technology Tools in education.
Publication | Focus area | Evaluation method | Target group | Sample size | Outcome | Limitations | General-izable outcome |
---|---|---|---|---|---|---|---|
Mathematics | Experimental investigation | Secondary school students | 60 | Multimedia aids the teaching of mathematics | Duration of the experiment was not stated. Two schools were chosen randomly, no definite number of sample size per group. | No | |
Physiology | Survey (online) | 2 year University Students | 231 | Technology affects students achievements | Study focused on students' interaction with curricular content, administrators, instructors, and other related personnel not considered. | Yes | |
Education | Experimental - comparison with traditional method | University female students | 40 (20 students for each group) | Significant difference observed between the average marks of the two methods | 40 out of 400 female students were used for the study, representing only 10%. | No | |
General course | Survey | University students | 234 | The amount of students learning significantly increased compared to traditional method. | Multimedia has no effect on participation and responsibility, team work, self- esteem and democracy skills of the students. | No | |
Physical education studies | Survey | Professor interview | Undisclosed | Multimedia has positive influence on college physical education. | The paper did not provide the methodology, sample space or size. | No | |
Science | Experimental (using animated cartoons) | 10–11 years | 179 | Motivations to learning aid to young people. | The scope of the multimedia solution is narrow. | Yes | |
Social science | Experimental:-Teaching with multimedia -Teaching without multimedia | 4th grade students | 67 | Multimedia technique increased the academic success. | Single lesson within social studies curriculum was considered Both groups were chosen randomly, no definite number of sample size per group. | No | |
Science | Experimental (using animated cartoons) | Elementary school | 76 | Significant difference was determined in favour of post-test scores | Quasi experimental design was adopted and no control group used for the testing. | No | |
Visual Art Education | Survey: in-depth interview | Secondary school teachers | 2 | Multimedia usage resulted in accelerated teaching and learning processes. | Very small sample size. | No | |
General Education | Survey | Academic staff | 6,139 | Restriction and limit on the use of social media among the academics | Low level of response rate, i.e. 10.5%. | No | |
Mathematics classes | Experimental: -Teaching with multimedia -Teaching without multimedia | University students | 50 (25 each for experimental and control groups) | Experimental group had significantly higher scores | Only two lessons considered: Isometric transformations and regular polyhedral. | No | |
Science | Experimental: multimedia-aided teaching (MAT) | Elementary students | 60 (30 students for each group) | Learners become active participants | No significant difference observed in academic performance. | No | |
General studies | Survey | Students | 272 | Students prefer structured texts with colour discrimi-nation. | No experiment undertaken to validate the outcome. | Yes |
Evaluation of multimedia technology used for teaching and learning is important in establishing the efficacy of the tool. For determination of the impact of a developed tool, an experimental evaluation is more meaningful over a survey. However, the results from the analysis showed that the survey method for evaluation was used nearly as equally as the experimental design.
Experimental based evaluation was conducted by Akinoso (2018) , Aloraini (2012) , Ilhan and Oruc (2016) , and Shah and Khan (2015) in order to determine the effectiveness of the multimedia tool they developed. Another group of experimental evaluations involved designing the research for teaching with or without multimedia aids not necessarily developed by the research team which involved exposing 10–11 year olds ( Dalacosta et al., 2009 ) and elementary school students ( Kaptan and İzgi, 2014 ) to animated cartoons. Another of such evaluation was done by Milovanovi et al. (2013) , who used an experimental and control group to evaluate the impact of teaching a group of university students with multimedia.
In contrast, the survey method was used to elicit the opinion of respondents on the impact of the use of multimedia in teaching and learning and the target group were university students ( Al-Hariri and Al-Hattami, 2017 ; Barzegar et al., 2012 ), secondary school students ( Akinoso, 2018 ; Maaruf and Siraj, 2013 ); one involved interviewing the Professors ( Chen and Xia, 2012 ), another involved 4–10 year olds ( Manca and Ranieri, 2016 ) and a sample of 272 students whose ages were not specified ( Ocepek et al., 2013 ).
The focus areas in which the evaluations were conducted ranged from the sciences including mathematics ( Akinoso, 2018 ; Al-Hariri and Al-Hattami, 2017 ; Dalacosta et al., 2009 ; Kaptan and İzgi, 2014 ; Milovanovi et al., 2013 ) to the social sciences ( Ilhan and Oruc, 2016 ) and the arts ( Maaruf and Siraj, 2013 ). There were evaluations focused on education as a subject as well ( Aloraini, 2012 ; Chen and Xia, 2012 ; Maaruf and Siraj, 2013 ; Manca and Ranieri, 2016 ). While positive outcomes were generally reported, Ocepek et al. (2013) , specified that students in their evaluation study preferred structured texts with colour discrimination.
Sample sizes used in the studies varied widely, from Maaruf and Siraj (2013) that based their conclusions on an in-depth interview of teachers, to Manca and Ranieri (2016) that carried out a survey with a sample of 6,139 academic staff. However, the latter study reported a low response rate of 10.5%. One notable weakness identified was that the findings from all but one of the studies could not be generalized. Reasons for this ranged from inadequate sample size, the exposure being limited to a single lesson, or the sampling method and duration of the experiment not explicitly stated.
The review revealed some challenges that could be barriers to the use of multimedia tools in teaching and learning. Some of these barriers, as found in the reviewed articles, are highlighted as follows:
The findings from the systematic review are discussed in this section with a view to answering the research questions posed. The questions bordered on identifying the existing multimedia tools for teaching and learning and the multimedia components adopted in the tools, the type of audience best suited to a certain multimedia component, the methods used when multimedia in teaching and learning are being evaluated and the success or failure factors to consider.
The review revealed that multimedia tools have been developed to enhance teaching and learning for various fields of study. The review also shows that multimedia tools are delivered using different technologies and multimedia components, and can be broadly categorized as web-based or standalone.
From the review, it was found that standalone multimedia tools were more than twice (64%) the number of tools that were web-based (36%). Standalone tools are a category of teaching and learning aids which are not delivered or used over the internet, but authored to be installed, copied, loaded and used on teachers or students' personal computers (PCs) or workstations. Standalone tools are especially useful for teaching and practicing new concepts such as 3D technology for modelling and printing ( Huang et al., 2017 ) or understanding augmented reality (AR) software ( Blevins, 2018 ). Microsoft Powerpoint is a presentation tool used in some of the reviewed articles and is usually done with standalone systems.
Standalone tools were favoured over web-based tools probably because the internet is not a requirement which makes the tool possible to deploy in all settings. This means that teachers and students in suburban and rural areas that are digitally excluded, can benefit from such a multimedia tool. This system is considered most useful because a majority of the populace in most developing countries are socially and educationally excluded due to a lack of the necessary resources for teaching and learning. The need to sustainably run an online learning environment may be difficult, and therefore, the standalone, provides a better fit for such settings. However, the problem with a standalone application or system is the platform dependency. For instance, a Windows based application can only run on a windows platform. Also, there will be slow convergence time when there is modification in the curricular or modules, since, each system will run offline and has to be updated manually or completely replaced from each location where the tool is deployed.
The other category, web-based multimedia tools, are authored using web authoring tools and delivered online for teaching and learning purposes. About one-third of the tools identified from the review were web-based although they were used largely in university teaching and learning. Examples of these tools are: online teaching and learning resource platform ( Zhang, 2012 ), graphic web-based application ( Bánsági and Rodgers, 2018 ), multimedia tool for teaching optimization ( Jian-hua & Hong, 2012 ), and educational videos on YouTube ( Shoufan, 2019 ).
One of the benefits of the web based multimedia solution is that it is online and centralized over the internet. Part of its advantages is easy update and deployment in contrast to the standalone multimedia system. The major requirements on the teachers and learners' side are that a web browser is installed and that they have an internet connection. Also, the multimedia web application is platform independent; it does not require any special operating system to operate. The same multimedia application can be accessed through a web browser regardless of the learners' operations system. However, when many people access the resource at the same time, this could lead to congestion, packet loss and retransmission. This scenario happens often when large classes take online examinations at the same time. Also, the data requirements for graphics or applications developed with the combination of video, audio and text may differs with system developed with only pictures and text. Hence, the web based system can only be sustainably run with stable high speed internet access.
A major weakness of web-based multimedia tools is the challenge posed for low internet penetration communities and the cost of bandwidth for low-income groups. As access to the internet becomes more easily accessible, it is expected that the advantages of deploying a web-based multimedia solution will far outweigh the disadvantages and more of such tools would be web-based.
The results from the review revealed that most of the existing multimedia tools in education consist of various multimedia components such as text, symbol, image, audio, video and animation, that are converged in technologies such as 3D ( Huang et al., 2017 ), Camtasia Studio 7 software ( Karel and Tomas, 2015 ), Macromedia Flash ( Zhang, 2012 ), HTML5, JavaScript, CSS ( Bánsági and Rodgers, 2018 ; Eady and Lockyer, 2013 ; Chen and Liu, 2008 ; Shah and Khan, 2015 ; Shoufan, 2019 ). As shown in Figure 3 , the analysis confirms that text (26.8%) is the predominant multimedia component being used in most of the educational materials while other components such as videos (19.5%), audios (18.3%), images (18.3%) and animation (11.0%) are fairly used in teaching and learning multimedia materials. However, annotation and 3D technologies are least incorporated.
Proportion of multimedia components in reviewed articles.
How these components are combined is shown in Figure 4 . Perhaps, the combination of these four major components (text, video, audio, image) provides the best outcome for the learner and points to the place of text as a most desired multimedia component. The components used also reflect the type of subject matter being addressed. For instance, the audio component is important for language classes while video and image components are stimulating in Biology classes, for example, due to the need for visual perception for the learners. It is, therefore, imperative to note that the choice of the combination of these components could yield variable impacts to learners. Hence, it can be deduced from the studies that most of the tools are applied either as teaching or/and learning aids depending on the nature of the audience and teacher.
Use of various multimedia combinations.
In Figure 4 , we provided the analysis of the component combination of the data set reviewed. The multimedia components combinations range from two to six. This was grouped based on the multimedia components combination employed in each of the data set. Group 1 (G1) represents the number of multimedia application with the combination of Text, Image, audio, Video, and 3D. G2 consists of video and audio, while G13 combines all the multimedia components except the 3D.
Furthermore, a majority of the multimedia applications considered four (4) and two (2) combinations of components in their design as shown in Figure 5 . Tools with five and six components were very few and as the figure reveals, all the tools used at least two components.
Multimedia tools and the number of components utilized.
These findings stress the fact that application of multimedia tools in education and the multimedia component incorporated, are audience, subject, curricula and teacher-specific and the tool needs to be well articulated and structured to achieve its goals.
Our systematic review also revealed that most multimedia solutions deployed for teaching and learning target the solution to the pedagogical content of the subject of interest (see Table 4 ) and the user audience of the solution ( Table 5 ). Several studies highlighted in Tables 4 and and5 5 showcase multimedia tools used for mathematics classes ( Akinoso, 2018 ; Milovanovi et al., 2013 ), Social science ( Ilhan and Oruc, 2016 ), Physiology ( Al-Hariri and Al-Hattami, 2017 ), Physics ( Jian-hua and Hong, 2012 ), in Chemical engineering ( Bánsági and Rodgers, 2018 ) and teaching of Chinese language ( Wu and Chen, 2018 ). In addition, multimedia tools were utilized for teaching specific principles such as in control theory ( Karel and Tomas, 2015 ) and teaching of arrays ( Kapi et al., 2017 ). That multimedia solutions are subject-based is not surprising given that multimedia involves relaying information using different forms of communication. It follows that multimedia solution developers need to incorporate some text, video, audio, still photographs, sound, animation, image and interactive contents in a manner that best conveys the desired content for teaching or to aid learning.
As stated earlier, the review revealed a variety of user types for the multimedia solutions reported. It is noteworthy that a large proportion of the studies where the target audience were university students, a mixture of graphics, text, audio, video and sometimes animation was utilized ( Aloraini 2012 ; Blevins, 2018 ; Huang et al., 2017 ; Shah and Khan, 2015 ). While a sizeable number of solutions were targeted at secondary school students (such as Maaruf and Siraj, 2013 , Kapi et al., 2017 , and Ilhan and Oruc, 2016 ), very few studies were identified that targeted students less than 15 years in age. Shah and Khan (2015) targeted a multimedia teaching aid that incorporated text, audio, video and animation. Perhaps the absence of multimedia tools targeted at very young children may be as a result of the inclusion criteria used for identifying articles for the review.
The success of the different multimedia tools that have been used on the various target groups and subjects can be attributed to the technologies and components embedded as shown in Tables 4 and and5. 5 . In most cases where text, audio, video, graphics and animations were the components of choice, significant improvements in teaching and learning are used, as reported in the studies reviewed ( Blevins, 2018 ; Huang et al., 2017 ; Zhang, 2012 ).
These studies also implemented technologies such as 3D modelling and printing; Macromedia flash version 8.0 and augmented reality (AR) software respectively. It is worthy of note that all the above-mentioned multimedia tools were applicable in both the teaching and learning processes. Another set of tools with components being text, audio, video and animation, excluding graphics, and equally applied in both the teaching and learning processes, adopted computer representation as their technologies ( Aloraini, 2012 ; Ilhan and Oruc, 2016 ; Milovanovic et al., 2013 ). Teaching and learning were equally greatly improved in these cases.
Our systematic review included a synthesis of the methodologies described by the reviewed articles for evaluating the multimedia tools that they present as shown in the summary in Table 5 . The evaluation methodologies appeared to be different depending on the type of multimedia tool, technology components, deployment strategies, and application area and target groups. However, two main evaluation methods were identified - experimental investigations and the survey methodology.
The experimental approach involved the use of an experimental group and a control group, where the assessment of the impact of the multimedia tool on the students' performance on the experimental group was compared with the performance of the control group who were taught the same content without the use of the multimedia tool. This experimental approach is a widely practiced evaluation method and has proven to be effective. It was deployed by Aloraini (2012) , Milovanovi et al. (2013) , Kaptan and İzgi (2014) , Shah and Khan (2015) , Ilhan and Oruc (2016) and Akinoso (2018) in their studies in the subject area of education, social sciences, general science, science, education and mathematics classes respectively.
Survey, as an evaluation approach which was used in 46% of the studies reviewed, involved the use of questionnaires that were administered to gather opinion on the perceived impact of the multimedia tool from a targeted group of respondents. From the systematic review, it was found that the questionnaire administration approach also varied. The data collection could be face-to-face interview ( Al-Hariri and Al-Hattami, 2017 ; Barzegar et al., 2012 ; Chen and Xia, 2012 ), or online survey ( Armenteros et al., 2013 ; Wang et al., 2020 ).
The difficulty of determining impact from a survey is related to the weaknesses associated with instrument design and sampling biases. It is our opinion that the perceived impact of the technology components used in the development of the multimedia tools may not be accurately ascertained using survey when compared with the actual deployment and experimentation with the multimedia tool that takes place in experimentation approach. Besides, in the survey approach, judgment is merely based on perceptions. Interestingly, the simplicity and ease of the survey method makes it a good option for evaluating larger target groups, and its findings can be generalised when the statistical condition is satisfied ( Krejcie and Morgan, 1970 ).
Although the evaluation studies analysed had publication dates as recently as 2015 to 2018, none reported any objective data collection such as from eye-tracking or other behavioural data. Perhaps, this may be due to our search keyword terms not being wide enough to identify multimedia evaluation studies that used objective data gathering. It could also be that the cost, time and effort needed to collect objective data means that many studies incorporating evaluation are avoiding this route.
Several barriers to multimedia use in teaching and learning were revealed as a result of the review. Such barriers include resistance to the adoption of ICT, lack of teachers' confidence in the use of technology, resistance to change on the part of teachers, a lack of ICT skills and lack of access to ICT resources. Other barriers identified were the lack of support, lack of time to learn new technologies, lack of instructional content, and the physical environment in which multimedia delivery took place. Some studies reported respondents that perceived no benefits from the use of multimedia. These barriers certainly affect both the integration of multimedia in teaching and learning and the uptake of the multimedia tool.
Most of the barriers identified could be classified into three groups with a major one being the fear or resistance to change. This means that change management must be an integral part of multimedia tools development and deployment in order to achieve the desired goal. Also, barriers such as lack of time and lack of resources should not be underestimated. Some of the studies reported providing the hardware for the multimedia application and such an approach should be considered. Most multimedia tools are ICT driven and as such the identified barrier of lack of ICT skills is an important aspect that must be addressed. This can be done as part of the change process and would also boost the confidence of teachers to incorporate multimedia for teaching.
It is important that the multimedia tool is designed and developed with the end-goal in mind. As indicated, some recipients of multimedia applications did not see any benefit in its use. This means that the multimedia tool should be designed to provide an experience that is worth the teachers and students' time, attention and effort.
A lot of work has been done to highlight the effectiveness of multimedia as a teaching and learning aid. This paper provides a systematic review of studies on the use of multimedia in education in order to identify the multimedia tools being commonly used to aid teaching and learning. The paper did a systematic review of extant literature that reported studies that have been carried out to determine the extent to which multimedia has been successful in improving both teaching and learning, and challenges of using multimedia for leaning and teaching.
We note, however, that our review, especially of the studies on evaluation of multimedia, leaned more to the outcome from the studies rather than the process. Some of the information that was not captured include how the classroom teacher's mastery of the technology influences the attractiveness of the tool to the learner, both visually and through the content and if the multimedia tool allowed for learners' participation. Also, while studies on multimedia evaluation was of interest to us, this search phrase was not part of the search phrases used. A future review could incorporate these for a richer perspective.
It is obvious from the review that researchers have explored several multimedia in order to develop teaching and learning tools either based on the web or standalone using different technologies. It is observed that there exist several multimedia tools in education, but the proliferation of the tools is attributed to the evolvement of technologies over the years and the continuous teachers' efforts to improving knowledge delivery with respect to the subject areas and target audience. It is also revealed that most multimedia solutions deployed for teaching and learning target the solution to the pedagogical content of the subject of interest and the user audience of the solution. The success of the different multimedia tools that have been used on the various target groups and subjects is also attributed to the technologies and components embedded.
Furthermore, the evaluation methodologies and learning outcomes of the deployment of multimedia tools appeared to be different depending on the type of multimedia tool, technology components, deployment strategies, and application area and target groups. The two main evaluation methodologies identified from the various studies reported in the articles we reviewed were the experimental investigations and the survey methodology.
Attitudes and beliefs towards the use of technology in education, lack of teachers' confidence and resistance to change, lack of basic knowledge and ICT skills, lack of technical, administrative and financial supports, lack of physical environment are some of the barriers identified in the various articles reviewed. These barriers affect the integration of multimedia in education.
For future work, efforts should be made to explore mobile technology with several multimedia components in order to enhance teaching and learning processes across a diverse group of learners in the primary, secondary, vocational, and higher institutions of learning. Such research efforts would be significant in increasing inclusiveness and narrowing the educational divide. Also, research into the change management process for overcoming the barriers to multimedia adoption would be of interest.
Author contribution statement.
All authors listed have significantly contributed to the development and the writing of this article.
This work was supported by Tertiary Education Trust Fund (TetFund), Ministry of Education, Federal Government of Nigeria 2016–2017 Institutional Based Research Grant.
The authors declare no conflict of interest.
No additional information is available for this paper.
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Email citation, add to collections.
Your saved search, create a file for external citation management software, your rss feed.
Affiliations.
Background: Eye-tracking technology is an established research tool within allied industries such as advertising, psychology and aerospace. This review aims to consolidate literature describing the evidence for use of eye-tracking as an adjunct to traditional teaching methods in medical education.
Methods: A systematic literature review was conducted in line with STORIES guidelines. A search of EMBASE, OVID MEDLINE, PsycINFO, TRIP database, and Science Direct was conducted until January 2017. Studies describing the use of eye-tracking in the training, assessment, and feedback of clinicians were included in the review.
Results: Thirty-three studies were included in the final qualitative synthesis. Three studies were based on the use of gaze training, three studies on the changes in gaze behavior during the learning curve, 17 studies on clinical assessment and six studies focused on the use of eye-tracking methodology as a feedback tool. The studies demonstrated feasibility and validity in the use of eye-tracking as a training and assessment method.
Conclusions: Overall, eye-tracking methodology has contributed significantly to the training, assessment, and feedback practices used in the clinical setting. The technology provides reliable quantitative data, which can be interpreted to give an indication of clinical skill, provide training solutions and aid in feedback and reflection. This review provides a detailed summary of evidence relating to eye-tracking methodology and its uses as a training method, changes in visual gaze behavior during the learning curve, eye-tracking methodology for proficiency assessment and its uses as a feedback tool.
PubMed Disclaimer
Full text sources.
NCBI Literature Resources
MeSH PMC Bookshelf Disclaimer
The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.
Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.
Scientific Reports volume 14 , Article number: 18186 ( 2024 ) Cite this article
155 Accesses
Metrics details
Patients with mental illnesses, particularly psychosis and obsessive‒compulsive disorder (OCD), frequently exhibit deficits in executive function and visuospatial memory. Traditional assessments, such as the Rey‒Osterrieth Complex Figure Test (RCFT), performed in clinical settings require time and effort. This study aimed to develop a deep learning model using the RCFT and based on eye tracking to detect impaired executive function during visuospatial memory encoding in patients with mental illnesses. In 96 patients with first-episode psychosis, 49 with clinical high risk for psychosis, 104 with OCD, and 159 healthy controls, eye movements were recorded during a 3-min RCFT figure memorization task, and organization and immediate recall scores were obtained. These scores, along with the fixation points indicating eye-focused locations in the figure, were used to train a Long Short-Term Memory + Attention model for detecting impaired executive function and visuospatial memory. The model distinguished between normal and impaired executive function, with an F 1 score of 83.5%, and identified visuospatial memory deficits, with an F 1 score of 80.7%, regardless of psychiatric diagnosis. These findings suggest that this eye tracking-based deep learning model can directly and rapidly identify impaired executive function during visuospatial memory encoding, with potential applications in various psychiatric and neurological disorders.
Introduction.
Psychiatric disorders commonly manifest from underlying brain dysfunctions, often resulting in cognitive deficits across various neuropsychological domains 1 . The Rey‒Osterrieth Complex Figure Test (RCFT) has been used as a neuropsychological measure in clinical and research settings to evaluate visuospatial memory and executive function, such as organizational strategy and planning 2 . Notably, impairment of these functions is commonly reported in both patients with psychotic disorders and those with obsessive‒compulsive disorder (OCD) 3 , 4 , 5 , 6 , 7 , 8 . Both patient groups often exhibit significantly lower RCFT immediate recall scores and organization scores than healthy controls (HCs), stemming from executive function deficits during visuospatial memory encoding 3 , 5 , 7 , suggesting that these deficits could serve as transdiagnostic markers across these disorders. These cognitive deficits are also often observed in patients with various psychiatric and neurological disorders, such as bipolar disorder, Alzheimer’s disease, Parkinson’s disease, closed head injury, autism, epilepsy, encephalitis, multiple sclerosis, and ischemic stroke, who exhibit brain dysfunctions 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 .
Although the well-established conventional RCFT is a valuable tool for assessing cognitive deficits across various psychiatric and neurological disorders, it has several limitations. The RCFT indirectly assesses executive function through a drawing task, which makes it challenging to link observable behavioral-level phenotypes, such as poor drawing, to underlying brain dysfunction. The RCFT can also be affected by a patient’s visuomotor function 17 ; in addition, the administration and scoring systems of the RCFT are time-consuming, labor-intensive, complex, and subject to scoring variability among clinicians owing to human biases 18 , 19 .
Considerable efforts have been made to overcome the limitations of the RCFT scoring system, including the development of an automated scoring system using photos of RCFT drawings and a deep learning algorithm 20 , 21 , 22 , 23 , the implementation of a tablet-based digital drawing assessment 24 , and the adoption of a simpler method for scoring organizational strategies (0 or 1 points) 25 . Although these approaches have made substantial advancements in addressing the complexity, labor intensity, and scoring variability of the scoring system, there are still limitations in its administration given that it is a time-consuming, visuomotor function-affected, indirect drawing test.
To address the remaining limitations, a previous study from our laboratory successfully identified an eye movement biomarker that can be used to rapidly and directly assess impaired organizational strategy during the RCFT in patients with OCD 26 . However, the application of the results of that study to other psychiatric and neurological disorders is limited because the biomarker relies solely on calculations of eye gaze distribution and is based on the assumption that patients with OCD exhibit weak central coherence, focusing on narrow details of the RCFT figure. Thus, there is a need to develop a data-driven deep learning-based assessment model that is not constrained by a single disease characteristic and exhibits increased speed, simplicity, and directness and to extend its applicability to a broader range of psychiatric and neurological disorders. The development of this model would also be consistent with recent interest in assessing cognitive functions such as visual memory and attention in patients with mental illnesses using eye tracking, as eye movements provide real-time insights into the cognitive activities involved in how gaze interacts with visuospatial stimuli during the visual encoding process 27 , 28 , 29 , 30 , 31 , 32 .
Therefore, this study aimed to establish an assessment model using eye tracking and deep learning in patients with early psychosis, including those with first-episode psychosis (FEP), patients at clinical high risk (CHR) for psychosis, patients with OCD and HCs. The primary aim was to explore the effectiveness of the model as a rapid, simple, and direct assessment of impaired executive function in these patients. FEP and OCD patients were selected because of their notable impairments in executive function and visuospatial memory 3 , 5 , whereas CHR patients were selected because they included both patients with early psychosis and those with nonspecific psychiatric symptoms, such as depression and anxiety 33 . We hypothesized that the RCFT assessment model developed in this study would be able to distinguish normal and impaired executive function as well as identify visuospatial memory impairment on the basis of eye movements during the memorization of the RCFT figure, regardless of the specific psychiatric diagnosis, with increased speed, simplicity, and directness.
The demographic and clinical characteristics of the participants in each diagnostic group are summarized in Table 1 . The demographic characteristics of the participants, stratified by normal or impaired executive function and normal or impaired visuospatial memory, are summarized in Table 2 . The participants with normal executive function, as measured by the organization T score, and normal visuospatial memory, as measured by the immediate recall T score, had a greater intelligence quotient (IQ) than did the subjects with impaired executive function (t = 2.801, p = 0.013) and impaired visuospatial memory (t = 6.832, p = < 0.001), respectively. There were more females than males with impaired executive function than with normal executive function (χ 2 = 4.620, p = 0.032). Age, years of education, handedness and the proportion of participants wearing glasses were not different between the groups with normal and impaired executive function or visuospatial memory.
Analysis of covariance (ANCOVA) with IQ as a covariate revealed that participants with normal executive function had higher organization T scores (F = 249.031, p < 0.001) than did participants with impaired executive function. ANCOVA using sex and IQ as covariates revealed that the participants with normal executive function had higher total organization scores (F = 50.382, p < 0.001), fragmentation scores (F = 4.301, p = 0.039), and planning scores (F = 102.373, p < 0.001) than did the participants with impaired executive function. Similarly, ANCOVA with IQ as a covariate revealed that participants with normal visuospatial memory had higher immediate recall T scores (F = 87.960, p < 0.001), total immediate recall scores (F = 200.415, p < 0.001), configural presence scores (F = 242.611, p < 0.001), configural accuracy scores (F = 70.500, p < 0.001), cluster presence scores (F = 89.203, p < 0.001), cluster accuracy scores (F = 52.323, p < 0.001), and detail presence scores (F = 39.270, p < 0.001) than did participants with impaired visuospatial memory. There was no difference in immediate recall time (F = 2.073, p = 0.151) between participants with normal and impaired visuospatial memory. The ANCOVA results for RCFT performance between HCs and patients with normal and impaired executive function and visuospatial memory are described in Table S1 in the Supplementary material.
Table 3 shows the deep learning performance results of the LSTM + Attention model 34 , 35 for the two classifications. The model achieved F 1 scores of 83.5 and 80.7% and area under the receiver operating characteristic curve (AUROC) values of 60.7% (Fig. 1 a) and 69.9% (Fig. 1 b) for distinguishing between normal and impaired executive function and between normal and impaired visuospatial memory, respectively.
Gaze fixation sequence map showing the order of eye movements during the 3-min memorization of the Rey‒Osterrieth Complex Figure Test (RCFT) figure. ( a ) Receiver operating characteristic (ROC) curve of the Long Short-Term Memory (LSTM) + Attention model classification for normal and impaired executive function. ( b ) ROC curve of the LSTM + Attention model classification for normal and impaired visuospatial memory. ( c ) The order of gaze fixation in the patient with the lowest organization T score (i.e., < 20). ( d ) The order of gaze fixation in the patient with the highest organization T score (i.e., 70). The numbers within the circles indicate the order of gaze fixation, starting from 0. Abbreviation: AUROC, area under the ROC curve.
Eye gaze sequence maps are displayed in Fig. 1 c and d. The order of gaze fixation was concentrated in a narrow area and horizontally distributed, with a low number of fixations in the patient with the lowest organization T score (i.e., < 20; Fig. 1 c). Conversely, the order of gaze fixation was widely and evenly distributed across the figure in the patient with the highest organization T score (i.e., 70; Fig. 1 d). The eye movement comparison results between the normal and impaired groups are provided in Tables 4 and 5 .
This study aimed to develop an eye-tracking and deep learning-based RCFT assessment model for evaluating impaired executive function during visuospatial memory encoding in the RCFT that is faster, simpler, and more direct. The model achieved high performance in assessing impairment in early psychosis and OCD patients on the basis of their sequential eye movements while they were memorizing the RCFT figure. This assessment is performed regardless of specific psychiatric diagnoses, as this impairment is shared across these disorders. These results indicate that eye movements during the encoding of highly complex figures reflect executive function during visuospatial memory encoding, serving as a transdiagnostic biomarker of impairment in early psychosis and OCD. Additionally, since the assessment model in this study utilized a data-driven deep learning technique that does not require handcrafted feature selection according to specific disease hypotheses, there is the potential to extend the use of this model beyond early psychosis and OCD to other psychiatric and neurological disorders that also exhibit impaired executive function, poor performance in the RCFT in the form of disorganized and fragmented drawings, and difficulties in visuospatial integration 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 .
The model also offered a rapid and simple measure of impaired executive function, taking only 3 min from administration to assessment through computerized eye movement recording and deep learning analysis. This process is considerably faster and simpler than the traditional RCFT scoring system and the automated scoring systems reported in previous studies, which still require a prior drawing process 20 , 21 , 22 , 23 . This improved model is beneficial and easy to apply in real-world clinical and research settings, saving a significant amount of labor and time and reducing human scoring variability. Moreover, the model enabled a more direct assessment, as sequential eye movements reflected real-time visuospatial information processing 36 and indicated how subjects strategically encoded, planned, and organized the figure. Thus, eye-tracking measurements can bridge the gap between the behavioral-level phenotype and brain dysfunction by capturing the inner workings of executive function during visuospatial memory encoding.
In the interpretation of the results of the deep learning model, the gaze fixation sequence maps in Fig. 1 c and d show distinct eye movement patterns between patients with the lowest and highest organization scores. In the patient with the lowest organization score, exploration of the figure was limited, and the patient lacked a strategy and focused simply on horizontal movements without attending to important areas. In contrast, in the patient with the highest organization score, exploration was more structured and extensive, capturing a larger picture and focusing sequentially on various critical areas. In addition, quantitative differences in eye movements revealed that participants with impaired executive function and visuospatial memory spent less time looking at the figure and exhibited fewer fixations and saccades than the participants in the normal group did, indicating less effective and comprehensive encoding of the visuospatial information within the figure (Tables 4 and 5 ). These ineffective, disorganized, and limited eye movement patterns in the impaired group seem to be consistent with previous research findings that patients with executive function deficits have difficulties processing the overall RCFT figure and utilize a fragmented and piecemeal approach 3 , 5 , 7 . Overall, these differences in eye movement patterns between the impaired and normal groups may have contributed to the ability of the deep learning model to distinguish between them.
There are several limitations in this study. First, this assessment model was initially developed using data from individuals with early psychosis and OCD and classified their functions into only normal and impaired, limiting its applicability to individuals with other psychiatric and neurological disorders. Additionally, the impairment criteria in this study were stringent, making it challenging to detect patients with mild impairment. Nonetheless, this model has the potential to expand beyond binary classification and include a wider range of psychiatric and neurological disorders. Second, the small number of participants in the impaired group resulted in highly imbalanced datasets, which limits the validity and reliability of the assessment model in this study. To address this, the split dataset was stratified to maintain class label proportions consistent with those of the original dataset, and synthetic minority oversampling technique (SMOTE) 37 data augmentation was implemented, as in previous studies with similarly imbalanced datasets 38 , 39 , 40 . However, our results should be interpreted with caution because of the relatively small and imbalanced sample size of the original dataset. Third, most patients were taking medication at the time of the eye-tracking RCFT. Therefore, it is necessary to consider the medication effect when interpreting the study results, as this study did not investigate the impact of medication on patients' RCFT performance or eye movement markers. However, given that the assessment model aims to encompass various psychiatric and neurological disorders in future research, these findings remain promising, as they indicate effectiveness of the model even in the presence of potential influences from medication. Fourth, there was a significant difference in IQ between the normal and impaired groups. Although the statistical group comparison was conducted with IQ as a covariate, the deep learning model, LSTM + Attention, does not account for or exclude the potential impact of cognitive function on eye movement markers in its classification.
Although the RCFT is a well-established tool for evaluating executive function during visuospatial memory encoding, its administration and scoring pose difficulties because of its time-consuming nature, indirect measurement, and scoring variability. While a previous study from our laboratory identified an eye movement biomarker to detect impaired executive function with enhanced speed and directness, its utility was limited to OCD because it was based on OCD-specific characteristics. Therefore, we developed an RCFT assessment model that integrated eye tracking and deep learning, which not only offered a more direct, rapid, and simplified evaluation of impaired executive function but also demonstrated the potential for wider applicability to other disorders, as it was data driven and did not rely on singular disease hypotheses. Future studies could benefit from including various psychiatric and neurological disorders and utilizing explainable artificial intelligence to identify key features distinguishing between individuals with normal and impaired executive function during visuospatial memory encoding.
We analyzed data from 408 participants, including 96 patients with FEP, 49 patients at CHR for psychosis, 104 patients with OCD, and 159 HCs. FEP patients and CHR individuals were recruited from both the inpatient and outpatient clinics of the Department of Neuropsychiatry and the Seoul Youth Clinic ( www.youthclinic.org ) at Seoul National University Hospital (SNUH). In this study, the FEP patients included individuals who were diagnosed with schizophrenia, schizoaffective disorder, or schizophreniform disorder according to the Structured Clinical Interview for the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Axis I Disorders (SCID-I) with an illness duration of less than 2 years. Psychotic symptoms were assessed using the Positive and Negative Syndrome Scale (PANSS). To confirm the CHR status of the participants, the Structured Interview for Prodromal Symptoms (SIPS) 41 was used. Prodromal symptoms were assessed using the validated Korean version of the Scale of Prodromal Symptoms (SOPS) 41 , 42 . Patients with OCD were recruited from the OCD clinic ( www.ocd.snu.ac.kr ) at SNUH and fulfilled the Diagnostic and Statistical Manual of Mental Disorders-IV (DSM-IV) criteria for OCD. The Yale‒Brown Obsessive Compulsive Scale (Y-BOCS) 43 was used to evaluate the severity of OCD. The Hamilton Rating Scale for Depression (HAM-D) 44 and the Hamilton Rating Scale for Anxiety (HAM-A) 45 were used to evaluate the severity of depressive and anxious symptoms, respectively. The HCs were recruited using internet advertisements. To screen for the presence of psychiatric disorders or symptoms, the HCs underwent an assessment employing the Structured Clinical Interview for DSM-IV Non-Patient Version (SCID-NP). HCs with a past or current axis I diagnosis or first- to third-degree biological relatives with a history of psychotic disorders were not eligible to participate in this study. We assessed participants' IQ using the Korean version of the Wechsler Adult Intelligence Scale (K-WAIS) 46 .
All participants were assessed according to the exclusion criteria, which included the presence of neurological conditions, significant head injuries, substance abuse or dependency (except for nicotine), and intellectual disability (IQ < 70). We provided thorough explanations of the research procedures and obtained written informed consent from all participants (IRB no. H-1110-009-380, H-1201-008-392). For participants younger than 18, consent was also obtained from their parents. This study adhered to the principles outlined in the Declaration of Helsinki (2013) and received approval from the Institutional Review Board of SNUH (IRB No. H-2306-210-1445).
Eye movement data were obtained during an eye-tracking experiment in which participants were instructed to view and memorize the RCFT figure for a duration of 3 min (Fig. 2 ), as detailed by Kim et al. 26 . Briefly, the RCFT figure was presented on a 19-inch monitor with a screen resolution of 1280 × 1024 pixels using Experiment Builder v.2.1.45 software (SR Research, Ottawa, Ontario, Canada). During the experiment, the participant's head was positioned on a chin rest in a room with low lighting. The distance between the chin rest and the monitor was 70 cm, and the participant had a horizontal viewing angle of 22° and a vertical viewing angle of 17°. Before eye movements were measured, a nine-point calibration and verification process was conducted. The data were collected at a 1,000-Hz sampling rate and exported through the EyeLink 1000 (SR Research) eye tracking device.
Eye tracking-based Rey‒Osterrieth Complex Figure Test (RCFT) procedure used in this study. Memorization of an RCFT figure for 3 min was followed by immediate recall of the figure.
The key parameters collected were gaze fixation point coordinates, indicating where the eyes briefly paused to focus and acquire new information 47 , and time in milliseconds. Upon the completion of the eye-tracking session, the participants were instructed to reproduce the RCFT figure from memory, akin to the immediate recall condition of the RCFT. During this drawing task, response times were recorded, and an experimenter systematically tracked the participant’s reproduction of the figure. This meticulous monitoring aimed to assess organizational strategies as a substitution for the RCFT copy condition. A skilled evaluator manually assessed the participants’ drawings using the Boston Qualitative Scoring System (BQSS) 48 . This assessment aimed to evaluate the participants' organizational and immediate recall performance in the RCFT. The organization and immediate recall scores were subsequently categorized according to the BQSS clinical interpretation criteria as normal (score: 40–70) or impaired (score: < 39). The participants were grouped as follows: a normal executive function group with normal organization scores, a normal visuospatial memory group with normal immediate recall scores, an impaired executive function group with impaired organization scores and an impaired visuospatial memory group with impaired immediate recall scores.
In this study, we utilized an LSTM model using Python and PyTorch 49 to analyze time series eye movement fixations during the memorization of the RCFT figure. The effectiveness of the LSTM model in handling sequential data was a key factor in its selection, especially since sequential temporal relationships might play a significant role in participants' effective memorization and organization of the RCFT figure. Additionally, the LSTM model was combined with an attention mechanism. Time series fixation sequences were input recursively into the model, facilitating the learning of patterns and relationships within sequential eye movement fixations. The acquired representations at each timestamp were summed by attention coefficients to obtain the final sequence representation. The resulting representations were fed into a single-layer classifier to determine the probabilities of sequences belonging to a specific class (normal or impaired). The model was trained with a sequence size of 32 fixation points, and the learning rate was set to α = 0.005 over the training course. The fixation dataset was split 70/30 into training and testing sets, and the split was stratified to preserve class label proportions similar to those of the original dataset. The evaluation metrics used in the LSTM + Attention model included recall (sensitivity), precision, AUROC, and F 1 score. The AUROC and F 1 score were utilized to determine the accuracy of the model in highly imbalanced datasets, as in this study. The overall modeling workflow is described in Fig. 3 .
Overall workflow of modeling in this study. Abbreviations: Org, organization; IR, immediate recall; SMOTE, synthetic minority oversampling technique.
In this study, the dataset was highly imbalanced, with a significant disparity between the majority class (e.g., 385 participants in the normal group) and the minority class (e.g., 23 participants in the impaired group). Imbalanced datasets cause problems for learning algorithms that expect an even distribution across classes, leading to bias favoring the majority class 50 . To address this, data augmentation is commonly employed to achieve an ideal balance, e.g., a 50:50 ratio, by artificially expanding the training dataset for enhanced reliability. Thus, the minority class (impaired group) in our training datasets was oversampled using SMOTE. The imbalance was maintained in the test dataset to represent the real-world distribution.
SPSS v.26.0 (IBM, Armonk, NY, USA) was used for the statistical analyses, and the significance level was set at p < 0.05. Comparisons of demographic and clinical characteristics across groups were performed using independent t tests or Welch's t tests if the variances were not equal for continuous variables and chi-square tests for categorical variables. Group comparisons of RCFT scores were performed using ANCOVA with IQ or IQ and sex as covariates.
Owing to the limited interpretability of the decision-making process of the LSTM + Attention model, additional visual interpretations and statistical analyses were conducted to interpret the results of the model in this study. First, a gaze fixation sequence map was created to explore participants' visuospatial information processing and organization of the RCFT figure in sequence. Second, eye movement measures, including the number of fixations, average duration of fixation (ms), average saccade amplitude and duration (ms), and number of blinks and saccades, were compared between the normal and impaired groups to identify quantitative differences in eye movements. Saccades refer to rapid eye movements between fixations.
The data that support the findings of this study are available from the corresponding author upon reasonable request.
Keefe, R. S. The contribution of neuropsychology to psychiatry. Am. J. Psychiatry 152 , 6–15 (1995).
Article CAS PubMed Google Scholar
Rey, A. L’examen psychologique dans les cas d’encephalopathie traumatique. Arch. Psychol. 28 , 215–285 (1941).
Google Scholar
Kim, M. S., Namgoong, Y. & Youn, T. Effect of organizational strategy on visual memory in patients with schizophrenia. Psychiatry Clin. Neurosci. 62 , 427–434 (2008).
Article PubMed Google Scholar
Penadés, R., Catalán, R., Andrés, S., Salamero, M. & Gastó, C. Executive function and nonverbal memory in obsessive-compulsive disorder. Psychiatry Res. 133 , 81–90 (2005).
Savage, C. R. et al. Organizational strategies mediate nonverbal memory impairment in obsessive–compulsive disorder. Biol. Psychiatry 45 , 905–916 (1999).
Scala, S. et al. Verbal and visual–spatial memory impairment in youth at familial risk for schizophrenia or affective psychosis: A pilot study. Schizophr. Res. 144 , 122–128 (2013).
Article PubMed PubMed Central Google Scholar
Seidman, L. J., Lanca, M., Kremen, W. S., Faraone, S. V. & Tsuang, M. T. Organizational and visual memory deficits in schizophrenia and bipolar psychoses using the Rey-Osterrieth complex figure: Effects of duration of illness. J. Clin. Exp. Neuropsychol. 25 , 949–964 (2003).
Shin, N. Y., Lee, T. Y., Kim, E. & Kwon, J. S. Cognitive functioning in obsessive-compulsive disorder: A meta-analysis. Psychol. Med. 44 , 1121–1130 (2014).
Bigler, E. D., Rosa, L., Schultz, F., Hall, S. & Harris, J. Rey-Auditory verbal learning and Rey-Osterrieth complex figure design performance in Alzheimer’s disease and closed head injury. J. Clin. Psychol. 45 , 277–280 (1989).
Cardillo, R., Lievore, R. & Mammarella, I. C. Do children with and without autism spectrum disorder use different visuospatial processing skills to perform the Rey-Osterrieth complex figure test?. Autism Res. 15 , 1311–1323 (2022).
Correa-Ghisays, P. et al. Visual memory dysfunction as a neurocognitive endophenotype in bipolar disorder patients and their unaffected relatives. Evidence from a 5-year follow-up Valencia study. J. Affect. Disord. 257 , 31–37 (2019).
Dimitrov, I. et al. Application of the Rey-Osterrieth complex figure test for assessment of cognitive impairment in multiple sclerosis. Scr. Sci. Med. 47 , 59–64 (2015).
Finke, C. et al. Evaluation of cognitive deficits and structural hippocampal damage in encephalitis with leucine-rich, glioma-inactivated 1 antibodies. JAMA Neurol. 74 , 50–59 (2017).
Grossman, M. et al. Visual construction impairments in Parkinson’s disease. Neuropsychology 7 , 536–547 (1993).
Article Google Scholar
Hernandez, M. T. et al. Attention, memory, and behavioral adjustment in children with frontal lobe epilepsy. Epilepsy Behav. 4 , 522–536 (2003).
Schaapsmeerders, P. et al. Long-term cognitive impairment after first-ever ischemic stroke in young adults. Stroke 44 , 1621–1628 (2013).
VanGilder, J. L., Lohse, K. R., Duff, K., Wang, P. & Schaefer, S. Y. Evidence for associations between Rey-Osterrieth complex figure test and motor skill learning in older adults. Acta Psychol. 214 , 103261 (2021).
Fastenau, P. S., Bennett, J. M. & Denburg, N. L. Application of psychometric standards to scoring system evaluation: Is “new” necessarily “improved”?. J. Clin. Exp. Neuropsychol. 18 , 462–472 (1996).
Tupler, L. A., Welsh, K. A., Asare-Aboagye, Y. & Dawson, D. V. Reliability of the Rey-Osterrieth complex figure in use with memory-impaired patients. J. Clin. Exp. Neuropsychol. 17 , 566–579 (1995).
Langer, N. et al. The AI neuropsychologist: Automatic scoring of memory deficits with deep learning. bioRxiv . https://doi.org/10.1101/2022.06.15.496291 (2022).
Park, J. Y., Seo, E. H., Yoon, H. J., Won, S. & Lee, K. H. Automating Rey complex figure test scoring using a deep learning-based approach: A potential large-scale screening tool for cognitive decline. Alzheimers Res. Ther. 15 , 145 (2023).
Vogt, J. et al. Automated scoring of the Rey-Osterrieth complex figure test using a deep-learning algorithm. Arch. Clin. Neuropsychol. 34 , 836–836 (2019).
Youn, Y. C. et al. Use of the clock drawing test and the Rey-Osterrieth complex figure test-copy with convolutional neural networks to predict cognitive impairment. Alzheimers Res. Ther. 13 , 85 (2021).
Article CAS PubMed PubMed Central Google Scholar
Petilli, M. A., Daini, R., Saibene, F. L. & Rabuffetti, M. Automated scoring for a Tablet-based Rey Figure copy task differentiates constructional, organisational, and motor abilities. Sci. Rep. 11 , 14895 (2021).
Article ADS CAS PubMed PubMed Central Google Scholar
Smith, S. R. et al. Psychometrics of a simple method for scoring organizational approach on the Rey-Osterrieth complex figure. J. Neuropsychol. 1 , 39–51 (2007).
Kim, M. et al. Eye movement as a biomarker of impaired organizational strategies during visual memory encoding in obsessive–compulsive disorder. Sci. Rep. 11 , 18402 (2021).
Yi, L. et al. Children with autism spectrum disorder scan own-race faces differently from other-race faces. J. Exp. Child Psychol. 141 , 177–186 (2016).
Bradley, M. C. et al. Obsessive–compulsive symptoms and attentional bias: An eye-tracking methodology. J. Behav. Ther. Exp. Psychiatry 50 , 303–308 (2016).
Haque, R. U. et al. VisMET: A passive, efficient, and sensitive assessment of visuospatial memory in healthy aging, mild cognitive impairment, and Alzheimer’s disease. Learn. Mem. 26 , 93–100 (2019).
Vail, A. K. et al. Visual attention in schizophrenia: Eye contact and gaze aversion during clinical interactions. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) 490–497 (IEEE, 2017).
Navalón, P. et al. Attentional biases to emotional scenes in schizophrenia: An eye-tracking study. Biol. Psychol. 160 , 108045 (2021).
Pavisic, I. M. et al. Eye-tracking indices of impaired encoding of visual short-term memory in familial Alzheimer’s disease. Sci. Rep. 11 , 8696 (2021).
Addington, J. et al. Comorbid diagnoses for youth at clinical high risk of psychosis. Schizophr. Res. 190 , 90–95 (2017).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9 , 1735–1780 (1997).
Mnih, V., Heess, N., Graves, A. & Kavukcuoglu, K. Recurrent models of visual attention. In NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Systems 2204–2212 (MIT Press, 2014).
Wolf, K. et al. Neurocognitive development of the resolution of selective visuo-spatial attention: Functional MRI evidence from object tracking. Front. Psychol. 9 , 1106 (2018).
Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. SMOTE: Synthetic minority over-sampling technique. J. Artif. Intell. Res. 16 , 321–357 (2002).
Ahmed, Z. A. T. et al. Applying eye tracking with deep learning techniques for early-stage detection of autism spectrum disorders. Data 8 , 168 (2023).
Priyadarshini, I. Autism screening in toddlers and adults using deep learning and fair AI techniques. Future Internet 15 , 292 (2023).
Sims, S. & Conati, C. A neural architecture for detecting confusion in eye-tracking data. arXiv preprint arXiv:2003.06434 (2020).
Miller, T. J. et al. Prodromal assessment with the structured interview for prodromal syndromes and the scale of prodromal symptoms: Predictive validity, interrater reliability, and training to reliability. Schizophr. Bull. 29 , 703–715 (2003).
Jung, M. H. et al. The reliability and validity of the Korean version of the structured interview for prodromal syndrome. Psychiatry Investig. 7 , 257–263 (2010).
Goodman, W. K. et al. The Yale-Brown obsessive compulsive scale. I. Development, use, and reliability. Arch. Gen. Psychiatry 46 , 1006–1011 (1989).
Hamilton, M. A rating scale for depression. J. Neurol. Neurosurg. Psychiatry 23 , 56–62 (1960).
Hamilton, M. The assessment of anxiety states by rating. Br. J. Med. Psychol. 32 , 50–55 (1959).
Kim, Z. Two-and four-subtest short forms of the Korean-Weschler Adult Intelligence. Scale Seoul J. Psychiatry 19 , 121–126 (1994).
Blignaut, P. Fixation identification: The optimum threshold for a dispersion algorithm. Atten. Percept. Psychophys. 71 , 881–895 (2009).
Somerville, J., Tremont, G. & Stern, R. A. The Boston qualitative scoring system as a measure of executive functioning in Rey-Osterrieth complex figure performance. J. Clin. Exp. Neuropsychol. 22 , 613–621 (2000).
Paszke, A. et al. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems 8026–8037 (Curran Associates Inc., 2019).
Fotouhi, S., Asadi, S. & Kattan, M. W. A comprehensive data level analysis for cancer diagnosis on imbalanced data. J. Biomed. Inform. 90 , 103089 (2019).
Download references
This research was supported by the Bio & Medical Technology Development Program, the Brain Science Convergence Research Program through the National Research Foundation of Korea (NRF) and the KBRI basic research program through the Korea Brain Research Institute, funded by the Ministry of Science & ICT (2021M3A9E408078412, RS-2023-00266120, and 21-BR-03-01).
These authors contributed equally: Minah Kim and Jungha Lee.
Department of Neuropsychiatry, Seoul National University Hospital, Seoul, Republic of Korea
Minah Kim, Jiseon Jang, Moonyoung Jang, Sunghyun Park & Jun Soo Kwon
Department of Psychiatry, Seoul National University College of Medicine, 101 Daehak-no, Chongno-gu, Seoul, 03080, Republic of Korea
Minah Kim, Moonyoung Jang & Jun Soo Kwon
Department of Brain and Cognitive Sciences, Seoul National University College of Natural Sciences, Seoul, Republic of Korea
Jungha Lee, Minji Ha, Inkyung Park & Jun Soo Kwon
Kim Jaechul Graduate School of Artificial Intelligence, KAIST, Daejeon, Republic of Korea
Soo Yong Lee
Institute of Human Behavioral Medicine, SNU-MRC, Seoul, Republic of Korea
Jun Soo Kwon
You can also search for this author in PubMed Google Scholar
Authors M.K., J.L., and J.S.K. contributed to the conception and design of the study. Authors M.K., J.L., M.H., I.P., J.J., M.J., and S.P. collected the data. Authors J.L. and S.Y.L. performed the data analysis. Authors M.K. and J.L. wrote the first draft of the manuscript. Authors S.Y.L., M.H., I.P., J.J., M.J., S.P., and J.S.K. interpreted the data and critically edited the manuscript. Authors M.K. and J.S.K. contributed to the conception of the study, interpreted the data, and provided critical comments regarding the manuscript. All the authors contributed to manuscript revision and read and approved the submitted version.
Correspondence to Jun Soo Kwon .
Competing interests.
The authors declare no competing interests.
Publisher's note.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information., rights and permissions.
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .
Reprints and permissions
Cite this article.
Kim, M., Lee, J., Lee, S.Y. et al. Development of an eye-tracking system based on a deep learning model to assess executive function in patients with mental illnesses. Sci Rep 14 , 18186 (2024). https://doi.org/10.1038/s41598-024-68586-2
Download citation
Received : 05 May 2024
Accepted : 25 July 2024
Published : 06 August 2024
DOI : https://doi.org/10.1038/s41598-024-68586-2
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.
Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.
262 Accesses
Explore all metrics
This comprehensive systematic review synthesizes thirty-four peer-reviewed articles published between 2010 and 2022, utilizing eye-tracking research within interactive language learning environments. Following the PRISMA scheme for article selection, this review illuminates both the affordances and challenges of eye-tracking technology in enhancing language learning outcomes. Through a methodical examination, including sensitivity and specificity analysis of relevant databases such as JSTOR, EBSCOHost, and ProQuest, this study not only underscores the potential of eye-tracking technology in identifying effective instructional strategies and personalizing instruction but also addresses significant challenges like equipment cost and complexity. Theoretically, this review advances our understanding of the cognitive processes involved in language learning by detailing how eye-tracking data can reveal patterns of attention allocation and information processing that are essential for effective CALL design. Pedagogically, it suggests that educators can leverage these insights to develop more engaging and effective language learning interventions that cater to the diverse needs of learners. By highlighting specific instances where eye-tracking technology has facilitated improved learning outcomes, this review sets a foundation for future research to explore innovative ways to integrate visual attention analysis in language education. Future research directions are proposed for continuing to harness eye-tracking technology’s utility in both theoretical exploration and practical application in language learning research and CALL design.
This is a preview of subscription content, log in via an institution to check access.
Subscribe and save.
Price includes VAT (Russian Federation)
Instant access to the full article PDF.
Rent this article via DeepDyve
Institutional subscriptions
Data availability.
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.
Alkan, S., & Cagiltay, K. (2007). Studying computer game learning experience through eye tracking. British Journal of Educational Technology , 38 (3), 538–542. https://doi.org/10.1111/j.1467-8535.2007.00721.x .
Article Google Scholar
Aryadoust, V., & Ang, B. H. (2021). Exploring the frontiers of eye tracking research in language studies: A novel co-citation scientometric review. Computer Assisted Language Learning , 34 (7), 898–933. https://doi.org/10.1080/09588221.2019.1647251 .
Bacca-Acosta, J., Tejada, J., Fabregat, R., et al. (2022). Scaffolding in immersive virtual reality environments for learning English: An eye tracking study. Educational Technology Research and Development , 70 , 339–362. https://doi.org/10.1007/s11423-021-10068-7 .
Bacca-Acosta, J., Tejada, J., Fabregat, R., Kinshuk, & Guevara, J. (2022b). Scaffolding in immersive virtual reality environments for learning English: An eye tracking study. Educational Technology Research and Development , 1–24. https://doi.org/10.1007/s11423-021-10068-7 .
Bahari, A. (2021). Computer-assisted Language Proficiency Assessment Tools and Strategies. Open Learning: The Journal of Open Distance and e-Learning , 36 (1), 61–87. https://doi.org/10.1080/02680513.2020.1726738 .
Article MathSciNet Google Scholar
Bahari, A. (2022). Affordances and challenges of Technology-assisted Language Learning for Motivation: A systematic review. Interactive Learning Environments https// . https://doi.org/10.1080/10494820.2021.2021246 .
Bahari, A. (2023). Challenges and affordances of cognitive load management in Technology-assisted Language Learning: A systematic review. International Journal of Human-Computer Interaction , 38 (3), 1–15. https://doi.org/10.1080/10447318.2021.2019957 .
Bahari, A., Wu, S., & Ayres, P. (2023). Improving computer–assisted Language Learning through the Lens of Cognitive load. Educational Psychology Review , 35 (53), 1–30. https://doi.org/10.1007/s10648-023-09764-y .
Benati, A. (2022). The effects of structured input and traditional instruction on the acquisition of the English causative passive forms: An eye-tracking study measuring accuracy in responses and processing patterns. Language Teaching Research , 26 (6), 1231–1251. https://doi.org/10.1177/1362168820928577 .
Boers, F., Warren, P., Grimshaw, G., & Siyanova-Chanturia, A. (2017). On the benefits of multimodal annotations for vocabulary uptake from reading. Computer Assisted Language Learning , 30 (7), 709–725. https://doi.org/10.1080/09588221.2017.1356335 .
Bolzer, M., Strijbos, J. W., & Fischer, F. (2015). Inferring mindful cognitive-processing of peer‐feedback via eye‐tracking: Role of feedback‐characteristics, fixation‐durations and transitions. Journal of Computer Assisted Learning , 31 (5), 422–434. https://doi.org/10.1111/jcal.12091 .
Chen, Y. T., Li, M., Huang, C. Q., Han, Z. M., Hwang, G. J., & Yang, G. (2022). Promoting deep writing with immersive technologies: An SVVR-supported Chinese composition writing approach for primary schools. British Journal of Educational Technology , 53 (6), 2071–2091. https://doi.org/10.1111/bjet.13247 .
Chen, Y. T., Li, M., & Cukurova, M. (2023a). Unleashing imagination: An effective pedagogical approach to integrate into spherical video-based virtual reality to improve students’ creative writing. Education and Information Technologies , 1–25. https://doi.org/10.1007/s10639-023-12115-7 .
Chen, Y., Li, M., Huang, C., Cukurova, M., & Ma, Q. (2023b). A Systematic Review of Research on Immersive Technology-enhanced writing education: The current state and a Research Agenda. IEEE Transactions on Learning Technologies . https://doi.org/10.1109/TLT.2023.3341420 .
Chen, F., Xia, Q., Feng, Y., Wang, L., & Peng, G. (2023c). Learning challenging L2 sounds via computer-assisted training: Audiovisual training with an airflow model. Journal of Computer Assisted Learning , 39 (1), 34–48. https://doi.org/10.1111/jcal.12724 .
Chen, Y. T., Li, M., Cukurova, M., & Jong, M. S. Y. (2024). Incorporation of peer-feedback into the pedagogical use of spherical video‐based virtual reality in writing education. British Journal of Educational Technology , 55 (2), 519–540. https://doi.org/10.1111/bjet.13376 .
Chien, K. P., Tsai, C. Y., Chen, H. L., Chang, W. H., & Chen, S. (2015). Learning differences and eye fixation patterns in virtual and physical science laboratories. Computers & Education , 82 , 191–201. https://doi.org/10.1016/j.compedu.2014.11.023 .
Chun, D., Kern, R., & Smith, B. (2016). Technology in language use, language teaching, and language learning. The Modern Language Journal , 100 (S1), 64–80. https://doi.org/10.1111/modl.12302 .
Clark, R. C., & Mayer, R. E. (2016). E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning . Wiley.
Cook, A. E., Wei, W., & Preziosi, M. A. (2018). The use of ocular-motor measures in a convergent approach to studying cognitive load. In R. Z. Zheng (Ed.), Cognitive load measurement and application: A theoretical framework for meaningful research and practice (pp. 112–128). Routledge.
Dagienė, V., Jasutė, E., & Dolgopolovas, V. (2021). Professional development of in-service teachers: Use of eye tracking for language classes, case study. Sustainability , 13 (22), 12504. https://doi.org/10.3390/su132212504 .
De Koning, B. B., Tabbers, H. K., Rikers, R. M., & Paas, F. (2010). Attention guidance in learning from a complex animation: Seeing is understanding? Learning and Instruction , 20 (2), 111–122. https://doi.org/10.1016/j.learninstruc.2009.02.010 .
Frenck-Mestre, C. (2005). Eye-movement recording as a tool for studying syntactic processing in a second language: A review of methodologies and experimental findings. Second Language Research , 21 (2), 175–198. https://doi.org/10.1191/0267658305sr257oa .
Gass, S., Winke, P., Isbell, D. R., & Ahn, J. (2019). How captions help people learn languages: A working-memory, eye-tracking study. Language Learning & Technology , 23 (2), 84–104.
Google Scholar
Godfroid, A., & Schmidtke, J. (2013). What do eye movements tell us about awareness? A triangulation of eye-movement data, verbal reports and vocabulary learning scores. In J. M. Bergsleithner, S. N. Frota, & J. K. Yoshioka (Eds.), Noticing and second language acquisition: Studies in honor of Richard Schmidt (pp. 183–205). University of Hawai‛i, National Foreign Language Resource Center.
Godfroid, A., Boers, F., & Housen, A. (2013). An eye for words: Gauging the role of attention in incidental L2 vocabulary acquisition by means of eye-tracking. Studies in Second Language Acquisition , 35 (3), 483–517. https://doi.org/10.1017/S0272263113000119 .
Godfroid, A., Loewen, S., Jung, S., Park, J. H., Gass, S., & Ellis, R. (2015). Timed and untimed grammaticality judgments measure distinct types of knowledge: Evidence from eye-movement patterns. Studies in Second Language Acquisition , 37 (2), 269–297. https://doi.org/10.1017/S0272263114000850 .
Godfroid, A., Ahn, J., Choi, I. N. A., Ballard, L., Cui, Y., Johnston, S., & Yoon, H. J. (2018). Incidental vocabulary learning in a natural reading context: An eye-tracking study. Bilingualism: Language and Cognition , 21 (3), 563–584. https://doi.org/10.1191/0267658305sr257oa .
Hegarty, M. (2010). Components of spatial intelligence. In Psychology of learning and motivation (Vol. 52, pp. 265–297). Academic Press. https://doi.org/10.1016/S00797421(10)52007-3 .
Hou, J., Lee, J. F., & Doherty, S. (2022). A study of the effects of mobile media on L2 text processing: Beyond offline comprehension accuracy measures. Computers & Education , 182 , 104466. https://doi.org/10.1016/j.compedu.2022.104466 .
Indrarathne, B., & Kormos, J. (2017). Attentional processing of input in explicit and implicit conditions: An eye-tracking study. Studies in Second Language Acquisition , 39 (3), 401–430. https://doi.org/10.1017/S027226311600019X .
Jarodzka, H., & Brand-Gruwel, S. (2017). Tracking the reading eye: Towards a model of real‐world reading. Journal of Computer Assisted Learning , 33 (3), 193–201. https://doi.org/10.1111/jcal.12189 .
Kang, H. (2014). Establishing wiki design principles to advance wiki-based learning: An eye tracking study. International Journal of Social Media and Interactive Learning Environments , 2 (3), 258–270. https://doi.org/10.1504/IJSMILE.2014.064206 .
Kang, H., Kweon, S. O., & Choi, S. (2022). Using eye-tracking to examine the role of first and second language glosses. Language Teaching Research , 26 (6), 1252–1273. https://doi.org/10.1177/1362168820928567 .
Kao, G. Y. M., Chiang, X. Z., & Foulsham, T. (2019). Reading behavior and the effect of embedded selfies in role-playing picture e-books: An eye-tracking investigation. Computers & Education , 136 , 99–112. https://doi.org/10.1016/j.compedu.2019.03.010 .
Knoblich, G., Ohlsson, S., & Raney, G. E. (2001). An eye movement study of insight problem solving. Memory & Cognition , 29 (7), 1000–1009. https://doi.org/10.3758/BF03195762 .
Kobylyanskaya, S. (2022). Speech and Eye Tracking features for L2 Acquisition: A Multimodal Experiment. In M. M. Rodrigo, N. Matsuda, A. I. Cristea, & V. Dimitrova (Eds.), Artificial Intelligence in Education. Posters and late breaking results, Workshops and tutorials, Industry and Innovation Tracks, practitioners’ and Doctoral Consortium. AIED 2022 (Vol. 13356). Springer. Lecture Notes in Computer Science https://doi.org/10.1007/978-3-031-11647-6_8 .
Koć-Januchta, M. M., Höffler, T. N., Prechtl, H., & Leutner, D. (2020). Is too much help an obstacle? Effects of interactivity and cognitive style on learning with dynamic versus non-dynamic visualizations with narrative explanations. Educational Technology Research and Development , 68 , 2971–2990. https://doi.org/10.1007/s11423-020-09822-0 .
Lai, M. L., Tsai, M. J., Yang, F. Y., Hsu, C. Y., Liu, T. C., Lee, S. W. Y., & Tsai, C. C. (2013). A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational Research Review , 10 , 90–115. https://doi.org/10.1016/j.edurev.2013.10.001 .
Lawson, A. P., & Mayer, R. E. (2022). The power of voice to convey emotion in multimedia instructional messages. International Journal of Artificial Intelligence in Education , 32 (4), 971–990. https://doi.org/10.1007/s40593-021-00282-y .
Liberman, L., & Dubovi, I. (2023). The effect of the modality principle to support learning with virtual reality: An eye-tracking and electrodermal activity study. Journal of Computer Assisted Learning , 39 (2), 547–557. https://doi.org/10.1111/jcal.12763 .
Liou, H. C. (2012). The roles of Second Life in a college computer-assisted language learning (CALL) course in Taiwan, ROC. Computer Assisted Language Learning , 25 (4), 365–382. https://doi.org/10.1080/09588221.2011.597766 .
Liu, P. L. (2014). Using eye tracking to understand learners’ reading process through the concept-mapping learning strategy. Computers & Education , 78 , 237–249. https://doi.org/10.1016/j.compedu.2014.05.011 .
Liu, T. S. W., Liu, Y. T., & Chen, C. Y. D. (2019). Meaningfulness is in the eye of the reader: Eye-tracking insights of L2 learners reading e-books and their pedagogical implications. Interactive Learning Environments , 27 (2), 181–199. https://doi.org/10.1080/10494820.2018.1451901 .
Mayer, R. E. (2010). Unique contributions of eye-tracking research to the study of learning with graphics. Learning and Instruction , 20 (2), 167–171. https://doi.org/10.1016/j.learninstruc.2009.02.012 .
Meißner, M., & Oll, J. (2019). The promise of eye-tracking methodology in organizational research: A taxonomy, review, and future avenues. Organizational Research Methods , 22 (2), 590–617. https://doi.org/10.1177/1094428117744882 .
Mohamed, A. A. (2018). Exposure frequency in L2 reading: An eye-movement perspective of incidental vocabulary learning. Studies in Second Language Acquisition , 40 (2), 269–293. https://doi.org/10.1017/S0272263117000092 .
Mohsen, M. A. (2016). The use of help options in multimedia listening environments to aid language learning: A review. British Journal of Educational Technology , 47 (6), 1232–1242. https://doi.org/10.1111/bjet.12305 .
Ozcelik, E., Arslan-Ari, I., & Cagiltay, K. (2010). Why does signaling enhance multimedia learning? Evidence from eye movements. Computers in Human Behavior , 26 (1), 110–117. https://doi.org/10.1016/j.chb.2009.09.001 .
Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist , 38 (1), 1–4. https://doi.org/10.1207/S15326985EP3801_1 .
Pellicer-Sánchez, A. (2016). Incidental L2 vocabulary acquisition from and while reading: An eye-tracking study. Studies in Second Language Acquisition , 38 (1), 97–130. https://doi.org/10.1017/S0272263115000224 .
Ponce, H. R., Mayer, R. E., Loyola, M. S., López, M. J., & Méndez, E. E. (2018). When two computer-supported learning strategies are better than one: An eye-tracking study. Computers & Education , 125 , 376–388. https://doi.org/10.1016/j.compedu.2018.06.024 .
Qu-Lee, J., & Balcetis, E. (2022). The call for ecological validity is right but missing perceptual idiosyncrasies is wrong. Behavioral and Brain Sciences , 45 (1), 1–15. https://doi.org/10.1017/S0140525X21000807 .
Russell, M. C. (2005). Hotspots and hyperlinks: Using eye-tracking to supplement usability testing. Usability News , 7 (2), 1–11.
Salmerón, L., Naumann, J., García, V., & Fajardo, I. (2017). Scanning and deep processing of information in hypertext: An eye tracking and cued retrospective think-aloud study. Journal of Computer Assisted Learning , 33 (3), 222–233. https://doi.org/10.1111/jcal.12152 .
Schugar, H. R., Smith, C. A., & Schugar, J. T. (2013). Teaching with interactive picture e-books in grades K–6. The Reading Teacher , 66 (8), 615–624. https://doi.org/10.1002/trtr.1168 .
Shadiev, R., & Li, D. (2023). A review study on eye-tracking technology usage in immersive virtual reality learning environments. Computers & Education , 196 , 104681. https://doi.org/10.1016/j.compedu.2022.104681 .
Siyanova-Chanturia, A., Conklin, K., & Schmitt, N. (2011). Adding more fuel to the fire: An eye-tracking study of idiom processing by native and non-native speakers. Second Language Research , 27 (2), 251–272. https://doi.org/10.1177/0267658310382068 .
Spit, S., Andringa, S., Rispens, J., & Aboh, E. O. (2022). The Effect of Explicit instruction on implicit and explicit linguistic knowledge in Kindergartners. Language Learning and Development , 18 (2), 201–228. https://doi.org/10.1080/15475441.2021.1941968 .
Tai, T. Y., & Chen, H. H. J. (2021). The impact of immersive virtual reality on EFL learners’ listening comprehension. Journal of Educational Computing Research , 59 (7), 1272–1293. https://doi.org/10.1177/0735633121994291 .
Takacs, Z. K., Swart, E. K., & Bus, A. G. (2015). Benefits and pitfalls of multimedia and interactive features in technology-enhanced storybooks: A meta-analysis. Review of Educational Research , 85 (4), 698–739. https://doi.org/10.3102/0034654314566989 .
Wang, J., Antonenko, P., & Dawson, K. (2020). Does visual attention to the instructor in online video affect learning and learner perceptions? An eye-tracking analysis. Computers & Education , 146 , 103779. https://doi.org/10.1016/j.compedu.2019.103779 .
Winke, P., Gass, S., & Sydorenko, T. (2013). Factors influencing the use of captions by foreign language learners: An eye-tracking study. The Modern Language Journal , 97 (1), 254–275. https://doi.org/10.1111/j.1540-4781.2013.01432.x .
Yaros, R. A., & Cook, A. E. (2011). Attention versus learning of online content: Preliminary findings from an eye-tracking study. International Journal of Cyber Behavior Psychology and Learning (IJCBPL) , 1 (4), 49–69. https://doi.org/10.4018/ijcbpl.2011100104 .
Download references
The authors declares that they have no relevant or material financial interests that relate to the research described in this paper.
Authors and affiliations.
School of Foreign Languages, Xinyang Normal University, Xinyang, Henan, 464000, China
Xiaojing Li
You can also search for this author in PubMed Google Scholar
Xiaojing Li is the sole author of the work.
Correspondence to Xiaojing Li .
Publisher’s note.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Reprints and permissions
Li, X. Eye-tracking research in interactive language learning environments: A systematic review. Educ Inf Technol (2024). https://doi.org/10.1007/s10639-024-12648-5
Download citation
Received : 24 August 2023
Accepted : 21 March 2024
Published : 18 April 2024
DOI : https://doi.org/10.1007/s10639-024-12648-5
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
Perform simple searches, like the ones you use in Google. A single term for a condition and another for an intervention may suffice.
For instance,
There is no real need of using a taxonomy, or boolean terms, such as AND or OR (they work, but are rarely needed. If you are really fond of this kind of search strategies, you can use our advanced search).
You can search in your own language or in English. You can even combine terms from different languages.
Export Citation
Create your own Matrix Beta
Broad syntheses, systematic reviews, primary studies, external links:, available languages for this document.
IMAGES
COMMENTS
This study provides a current systematic review of eye tracking research in the domain of multimedia learning. The particular aim of the review is to explore how cognitive processes in multimedia learning are studied with relevant variables through eye tracking technology. To this end, 52 articles, including 58 studies, were analyzed.
In their systematic review of eye-tracking research in multimedia learning, [14] found that most studies have been conducted with university students, providing little empirical evidence for ...
A review of eye tracking research on video-based learning. This review sought to uncover how the utilisation of eye tracking technology has advanced understandings of the mechanisms underlying effective video-based learning and what type of caution should be exercised when interpreting the findings of these studies.
The most challenging task in eye-tracking-based multimedia research is to establish a relationship between eye-tracking metrics (or cognitive processes) and learners' performance scores. Additionally, there are current debates about the effectiveness of animations (or simulations) in promoting learning in multimedia settings.
The particular aim of the review is to explore how cognitive processes in mult... Highlights •Multimedia learning research with eye tracking technology is on the rise.•Eye movements of college students using science materials were mainly analyzed.•Eye movements were associated w...
Objectives: As a result, the current study aimed to review eye tracking-based. research on learners' cognitive processes in the animated/simulated multimedia. learning domain. Method: For this ...
Eye tracking technology is increasingly used to understand individuals' non-conscious, moment-to-moment processes during video-based learning. This review evaluated 44 eye tracking studies on video-based learning conducted between 2010 and 2021. Specifically, the review sought to uncover how the utilisation of eye tracking technology has advanced understandings of the mechanisms underlying ...
A systematic review of eye tracking research on multimedia learning, Computers & Education (2018), doi: 10.1016/j.compedu.2018.06.023. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting ...
A systematic review of eye tracking research on multimedia learning; A systematic review of eye tracking research on multimedia learning. Ecenaz Alemdag. Ecenaz Alemdag; Kursat Cagiltay. Kursat Cagiltay; Top Cited Papers. Publisher Website . Google Scholar . Cite Download Share Download. 1 October 2018;
This study provides a current systematic review of eye tracking research in the domain of multimedia learning. The particular aim of the review is to explore how cognitive processes in multimedia learning are studied with relevant variables through eye tracking technology. To this end, 52 articles, including 58 studies, were analyzed. Remarkable results are that (1) there is a burgeoning ...
Alemdag and Cagiltay (2018) conducted a systematic review of eye-tracking research on multimedia learning and found that while this research method was on the rise it was mainly used to understand the effects of multimedia use among higher education students. They also identified that although eye movements were linked to how students select ...
Even though recent eye tracking studies used the analysis of scanpaths or gaze patterns to investigate (meta)cognitive processes (e.g., Bühler et al., 2024; Stark et al., 2024; Tjon et al., 2023), the use of simple eye tracking indicators is widespread in research on multimedia learning as they are easy to calculate.
this gap by presenting a systematic review of eye-tracking research in interactive language learning environments. It aims to critically analyze existing literature, pose ... ries of multimedia learning and theories specific to language acquisition. Theo-retical frameworks were critically analyzed to understand how they influenced
Alemdag and Cagiltay (2018) conducted a systematic review of eye-tracking for the research of multimedia learning. This work reported the deploy-ment of temporal and count scales of eye-tracking in studying the selection, organization, and integration of multimedia information. The authors advocated more research on using
2.1. Eye tracking as a method of measuring learning in multimedia environments. Eye tracking technology is a non-invasive technique that facilitates the recording and measurement of certain cognitive processes, as well as the inference of metacognitive processes that occur during the learning process (Asish et al., Citation 2022; Tong & Nie, Citation 2022; van Marlen et al., Citation 2022).
The present chapter summarizes the state of the art of using eye tracking in research on multimedia learning. It first provides an overview the various eye tracking parameters that have been used ...
A Systematic Review of Eye Tracking Research on Multimedia Learning Abstract This study provides a current systematic review of eye tracking research in the domain of multimedia learning. The particular aim of the review is to explore how cognitive processes in multimedia learning are studied with relevant variables through eye tracking technology.
Background: Eye-tracking technology is an established research tool within allied industries such as advertising, psychology and aerospace. This review aims to consolidate literature describing the evidence for use of eye-tracking as an adjunct to traditional teaching methods in medical education. Methods: A systematic literature review was ...
These findings suggest that this eye tracking-based deep learning model can directly and rapidly identify impaired executive function during visuospatial memory encoding, with potential ...
This comprehensive systematic review synthesizes thirty-four peer-reviewed articles published between 2010 and 2022, utilizing eye-tracking research within interactive language learning environments. Following the PRISMA scheme for article selection, this review illuminates both the affordances and challenges of eye-tracking technology in enhancing language learning outcomes. Through a ...
"Eye-tracking research provides valuable data to improve the learning process, making it more personalized, effective and engaging," says the KTU Ph.D. student, a co-author of the study.
This study provides a current systematic review of eye tracking research in the domain of multimedia learning. The particular aim of the review is to explore how cognitive processes in multimedia learning are studied with relevant variables through eye tracking technology. To this end, 52 articles, including 58 studies, were analyzed.
Eye tracking technology on children's mathematical education: systematic review. Frontiers in Education , 2024; 9 DOI: 10.3389/feduc.2024.1386487 Cite This Page :