Rashmi Khazanchi

Use of artificial intelligence in education (AIED) is growing rapidly and significantly influencing the teaching and learning processes. AI-based systems can provide immediate feedback and adapt instructions in real time, creating opportunities for interaction and engagement in the learning process that are often difficult to achieve in traditional classroom settings, where teachers cannot always offer individualized support to struggling learners. Several research studies have shown promising results when AI-based systems are used as a supplemental tool in K-12 classrooms.

AI-based systems such as ITS and ALS have the potential to deliver adaptive and personalized learning to students, particularly in STEM subjects like mathematics, which many students find challenging. This is more evident among disadvantaged students from rural areas who frequently fall behind grade-level expectations due to limited resources, a lack of highly qualified teachers, and the effects of poverty.

The learning loss caused by COVID -19 pandemic has significantly impacted students’ mathematics performance, further widening existing achievement gaps. To address this issue and narrow the achievement gap, several schools across the United States have implemented AI-based tools to support mathematics learning. A rural high school in the southern United States piloted ALEKS, an ITS, during the 2021-2022 academic year to support struggling eighth-grade students. In the 2022–2023 academic year, the school adopted Edmentum Exact Path, an ALS, to continue providing support.

This thesis consists of four chapters: Chapter 1 discusses a study conducted with ALEKS; Chapter 2 extends the first study for journal publication; Chapter 3 examines the use of Edmentum Exact Path; and Chapter 4 presents a systematic literature review on AI-based systems and their impact on mathematics achievement.

Chapter 1 focuses on the effect of intelligent tutoring systems on the mathematics achievement of underachieving students. This quasi-experimental study used ALEKS to evaluate its impact on 158 eighth-grade students, 62–68% of whom were economically disadvantaged, with only 12% proficient in mathematics. This study aimed to 1) Compare the effectiveness of ALEKS versus traditional instruction in improving students’ mathematics achievement. 2) Assess students’ progress on grade-level math standards over one academic year using ALEKS. 3) Analyze differences in achievement across class periods with ALEKS implementation.

We compared the results of pretest and posttest from teacher-led instructions and ALEKS -led instructions across two consecutive years. In the first year, only McGraw math curriculum Reveal was used. In the second year, ALEKS was implemented as a supplemental tool in a math support class for 50 minutes every other day, alongside the Reveal Math curriculum. We also analyzed five years of End-of-Grade (EOG) state assessment data (without ALEKS) and compared it to EOG data from the year ALEKS was implemented.

Findings showed that students receiving teacher-led instructions showed greater mathematics achievement than those using ALEKS-led instructions. This outcome may be attributed to ALEKS being in its pilot stage, with teachers still learning how to use it effectively. Many students were working on prerequisite skills as they were below grade level. The COVID-19 pandemic likely amplified this effect, as students were promoted to eighth grade without taking state or school exams after missing much of their seventh-grade instructions due to school closures.

This study also found that ALEKS contributed to improvement across all eleven mathematics standards within five math domains. However, most students were unable to complete all standards because ALEKS is mastery-based, requiring students to achieve 80% accuracy on prerequisite skills before progressing further. Limited access to ALEKS (only on alternate days) also restricted completion rates. Another finding indicated that high-achieving students with strong work ethics performed better compared to mixed-ability groups, which included students with disabilities.

Chapter 2 was extended for journal publication by incorporating statistical analysis, including paired t-tests and ANOVA, to evaluate the efficacy of ALEKS on students' mathematics achievement. A literature review and null hypothesis were also added. The results indicated that teacher-led instruction was more effective, showing higher test scores and lower variance compared to ALEKS. The study had two main objectives: 1) to examine whether the use of ALEKS show a statistically significant improvement in students' mathematics achievement compared to traditional teacher-led instruction, and 2) to determine whether the use of ALEKS show statistically significant improvement across grade-level mathematics standards over one academic year. The analysis found that both ALEKS-led and teacher-led instructions were statistically significant, with teacher-led instruction being more effective.

A similar pattern was observed when comparing five years of End of Grade (EOG) data with and without ALEKS. While the use of ALEKS significantly improved all mathematics standards, the gains varied, likely due to its mastery-based learning, which requires 80% mastery before progressing to the next topic. Since students only used ALEKS on alternate days, they were unable to complete all eleven standards.

Overall, the findings from this study provide valuable insights into the use of ITSs in K-12 classrooms. Mathematics ITSs like ALEKS offer adaptive and personalized learning opportunities and can significantly enhance achievement among underperforming students.

In Chapter 3, we examined the effectiveness of Edmentum Exact Path, an AI-based instructional system, in enhancing mathematics achievement and engagement (affective and cognitive) among 8th-grade students in the Southern United States. This quasi-experimental study included 78 students from socioeconomically disadvantaged backgrounds. We compared an experimental group that received both traditional teacher-led instruction and Edmentum Exact Path-led instructions to a control group that received only traditional teacher-led instruction.

The three objectives for this study were 1) To compare the efficacy of Edmentum Exact Path and traditional teacher-led instruction on students’ mathematics achievement. 2) To investigate differences in students’ affective engagement between Edmentum Exact Path instruction and traditional teacher-led instruction. 3) To examine differences in students’ cognitive engagement between Edmentum Exact Path instruction and traditional teacher-led instruction.

This intervention lasted five weeks, with a daily session of 50 minutes each. Both groups used the McGraw-Hill math curriculum Reveal, 8th-grade math curriculum and incorporated Edmentum Exact Path as a supplemental tool for the experimental group. The experimental group also attended math support classes, where they worked on individualized learning pathways in Edmentum Exact Path, created based on diagnostic assessments administered at the beginning of the school year. Mathematics achievement was measured using pretests and posttests, while student engagement was measured using a 35-item, 5-point Likert-scale Student Engagement Instrument (SEI), administered following the posttest to assess affective and cognitive engagement. A significant limitation of this study is the absence of a pre-intervention SEI survey, which restricts the ability to measure changes in engagement over time.

Data were analyzed using t-tests and ANOVA. The result showed that both the experimental and control groups showed statistically significant improvements in mathematics achievement. However, the control group showed greater gains in affective engagement, whereas no statistically significant differences were observed in cognitive engagement between the two groups.

These results suggest that integrating AI-based systems like Edmentum Exact Path may enhance mathematics achievement and cognitive engagement by addressing individual learning needs. However, such tools may be less effective in increasing affective engagement, possibly due to a lack of emotional responsiveness. Further research is needed to better understand the role of AI in promoting student engagement, particularly among the underserved population in rural areas.

In Chapter 4, we conducted a systematic literature review to investigate the impact of AI-based systems on mathematics achievement in K-12 classrooms. The review was guided by the following objectives: 1) To identify what types of AI-based systems are used in mathematics education, and the educational level at which they are implemented. 2) To identify the impact of AI-based systems on students' mathematics performance in K-12 classrooms. 3) To explore whether AI-based systems help reduce the mathematics achievement gap among students from low socioeconomic backgrounds, and which system features contribute to this effect.

We followed the PRISMA guidelines and searched six major databases: ACM Digital Library, ERIC (EBSCO), JSTOR, Wiley, ScienceDirect (Elsevier), and SpringerLink to locate peer-reviewed articles published between 2008 and 2023. An initial pool of 1,945 studies was identified based on predefined inclusion and exclusion criteria. After screening, 42 articles were selected for in-depth analysis. Data was organized and analyzed using spreadsheets.

The findings indicate that AI-based systems are widely used in K-12 classrooms across various countries to provide personalized and adaptive learning experiences to support students’ mathematics learning. Most studies were conducted in the United States. Both ITS and ALS are used at the elementary, middle, and high school levels, with more frequent implementation at the middle school level and less at the high school level. Several studies also reported the use of adaptive learning games, such as Lynnette, DragonBox, Woot Math Adaptive Learning (WMAL), and Math Whizz. The most commonly used AI-based systems in the U.S. include ALEKS, CTA1, ASSISTments, HALF, Math IVLE, MathSpring, and Decimal Point. Studies from other countries reported the use of AI-based systems such as MIT, dialogue-based tutors, ACALS, Adaptive CER-based mathematics games, PEDALE, PAT2Math, ZPDES, RiaRiT, AmritaITS, UZWEBMAT, APPEAL, and HINTS.

Overall, the findings suggest a moderately positive effect of AI-based systems on students’ mathematics achievement. Most studies reported moderate to significant improvements in student performance, engagement, and retention. Several AI-based systems, such as ALEKS, MathSpring, and eFit were associated with improved outcomes among students from low socioeconomic backgrounds, highlighting their potential to support educational equity.

The discussion presents the findings of each study of this doctoral thesis, along with the contributions and the limitations of this research. Grounded in the integration of AI-based instructional systems and personalized learning frameworks, the studies demonstrate how AI-based systems can enhance student learning outcomes in mathematics. These AI-based systems show potential in enhancing students’ cognitive engagement and academic performance. However, the findings also reveal limitations in promoting affective engagement, highlighting the challenges AI-based systems face in replicating the emotional connection of human teaching. The study shows limitations in affective engagement, showing the challenges of AI-based systems to replicate the emotional and relationale aspects of human instruction. This dissertation adds to the broader discourse within the AIED communities, providing empirical evidence on the pedagogical impact of AI-based instructions on students mathematics achievement and engagement in underserved population within rural educational contexts.

Future research should investigate the impact of various AI-based systems by comparing their effectiveness with ITS and ALS, such as ALEKS, and Edmentum Exact Path. It should consider factors like instructional design, student characteristics, and specific learning outcomes. By leveraging the strengths of these AI-based systems, educators and policymakers make informed decisions regarding their integration, ultimately enhancing student achievement in mathematics and related disciplines.