FAV researchers are testing artificial intelligence based on college-level mathematics and logic

A team from the Faculty of Applied Sciences at the University of West Bohemia tested current large language models on 18 complex math problems. The largest models performed best, but their reasoning was not always correct, and results varied across mathematical fields.

What are the limits of current artificial intelligence in an academic setting? Students from the Faculty of Applied Sciences at the University of West Bohemia in Pilsen (FAV) sought to answer this question. They developed a system that automatically feeds examples to large language models, records the results along with the solution processes, and thus enables the ongoing evaluation of their success. A team composed of students and academics tested four different language models on eighteen tasks. Preliminary data from the first phase of testing already show clear trends.

Of the more than 120 responses evaluated, the largest models achieved a 98 percent success rate, while smaller variants achieved roughly 68 percent. According to Kristian Tichota, the leader of the student portion of the team, the difference is partly due to technical limitations: smaller models lacked access to external tools or encountered text length limits.

Furthermore, the tests confirmed that a correct result does not automatically imply a flawless process. “Only 50 to 75 percent of the responses from the large models had completely correct logic, and at most half of all tested outputs offered a truly effective, straightforward solution,” said Tichota. He also pointed out differences in solution success rates across various mathematical disciplines. According to project mentor Petr Girg from the Department of Mathematics at the Faculty of Applied Sciences (FAV), current large language models are already reaching the level of teachers in some of these disciplines.

The development and operation of the testing platform brought together students from the departments of computer science, cybernetics, and mathematics. Academics are responsible for the expert evaluation of the results, including verifying procedures. According to them, this interdisciplinary collaboration clearly demonstrated how effectively software engineering can be combined with mathematics. “The resulting system continuously provides us instructors with accurate data on the current limits of artificial intelligence in an academic setting and allows us to track the performance of large language models over time. It also offers students valuable practical experience,” said Petr Girg. The team has already presented its findings at student conferences in the Czech Republic and abroad.

Gallery

Faculty of Applied Sciences	Petr Girg, Martina Batková
29. 07. 2026

The pilot phase of the project was made possible thanks to university funding for small internal projects focused on the adoption and use of artificial intelligence at UWB. As part of this initiative, the pilot phase was supported in 2025 under the title A New Era of Mathematics: How LLMs Will Affect Teaching and Research.