Evaluation of Quality of Slovak Language Use in LLMS

This study explores the rapid advancement of Large Language Models (LLMs) and their growing support for languages beyond English. Despite this progress, the LLM ecosystem remains predominantly focused on English, particularly in key areas such as tokenizers and evaluation metrics. Traditional metrics like BLEU and ROUGE, designed for English, are less effective for languages with different linguistic structures, such as Slovak. This paper highlights the limitations of current metrics and the need for language-specific evaluation methods. The performance of three LLMs—Mistral 7B, Mistral 7B fine-tuned on Slovak, and Gemma 7B—was assessed using both human evaluators and automated evaluations by ChatGPT-4. Results show that fine-tuning on Slovak improves model performance, though errors in grammar and syntax persist. The study underscores the need for multilingual optimization and more inclusive AI evaluation tools to ensure effective language model performance across diverse linguistic contexts.