Large language models (LLMs) are increasingly used to evaluate text quality, acting as automated judges in various applications. But how well do these AI judges perform when faced with the nuances of multiple languages? New research introduces MM-Eval, a benchmark designed to test the multilingual meta-evaluation capabilities of LLMs. Essentially, it assesses how well AI can judge other AIs across a diverse range of languages and tasks. The benchmark covers 18 languages and six categories, including chat, reasoning, safety, and even detecting language hallucinations. The results reveal a mixed bag. While some LLMs perform admirably, especially on tasks like detecting hallucinations, others struggle, particularly with safety and linguistic nuances in lower-resource languages. This uneven performance highlights a crucial challenge: LLMs tend to assign middle-ground scores in these lower-resource languages, blurring the lines between good and bad responses. This tendency suggests that current LLMs aren't equipped to handle the complexities of truly multilingual evaluation. They may overvalue poor responses and undervalue high-quality ones, especially in languages with less training data. This research underscores the need for more robust multilingual training and evaluation methods. As AI expands its reach across the globe, its ability to understand and judge quality across diverse languages is paramount, ensuring fairness and accuracy in applications from automated feedback systems to content moderation.