Large language models (LLMs) are increasingly used as automated evaluators in various fields, but how reliable are their judgments? New research reveals some surprising biases in how LLMs assess content, raising questions about their fairness and consistency. One key finding is "familiarity bias": LLMs tend to favor text they're already familiar with, similar to how humans sometimes prefer the known over the unknown. This bias is evident in how LLMs assign higher scores to summaries with lower perplexity, meaning text that is more predictable or common. The research also uncovered scoring biases, where LLMs overuse certain scores, like round numbers, and underuse others. This preference for specific numerical values further highlights the quirks in their evaluation process. Another intriguing finding is the "anchoring effect." When LLMs evaluate multiple aspects of a summary, their judgment on one aspect can be unduly influenced by their assessment of a previous aspect. This suggests that LLMs don't always approach evaluation with a fresh perspective, potentially leading to skewed results. The study also found inconsistencies in LLM evaluations. Their judgments can fluctuate significantly depending on minor prompt changes or even random sampling, indicating a lack of stability in their decision-making. These findings have real-world implications for using LLMs in tasks like grading student essays, evaluating job applications, or assessing creative writing. If LLMs are susceptible to biases and inconsistencies, their use in such sensitive applications could perpetuate unfairness. To address these limitations, the researchers suggest several strategies, including widening the scoring range, avoiding certain prompting techniques, and evaluating only one attribute at a time. These practical recommendations offer a path towards more robust and impartial LLM evaluations. While LLMs hold immense potential as automated evaluators, this research underscores the importance of understanding and mitigating their biases to ensure fair and consistent judgments.