Crowdsourcing is a powerful tool, but ensuring quality can be a challenge. Imagine ten people translating the same Japanese sentence into English—you'll likely get ten different versions, some better than others. How do you distill the most accurate translation from this jumble? Traditionally, methods like majority voting or picking the answer most similar to others have been used. But what if we could combine human intelligence with the power of AI? New research explores exactly this, introducing a 'human-LLM hybrid' approach where large language models (LLMs) act as 'aggregators,' sifting through crowdsourced text answers like translations to identify the best option. The study used real-world translation datasets and found that LLMs like GPT-4 excel at spotting high-quality answers within a crowd, often outperforming standard aggregation methods. However, while LLMs are good, they aren’t perfect. The most effective approach? Combining human and LLM aggregators. This hybrid system leverages both human intuition and AI's analytical capabilities to create a more accurate and robust aggregation process. Think of it as a team effort, where humans and AI work together to refine the 'wisdom of the crowd.' This research has implications beyond translation, opening doors for improving quality control in any crowdsourced text-based task, from surveys and questionnaires to content moderation and data labeling. While the study primarily focused on GPT-4 and Gemini, future research will explore the potential of other LLMs, potentially leading to even more effective human-AI partnerships in the quest for higher-quality crowdsourced data.