Twitter uses human "judges" to evaluate the importance of trending tweets in order to be correctly categorized for accurate searching and ad serving.

Twitter opens a window to the world in real-time. An event happens, and seconds later, people share it across the planet. When an important event happens, people instantly come to Twitter, and in particular search on Twitter to discover what was happening.

From a search and advertising perspective, however, these sudden events pose several challenges for Twitter. The queries people perform have probably never before been seen, so it's impossible to know without very specific context what they mean. In addition, these "spikes" in search queries are so short-lived, there's only a small window of opportunity to learn what they mean. So Twitter need to teach its systems what these queries mean quickly because in just a few hours, the search spike will be gone.

To cope with these issues, Twitter has built a real-time "human computation engine" to help the company identify search queries as soon as they're trending, send these queries to real humans to be judged, and then incorporate the human annotations into the company's back-end models.

In short, first Twitter monitors for which search queries are currently popular. Behind the scenes, Twitter runs a Storm topology that tracks statistics on search queries. As soon as Twitter discovers a new popular search query, it is sent to its human evaluators, who are asked a variety of questions about the query.

For example, as soon as Twitter notices a tweet spiking, it asks judges to categorize the query, or provide other information (e.g., whether there are likely to be interesting pictures of the query, or whether the query is about a person or an event) that helps Twitter serve relevant Tweets and ads.

After the human response is received, the information is then fed in to the back end systems to improve the context of the search response the next time the term is searched for.

Tasks are sent to Amazon?s Mechanical Turk, a vast available crowdsourced workforce on hand around the world to help with queries, categorising them and putting them in context.

Using crowdsourced workers enables Twitter to select 'highly trusted' judges from 'the best of Mechanical Turk' across all of the languages used by Twitter. Twitter says that it is easier to scale the workforce when required.

Using humans to categorise search queries correctly ensures that it delivers the most relevant ads to users. Having human intervention ensures that searching will get you the results you want.