Friday, August 18, 2017
Search
  
Submit your own News for
inclusion in our Site.
Click here...
Breaking News
Google Home Now Supports Free Calls
Asus Unveils the ZenFone 4 Pro, ZenFone 4, ZenFone 4 Selfie Pro, and ZenFone 4 Selfie
Nokia 8 Shipped With ZEISS Optics
Apple is Getting Serious in TV Shows and Film Prospect
Acer's New 4K Projectors Bring the Benefits of Cinema Home
Fiat Chrysler Joins BMW, Intel, Mobileye in Autonomous Driving Team
Kingston Adds Lower 4GB and 8GB Capacities to DataTraveler 2000 Encrypted USB
Intel Ice Lake Architecture Will Find its Way to 10nm+ Chips
Active Discussions
Which of these DVD media are the best, most durable?
How to back up a PS2 DL game
Copy a protected DVD?
roxio issues with xp pro
Help make DVDInfoPro better with dvdinfomantis!!!
menu making
Optiarc AD-7260S review
cdrw trouble
 Home > News > General Computing > Twitter...
Last 7 Days News : SU MO TU WE TH FR SA All News

Friday, October 08, 2010
Twitter's New Search Architecture


Twitter launched a new backend for search on twitter.com during the last few weeks.

Twitter?s real-time search engine was, until very recently, based on the technology that Summize originally developed. However, scaling the old MySQL-based system had become increasingly challenging. About 6 months ago, Twitter decided to develop a new search architecture that is based on a highly efficient inverted index instead of a relational database. Twitter chose Lucene, a search engine library written in Java, as a starting point.

Twitter's demands on the new system are immense: With over 1,000 TPS (Tweets/sec) and 12,000 QPS (queries/sec) = over 1 billion queries per day (!) Twitter already put a very high load on our machines. In addition to these scalability requirements, Twitter also need to support extremely low indexing latencies (the time it takes between when a Tweet is tweeted and when it becomes searchable) of less than 10 seconds. Since the indexer is only one part of the pipeline a Tweet has to make it through, Twitter needed the indexer itself to have a sub-second latency.

However, Lucene has several shortcomings for real-time search. That?s why Twitter rewrote big parts of the core in-memory data structures, especially the posting lists, while still supporting Lucene?s standard APIs. This allows Twitter to use Lucene?s search layer almost unmodified. Some of the highlights of the changes include:

* significantly improved garbage collection performance
* lock-free data structures and algorithms
* posting lists, that are traversable in reverse order
* efficient early query termination

Twitter estimates that it is only using about 5% of the available backend resources. Twitter's new indexer could also index roughly 50 times more Tweets per second than Twitter currently gets.

The first difference users might notice is the bigger index, which is now twice as long -- without making searches any slower. And, maybe most importantly, the new system is versatile and extensible, which will allow Twitter to build new features faster and better.


Previous
Next
Firefox 4 Beta for Android and Maemo is Now Available        All News        Opera To Participate In in Open Screen Project
Google Testing Self-driving Cars     General Computing News      Opera To Participate In in Open Screen Project

Get RSS feed Easy Print E-Mail this Message

Related News
Twitter Struggles to Attract New Users
Twitter Lets you Mute Notifications From Unknown Accounts
Facebook, Microsoft, Twitter and YouTube Team Up To Counter Terrorism
Twitter Makes Changes to its Website, Mobile Apps
Twitter's Privacy Update Means You Are Officially Tracked
Twitter's Revenue Drops Despite User Growth
U.S. Government Withdraws Demand to Unmask Anti-Trump Account on Twitter
Twitter Releases 'Lite' Version For Emerging Markets
The European Commission Ask Social Media Companies To Comply With EU Consumer Rules
Twitter To Use Algorithms to Tackle Abusive Content
Twitter Pace of Growth Slows
Twitter To Filter Abusive Tweets

Most Popular News
 
Home | News | All News | Reviews | Articles | Guides | Download | Expert Area | Forum | Site Info
Site best viewed at 1024x768+ - CDRINFO.COM 1998-2017 - All rights reserved -
Privacy policy - Contact Us .