Large scale web-services such as Twitter need thousands of servers to manage the flow of updates that are sent out by their millions of users. As a result, keeping all these servers updated with the latest data can be time and resource intensive.
To improve the deployment of files across their servers, Twitter is calling in the help of BitTorrent. With BitTorrent, Twitter is planning to distribute files faster and more efficiently, saving time and precious resources and improving the scalability of Twitter’s operation.
Twitter’s new project, codenamed ‘Murder’, will not use the bandwidth of Twitter users. Instead, it will transform the site’s servers into a large BitTorrent swarm that will distribute file updates using BitTorrent technology.
The setup is pretty straightforward. Murder uses a ’seeder’ server where the new files will be distributed to thousands of ‘peer’ servers.
Because all servers assist in the deployment of the files, it will only take a fraction of the time it would otherwise take when files are distributed from a central server. This server-to-server BitTorrent technology also explains the name ‘Murder’ which is used to describe a flock of crows.
The Murder project is developed by Twitter’s Larry Gadea and built based on the BitTornado BitTorrent client, which was the first client to implement web-seeding. The code is open to the public and licensed under the free software Apache License.
Eric Klinker, CEO of BitTorrent Inc. is excited about Twitter’s adoption of BitTorrent. “We are thrilled to collaborate with them on this and hope that more Web Monsters out there look to this and other applications of BitTorrent in solving some of the hard problems of the Internet,” he said.
“The distributed nature of BitTorrent means an operation that once took many dozens of minutes, now happens in less than a dozen seconds. These efficiencies will reduce maintenance windows, site downtime and exposure to security vulnerabilities,” Klinker added. Although Klinker speaks about “collaboration” with Twitter it is not clear what role BitTorrent Inc. will play in the Murder project.
At this point it is still unclear when Murder will be implemented or released to the public but Twitter is expected to release more details on the upcoming project in the weeks to come.
Google, Yahoo, Amazon, and Facebook are some of the internets biggest names. All of them have huge online presence with millions of hits every single day. Their names have become so well known that even elderly people who have never sat at a computer know what they are. Ever think about their electricity bills?
Me neither, until I came across a report by MIT which states that they are easily spending $30 million per annum on electricity. This is to keep their servers up and running. First they have to power the servers. Then comes the technology that runs them and all the cooling and networking systems in between. A tall task with an even taller bill.
This report suggests a way in which these companies could save on their electricity bills.
It doesn’t offer a green solution of cutting back, rather it suggests that they constantly change which servers power their machines on a daily, even hourly basis.
So for example, the cost of electricity could rise in the US for any given reason so they shift the workload to servers in Belgium where the electricity is cheaper on that particular day.
While this would be a costly investment to begin with, it should pay off. After all, the total energy usage by the internet’s servers is set to go up four times within the next decade. Time for the big lads to make a big decision I think.
Over the last few months the market of ad serving has really heated up. Firstly, it was the acquisition of DoubleClick by Google, which was really supposed to accelerate Google into the display advertising market. While at the same time Open X announced funding, basically so they could turn their ad server from self hosted to hosted and controlled by Open X. Then comes the very recent announcement that Google is launching Ad Manager. Finally, this week it was announced that Yahoo are entering this competitive market and are soon to launch Yahoo AMP!.
The overall fight seems to be about control and more importantly information. Google has always been a company that is hungry for knowledge and information, but it seems that everyone wants to know all about advertising, be it long tail or short. Each option has its own unique selling point, DoubleClick is for huge volume clients, Google’s Ad Manager is just simple to use (especially if they have Google Adsense as defaults), Open X has a great free service for self hosted companies and with the inclusion of a hosted option im sure they will see even more interest, then finally there is Yahoo and AMP!, this service has had amazing early reviews and im very keen to get my hands on it.
Thus, who will win this race? Well, it has always been hard to look away from Google, but I do think that Yahoo might be on to something here. Yahoo has a very strong display advertising business already, and if they make a Ad Manager that ties in hundreds of other ad networks and offers great transparency, then im sure they will be in the box seat.
Interesting news has come our of Technorati today: former boss David Sifry has announced changes that will help bloggers again, like in the old days of Technorati.
Changes include:
- Change of design and results on search pages.
- Authority filtering. You can now narrow your results based on authority, which really only goes so far when tryign to evaluate the usefulness a blog, or if it is a spam blog.
- Charts. You can now see the popularity of a word over time.
- New Server Farm. They recently moved from 365 Main to a new colocation center.
It is good news to hear that Technorati is back and trying to satisfy their core customer base that helped them get to where they are today. Recent redesigns have completely turned away several numbers of bloggers from the service and although these changes are going in the right direction, im not too sure if they will be able to get all the bloggers back. Time will tell! Technorati stick to your core customer base, cause Google owns the rest of the blog search market!
