« Svante Pettersson journalistar om översättningar och John Eje Thelin | Main | Game Theory for Swingers: What states should the candidates visit before Election Day? »
oktober 24, 2004
Föredrag: Google under huven
Googles Jeff Dean har ett intressant föredrag om tekniken bakom Google: Google: A Behind-the-scenes Look (WMV, cirka 1 timme). Läs en blänkare om föredraget här. Det är alltså mest teknik, både hård- och mjukvara.
Efter cirka 30 minuter börjar det avsnitt som jag tyckte var mest intressant: "Fun with Data" där det bl.a. visas en demo av ett system för automatisk ordklustring och som förhoppningsvis kommer att föras in i systemet för att få ännu bättre sökresultat.
Cirka 25 minuter in i föredraget, beskrivs MapReduce-modellen, som beskrivits mer detaljerat i papret Jeffrey Dean and Sanjay Ghemawat: MapReduce: Simplified Data Processing on Large Clusters
MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.
Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.
Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers and the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google's clusters every day.
Via Google Blogoscoped.
Posted by hakank at oktober 24, 2004 07:42 EM Posted to Sökmotorer