Large-scale Incremental Processing Using Distributed Transactions and Notifications (Percolator)¶
Primary objective: prepare web pages for inclusion in the live web search index. Individual documents processed as they are crawled. System has also been used to render pages into images. Links are maintained between web pages and resources they depend on - pages can be reprocessed when any depended-upon resources change.
Extends Bigtable with multi-row, distributed transactions. Provides an observer interface to allow applications to be structured around notifications of changed data.
Imperative language (C++). - basicamente uma transacção, com commit() - falha, retry com backoff. Parallelism achieved by running many transactions simultaneously in a thread pool.
Notifications: By way of Observers. Triggered by changes to the table. Each observer registers a function and a set of columns with Percolator. Percolator invokes the function after data is written to one of those columns in any row.
Percolator application is structured as a series of observers. Observer completes a task, and writes to the table, thus enabling the next observer.
e.g. In current indexing system, a MapReduce operation loads crawled documents into Percolator. This triggers a document processor to index the document (parse, extract links, etc). This triggers a clustering transaction which, in turn, triggers transactions to export changed document clusters to the serving system.
Notifications similar to triggers/events. Very few observers (Google indexing has roughly 10). Observers can ‘observe’ the same column(s). No safeguard against ‘infinite notifications’.
Guarantee: at most one observer’s transaction will commit for each change of an observed column. Multiple writes to an observed column may cause an observer to be invoked only once -> no caso do percolator faz sentido -> e.g. causa um processamento periódico de http://www.google.com, em vez de a cada modificação feita - poupa no processamento.
Tem muitos locks, delay de locks, mensagens (cerca de 50 operações Bigtable para processar um único documento).
Load distribution: Basicamente, como mencionado antes, se algo tem muito peso, o processamento é adiado, e apenas se corre uma vez - no caso da solução para a Google, funciona como querem. É um problema no caso de se querer efectuar todas as operações pedidas. Além disso apenas corre sobre 1 datacenter (não tem replicação sobre múltiplos datacenters).
Final Observations: