Imperial College London
Portrait Picture

ProfessorPeterPietzuch

Faculty of Engineering, Department of Computing

Professor of Distributed Systems
 
 
 
//

Contact

 

+44 (0)20 7594 8314prp Website

 
 
//

Location

 

442Huxley BuildingSouth Kensington Campus

//

Summary

 

Publications

Citation

BibTex format

@article{Rupprecht:2017,
author = {Rupprecht, L and Culhane, WJ and Pietzuch, P},
journal = {Proceedings of the VLDB Endowment},
pages = {1250--1261},
title = {SquirrelJoin: network-aware distributed join processing with lazy partitioning},
url = {http://hdl.handle.net/10044/1/49766},
volume = {10},
year = {2017}
}
Download

RIS format (EndNote, RefMan)

TY  - JOUR
AB  - To execute distributed joins in parallel on compute clusters, systemspartition and exchange data records between workers. With largedatasets, workers spend a considerable amount of time transferringdata over the network. When compute clusters are shared amongmultiple applications, workers must compete for network bandwidthwith other applications. These variances in the available networkbandwidth lead tonetwork skew, which causes straggling workersto prolong the join completion time.We describeSquirrelJoin, a distributed join processing techniquethat useslazy partitioningto adapt to transient network skew inclusters. Workers maintain in-memorylazy partitionsto withhold asubset of records, i.e. not sending them immediately to other work-ers for processing. Lazy partitions are then assigned dynamicallyto other workers based on network conditions: each worker takesperiodic throughput measurements to estimate its completion time,and lazy partitions are allocated as to minimise the join completiontime. We implement SquirrelJoin as part of the Apache Flink dis-tributed dataflow framework and show that, under transient networkcontention in a shared compute cluster, SquirrelJoin speeds up joincompletion times by up to 2.9× with only a small, fixed overhead.
AU  - Rupprecht,L
AU  - Culhane,WJ
AU  - Pietzuch,P
EP  - 1261
PY  - 2017///
SN  - 2150-8097
SP  - 1250
TI  - SquirrelJoin: network-aware distributed join processing with lazy partitioning
T2  - Proceedings of the VLDB Endowment
UR  - http://hdl.handle.net/10044/1/49766
VL  - 10
ER  -
Download