Top Banner

Click here to load reader

Apache Hadoop: The Scalability Update Building a Cloud ... · PDF file Apache Hadoop: The Scalability ... Hadoop cluster, at Yahoo!, and is now working to build Hadoop clusters at

Aug 25, 2020

ReportDownload

Documents

others

  • t h e a d v a n c e d c o m p u t i n g s y s t e m s a s s o c i a t i o n

    j u n e 2 0 1 1 v o l . 3 6 , n o . 3

    Apache Hadoop: The Scalability Update K o n s t a n t i n v . s h v a c h K o

    Building a Cloud File System j e f f d a r c y

    Testing the Transport Side of Highly Available Hosts s t u a r t K e n d r i c K

    Conference Reports from FAST ’11, NSDI ’11, and more

    t h e a d v a n c e d c o m p u t i n g s y s t e m s a s s o c i a t i o n

    usenix_login_june11_covers.indd 1 5.16.11 11:51 AM

  • 20th USENIX Security Symposium (USENIX Security ’11) August 10–12, 2011, San Francisco, CA, USA http://www.usenix.org/sec11

    Workshops co-located with USENIX Security ’11 include:

    2011 Electronic Voting Technology Workshop/ Workshop on Trustworthy Elections (EVT/ WOTE ’11) S P O N S O R E D B Y U S E N I X , A C C U R AT E , A N D I AV O S S

    August 8–9, 2011 http://www.usenix.org/evtwote11

    4th Workshop on Cyber Security Experimenta- tion and Test (CSET ’11) August 8, 2011 http://www.usenix.org/cset11

    USENIX Workshop on Free and Open Communications on the Internet (FOCI ’11) August 8, 2011 http://www.usenix.org/foci11

    5th USENIX Workshop on Offensive Technolo- gies (WOOT ’11) August 8, 2011 http://www.usenix.org/woot11

    2nd USENIX Workshop on Health Security and Privacy (HealthSec ’11) August 9, 2011 http://www.usenix.org/healthsec11

    6th USENIX Workshop on Hot Topics in Security (HotSec ’11) August 9, 2011 http://www.usenix.org/hotsec11

    23rd ACM Symposium on Operating Systems Principles (SOSP 2011) S P O N S O R E D B Y A C M S I G O P S I N C O O P E R AT I O N W I T H U S E N I X

    October 23–26, 2011, Cascais, Portugal http://sosp2011.gsd.inesc-id.pt

    25th Large Installation System Administration Conference (LISA ’11) S P O N S O R E D B Y U S E N I X I N C O O P E R AT I O N W I T H L O P S A A N D S N I A

    December 4–9, 2011, Boston, MA, USA http://www.usenix.org/lisa11

    2011 USENIX Federated Conferences Week June 14–17, 2011, Portland, OR, USA http://www.usenix.org/fcw11 Events include:

    3rd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud ’11) June 14–15, 2011 http://www.usenix.org/hotcloud11

    3rd USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage ’11) June 14, 2011 http://www.usenix.org/hotstorage11

    3rd Workshop on I/O Virtualization (WIOV ’11) June 14, 2011 http://www.usenix.org/wiov11

    2011 USENIX Annual Technical Conference (USENIX ATC ’11) June 15–17, 2011 http://www.usenix.org/atc11

    2nd USENIX Conference on Web Application Development (WebApps ’11) June 15–16, 2011 http://www.usenix.org/webapps11

    3rd USENIX Workshop on the Theory and Practice of Provenance (TaPP ’11) June 20–21, 2011, Heraklion, Crete, Greece http://www.usenix.org/tapp11

    The 9th Annual International Conference on Mobile Systems, Applications, and Services J O I N T LY S P O N S O R E D B Y U S E N I X A N D A C M S I G M O B I L E I N C O O P E R AT I O N W I T H A C M S I G O P S

    June 28–July 1, 2011, Washington, DC, USA http://www.sigmobile.org/mobisys/2011/

    The 2nd ACM SIGOPS Asia-Pacific Workshop on Systems (APSys 2011) S P O N S O R E D B Y A C M S I G O P S I N C O O P E R AT I O N W I T H U S E N I X

    July 11–12, 2011, Shanghai, China http://apsys11.ucsd.edu/

    f o r a c o m p l e t e l i s t o f a l l u s e n i X a n d u s e n i X c o - s p o n s o r e d e v e n t s , s e e h t t p : // W W W . u s e n i X . o r g / e v e n t s

    U P C O M I N G E V E N T S

    usenix_login_june11_covers.indd 2 5.16.11 11:51 AM

  • E d i t o r

    Rik Farrow [email protected]

    M a n a g i n g E d i t o r

    Jane-Ellen Long [email protected]

    C o p y E d i t o r

    Steve Gilmartin [email protected]

    p r o d u C t i o n

    Casey Henderson Jane-Ellen Long Jennifer Peterson

    t y p E s E t t E r

    Star Type [email protected]

    u s E n i X a s s o C i at i o n

    2560 Ninth Street, Suite 215, Berkeley, California 94710 Phone: (510) 528-8649 FAX: (510) 548-5738

    http://www.usenix.org http://www.sage.org

    ;login: is the official magazine of the USENIX Association. ;login: (ISSN 1044-6397) is published bi-monthly by the USENIX Association, 2560 Ninth Street, Suite 215, Berkeley, CA 94710.

    $90 of each member’s annual dues is for a subscription to ;login:. Subscriptions for nonmembers are $125 per year. Periodicals postage paid at Berkeley, CA, and additional offices.

    POSTMASTER: Send address changes to ;login:, USENIX Association, 2560 Ninth Street, Suite 215, Berkeley, CA 94710.

    ©2011 USENIX Association USENIX is a registered trademark of the USENIX Association. Many of the designa- tions used by manufacturers and sellers to distinguish their products are claimed as trademarks. USENIX acknowledges all trademarks herein. Where those designations appear in this publication and USENIX is aware of a trademark claim, the designations have been printed in caps or initial caps.

    J u n E 2 0 1 1 , v o l . 3 6 , n o . 3

    o p i n i o n

    Musings R I k F A R R O W . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

    f i l E s y s t E M s

    Apache Hadoop: The Scalability Update k O N S TA N T I N V. S H VA C H k O . . . . . . . . . . . . . . . . . . . . . . .7

    Building a Cloud File System J E F F D A R C y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

    h a r d w a r E

    System Impacts of Storage Trends: Hard Errors and Testability S T E V E N R . H E T z L E R . . . 22

    s y s a d M i n

    Testing the Transport Side of Highly Available Hosts S T U A R T k E N D R I C k . . . . . . . . . . . . . . . . 33

    C o l u M n s

    Practical Perl Tools: Give as Good as You Get, My Tiny Dancer D AV I D N . B L A N k - E D E L M A N . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

    iVoyeur: A Handful of Gems D AV E J O S E P H S E N . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

    /dev/random R O B E R T G . F E R R E L L . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

    b o o k s

    Book Reviews E L I z A B E T H z W I C k y, W I T H E VA N T E R A N A N D R yA N M A C A R T H U R . . . . . . . . . 56

    C o n f E r E n C E s

    FAST ’11: 9th USENIX Conference on File and Storage Technologies . . . . . . . . . . . . . . . . . . . . . 59

    NSDI ’11: 8th USENIX Symposium on Networked Systems Design and Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

    LEET ’11: 4th USENIX Workshop on Large-Scale Exploits and Emergent Threats . . . . . 100

    Hot-ICE ’11: Workshop on Hot Topics in Management of Internet, Cloud, and Enterprise Networks and Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .106

    JUNE_11_articles_updtdfile.indd 1 5.16.11 11:53 AM

  • 2 ;login: VOL. 36, NO. 3

    Rik is the editor of ;login:.

    [email protected]

    I just got back from NSDI in Boston . Wintry weather (where’s spring?) and interesting talks, as usual . I really liked the LEET talks [1] about spam and botnets . The scale of these operations is just amazing . It is also amazing to learn just what can be done with a handful of servers, even if they are working to send out over a trillion spam emails . The NSDI talks also covered issues of scalability, with one of the best papers describing how GPUs can be used to support parallel SSL operations .

    Scalability of today’s systems remains an intriguing question, one that Konstantin Shvachko takes on in this issue . Konstantin has worked on the largest known Hadoop cluster, at Yahoo!, and is now working to build Hadoop clusters at eBay . In his article, he examines the limits to scalability .

    The Factory Model

    I am always fascinated—like many computer enthusiasts, I imagine—by the latest offerings of Intel and AMD . CPU clock speeds have reached approximate upper limits, and there are real reasons for this . But it is hard to think clearly about these limits, when processors and I/O busses are just so darn fast . Clock speeds in gigahertz, where one gigahertz means a cycle time of one nanosecond, are far afield from the human sensory realm . So I decided to think about a factory model instead .

    My imaginary factory has an assembly line that can crank out 3 .4 thousand widgets

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.