CineGrid @ ASIS&T 2010
Building a Collaborative Testbed for Very High-Quality Media Applications Using Very High-Speed Networks
April 9, 2010
Laurin HerrPresident, Pacific Interface Inc.
What is CineGrid? Formed 2004 – non-profit international membership organization
Mission – to build an interdisciplinary community Focus – the research, development & demonstration of
networked collaborative tools To enable – the production, use, preservation & exchange of
very high-quality digital media over high-speed photonic networks
Members – media arts schools, research universities, scientific labs, post-production facilities & hardware and software developers around the world
Connected via 1 G Ethernet & 10 G Ethernet networks For – research & education
Convergence Motivates CineGrid
Media technology historically driven by 3 sectors Entertainment, media, art and culture Science, medicine, education and research Military, intelligence, security and police
Adoption of digital media means all 3 face converging needs Fast networking for distributed applications Access to shared devices Specialized computers and massive storage Collaboration tools for distributed teams Robust security for intellectual property Higher quality sound and picture A next generation of trained professionals
CineGrid: A Scalable Approach
4K x 24
2K x 24
HD2 x 24/25/30
HDTV x 24/25/30/60
HDV x 24/25/30/60
4K2 x 24/30/60
2K2 x 24
8K x 60
Consumer HD
HDTV
Stereo HD
Digital Cinema
Stereo 4K (future)
UHDTV (far future)
5 - 25 Mbps
20 Mbps - 1.5 Gbps
200 Mbps - 3 Gbps
250 Mbps - 7.6 Gbps
500 Mbps - 15.2 Gbps
1 - 24 Gbps
More
SHD x 24/30/60
Tiled DisplaysCamera Arrays
SHD (Quad HD)250 Mbs - 6 Gbps
UHDTV(8K)
108
0 1920
432
0
7680
0.75 × Picture Height
1.5 × Picture Height
3.0 × Picture Height
UHDTV(4K)
HDTV (2K)
3840
2160
100º
30º
Visual acuity=1.0=20/20
Standard viewing distance
60ºYutaka TANAKASHARP CORPORATIONAdvanced Image Research Laboratories
Why is more resolution is better?1. More Resolution Allows Closer Viewing of Larger Image2. Closer Viewing of Larger Image Increases Viewing Angle3. Increased Viewing Angle Produces Stronger Emotional Response
Saint Issac’s cathedral in St. Petersburg, Russia
Original art is 3 meters tall by 1 meter wide
Original photograph shot from about 3 meters distance.
Camera resolution of
12 Mpixels (4K x 3K)
Original shot from about 1 meters distance
Original shot from about 0.3 meters distance
1981 Francis Ford Coppola with Dr. Takashi Fujio
“First Look” at Electronic Cinema
2001NTT Network Innovations Laboratory
“First Look” at 4K Digital Cinema
2004 OptIPuter Vision for the Next DecadeGigapixels @ Terabits/sec
Source: Jason Leigh, EVL
Augmented RealityNo Glasses
4K Streaming Video
GigapixelWall Paper
1 GigaPixel x 3 bytes/pixel x 8 bits/byte x 30 frames/sec ~ 1 Terabit/sec!
CineGrid projects run over the Global Lambda Integrated Facility (GLIF) backbone
2008 GLIF Visualization by Bob Patterson, NCSA/UIUC
In California, CineGrid access primarily via CENIC
“Learning by Doing” Early CineGrid Projects
CineGrid @ iGrid 2005 CineGrid @ AES 2006
CineGrid @ GLIF 2007CineGrid @ Holland Festival 2007
CineGrid Exchange
TERABYTES PILING UP. To store & distribute its own collection of digital media assets. Members access materials for experiments and demonstrations.
THE DIGITAL DILEMMA. Report published by AMPAS – lessons learned by NDIPP and NARA + pioneering research at Stanford (LOCKSS) & UCSD (SRB and iRODS)
GLOBAL SCALE TESTBED = distributed storage + fast networks + high quality digital media assets. Explore strategic issues in digital media storage, access, distribution and preservation.
• Movie-making is going digital and going global. Local talent is key!
• Post-production is almost all digital already. Increasingly distributed!
• Many movies still shot using 35mm film cameras, then scanned as the first step in the “digital intermediate” process, but digital cameras are getting better and better!
• Digital cinema projection technology is enabling better picture quality: From 2K digital to 4K digital, plus a new generation of 3D stereo movies… Avatar !
Trends in Cinema
• Cinema sound quality is improving. From stereo to compressed 5.1, to uncompressed 5.1, to uncompressed 7.1 surround sound!
• Of the roughly 100,000 screens worldwide, only 15% are digital today. But recent financing announcements in the USA lead to forecasts that 60-70% of American screens will be digital by 2013!
• Fast networks are becoming critical infrastructure for media companies involved in production, post-production and distribution!
• Education of the next generation of media professional is critical!
Trends in Cinema
Cinema Archives vs. Libraries
• Archive• Goal: preservation without errors, access without end• Access is “WORSE” case: Write Once Read Seldom if Ever• Long-term “insurance” for expensive intellectual property• Mixed motivations
• Potential for future revenue• Corporate asset protection• Cultural record
• Library• Goal: customer service, active management, temporary storage• Access is “BEST” case: Best Effort Sells Titles• Mixed inventory: film, video, digital in many formats• Increasing going online/near-line
• quicker search of inventory• faster time-to-market• lower cost of replication & distribution
The Digital Dilemma of Cinema
• 35mm film is the quality standard for motion pictures• Properly stored lasts 50-100 years• Survives “benign neglect” gracefully• Directly readable without reliance on “software stack”• Very cost effective storage media
• 4K digital cinema needed to match 35mm analog film quality
• More than 8 TB per master version of 2 hour theatrical movie
• More than 5 million objects, ~ 2 PB total all elements per production
• Hollywood produce approx. 400 motion pictures per year
• Hollywood archives are privately funded for future profit
• Digital cinema preservation relying on endless migration is less reliable and much more expensive (TCO) than film preservation
CineGrid Exchange 2008Geographically Distributed Repositories + Fast
Networks
San Diego @ UCSD/Calit240 TB with 10 Gbps connectivity
Amsterdam @ UvA 30 TB with 10 Gbps connectivity
Tokyo @ Keio/DMC 6 TB with 10 Gbps connectivity
Total storage = 76 TB
High-quality digital media assets: 4K, 2K, HD, mono & stereo, still & motion pictures + audio
CineGrid Exchange 2009Building on Basic Concept
• Receive “seed” funding from AMPAS to start building multi-layer open source asset management & user access framework for distributed digital media repository
• Establish Working Group• Bi-annual workshops• Weekly conference calls• Basecamp for online coordination
• Write CineGrid Exchange functional requirements document
• Expand CineGrid Exchange storage capacity with more nodes
• Diversify to reduce risk!
CineGrid Exchange 2009Phase 1 Development Goals
• Define basic workflows for ingest, preservation, distribution, & deletion• Implement basic workflows to automate management of collection
• Expand distributed repository: more nodes, more assets in collection
• Deploy consistently at all CineGrid Exchange nodes
• Enable easier large data transfers between nodes
• Prepare for integration of cataloging application + distributed repository
• Define and validate CineGrid Exchange metadata schema
CineGrid Exchange Development Multi-layer Open Software Stack
User Interface & Access
Open Source Application for Collections Management & On-Line Access
Open Source Digital Repository Interface
Open Source Middleware for Rule-based Management of Distributed Digital
Repository Resources & Asset
Testbed Infrastructure ofDistributed Storage and Network Links
iRODS by DICEFor CineGrid By Calit2
Collective AccessExtended by Whirl-I-Gig for AMPAS
CineGrid Exchange Access Portal For CineGrid by CineGrid
Resource Description FrameworkFor GLIF and CineGrid by UvA
• Programmable micro-services to implement management policies as non-ambiguous “rules” for automated operations across distributed storage repository
• Transparent management of CX files in a distributed repository
• Dedicated CX storage
• Non-dedicated CX storage
• Centralized catalog (iCAT)
• File transfer using parallel TCP & UDP
• Suitable for experimenting with digital media preservation strategies
CineGrid Exchange (CX) MiddleWare iRODS (Integrated Rules Oriented Data Services)
CineGrid Exchange 2010
Build multi-layer open-source framework for distributed digital media repository
Refine iRODS middleware “rules” for robustness
Integrate iRODS + Collective Access
CX Node Site Storage Type CX Allocation
UCSD/Calit2, San Diego, USA Sun Thumper (x4540) 66 TB
UvA, Amsterdam, Netherlands Sun Thumper (x4540) 30 TB
UIC/EVL, Chicago, USA RAID Array 10 TB
Keio U./DMC, Tokyo, Japan RAID Array 8 TB
CESNET, Prague, Czech Republic
Sun Thumper (x4540) 48 TB
Ryerson U, Toronto, Canada Sun Thumper (x4540) 57 TB
AMPAS, Los Angeles, USA Sun Thumper (x4540) 24 TB
Total CineGrid Exchange Capacity 243 TB
Acknowledgements
• Academy of Motion Picture Arts and Sciences, STC• Naval Postgraduate School, MOVES Institute• University of California San Diego, Calit2• University of Illinois Chicago, EVL• University of Washington, Research Channel• Ryerson University, Rogers Communications Ctr.• CESNET• NTT Network Innovation Laboratory• Keio University, DMC• University of Amsterdam• Pacific Interface• DICE at UNC & SDSC
CineGrid International WorkshopDecember 13-15, 2010 @ UCSD