CineGrid @ ASIS&T 2010 Building a Collaborative Testbed for Very High-Quality Media Applications Using Very High-Speed Networks April 9, 2010 Laurin Herr.

CineGrid @ ASIS&T 2010

Building a Collaborative Testbed for Very High-Quality Media Applications Using Very High-Speed Networks

April 9, 2010

Laurin HerrPresident, Pacific Interface Inc.

[email protected]

What is CineGrid? Formed 2004 – non-profit international membership organization

Mission – to build an interdisciplinary community Focus – the research, development & demonstration of

networked collaborative tools To enable – the production, use, preservation & exchange of

very high-quality digital media over high-speed photonic networks

Members – media arts schools, research universities, scientific labs, post-production facilities & hardware and software developers around the world

Connected via 1 G Ethernet & 10 G Ethernet networks For – research & education

Convergence Motivates CineGrid

Media technology historically driven by 3 sectors Entertainment, media, art and culture Science, medicine, education and research Military, intelligence, security and police

Adoption of digital media means all 3 face converging needs Fast networking for distributed applications Access to shared devices Specialized computers and massive storage Collaboration tools for distributed teams Robust security for intellectual property Higher quality sound and picture A next generation of trained professionals

CineGrid: A Scalable Approach

4K x 24

2K x 24

HD2 x 24/25/30

HDTV x 24/25/30/60

HDV x 24/25/30/60

4K2 x 24/30/60

2K2 x 24

8K x 60

Consumer HD

HDTV

Stereo HD

Digital Cinema

Stereo 4K (future)

UHDTV (far future)

5 - 25 Mbps

20 Mbps - 1.5 Gbps

200 Mbps - 3 Gbps

250 Mbps - 7.6 Gbps

500 Mbps - 15.2 Gbps

1 - 24 Gbps

More

SHD x 24/30/60

Tiled DisplaysCamera Arrays

SHD (Quad HD)250 Mbs - 6 Gbps

UHDTV(8K)

108

0 1920 　

432

0

7680 　

0.75 × Picture Height

1.5 × Picture Height

3.0 × Picture Height

UHDTV(4K)

HDTV (2K)

3840 　

2160

100º

30º

Visual acuity=1.0=20/20

Standard viewing distance

60ºYutaka TANAKASHARP CORPORATIONAdvanced Image Research Laboratories

Why is more resolution is better?1. More Resolution Allows Closer Viewing of Larger Image2. Closer Viewing of Larger Image Increases Viewing Angle3. Increased Viewing Angle Produces Stronger Emotional Response

Saint Issac’s cathedral in St. Petersburg, Russia

Original art is 3 meters tall by 1 meter wide

Original photograph shot from about 3 meters distance.

Camera resolution of

12 Mpixels (4K x 3K)

Original shot from about 1 meters distance

Original shot from about 0.3 meters distance

1981 Francis Ford Coppola with Dr. Takashi Fujio

“First Look” at Electronic Cinema

2001NTT Network Innovations Laboratory

“First Look” at 4K Digital Cinema

2004 OptIPuter Vision for the Next DecadeGigapixels @ Terabits/sec

Source: Jason Leigh, EVL

Augmented RealityNo Glasses

4K Streaming Video

GigapixelWall Paper

1 GigaPixel x 3 bytes/pixel x 8 bits/byte x 30 frames/sec ~ 1 Terabit/sec!

CineGrid projects run over the Global Lambda Integrated Facility (GLIF) backbone

2008 GLIF Visualization by Bob Patterson, NCSA/UIUC

In California, CineGrid access primarily via CENIC

“Learning by Doing” Early CineGrid Projects

CineGrid @ iGrid 2005 CineGrid @ AES 2006

CineGrid @ GLIF 2007CineGrid @ Holland Festival 2007

CineGrid Exchange

TERABYTES PILING UP. To store & distribute its own collection of digital media assets. Members access materials for experiments and demonstrations.

THE DIGITAL DILEMMA. Report published by AMPAS – lessons learned by NDIPP and NARA + pioneering research at Stanford (LOCKSS) & UCSD (SRB and iRODS)

GLOBAL SCALE TESTBED = distributed storage + fast networks + high quality digital media assets. Explore strategic issues in digital media storage, access, distribution and preservation.

• Movie-making is going digital and going global. Local talent is key!

• Post-production is almost all digital already. Increasingly distributed!

• Many movies still shot using 35mm film cameras, then scanned as the first step in the “digital intermediate” process, but digital cameras are getting better and better!

• Digital cinema projection technology is enabling better picture quality: From 2K digital to 4K digital, plus a new generation of 3D stereo movies… Avatar !

Trends in Cinema

• Cinema sound quality is improving. From stereo to compressed 5.1, to uncompressed 5.1, to uncompressed 7.1 surround sound!

• Of the roughly 100,000 screens worldwide, only 15% are digital today. But recent financing announcements in the USA lead to forecasts that 60-70% of American screens will be digital by 2013!

• Fast networks are becoming critical infrastructure for media companies involved in production, post-production and distribution!

• Education of the next generation of media professional is critical!

Trends in Cinema

Cinema Archives vs. Libraries

• Archive• Goal: preservation without errors, access without end• Access is “WORSE” case: Write Once Read Seldom if Ever• Long-term “insurance” for expensive intellectual property• Mixed motivations

• Potential for future revenue• Corporate asset protection• Cultural record

• Library• Goal: customer service, active management, temporary storage• Access is “BEST” case: Best Effort Sells Titles• Mixed inventory: film, video, digital in many formats• Increasing going online/near-line

• quicker search of inventory• faster time-to-market• lower cost of replication & distribution

The Digital Dilemma of Cinema

• 35mm film is the quality standard for motion pictures• Properly stored lasts 50-100 years• Survives “benign neglect” gracefully• Directly readable without reliance on “software stack”• Very cost effective storage media

• 4K digital cinema needed to match 35mm analog film quality

• More than 8 TB per master version of 2 hour theatrical movie

• More than 5 million objects, ~ 2 PB total all elements per production

• Hollywood produce approx. 400 motion pictures per year

• Hollywood archives are privately funded for future profit

• Digital cinema preservation relying on endless migration is less reliable and much more expensive (TCO) than film preservation

CineGrid Exchange 2008Geographically Distributed Repositories + Fast

Networks

San Diego @ UCSD/Calit240 TB with 10 Gbps connectivity

Amsterdam @ UvA 30 TB with 10 Gbps connectivity

Tokyo @ Keio/DMC 6 TB with 10 Gbps connectivity

Total storage = 76 TB

High-quality digital media assets: 4K, 2K, HD, mono & stereo, still & motion pictures + audio

CineGrid Exchange 2009Building on Basic Concept

• Receive “seed” funding from AMPAS to start building multi-layer open source asset management & user access framework for distributed digital media repository

• Establish Working Group• Bi-annual workshops• Weekly conference calls• Basecamp for online coordination

• Write CineGrid Exchange functional requirements document

• Expand CineGrid Exchange storage capacity with more nodes

• Diversify to reduce risk!

CineGrid Exchange 2009Phase 1 Development Goals

• Define basic workflows for ingest, preservation, distribution, & deletion• Implement basic workflows to automate management of collection

• Expand distributed repository: more nodes, more assets in collection

• Deploy consistently at all CineGrid Exchange nodes

• Enable easier large data transfers between nodes

• Prepare for integration of cataloging application + distributed repository

• Define and validate CineGrid Exchange metadata schema

CineGrid Exchange Development Multi-layer Open Software Stack

User Interface & Access

Open Source Application for Collections Management & On-Line Access

Open Source Digital Repository Interface

Open Source Middleware for Rule-based Management of Distributed Digital

Repository Resources & Asset

Testbed Infrastructure ofDistributed Storage and Network Links

iRODS by DICEFor CineGrid By Calit2

Collective AccessExtended by Whirl-I-Gig for AMPAS

CineGrid Exchange Access Portal For CineGrid by CineGrid

Resource Description FrameworkFor GLIF and CineGrid by UvA

• Programmable micro-services to implement management policies as non-ambiguous “rules” for automated operations across distributed storage repository

• Transparent management of CX files in a distributed repository

• Dedicated CX storage

• Non-dedicated CX storage

• Centralized catalog (iCAT)

• File transfer using parallel TCP & UDP

• Suitable for experimenting with digital media preservation strategies

CineGrid Exchange (CX) MiddleWare iRODS (Integrated Rules Oriented Data Services)

CineGrid Exchange 2010

Build multi-layer open-source framework for distributed digital media repository

Refine iRODS middleware “rules” for robustness

Integrate iRODS + Collective Access

CX Node Site Storage Type CX Allocation

UCSD/Calit2, San Diego, USA Sun Thumper (x4540) 66 TB

UvA, Amsterdam, Netherlands Sun Thumper (x4540) 30 TB

UIC/EVL, Chicago, USA RAID Array 10 TB

Keio U./DMC, Tokyo, Japan RAID Array 8 TB

CESNET, Prague, Czech Republic

Sun Thumper (x4540) 48 TB

Ryerson U, Toronto, Canada Sun Thumper (x4540) 57 TB

AMPAS, Los Angeles, USA Sun Thumper (x4540) 24 TB

Total CineGrid Exchange Capacity 243 TB

Acknowledgements

• Academy of Motion Picture Arts and Sciences, STC• Naval Postgraduate School, MOVES Institute• University of California San Diego, Calit2• University of Illinois Chicago, EVL• University of Washington, Research Channel• Ryerson University, Rogers Communications Ctr.• CESNET• NTT Network Innovation Laboratory• Keio University, DMC• University of Amsterdam• Pacific Interface• DICE at UNC & SDSC

CineGrid International WorkshopDecember 13-15, 2010 @ UCSD

www.cinegrid.org