Cloud Libraries

Cloud libraries

April 2008

cloud computing concepts

get fuzzier over time ...

many different broadly similardefinitions abroad

generally means:widely distributed, network based -- storage,- computation,- SaaS models

single datacenter or -hosted apps >

moving to network based solutions

usually provided by external parties on a quanta basis, e.g. compute cycle or byte traffic

a concept of “virtualization”

of infrastructure or web serviceor application

Poster Child:Poster Child:

Amazon’s services --S3 (simple storage system) and EC2 (elastic compute cloud)

NY Times converted its TIFF backlist to PDFwith Amazon services (EC2 / S3)and bill was ~ $500

“I then began some rough calculations and determined that if I used only four machines, it could take some time to generate all 11 million article PDFs. But thanks to the swell people at Amazon, I got access to a few more machines and churned through all 11 million articles in just under 24 hours using 100 EC2 instances, and generated another 1.5TB of data to store in S3.” - Derek Gottfrid, NYT

services are offeredor increasinglydefined by

Google, Amazon,Microsoft (maybe), IBM and others.

Google’s MapReduce

break up a problemallocate to many machinesreassemble for use

[1, 2, 3].map{|i| i * 10}

map{} returns new array[10, 20, 30]

map{} could be

{image processing}{text search}{metadata extraction}

reduce{} transforms output of map{}

It doesn’t matter how many parameters in listevery item handled independentlyfrom every other item

• means computation • over each item in list• can be distributed• to many computers• eg one VM per item

reduce{} combines results

“... a new super-group of tech companies [will] dominate the cloud computing market, each of them managing what amounts to a giant centralised computer made up of a number of big datacentres.”

- Steve Ballmer, MSFT

“NSF Partners With Google and IBM to Enhance Academic Research Opportunities”(Press Release 08-031)

even very high end universitieslag behind corporate computing

“Fundamental changes in computer architecture and increases in network capacity are encouraging software developers to take new approaches to computer-science problem solving.”

“In order to bridge the gap between industry and academia, it is imperative that academic researchers are exposed to the emerging computing paradigm behind the growth of "Internet-scale" applications.”

IBM is deeply committed

IBM Kittyhawk

http://domino.research.ibm.com/comm/research_projects.nsf/pages/

kittyhawk.index.html

IBM Kittyhawk

"We postulate that efficient, balanced machines with high-performance internal networks such as Blue Gene are not only significantly better choices for web-scale companies but can form the building blocks of one global-scale shared computer. Such a computer would be capable of hosting not only individual web-scale workloads but the entire Internet. ...”

in other words:

not your father’sdatacenter

“Hiding in Plain Sight, Google Seeks More Power”

Photo Credit: Melanie Conner for The New York Times

cloud computinghas already revolutionized many content basedinformation services

Google Books virtualizes libraries

Flickr virtualizes image collections

iTunesvirtualizesaudio recordings

YouTubevirtualizes video

Is there space for HE to contribute to

virtualization layers ?

Most new initiativesdo not directly involveresearch libraries as central constituents

OpenLibrary

Zotero and Internet Archive

LibraryThing

If (ARTStor) did (Flickr) ...

libraries provide fairlylightweight applicationsor “soft infrastructure”

e.g. faculty repository systemsonline publishing platformsboutique special collections

not fundamental infrastructureprovision

power is in aggregation

but there are problems with library collaboration at scale

(it doesn’t happen)

similar to classic political science issueson sovereignty

EU one of the most successfultrans-national political efforts

Europe tightly integrated due to - physical and - historical and - economic issues

consider the difficulties establishing

League of Nations

and post WW-II,

United Nations

United Nations (as it exists today)not (at all) what itsplanners wished

Tremendous conflict arose in San Francisco between Russia (Molotov)other Allied war parties

Even in the face of the possibility of world-ending

NUCLEAR WAR

Libraries have little beyond established workflow

interoperation like ILL

a few dominance hierarchieslike the CICor OhioLinkor the CDL

which do well in core operations

OCLC

I see primarily as a vendorin important but established bibliographic service space

Difficult to establish - much less maintain - a of large libraries

with serious long term resource commitments

maybe the best model issupport of the equivalent of something like the IAEA

contribute - dollars - staff - expertise- technology

workingwith othersbut relying onan independentorganizationalframework for action

a possible role for the DLF

“marriage broker”

make introductionsbetween nascent efforts and potential partners

like libraries and museumsand archives and tech firmsand societies

figure out domain- or discipline-based early-stage faculty teams seeking to develop new services

aid and abet --

encourage faculty and professional societiesto assume leadership

current thrusts for the DLF ...

mobile apps virtual worlds moving images science CI

focus today on mobile :fundamentally reliant on net- scale applications

mantra

connectcommunicatecollaborate

(CCC)

mobile apps

iPhone Android

wholly game changingdevelopment platforms

coming :social collaborative learning environments

wild assed designs• portable • networked• computing• interfaces

petiteinvention

areas to watch

Rudy de Waele Plugg.eu - March 19, 2008 dot.Open

sensoringbiometrics

transactionslifestreaming

recommendationimage recognitionaugmented reality

mobile connected gameslocation based social media

retail proximity media consumption

Rudy de Waele Plugg.eu - March 19, 2008 dot.Open

in ...

where ? librariesin the future

university libraries will not bandtogether in virtual organizationsto establish

bigger, better librariesoffering cloud based information services

probably this is against theirinstitutional nature

a significant difficulty inmoving into wholly differentdomain areas (like mobile)

IN NEW ecological niches

– zero platform expertise– no collaboration : industry– a paradigm shift in IT

libraries MUSTMUST

STOPSTOP defining solutions

(in terms of) libraries

libraries must learn how to :

- partner ||- collaborate || - contribute ||- coordinate

via organizational frameworks

- external to us - friendly to us - supported by us- sponsored by us(but not us.

that’s a differentkind of library

more diversemore opportunistic more open

it’s a smaller libraryit’s a more nimble library it’s a library of pieces

broken up from somethingunwieldy lethargic stifling

( It should bea

more funmore fun

LIBRARY )

thanks

peter at diglib.orgnaypinya {twitter, twine, skype}