1 The NSDL Program Stephen Griffin National Science Foundation.

1

The NSDL Program

Stephen GriffinNational Science Foundation

2

The NSDL Program

1996 Vision articulated by NSF's Division of Undergraduate Education

1997 National Research Council workshop

1998 Preliminary grants through Digital Libraries Initiative 2

1998 SMETE-Lib workshop

1999 NSDL Solicitation

2000 6 Core Integration demonstration projects + 23 others funded

2001 1 large Core Integration System project funded

2002 More than 60 independent projects funded

3

NSF-funded Research Programs

NSF

Solicitation

Proposals

Research

New ideas

New ideas

4

The NSDL Program

NSF's objective

Build a comprehensive digital library for all aspects of science education

NSF's approach

Solicitation encouraged wide diversity of proposals divided into general categories

Best 60+ proposals funded -- more to follow

Grants allow projects flexibility

Result

A splendid set of projects

A challenge in interoperability!

5

The NSDL:The Challenge of Scale

William Y. ArmsCornell University

6

Core Integration Philosophy

Scientific and technical information

Materials used in education

Materials tailored toeducation

7

It is possible to build a very large digital library with a small staff.

But ...

Every aspect of the library must be planned with scalability in mind.

Some compromises will be made.

Core Integration Philosophy

8

All branches of science, all levels of education, very broadly defined:

Five year targets

1,000,000 different users

10,000,000 digital objects

10,000 to 100,000 independent sites

How Big might the NSDL be?

9

Resources for Core Integration

Core Integration

Budget $4-6 million

Staff 25 - 30

Management Diffuse How can a small team, without direct management control, create a very large-scale digital library?

10

Collections: the Basic AssumptionThe Core Integration team will not manage any collections

11

Collections The NSDL program funds only a fraction of the relevant collections.

12

Every Collection is Different

13

... to provide a coherent set of collections and services across

great diversity.

The Core Integration Task ...

14

Interoperability

The Problem

Conventional approaches to interoperability require partners to support agreements (technical, content, and business

But NSDL needs thousands of very different partners

... most of whom are not directly part of the NSDL program

The Approach

A spectrum of interoperability

15

Levels of interoperability

Level Agreements Example

Federation Strict use of standards AACR, MARC(syntax, semantic, Z 39.50and business)

Harvesting Digital libraries expose Open Archivesmetadata; simple metadata harvesting

protocol and registry

Gathering Digital libraries do not Web crawlerscooperate; services must and search enginesseek out information

16

What to Index?

When possible, full text indexing is excellent, but full text indexing is not possible for all materials (non-textual, no access for indexing).

Comprehensive metadata is an alternative, but available for very few of the materials.

What Architecture to Use?

Few collections support an established search protocol (e.g., Z39.50)

Searching

17

Broadcast Searching does not Scale

User interfaceserver

User

Collections

18

Users

Collections

Metadata repository

The Metadata Repository

Services

The metadata repository is a resource for service providers.

It holds information about every collection and item known to the NSDL.

19

Search Architecture

Portal

Portal

Portal

Search andDiscoveryServices Collections

SDLIP OAI

http

Metadata repository

James Allan, Bruce Croft (University of Massachusetts, Amherst)

20

The Metadata Repository as a Resource

Records are exposed through Open Archives Initiative harvesting protocol.

Core Integration team will provide some services based on the metadata repository.

The architecture encourages others to build services.

Support for Service Providers

21

Metadata StrategyMetadata is expensiveThe NSDL cannot afford to create it manually

22

Metadata Strategy

• Support eight standard formats

• Collect all existing metadata in these formats

• Provide crosswalks to Dublin Core

• Expose records in the metadata repository for others to harvest

• Concentrate on collection-level metadata

• Use automatic generation to augment item-level metadata

23

Collection-level OperationsMaterial in the NSDL is selected and managed as collections:

• Alexandria Digital Library• Cornell course web sites• JISC Resource Discovery Network• Joe's web page

Human effort will be used to select and integrate major collections.

Automated methods (e.g., web crawling) will be used to identify and integrate additional collections.

24

The Problem

Material in the NSDL should be relevant.

But we cannot select each item individually.

The Approach

Most selection and quality control decisions are made at a collection level, not at an item level.

Information about quality will be maintained in a collection-level metadata record, which is stored in a central metadata repository.

This metadata is made available to NSDL service providers.

User interfaces can display quality information.

Quality Control

25

User Interfaces

The Problem

Cannot handcraft every web page

Must be usable on a very wide range of equipment and with a very diverse group of users

The Solution

Data driven portals using a channel architecture.

Interfaces guide the user to understand the library.

One library, many portals.

26

27

28

The Mortal behind the Portal

[This space left intentionally blank.]

29

Where is the Center of the Universe?

NSDL

Alexandria

Elsevier

Informedia

Library of Congress

Joe's PicturesMath DL

30

Where is the Center of the Universe?

NSDL

British Library

Elsevier

OCLC

Library of Congress

Internet Archive

Harvard

31

Where is the Center of the Universe?

NSDL

email

Course web sites

News and weather

Bill Arms

Office

Technical documentation

Google

Directories

32

The NSDL is a program of the National Science Foundation's Directorate for Education and Human Resources, Division of Undergraduate Education.

The NSDL Core Integration is a collaboration between the University Center for Atmospheric Research (Dave Fulker), Columbia University (Kate Wittenberg) and Cornell University (Bill Arms). The Technical Director is Carl Lagoze (Cornell University).

Acknowledgement and Disclaimer

33

The NSDL:The Challenge of Scale

William Y. ArmsCornell University