National Digital Repository ® Preserving the imperfect: reflections from NDAD and elsewhere Kevin Ashley Head of Digital Archives Group ULCC.

National Digital Repository®

Preserving the imperfect: reflections from NDAD and

elsewhere

Kevin Ashley

Head of Digital Archives Group

ULCC

2007-03-23 Presdb07 - Edinburgh 2


Overview

• Issues that arise when databases are records• Informing (expensive, important) decisions• Tensions between ideal formats and non-ideal

data• Representation mechanisms for access control

and absent data• Concentrating on R&D issues





What is NDAD?

• A service for UK government records which exist as ‘structured information’

• Contains data + contextual information• Established in 1997 - service in March 1998• First service by a national archive to provide

online public access to preserved material• Selection undertaken by National Archives and

government departments• Everything else at ULCC: under contract to TNA





Preservation

• Data transformed to canonical form - originals kept

• Paper documentation digitised• Technical metadata produced or transformed• Consistency checks applied:

For transformation processAgainst original systemAgainst published information Internal cross-checks



Consequences

• Preservation far removed from creation• Unlike actively curated systems: preservation

and use can take place simultaneously• Multiple use scenarios - more than views



Where are the problems?

Management



Perfect Preservation Formats?

• DDI: XML-basedgood for survey/social science dataNot so good for complex relational stuffLikes clean data

• XML representationsMore flexibleNot so good when data is unclean

• As SQLMuch metadata or needs another schemeUseless for unclean data



How bad is bad?

• Data out of range is a quality problem, not a preservation problem (e.g. ‘Age’ of 230)

• But…Age = -20?Age = B0 ?Age = Thursday?

• All present problems if ‘Age’ is a positive integer in our preservation schema

• Date = ‘31 Feb 2007’ is syntactically but not semantically valid



More bad stuff

• Absent key fields or mandatory fields• Encoded data that uses bad codes

if days of week are 1 - 7, what is day 9? Day X ?

• ‘Encoded’ data which is stored translated• 1 - 1 mappings that aren’t



What’s the problem?

• Must preserve errors - their nature is informative• Would like to understand original system

behaviour with these errors• Don’t want to use tools that force all fields to be

text• Want a datatype like ‘almost always integer’ or

‘often a date’ - and intelligent behaviour when it isn’t.



How does it get that way?

• Data validation often in application, not database Isn’t always well-implemented

• People hack around the application• Past migrations were poor



Missing and absent values

• Common occurrence in survey and experimental data

• Different types of ‘missing’:No informationKnown to be unreadableRefused to answerSubject didn’t know

• All mechanisms for representation ad-hoc• Knowledge in application, not database• Query engines don’t understand concept



Access: restricted viewingID Name Fname Office

10246 Ashley Kevin 179

10579 Mouse Mickey 188

ID REG Date From To10246 X111ABC 1 Oct 98 Clapham Deptford10579 H179JKL 1 Apr 99 Land’s

EndJohnO’Groats

56999 A217HGB 23 Dec 97 Poole Sandy

REG Make Year ColourX111ABC Yugo 1999 Grey

H179JKL Trabant 1957 Brown

People

Trips

Vehicles

Not available until 2050



Access - goal

• Duplicate original system• Advanced analysis tools• Simple viewing via a generic tool• Multimedia datatypes • Extensible via object-like design

• Traditional database systems not up to task without significant additional effort

• Hence much software home-grown



New issues from temporal GIS

• Temporal GIS allows one system to represent changing features and knowledge

• Queries like:Which features are newer than feature X?What did area Y look like 10 years ago?What present-day names correspond to ‘Hetfelle’?

• In a preserved temporal GIS:What would the answer to question 2 have been if I

asked it 5 years ago?



Inconsistencies and errors

• Schools census - 4 datasets per year for different school types

• But 1976 only has 3 - no nursery schools• Further examination shows files have been

merged• Confirmation came from completed census

forms held by schools - not by government department



Cornell’s DP model

National Digital Repository ® Preserving the imperfect: reflections from NDAD and elsewhere Kevin Ashley Head of Digital Archives Group ULCC.

Documents

national digital repository

presdb07 edinburgh

preservation data

unclean data slide

national archives

absent data

encoded data

views slide