Europeana cloud services

Europeana Cloud Services

June 10 2015Europeana Cloud Plenary Meeting Validating Europeana Cloud

University of Edinburgh Edinburgh

Bram Lohman (Europeana)Manos Koufakis (Europeana)

Lucas Anastasiou (The Open University)

Content

Data storage

Data processing

Development

Cloud

Cloud

Storage Processing

Storage

You and the cloud

Data storageData processing

Development

Storage

httprecordsCLOUDIDhttprecordsCLOUDID

Creating records (IDs)

Data storageData processing

Development

httprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEhttprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEhttprecordsCLOUDIDrepresentationsREPRESENTATIONNAME

Storage

Adding representations to records

Data storageData processing

Development

Storage

Adding versions (and files) to representations

httprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEversionsVERSIONfiles

httprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEversionsVERSIONfilesFILENAMEhttprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEversionsVERSIONfilesFILENAMEhttprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEversionsVERSIONfilesFILENAME

Data storageData processing

Development

httprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEversionsVERSIONfiles

httprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEversionsVERSIONfilesFILENAMEhttprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEversionsVERSIONfilesFILENAMEhttprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEversionsVERSIONfilesFILENAME

Storage

Completing the record

Data storageData processing

Development

httprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEversionsVERSIONfiles

Storage

Add a new file to a version

Data storageData processing

Development

Storage

Versioning the representations

httprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEversionsVERSIONcopy

httprecordsCLOUDIDrepresentationsREPRESENTATIONNAMEversionsVERSIONpersist

Data storageData processing

Development

httpdata-providersDATAPROVIDERdata-setsDATASEThttpdata-providersDATAPROVIDERdata-setsDATASEThttpdata-providersDATAPROVIDERdata-setsDATASET

httpwwwexamplecomdata-providersDATAPROVIDERdata-setsDATASETassignmentshttpwwwexamplecomdata-providersDATAPROVIDERdata-setsDATASETassignments

Storage

Assigning records to sets

Data storageData processing

Development

Data processing

Data storageData processing

Development

Data processing

a How generic is it

b What kind of data does it support

c Do I have to move my data

d Can I make my own plugins How

e How can I trigger such plugins

f Can you monitor the plugins

g What is the point of this

Data storageData processing

Development

What can you do with this How generic is it

1 Validations

2 Transformations

3 Enrichments

4 Data analysis

5 Data extraction from digital objects

xslt transformation

Data storageData processing

Development

Can I make my own plugins How

xslt-plugin

Data storageData processing

Development

Submitting your plugin to Europeana Cloud

xslt-plugin

Data storageData processing

Development

Submitting your plugin to Europeana Cloud

Data storageData processing

Development

Submitting your plugin to Europeana Cloud

Europeana Cloud Scales Deploys Monitors

You Focus on the code that matters

Data storageData processing

Development

How does the API look like Is it scary

Data storageData processing

Development

Data storageData processing

Development

Starting a taskcurl -X POST --user adminadmin -H Content-Type applicationjson -d

inputData

FILE_URLS[httprecordsL9WSPSMVQ85representationsedmversionsb17c4f60filesaf7d3a77-4b00 ]

parameters

XSLT_URLhttpsample_xsltxslt

taskId11963taskNamexslt_1 -i httptopologiesxslt_transformation_plugintasks

Data storageData processing

Development

Monitoring the progress

curl --user adminadmin

httptopologiesxslt_transformation_plugintasks1196333270517368419progress

Putting DPS to work Or how to build a recommender system on top of DPS

Can we discover similar records

Can we discover duplicate records

Can we rank according to semantic distance

Data storageData processing

Development

Cross-datasets similarities

Reference record

Same articledifferent cover

Similar style different content

Similarity 98

Similarity 56

Data storageData processing

Development

Preprocessing datasets (1)

httpDPStopologytext-stripping inputData parameters EXTRACT_TEXT true PROVIDER_ID provider_name DATASET_ID my_dataset taskId 12345 taskName strip_my_dataset_task

Data storageData processing

Development

Preprocessing datasets (2)A simple flat example record in the dataset looks like

Record

Representations

Versions

Files

aXef45XX

Metadata Content

v1 v1

Data storageData processing

Development

Record

Representations

Versions

Files

aXef45XX

Metadata Content

v1 v1

Processed

v1

Preprocessing datasets (3)After execution finishes

Data storageData processing

Development

Similarity Service API httpSimSersimilaraXef45XX

Reply

results [ cloudId bYqwe57T

providerId other_provider datasetId other_providers_dataset

similarity_score 098 ] nextSlice s4FGX78qsss0rt

Data storageData processing

Development

IIIF on the cloud

Data storageData processing

Development

Experiment Enrichment

Data storageData processing

Development

Indexing

Data storageData processing

Development