(a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”) DAMS · A brief history of the FMNH EMu/DAMu-verse Digitization Projects FMNH + EMu the Future... 1,500,000 2,500,000 MM

DAMS(a.k.a “DAMNS”, “DAMBS”, “DAMu”, “DAAAMMS”)

Kate WebbinkSharon Grant

Janeen JonesPete Herbst

Rob ZschernitzNorth American User Conference

2019

[EMu Users 2019]

the silence of the DAMS

A brief history of the FMNH EMu/DAMu-verse

Digitization

Projects

FMNH + EMu

the Future...

1,500,000

2,500,000

MM opened to

all users

2008

~

Catalogue

Multimedia

DAMS…

Standardization

Project…

2010

~

2012

~

2014

~

2016

~

2018

~

2015

~

2013

~~~ ~

2019

~

A brief history of the FMNH EMu/DAMu-verse

Digitization

Projects

FMNH + EMu

the Future...

1,500,000

2,500,000

MM opened to

all users

2008

~

Catalogue

Multimedia

DAMS…

Standardization

Project…

2010

~

2012

~

2014

~

2016

~

2018

~

2015

~

2013

~~~ ~

2019

~

20 TB

$85 million

DAMu usage timelines

2008 - 2018...taking stock of ~10 year’s worth of Multimedia:

- File formats… not so standard- e.g., XLSX, 32 flavors of TIFF

- Filenames… also not so standard- e.g.: “DÈRP’s b.123(4).jpg”

- Files themselves… - sometimes go missing between EMu & storage → [broken icon]- dupdupduplicates

- Keywords… how to standard?- there was yelling...

[the screaming of the DAMS]

...not so standard...

...some proprietary…

...reflect a user’s local registry...

File formats

...not so standard...

...some proprietary…

...reflects user’s local registry...

File formatsDocumented standards…

Not set in stone, but reference-able

...also not so standard

- “DÈ[R]P’s b.123 (#4).jpg”

= an example of a file that will disappear itself from a Multimedia export.

- ...mostly from the “È” character with the diacritic, - but if we’re fixing that, may as well pull out other special characters)

Filenames

ÁàÀâÂäÄãÃåÅæÆçÇéÉèÈêÊëËíÍìÌîÎïÏñÑóÓòÒôÔöÖõÕøØœŒßúÚùÙûÛüÜ\]\*'

-----------sh*tty-filename-query:

...MulIdentifier contains '\*\[\-\\ ()~!@#$%^&*`,[]{}<>/?\;\:\'"\]\*'orMulIdentifier contains '\*\.\*\.\*'or MulIdentifier contains '\==\*\[áÁàÀâÂäÄãÃåÅæÆçÇéÉèÈêÊëËíÍìÌîÎïÏñÑóÓòÒôÔöÖõÕøØœŒßúÚùÙûÛüÜ\]\*'orSupIdentifier contains '\*\[\-\\ ()~!@#$%^&*`,[]{}<>/?\;\:\'"\]\*'orSupIdentifier contains '\*\.\*\.\*'orSupIdentifier contains '\==\*\[áÁàÀâÂäÄãÃåÅæÆçÇéÉèÈêÊëËíÍìÌîÎïÏñÑóÓòÒôÔöÖõÕøØœŒßúÚùÙûÛüÜ\]\*'...

Filenames

Files themselves - when they’re over 2GB…

Files themselves - when they’re over 2GB…get a little mangled

...into the mystery-void between EMu & storage → [broken icon]

Files themselves - sometimes go missing

/emufmnh/multimedia//1183/101//1183/102//1183/103/ (...!?)/1183/105//1183/106/

...into the mystery-void between EMu & storage → [broken icon]

Files themselves - sometimes go missing

- On each side of the void is an audit trail, (EMu audit + Filer log)

- but without cross-checking between systems…

EMu Filer

(the

void

)

...go missing

...into the mystery-void between EMu & storage → [broken icon]

- On each side of the void is an audit trail, (EMu audit + Filer log)- but without cross-checking between systems…

Files themselves...

github.com / fieldmuseum / Collections-Scripts

- On each side of the void is an audit trail(EMu audit + Filer log)

- but without cross-checking between systems…

...get re-re-dup-duplicated

- sometimes folks don’t look before they leap- Or they do, but their workflows get in the way

- (e.g., checking for dups by name...after renaming dups)

Files themselves...

Out-of-sync duplicated data

Keywords

- Taxon Coverage: Text field, [S], read-through, limit = 500 characters

- Read-through of ObjObjectsRef_tab.DarScientificName

- Logic: Pipe delimited if Multimedia is attached to multiple Catalog records, and remove duplicates

- Vernacular Name: Text field, [S], read-through, limit = 500 characters

- Read-through of ObjObjectsRef_tab.IdeTaxonRef_tab.ComName_tab(1), where

ObjObjectsRef_tab.IdeFiledAs_tab = Yes

- Logic: Pipe delimit if multiple values are returned, and remove duplicates

- Related Geography: Text field, [S], read-through, limit = 500 characters

- Read-through of ObjObjectsRef_tab.DarCountry, DarStateProvince, DarCounty, DarOceanContinent

- Logic: Pipe delimit if MM attached to multiple Catalog records, and remove duplicates

- Temporal Coverage: Text field, [S], read-through, limit = 500 characters

- Read-through of ObjObjectsRef_tab.PalStrStratigraphyRef_tab.SummaryData

- Logic: Pipe delimit if multiple values are returned, and remove duplicates

- Sex: Text field, [S], read-through, limit = 100 characters

- Logic: Read-through of ObjObjectsRef_tab(1).DarSex

- Logic: Pipe delimit if MM attached to multiple Catalog records, and remove duplicates

- Life Stage: Text field, [S], read-through, limit = 100 characters

- Read-through of ObjObjectsRef_tab(1).DarLifeStage

- Logic: Pipe delimit if MM attached to multiple Catalog records, and remove duplicates

- Subject Orientation: Lookup List Text field (100 Chars), [S] ,

- Lookup list: NEW Subject Orientation Level 1

- Subject Part: Lookup List Text field (100 chars), [M]

- Lookup list: NEW, “Parts” - Level 2

- Capture Device: Lookup List Text field (255 chars), [S]

- Lookup list: NEW Capture Device - Level 1

- Access URI

- Logic: if AdmPublishWebNoPassword = Yes, and if MulDocumentType = “M” (not U or R), then

Concatenate “http://fm-digital-assets.fieldmuseum.org/” + [Multimedia] -- e.g.,

“http://fm-digital-assets.fieldmuseum.org/1801/052/UC_22544_single.jpg”

- Functionality: Display web link icon

- Related Resource ID: Text field, [S] , lookup list, limit = 100 characters

- Logic: If AssAssociatedRef_tab is not null, pipe-delimited AssAssociatedRef_tab.AdmGlobalUniqueID when

AssAssociatedRef_tab.AdmPublishWebNoPassword = Yes

- Derived From: Text field, [S], read-through, limit = 100 characters

- Logic: If RelParentRef is not null display RelParentMediaRef.[new Multimedia Identifier field] when

RelParentRef.AdmPublishWebNoPassword = Yes

- Funding Attribution: Text field, [S], read-through, limit = 100 characters

- Where DetContributorRole_tab = “Funder” and DetContributorRole_tab.AdmPublishWebNoPassword = Yes, pipe

delimited list of DetContributorRef_tab.SummaryData

- Identifier- AdmGlobalUniqueIdentifier with AdmGUIDPreferred = Yes

- Associated Specimen

- Logic: If ObjObjectsRef_tab.ColCollectionEventRef.irn is not null, show pipe delimited list of

ObjObjectsRef_tab.DarGlobalUniqueIdentifier where ObjObjectsRef_tab.AdmPublishWebNoPassword = Yes and

ObjObjectsRef_tab.CatSightingsEventsRef_tab.irn is NULL

- Associated Observation

- Logic: If ObjObjectsRef_tab.ColSightingsEventRef_tab.irn is not null, show pipe delimited list of

ObjObjectsRef_tab.DarGlobalUniqueIdentifier where ObjObjectsRef_tab.AdmPublishWebNoPassword = Yes and

ObjObjectsRef_tab.CatSightingsEventsRef_tab.irn is NOT NULL

-Taxon Coverage: Text field, [S], read-through, limit = 500 characters

-Read-through of ObjObjectsRef_tab.DarScientificName

-Logic: Pipe delimited if Multimedia is attached to multiple Catalog records, and remove duplicates

-Vernacular Name: Text field, [S], read-through, limit = 500 characters

-Read-through of ObjObjectsRef_tab.IdeTaxonRef_tab.ComName_tab(1), where

ObjObjectsRef_tab.IdeFiledAs_tab = Yes

-Logic: Pipe delimit if m

ultiple values are returned, and remove duplicates

-Related Geography: Text field, [S], read-through, limit = 500 characters

-Read-through of ObjObjectsRef_tab.DarCountry, D

arStateProvince, DarCounty, DarOceanContinent

-Logic: Pipe delimit if M

M attached to multiple Catalog records, and remove duplicates

-Temporal Coverage: Text field, [S], read-through, limit = 500 characters

-Read-through of ObjObjectsRef_tab.PalStrStratigraphyRef_tab.SummaryData

-Logic: Pipe delimit if m

ultiple values are returned, and remove duplicates

-Sex: Text field, [S], read-through, limit = 100 characters

-Logic: Read-through of ObjObjectsRef_tab(1).DarSex

-Logic: Pipe delimit if M

M attached to multiple Catalog records, and remove duplicates

-Life Stage: Text field, [S], read-through, limit = 100 characters

-Read-through of ObjObjectsRef_tab(1).DarLifeStage

-Logic: Pipe delimit if M

M attached to multiple Catalog records, and remove duplicates

-Subject Orientation: Lookup List Text field (100 Chars), [S] ,

-Lookup list: NEW Subject Orientation Level 1

-Subject Part: Lookup List Text field (100 chars), [M]

-Lookup list: NEW, “Parts” - Level 2

-Capture Device: Lookup List Text field (255 chars), [S]

-Lookup list: NEW Capture Device - Level 1

-Access URI

-Logic: if AdmPublishWebNoPassword = Yes, and if MulDocumentType = “M” (not U or R), then

Concatenate “http://fm-digital-assets.fieldmuseum.org/” + [Multimedia] -- e.g.,

“http://fm-digital-assets.fieldmuseum.org/1801/052/UC_22544_single.jpg”

-Functionality: Display web link icon

-Related Resource ID: Text field, [S] , lookup list, limit = 100 characters

-Logic: If AssAssociatedRef_tab is not null, pipe-delimited AssAssociatedRef_tab.AdmGlobalUniqueID when

AssAssociatedRef_tab.AdmPublishWebNoPassword = Yes

-Derived From: Text field, [S], read-through, limit = 100 characters

-Logic: If RelParentRef is not null display RelParentMediaRef.[new Multimedia Identifier field] when

RelParentRef.AdmPublishWebNoPassword = Yes

-Funding Attribution: Text field, [S], read-through, limit = 100 characters

-Where DetContributorRole_tab = “Funder” and DetContributorRole_tab.AdmPublishWebNoPassword = Yes, pipe

delimited list of DetContributorRef_tab.SummaryData

-Identifier

-AdmGlobalUniqueIdentifier with AdmGUIDPreferred = Yes

-Associated Specimen

-Logic: If O

bjObjectsRef_tab.ColCollectionEventRef.irn is not null, show pipe delimited list of

ObjObjectsRef_tab.DarGlobalUniqueIdentifier where ObjObjectsRef_tab.AdmPublishWebNoPassword = Yes and

ObjObjectsRef_tab.CatSightingsEventsRef_tab.irn is NULL

-Associated Observation

-Logic: If O

bjObjectsRef_tab.ColSightingsEventRef_tab.irn is not null, show pipe delimited list of

ObjObjectsRef_tab.DarGlobalUniqueIdentifier where ObjObjectsRef_tab.AdmPublishWebNoPassword = Yes and

ObjObjectsRef_tab.CatSightingsEventsRef_tab.irn is NOT NULL

Out-of-sync duplicated dataMapping reverse-attached fields to Audubon Core:

Keywords Audubon Core

- Taxon Coverage- Vernacular Name- Related Geography- Temporal Coverage- Sex- Life Stage- Subject Orientation- Subject Part- Capture Device- Access URI- Related Resource ID- Derived From- Funding Attribution- Identifier- Associated Specimen- Associated Observation

2018 - 2019...taking stock of 5 year’s worth of Multimedia + 1 year of cleanup:

- File formats… “stabilized”?- XLSX → open-docs formats; still need conversion tools for DNGs

- Filenames… renamed- “DÈRP’s b.123(4).jpg” → sh*ty-file-renamer.pl → “DRPs_b_123_4_.jpg”

- Files themselves…- sometimes go missing → check/compare logs between EMu & filer- Duplicates → backlog deleted + uniqueness on ChaMD5sum

- Keywords… mapped Audubon Core reverse attachments- there is less yelling...for now...

[the silence of the DAMS?]

2019 - Setting up the DAMS server...So far, we’ve:

- set up the DAMS server (“Palmer”)- pointed it at our EMu server (“Shackleton)- [...testing stuff… (users / eevents / photo archives workflow?) ]

All clear going forward?Or just the calm before the next storm?

(And who exactly is ‘Hannibal Lecter’ in this scenario?)

(And how many different misspellings of “DAMS” will there be?)

DD

Citations/Sources- Clarice EMu / JodiEMu Foster - https://animals.sandiegozoo.org/animals/emu- Nathaniel B Palmer - - Ernest J Shackleton - - Shackleton crew with puppies - - Silence of the Lambs poster - © Orion Pictures - Screaming Lambs - https://twitter.com/screaminglxmb - Screaming Charlton Hestons - Soylent Green - © Metro-Goldwyn-Mayer- Death’s Head Moth - http://twmoth.tesri.gov.tw/peo/MothInfo/V19-20150515-174

- (Acherontia lachesis - occurrenceID: V19-20150515-174)

Questions, Comments, Thought?(No real lambs were harmed in the making of this talk.)

Kate WebbinkSharon Grant

Janeen JonesPete Herbst

Rob ZschernitzNorth American User Conference

2019

the silence of the DAMS

Jodie Foster = EMu?

DAMu file types...