Strata NY: Best Practices for Publishing Data

F I N D A N D U N D E R S TA N D D ATA

October, 2012Hjalmar Gislason, founder & CEO - [email protected]

Best Practices for

Publishing Data

Founder and CEO

HjalmarGislason

Twitter: @datamarketSlides: http://blog.datamarket.com/

HeavyData Consumers

Providers of

Data Delivery Technology

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Computers Humans

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Computers

• Structure

Humans

• Understand and use

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Computers

• Structure

Humans

• Understand and use

1. Simple formats2. Indexes, unique IDs and meta-data3. FAQs and feedback channels

Publishing for Computers

"Don't anthropomorphize computers - they hate it."

- Unknown

Simple Formats

Simple Formats

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Simple Formats:Tim Berners-Lee’s Five Stars

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Simple Formats:You lost me at “Semantics”

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Standards will emerge and there will be more and more of them

• RDF•OData vs. GData•DSPL• SDMX

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Indexes, unique ids and meta-data

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Indexes, unique ids and meta-data

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Indexes, unique ids and meta-data

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Indexes, unique IDs and meta-data

• Must: Unique ID, Title, Last updated• Should: Meta-data

• Why?• No need for scraping

• Less load on your end• Ensures full coverage• Ensures content removal and updates

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Indexes, unique IDs and meta-data

• Hard to emphasize enough!

• Unique IDs for everything: Datsets, columns, entities, ...

• Why?• Continuity: A small change for a man = giant leap for a

computer

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Indexes, unique IDs and meta-data

• Any relevant contextual information• URL(s), descriptions, methodology, next updated, authors,

keywords, units, license information, ...

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

FAQs and feedback channels

#1 reason for not publishing data:

“There are errors in the data and I don'twant others to discover them”

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

FAQs and feedback channels

#1 reason for not publishing data:

“There are errors in the data and I dowant others to discover them”

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

FAQs and feedback channels

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

FAQs and feedback channels

1. Simple formats2. Indexes, unique IDs and meta-data3. FAQs and feedback channels

Publishing for Computers

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Computers

• Structure

Humans

• Understand and use

1. Search / Discovery2. Visualization3. Download

Publishing for Humans

Search / Discovery

• Requirements differ from web/text search• A lot less textual content to base on

• Synonyms, dictionaries, autocomplete• But (hopefully) good meta-data = facets and filtering

• Give people ways to browse• Categories vs. tags vs. search• Serendipity: Random, related, interesting...

Search / Discovery

Visualize

109 columnsx

340 lines=

37.060 cells

Visualize

• What you should offer depends on the data

• Statistical data• Focus on the most common charts and get them right• Do NOT invent new visualizations or chart types

• Use standards compatible technologies• No Flash!• Charting and visualization libraries

Visualize

Visualize

Download

• Make it easy to use your data outside your tools• Play nicely with those providing functionality beyond what

you can offer: Tableau, R, SAS, MathLab, Mathematica, SPSS, ...

• Provide downloads in the formats most commonly used by your users:• Raw data: Excel, CSV, feeds (R, Excel live feeds, APIs)• Charts and visualizations: Bitmap, vector, PPT, embeds?

| BEST PRACTICES for PUBLISHING DATA | Hjalmar Gislason, [email protected] | October 2012

Computers

• Structure• Simple formats• Indexes, unique IDs and

meta-data• FAQs and feedback

channels

Humans

• Understand and use• Search / Discovery• Visualization• Download

F I N D A N D U N D E R S TA N D D ATA

Twitter: @datamarket · Facebook: DataMarket · E-mail: [email protected]

Hjalmar Gislason, founder & CEO