Government Legislative Data, The Other Great White Fail Whale & How To Avoid It OSCON 2011.

Government Legislative Data, The Other Great White Fail Whale & How To Avoid It

http://vimeo.com/3841614

OSCON 2011

Presenters

Jared WilliamsNew York State Senate

Graylin KimNew York State Senate

Noel Hidalgo, aka @noneck

OSCON 2011

History of Open Government at NY State Senate

OSCON 2011

OSCON 2011

OSCON 2011

OSCON 2011

OSCON 2011

OSCON 2011

OpenLeg "Yahoo Edition"

OSCON 2011

OpenLeg Mobile Edition

OSCON 2011

OpenLeg v2 prototype

OSCON 2011

Open Legislation v2 "Google Edition"

OSCON 2011

OSCON 2011

OSCON 2011

How does it work?

OSCON 2011

OSCON 2011

Pull - How we get the data

 2011S00607B402/01/11 ADVANCED TO THIRD READING

OSCON 2011

Bill Structure

Process

OSCON 2011

Process

We needed a buffer between pulling and pushing:• data quality• accountability

Flat files are convenient:• backups• human read/editable

It also opens the door for a few good ideas:• firehose• version control

OSCON 2011

Push

OSCON 2011

Publish

OSCON 2011

OSCON 2011

Story Time

OSCON 2011

Infinite Beta

Data Quality issues cropped up in many places

Users noticed this (and we fight for the users)• internally steps towards greater senate integration were

minimized until openleg could be proven acurate• externally users praised the idea but were clearly frustrated

with the quality of the service

We're serving NYS legislative data• bad data during initial development wasn't great but seen as

acceptable• 1.5 years after it's initial release people were getting worried

OSCON 2011

The Road Home

We had to be able to guarantee accuracy and rebuild our integrity• LBDC is the expert with this data and they've been a great

resourceo meet on a regular basiso set up a biweekly, automated diff process

• Created a simple data quality dashboard• Mutual gain from these mechanisms• In the beginning: nearly 1000 documents with issues• Currently we're hovering at ~20 known issues

o out of ~18K bills that isn't too bad!

OSCON 2011

???

OSCON 2011

Profit!

Just kidding. Its an Open Source project for the public good.

Seriously though.  Where does that leave us?

To start the public profits from the service being:• free• near-real time• linkable• a primary source of information

Integrates with internal tools and has potential for use in external tools

OSCON 2011

A Place to Disqus

We use Disqus for comments• People trust this: commenters are knowledgable, passionate

and dedicated• 3613 comments• Popular bills:

o S7234  - 136 commentso S2994 - 99 commentso A9529  - 97 comments

    (as of 7/22/11)

Created an aggregate service: BillBuzz• Follow member's legislation• receive updates with new comments• informative for constituents, GREAT for senators/staffers

OSCON 2011

Powerful Search API

Voting Correlations:

aye:(senX AND senY) AND nay:(senZ) 

Bills Passed 

actions:signed AND sponsor:senX 

Bills Abandoned 

stricken:true AND full:term 

Null values 

otype:bill AND NOT summary:[A* TO Z*]

OSCON 2011

Open Legislation on the Wild

• Increasing Usage 

• Google likes it (#1 source of traffic) 

• Linked from various different placeso twittero facebooko NYTimes

OSCON 2011

Thank you...

Now, Questions and Answers...

OSCON 2011