Top Banner
Supporting Collaborative Online Arguing Jodi Schneider Frontiers and Connections between Argumentation Theory and Natural Language Processing BiCi seminar series, Bertinoro (Forlì- Cesena), Italy 24 July 2014
41

Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Aug 29, 2014

Download

Technology

jodischneider

Seminar talk: Frontiers and Connections between Argumentation Theory and Natural Language Processing
BiCI seminar series, Bertinoro (Forlì-Cesena), Italy, 2014-07-24
http://www-sop.inria.fr/members/Serena.Villata/BiCi2014/program/index.html

Topics:
- Examples of argumentation support
- Supporting Collaborative Online Arguing
- Structuring scientific argument: Micropublications model

My paper covers related (but not identical) ground:
http://jodischneider.com/pubs/frontiersargnlp2014.pdf


Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Supporting Collaborative Online Arguing

Jodi Schneider

Frontiers and Connections between Argumentation Theory and Natural Language ProcessingBiCi seminar series, Bertinoro (Forlì-Cesena), Italy24 July 2014

Page 2: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Topics

• Examples of argumentation support• Supporting Collaborative Online Arguing• Structuring scientific argument:

Micropublications model

Page 3: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Create new spaces for civic debatehttp://Consider.It

"Supporting reflective public thought with ConsiderIt." CSCW 2012Travis Kriplean, Jonathan Morgan, Deen Freelon, Alan Borning, and Lance Bennett.

Page 4: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Create new spaces for civic debatehttp://Consider.It

"Supporting reflective public thought with ConsiderIt." CSCW 2012Travis Kriplean, Jonathan Morgan, Deen Freelon, Alan Borning, and Lance Bennett.

Page 5: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Choosetask-

appropriateformalisms:

SEAS

"Template-based structured argumentation." In Knowledge Cartography, Springer London, 2008.John Lowrance, Ian Harrison, Andres Rodriguez, Eric Yeh, Tom Boyce, Janet Murdock, Jerome Thomere, and Ken Murray.

Page 6: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Choosetask-

appropriateformalisms

discoursedb.org

Page 7: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Support incremental

formalization

argunet.org

Page 8: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

ClimateCoLabclimatecolab.org

http://www.climatecolab.org/ Circa 2010

Page 9: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Topics

• Examples of argumentation support• Supporting Collaborative Online Arguing• Structuring scientific argument:

Micropublications model

Page 10: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

General Approach (from Informatics)

1. Analyze requirements2. Consider which argumentation models to use3. Build a prototype support tool4. Evaluate and iterate

Page 11: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

500 Wikipedia debates each week: Should we delete this article?

Page 12: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Requirements: Support argumentation tasks

• Convince others of your position, using community norms

• Determine the overall consensus decision

Page 13: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Compare two argumentation theories

• Walton’s Argumentation Schemes (Walton, Reed, and Macagno 2008)

– Informal argumentation (philosophical & computational argumentation)

– Identify & prevent errors in reasoning (fallacies)– 60 patterns

• Factors/Dimensions Analysis (Ashley 1991; Bench-Capon and Rissland, 2001)

– Case-based reasoning– E.g. factors for deciding cases in trade secret law,

favoring either party (the plaintiff or the defendant).

Page 14: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Walton’s Argumentation SchemesExample Argumentation Scheme: Argument from Rules – “we apply rule X”

Critical Questions1. Does the rule require carrying out this type of action?

2. Are there other established rules that might conflict with or override this one?

3. Are there extenuating circumstances or an excuse for noncompliance?

Walton, Reed, and Macagno 2008

Page 15: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

“Rule” Argumentation Scheme

“Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups” CSCW 2013

Page 16: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

“Evidence” Argumentation Scheme

“Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups” CSCW 2013

Page 17: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Evidence + Rule -> Conclusion

“Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups” CSCW 2013

Page 18: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Supporting Tasks with Walton

• Convince others of your position, using community norms– To win an argument, use popular schemes:

• Argument from Evidence to Hypothesis (19%)• Argument from Rules (17%)

• Determine the overall consensus decision– Ask critical questions to check others' arguments

Page 19: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Factors/Dimensions Analysis

• Factors (case-based reasoning)– All or nothing• Either present ("applicable") or absent• When present, a factor always favors the same side

• Dimensions– More complex/subtle• Can be applicable to a varying degree ("sliding scale")• Favor plantiff on one extreme; defendant on the other

Ashley 1991; Bench-Capon and Rissland, 2001

Page 20: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Example factors analysis (Aleven 1997)

Aleven 1997

Page 21: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Wikipedia Factors Analysis

Factors determined by iterative annotation

4 Factors cover– 91% of comments– 70% of discussions

“Other” as 5th catchall

Factor Example (used to justify `keep')

Notability Anyone covered by another encyclopedic reference is considered notable enough for inclusion in Wikipedia.

Sources Basic information about this album at a minimum is certainly verifiable, it's a major label release, and a highly notable band.

Maintenance …this article is savable but at its current state, needs a lot of improvement.

Bias It is by no means spam (it does not promote the products).

**Other I'm advocating a blanket "hangon" for all articles on newly-drafted players

Page 22: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Wikipedia Factors Analysis

Factors determined by iterative annotation

4 Factors cover– 91% of comments– 70% of discussions

“Other” as 5th catchall

Factor Example (used to justify `keep')

Notability Anyone covered by another encyclopedic reference is considered notable enough for inclusion in Wikipedia.

Sources Basic information about this album at a minimum is certainly verifiable, it's a major label release, and a highly notable band.

Maintenance …this article is savable but at its current state, needs a lot of improvement.

Bias It is by no means spam (it does not promote the products).

**Other I'm advocating a blanket "hangon" for all articles on newly-drafted players

Page 23: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Wikipedia Factors AnalysisFactor Example (used to justify 'keep') Example (used to justify 'delete'Notability Anyone covered by another

encyclopedic reference is considered notable enough for inclusion in Wikipedia.

There is simply no coverage in reliable sources to establish notability.

Sources Basic information about this album at a minimum is certainly verifiable, it's a major label release, and a highly notable band.

There are no independent secondary sources (books, magazine articles, documentaries, etc.) about her.

Maintenance …this article is savable but at its current state, needs a lot of improvement.

Too soon for a page likely to be littered with rumour and speculation.

Bias It is by no means spam (it does not promote the products).

The article seems to have been created by her or her agent as a promotional device.

**Other I'm advocating a blanket "hangon" for all articles on newly-drafted players

it appears to be original research by synthesis

Deletion Discussions in Wikipedia: Decision Factors and Outcomes. WikiSym 2012.

Page 24: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Supporting Tasks with Factors

• Convince others of your position, using community norms– To win an argument, talk about the right topics

• Notability, Sources, Maintenance, Bias

• Determine the overall consensus decision– Group messages by factor– Summarize prevalence

Page 25: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Factor-based Summarization

Page 26: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Argument Schemes vs. Factors?

• Argument Schemes (kappa=.48)Details of how to put together an argument– Could support WRITING detailed arguments– Critical Questioning

• Factors (kappa=.64-.82, based on factor)Topics of discussion– Basic support for writing arguments– Summarization supports decision-making

Page 27: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Argument prevalence depends on the corpus

• Wikipedia– Argument from Evidence to Hypothesis (19%)– Argument from Rules (17%)

• Arucaria– Argument from example (38%)– Argument from cause to effect (27%)– Practical reasoning (14%)– Argument from consequences (11%)– Argument from verbal classification (10%)

Page 28: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Topics

• Examples of argumentation support• Supporting Collaborative Online Arguing• Micropublications model: Structuring

scientific argument

Page 29: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Micropublications

Page 30: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Model Data, Methods, Materials, References

Micropublications: a Semantic Model for Claims, Evidence, Arguments and Annotations in Biomedical CommunicationsTim Clark, Paolo N. Ciccarese, Carole A. Goblehttp://arxiv.org/abs/1305.3506

Page 31: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Direct Annotation with Domeo

http://swan.mindinformatics.org/ Paolo N Ciccarese

Page 32: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Micropublication: Claim + Support (e.g. Attribution)

Micropublications: a Semantic Model for Claims, Evidence, Arguments and Annotations in Biomedical CommunicationsTim Clark, Paolo N. Ciccarese, Carole A. Goblehttp://arxiv.org/abs/1305.3506

Page 33: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Constructs claim-argument network across scientific papers

Micropublications: a Semantic Model for Claims, Evidence, Arguments and Annotations in Biomedical CommunicationsTim Clark, Paolo N. Ciccarese, Carole A. Goblehttp://arxiv.org/abs/1305.3506

Page 34: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Argumentation Mining papersArguing on Wikipedia • “Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online

Task Groups” CSCW 2013.• “Deletion Discussions in Wikipedia: Decision Factors and Outcomes” WikiSym2012.Arguing in Social Media• “Dimensions of Argumentation in Social Media" EKAW 2012• “Why did they post that argument? Communicative intentions of Web 2.0 arguments.” Arguing on the

Web 2.0 at ISSA 2014Arguing in Reviews• “Identifying Consumers' Arguments in Text” SWAIE 2012• “Semi-Automated Argumentative Analysis of Online Product Reviews" COMMA 2012• “Arguing from a Point of View” Agreement Technologies 2012Structuring Arguments on the Social Semantic Web• “A Review of Argumentation for the Social Semantic Web” Semantic Web – Interoperability, Usability,

Applicability, 2013.• “Identifying, Annotating, and Filtering Arguments and Opinions in Open Collaboration Systems" 2013

Thesis: purl.org/jsphd• “Modeling Arguments in Scientific Papers” at ArgDiaP 2014

http://jodischneider.com/jodi.html

Page 35: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Argumentation mining today

• No unified vision of the field. Multiple:– Interrelated problems– Application domains– Tools handling one aspect of annotation

• Few corpora• Need for– Common definition(s) of argumentation– "Challenge problems"– Shared corpora– Applications

Page 36: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24
Page 37: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Example: "Stop at a red light"

1. Does the rule require carrying out this type of action?Were you driving a vehicle?

2. Are there other established rules that might conflict with or override this one?Did a police officer direct you to continue without stopping?

3. Are there extenuating circumstances or an excuse for noncompliance?Were you driving an ambulance with its siren on?

Critical Questions from Argument from Rules based on Walton, Reed, and Macagno 2008

Page 38: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

None of Wikipedia's top-used schemes are prevalent in Arucaria.

Classifying Arguments by Scheme. Vanessa Wei Feng. Master's thesis, Toronto, 2010.

Page 39: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Goal: large-scale arguing

• Search for issues, claims, and opinion clusters• Link to evidence when writing your opinion• Publish and navigate claims networks

Page 40: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Online argumentation support interfaces can:

• Promote "listening" in online conversations• Support incremental formalization• Slice and dice the views• Collect crisp examples• Support distributed sensemaking

Page 41: Supporting collaborative online arguing -- talk for BICI Frontiers and Connections between Argumentation Theory and Natural Language Processing 2014-07-24

Argumentation mining could be the basis for support tools

• Help participants write persuasive arguments– How: provide personalized feedback on drafts– Requires: knowing which arguments are accepted;

identifying argumentation in a drafts • Find weaknesses in others’ arguments– How: suggest & instantiate relevant critical questions– Requires: identifying argumentation schemes

• Summarize the overall conclusions of the debate– How: identify the winning and losing rationales– Requires: identifying rationales and contradictions