Text Classification in Python – using Pandas, scikit-learn, IPython Notebook and matplotlib

Text Classification in Python – using Pandas, scikit-learn, IPython

Notebook and matplotlib Jimmy Lai

r97922028 [at] ntu.edu.tw http://tw.linkedin.com/pub/jimmy-lai/27/4a/536

2013/02/17

Critical Technologies for Big Data Analysis

• Please refer http://www.slideshare.net/jimmy

_lai/when-big-data-meet-python for more detail.

2

Collecting

User Generated Content

Machine Generated Data

Storage

Computing

Analysis

Visualization

Infrastructure C/JAVA

Python/R

Javascript

Fast prototyping - IPython Notebook

• Write python code in browser:

– Exploit the remote server resources

– View the graphical results in web page

– Sketch code pieces as blocks

– Refer http://www.slideshare.net/jimmy_lai/fast-data-mining-flow-

prototyping-using-ipython-notebook for more introduction.

Text Classification in Python 3

Demo Code

• Demo Code: ipython_demo/text_classification_demo.ipynb in https://bitbucket.org/noahsark/slideshare

• Ipython Notebook: – Install

$ pip install ipython

– Execution (under ipython_demo dir)

$ ipython notebook --pylab=inline

– Open notebook with browser, e.g. http://127.0.0.1:8888

Text Classification in Python 4

Machine learning classification

• 𝑋𝑖 = [𝑥1, 𝑥2, … , 𝑥𝑛], 𝑥𝑛 ∈ 𝑅

• 𝑦𝑖 ∈ 𝑁

• 𝑑𝑎𝑡𝑎𝑠𝑒𝑡 = 𝑋, 𝑌

• 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑟 𝑓: 𝑦𝑖 = 𝑓(𝑋𝑖)

Text Classification in Python 5

Text classification

Text Classification in Python 6

Feature Generation

Feature Selection

Classification Model Training

Model Parameter

Tuning

From: [email protected] (zhenghao yeh) Subject: Re: Newsgroup Split Organization: University of Southern California, Los Angeles, CA Lines: 18 Distribution: world NNTP-Posting-Host: caspian.usc.edu In article <[email protected]>, [email protected] (Chris Herringshaw) writes: |> Concerning the proposed newsgroup split, I personally am not in favor of |> doing this. I learn an awful lot about all aspects of graphics by reading |> this group, from code to hardware to algorithms. I just think making 5 |> different groups out of this is a wate, and will only result in a few posts |> a week per group. I kind of like the convenience of having one big forum |> for discussing all aspects of graphics. Anyone else feel this way? |> Just curious. |> |> |> Daemon |> I agree with you. Of cause I'll try to be a daemon :-) Yeh USC

Dataset: 20 newsgroups

dataset

Text Classification in Python 7

Text

Structured Data

Dataset in sklearn

• sklearn.datasets

– Toy datasets

– Download data from http://mldata.org repository

• Data format of classification problem

– Dataset

• data: [raw_data or numerical]

• target: [int]

• target_names: [str]

Text Classification in Python 8

Feature extraction from structured data (1/2)

• Count the frequency of keyword and select the keywords as features: ['From', 'Subject', 'Organization', 'Distribution', 'Lines']

• E.g. From: [email protected] (where's my thing)

Subject: WHAT car is this!?

Organization: University of Maryland, College Park

Distribution: None

Lines: 15

Text Classification in Python 9

Keyword Count Distribution 2549 Summary 397 Disclaimer 125 File 257 Expires 116 Subject 11612 From 11398 Keywords 943 Originator 291 Organization 10872 Lines 11317 Internet 140 To 106

Feature extraction from structured data (2/2)

• Separate structured data and text data

– Text data start from “Line:”

• Transform token matrix as numerical matrix by sklearn.feature_extractionDictVectorizer

• E.g.

[{‘a’: 1, ‘b’: 1}, {‘c’: 1}] => [[1, 1, 0], [0, 0, 1]]

Text Classification in Python 10

Text Feature extraction in sklearn

• sklearn.feature_extraction.text

• CountVectorizer

– Transform articles into token-count matrix

• TfidfVectorizer

– Transform articles into token-TFIDF matrix

• Usage:

– fit(): construct token dictionary given dataset

– transform(): generate numerical matrix

Text Classification in Python 11

Text Feature extraction

• Analyzer – Preprocessor: str -> str

• Default: lowercase

• Extra: strip_accents – handle unicode chars

– Tokenizer: str -> [str] • Default: re.findall(ur"(?u)\b\w\w+\b“, string)

– Analyzer: str -> [str] 1. Call preprocessor and tokenizer

2. Filter stopwords

3. Generate n-gram tokens

Text Classification in Python 12

Text Classification in Python 13

Feature Selection

• Decrease the number of features:

– Reduce the resource usage for faster learning

– Remove the most common tokens and the most rare tokens (words with less information):

• Parameter for Vectorizer: – max_df

– min_df

– max_features

Text Classification in Python 14

Classification Model Training

• Common classifiers in sklearn:

– sklearn.linear_model

– sklearn.svm

• Usage:

– fit(X, Y): train the model

– predict(X): get predicted Y

Text Classification in Python 15

Cross Validation

• When tuning the parameters of model, let each article as training and testing data alternately to ensure the parameters are not dedicated to some specific articles.

– from sklearn.cross_validation import KFold

– for train_index, test_index in KFold(10, 2):

• train_index = [5 6 7 8 9]

• test_index = [0 1 2 3 4]

Text Classification in Python 16

Performance Evaluation

• 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑡𝑝

𝑡𝑝+𝑓𝑝

• 𝑟𝑒𝑐𝑎𝑙𝑙 =𝑡𝑝

𝑡𝑝+𝑓𝑛

• 𝑓1𝑠𝑐𝑜𝑟𝑒 = 2𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛×𝑟𝑒𝑐𝑎𝑙𝑙

𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑟𝑒𝑐𝑎𝑙𝑙

• sklearn.metrics

– precision_score

– recall_score

– f1_score

Text Classification in Python 17

Source: http://en.wikipedia.org/wiki/Precision_and_recall

Visualization

1. Matplotlib

2. plot() function of Series, DataFrame

Text Classification in Python 18

Experiment Result

• Future works:

– Feature selection by statistics or dimension reduction

– Parameter tuning

– Ensemble models

Text Classification in Python 19