Nhu Nhom8 Chuong3 PPNCKH

B GIO DC V O TO

My hc v vic dy my tnh phn loi vn bn ting VitKhoa hc my tnh

TRNG I HC TN C THNGKHOA HC MY TNH

PHAN NGUYN CNG

CNGLUN VN THC S KHOA HC MY TNH

Thnh ph H Ch Minh, nm hc: 2015

TRNG I HC TN C THNGKHOA HC MY TNH

PHAN NGUYN CNG

Chuyn ngnh: Khoa hc my tnhM chuyn ngnh:Ngi hng dn khoa hcPGS. TS. Nguyn Thi TrungThnh ph H Ch Minh, nm hc: 2015

TRNG I HC TN C THNG CNG HA X HI CH NGHA VIT NAM

KHOA O TO SAU I HC c lp T do Hnh phc

---o0o---((Tp. HCM, ngy 8 thng 3 nm 2015NHIM V LUN VN THC SH tn hc vin: Phan Nguyn Cng Gii tnh: Nam Ngy sinh: 29/10/1988Ni sinh: Bn TreM hc vin: 910768MI. Tn ti: II. Nhim v lun vn: Lm r l thuyt my hc s p dng trong vic dy my tnh phn loi cc vn bn ting Vit.

Hiu r thut ton cy quyt nh v th vin weka h tr cho vic dy my tnh phn loi cc vn bn. Quan trng nht l tm hiu cc thut ton dng loi b stopword thc hin vic phn loi vn bn ting Vit.III. Ngy giao nhim v: 8/2/2014

IV. Ngy hon thnh nhim v: 15/3/2014

V. H v tn cn b hng dn: PGS. TS Nguyn Thi Trung

Ni dung v cng lun vn thc s c Hi ng chuyn ngnh thng qua.

CN B HNG DNCH NHIM B MNKHOA QL CHUYN NGNH

Ph lc4Ph lc

5Chng 1 M u

5I.Tn ti

5II.t vn v mc ch nghin cu

6III.Tng quan ti liu:

7IV.Mc tiu nghin cu:

8V.i tng v phm vi nghin cu

8VI.Phm vi nghin cu:

9Chng 2: C S L THUYT

9I.L thuyt my hc

11II.Weka

111.S lc v weka

122.Cc ch nng chnh

16III.Thut ton cy quyt nh

161.nh ngha:

162.im mnh ca cy quyt nh:

173.V d

20Chng 3: Phng php nghin cu

20I.S v hot ng ca h truy hi thng tin ting Vit

21II.Phn tch cu v t ting Vit

211.Phn tch cu:

212.Phn tch t:

21III.Xy dng cng c h tr tin x l

221.Thut ton 1: Chun tp tin ting Vit

222.Thut ton 2: Chuyn cu ting Vit sang dng k s

233.Thut ton 3: Chuyn cu dng k s sang ting Vit

24IV.Thc nghim

26V.Kt lun

28Chng 4: D kin kt qu t c v ngha

28VI.D kin kt qu nghin cu t c

28VII. ngha khoa hc ca ti

28VIII. ngha thc tin ca ti

29Chng 5: K hoch thc hin

Chng 1 M uI. Tn ti

II. t vn v mc ch nghin cu

Trong cuc sng i thng ca chng ta, vic ng dng cng ngh thng tin mang li rt nhiu li ch to ln trong hu ht cc lnh lc ngnh ngh khc nhau (gio dc, truyn thng, qun l, y t, .) qua cc phn mm ng dng. Cc phn mm c lp trnh t cc ngn ng lp trnh khc nhau nh C, C++, Java, Cng vic lp trnh c hiu nh l s dng mt ngn ng lp trnh no din t nhng iu chng ta mun my tnh lm. Tuy vy, khng phi l tt c nhng g cng ngh thng tin c th lm c m hn th na con ngi vn mong rng cng ngh thng tin c th cung cp nhng ng dng c th t suy lun v a ra kt qu t vic suy lun.

Trong nhng nm gn y, ngnh cng ngh thng tin trn ton th gii pht trin mnh m vi tc rt nhanh. Song song vi iu chng ta cng phi i mt vi mt thch thc mi l s bng n v lng thng tin. Tuy nhin, mt thc t din ra rt ph bin l mc d c mt lng d liu rt ln nhng tri thc chng ta c rt t, nhng hiu bit thc s ca chng ta v lng d liu c cn rt hn ch.

Xut pht t thc t m trong mt vi nm gn y cc nh nghin cu v ng dng tin hc phi nghin cu, tm kim nhng phng php mi khai thc nhng tri thc tm n c trong c s d liu khng l . T cui nhng nm 1980 khi nim pht hin tri thc trong c s d liu ln u tin c ni n, y l qu trnh pht hin tri thc tim n, tri thc cha bit trc v tri thc hu ch trong cc c s d liu ln.

Hin nay cng vi s pht trin ca cng ngh thng tin ngy nay, khm ph tri thc trong cc c s d liu ln l mt trong nhng lnh vc c rt nhiu nh nghin cu v ng dng tin hc c bit quan tm. Vic nghin cu nhng phng php c th t ng pht hin nhng tri thc mi trong c s d liu trn my tnh t ra thc s hu ch trong vic h tr quyt nh cho con ngi.

Vi s pht trin nhanh chng ca internet, c bit l world wide web, gip cho ngi dng chia s cc thng tin vi nhau d dng hn. Cc trang web c phn b hu ht tt c cc quc gia, m hu ht cc trang web gn nh l thng tin dng vn bn.

Mi ti liu, thng tin c ngn ng, vn phong, t vng khc nhau ph thuc vo ngun gc quc gia, ngn ng, tc gi,... v th, c th ni, thng tin trong th gii world wide web l rt phong ph v a dng. Chnh s pht trin mnh m v a dng ny gy kh khn cho ngi dng trong vic tm kim thng tin lin quan n lnh vc cn tm. p ng c nhu cu tm kim thng tin lin quan n lnh vc cn tm ca ngi dung trong kho thng tin khng l, phong ph v a dng th cn phi c mt cng c tr gip ngi dng trong qu trnh tm kim. Cng c h tr ngi dng trong qu trnh tm kim chnh l cc my tm kim (search engine).

Hin nay, cc my tm kim c pht trin nhm h tr ngi dng trong qu trnh

tm kim thng tin nh Google, Yahoo,... Cc my tm kim ny v ang h tr c lc trong vic tm kim thng tin ngi dng quan tm v kt qu c chnh xc v bao ph cao. Tuy nhin, cc my ny ch h tr tt cho cc ngn ng m khong trng l du hiu nhn dng t. Ting Vit l mt trong nhng ngn ng m khong trng khng phi l du hiu nhn dng t. Chnh iu ny lm nh hng rt ln n chnh xc v bao ph ti liu lin quan.

Mc d, cc my tm kim ny vn h tr cho ting Vit trong vic tm kim thng tin, nhng vn cn nhiu thch thc nh kt qu tr v cha nhiu ti liu m ngi dng

khng quan tm. Do , ngi dng phi tn nhiu thi gian trong qu trnh chn lc

thng tin lin quan.Trn th gii c rt nhiu thut ton khm ph tri thc bng cch phn lp v ri rc d liu nh: S dng cy quyt nh, phng php thng k, cc mng neural, thut ton di truyn,mi thut ton u c u v khuyt im ring ca n v ty vo tng trng hp, lnh vc c th m c th p dng nhng thut ton cho ph hp.

V khi th gii khai ph c nhiu tri thc hu ch th chng ta phi tn dng nhng tri thc , mt yu cu mi l dy cho my tnh hc nhng tri thc my tnh hiu v a ra nhng quyt nh ng n cho mi s vic, s kin da trn nhng tri thc c khai kh.V trong lun vn ny chng ta cng tm hiu vi lnh vc dy my hc, thut ton cy quyt nh, ng dng vic dy my hc bng(qua, v) thut ton cy quyt nh khai ph trn th vin Weka thng qua phn mm Classify_Test8III. Tng quan ti liu:M ngun ca cc my tm kim nh Google, Yahoo,... khng c cng b. Cng nh cc my tm kim Google, Yahoo th m ngun cc my tm kim h tr cho ting Vit cng khng c cng b. Do vy, cc t chc mun s dng chc nng tm kim phc v cho mc ch ring ca mnh th hu nh u phi t xy dng v pht trin my tm kim t u. xy dng hon thin mt my tm kim th rt mt nhiu cng sc cng nh ti chnh. Mt trong nhng gii php ph hp l s dng cc m ngun m v my tm kim (hoc h truy hi thng tin - Information Retrieval) cng b. H truy hi thng tin khc vi my tm kim l khng c b d tm (web crawler) [5].Xy dng my tm kim/h truy hi thng tin hin nay c rt nhiu nhm nghin cu. Cng trnh ca nhm tc gi Phc v cng s l my tm kim ting Vit, h tr tm kim thng tin thuc lnh vc cng ngh thng tin. My tm kim ny c h tr tm kim thm cc t hoc cm t gn ngha vi cu truy vn ban u da vo ontology thuc lnh vc tin hc. Trong cng trnh ca mnh, nhm tc s dng Oracle text nh ch mc, y l mt trong nhng u im ca my ny. Tuy nhin, trong qu trnh to kho ng liu (corpus) phc v cho vic nh ch mc, nhm tc gi khng loi b cc t trong danh sch stopword ting Vit, iu ny c th dn n kho ng liu nh ch mc c ni dung ln. Cng trnh ca nhm tc gi Hunh c Vit v cng s [4] l h thng tm kim cng vn ting Vit pht trin da trn m ngun m Lucene. H tm kim ny h tr hai ng dng l: ng dng th nht cho php ngi qun tr to v cp nht ch mc trn my ch; ng dng th hai cho php ngi dung tm kim trn mi trng LAN v mi trng Internet. H thng tm kim cng vn ting Vit ch h tr cc chc nng x l ting Vit trong giai on nh ch mc nh loi b cc t loi b ting Vit, chun ha,... m khng cp n vic x l ting Vit nh tch cu, tch t ting Vit trong vic nh ch mc cng nh tm kim. My tm kim VSearch ca tc gi Nh nh Thun [3] l my h tr ting Vit. My ny s dng VietSpider, VSearch Crawler sng lc, bc tch ni dung cc trang web. VietSpider, VSearch Crawler l im mnh ca VSearch nn my ny khng cn nhiu n cc k thut xp hng (pagerank) phc tp. My Vsearch s dng m ngun m Solr v l mt my tm kim thng mi nn m ngun khng c cng b. Cng trnh Sarsarn Look ca NECTEC [5,7] l my tm kim h tr ting Thi.

My ny s dng m ngun m Lucene pht trin. h tr cho ting Thi, NECTEC xy dng cng c ThaiAnalyzer h tr tin x l ting Thi. My v ang pht trin vi nhiu phin bn khc nhau, vi mi phin bn th my c b sung thm cc thnh phn nh tm kim ng ngha da trn ontology, truy vn xuyn ng Thi-Anh,...IV. Mc tiu nghin cu:

Nghin cu c thc hin nhm gii quyt cc vn sau:

Hiu r vn dy dy my hc ng dng trong cng ngh thng tin ngy cng m rng hn v su sc hn.

Hiu r thut ton cy quyt nh gip khai ph d liu c cc tri thc hu ch phc v cho vic dy my hc.

Hiu r vic phn loi ti liu ting Vit l ng dng vic dy my hc thng qua phn mm Classify_test8 gm c thut ton cy quyt nh khai ph tri thc trn th vin Weka, cc thut ton loi b stopword trong ting Vit.

V. i tng v phm vi nghin cu

i trng nghin cu l my tnh c th suy lun ra tri thc hu dng t nhng d liu c bn c hc trc.

VI. Phm vi nghin cu:

Phn tch vn my c th suy lun tri thc hu dng trong ti ny c thc hin trong phm vi nghin cu sau:

Dy my hc nh th no?

V p dng th nghim trong vic dy my hc cch phn loi vn bn n gin c trc.

Tm ra thut ton loi b stopword trong ting Vit.Chng 2: C S L THUYTI. L thuyt my hc

Alan Turing (23 thng 6, 1912 7 thng 6, 1954) l mt nh ton hc, logic hc v mt m hc ngi Anh va thng c xem l cha ca ngnh khoa hc my tnh. Php th Turing (Turing test) l mt trong nhng cng hin ca ng trong ngnh tr tu nhn to. nh ton hc li lc ngi Anh, ngi c xem l cha ca Tin hc do a ra cch hnh thc ha cc khi nim thut ton v tnh ton trn my Turingmt m hnh my tnh tru tng m t bn cht vic x l cc k hiu hnh thcc mt ng gp quan trng v th v cho tr tu nhn to (TTNT) vo nm 1950, gi l php th Turing.php th Turing (Turing test). Php th c pht biu di dng mt tr chi. Theo , c ba i tng tham gia tr chi (gm hai ngi v mt my tnh). Mt ngi (ngi thm vn) ngi trong mt phng kn tch bit vi hai i tng cn li. Ngi ny t cc cu hi v nhn cc cu tr li t ngi kia (ngi tr li thm vn) v t my tnh. Cui cng, nu ngi thm vn khng phn bit c cu tr li no l ca ngi, cu tr li no l ca my tnh th lc c th ni my tnh c kh nng "suy ngh" ging nh ngi. Php th ny c ngha rt ln v cho thy c kh nng giao tip ca my tnh vi con ngi, chnh l mt biu hin c bn ca tr tu nhn to.

Php th Turing l mt cch tr li cu hi my tnh c bit ngh khng?, c pht biu di dng mt tr chi.Hnh dung c ba ngi tham gia tr chi, mt ngi n ng (A), mt ngi n b (B) v mt ngi chi (C). Ngi chi ngi mt phng tch bit vi A v B, khng bit g v A v B (nh hai i tng n X v Y) v ch t cc cu hi cng nh nhn tr li t A v B qua mt mn hnh my tnh. Ngi chi cn kt lun trong X v Y ai l n ng ai l n b. Trong php th ny, A lun tm cch lm cho C b nhm ln v B lun tm cch gip C tm c cu tr li ng. Php th Turing thay A bng mt my tnh, v bi ton tr thnh liu C c th phn bit c trong X v Y u l my tnh u l ngi n b. Php th Turing cho rng my tnh l thng minh (qua c php th) nu nh bit cch lm sao cho C khng th chc chn kt lun ca mnh l ng. Tuy php th Turing n nay vn c xem c tm quan trng lch s v trit hc hn l gi tr thc t (v con ngi vn cha lm c my hiu ngn ng v bit lp lun nh vy), ngha rt ln ca n nm ch nhn mnh rng kh nng giao tip thnh cng ca my vi con ngi trong mt cuc i thoi t do v khng hn ch l mt biu hin chnh yu ca tr thng minh nhn to.Nhng nm u thp k 80 ca th k trc chng kin s bt u ca mt giai on qung 15 nm ca s hi sinh, bng n v thi ua quc t trong ngnh TTNT. tng c bn pht trin TTNT khi ny l s thng minh ca my tnh khng th ch da trn vic suy din logic m phi da c vo tri thc con ngi, v dng kh nng suy din ca my khai thc cc tri thc ny. Ct li caTTNT c th din gii bi cng thcTTNT = Tri thc + Suy dinT 1956 n nay l chng ng 50 nm. Trong 40 nm u c th gi tn l m c, tht bi v hi sinh. Vy hn mi nm qua TTNT ra sao?

Liu my c hiu c ting ni v vn bn ca con ngi? Hiu ngn ng l mt c trng tiu biu ca tr tu v vic lm chomy hiu c ngn ng l mt trong vi vn kh nht ca TTNT ni ring v CNTT ni chung. Ta ly th d ca Marvin Minsky nm

1992 khi l gii ti sao vn ny li kh v lnh vc ny tin chm:Xt mt t, chng hn si dy. Ngy nay khng mt my tnh no c th hiu ngha t ny nh con ngi. Ta c th ko mt vt bng mt si dy, nhng khng th y mt vt bng si dy. Ta c th

gi mt gi hng hoc th diu bng mt si dy, nhng khng th n si dy ny. Trong vi pht, mt a tr nh c th ch ra hng trm cch dng hoc khng dng mt si dy, nhng khng my tnh no c th lm vic ny. hiu ngha mt cu, my khng ch cn bit ngha tng t, m trc ht phi bit phn tch c cu ny v mt ng php. lm vic ny, i th my phi tch cu thnh cc t n l hay cm t, nhn bit chng l cc loi t g, ri xc nh cu trc ca cu, on ngha ca tng t, v gii ngha c cu. Ngn ng thng nhp nhng a ngha v iu

ny tr nn v cng kh vi my. Ly mt th d quen thuc ca cu n gin ng gi i

nhanh qu. Vi hai cch phn tch t v cm t thnh (ng gi)(i)(nhanh qu) v

(ng)(gi i)(nhanh qu), vi cc ngha khc nhau ca ng t i, ca cm t ng gi,

ta cng c dm cch hiu cu ni trn. Lm sao my t ng hiu ng ngha mt

cu ni bt k cn l mt thch thc lu di ca ngnh TTNT.Nhng g TTNT ang to ra u th k 21 ny ang tng bc i vo cuc sng hng ngy ca con ngi. Hiu r v qu kh, con ngi ang thit k v thc hin nhng chng trnh nghin cu ln v nh hng, nh khoa hc v b no. Nhng g Alan Turing ni nm 1950 vn c ngha trong th k 21 ny: Chng ta ch c th nhn thy mt qung ng ngn trc mt, nhng chng ta c th thy rt nhiu vic lm. V vi nhng g con ngi ang lm, chng ta c quyn ngh n mt ngy my s qua c php th Turing, trc khi TTNT i ht chng ng mt th k.II. Weka

1. S lc v weka

Weka l mt b cng c phn mm m ngun m rt ni ting trong lnh vc khai thc d liu v my hc c pht trin bi i hc Waikato New Zealand. N l tp hp ca rt nhiu thut ton my hc ph v mc ch khai thc d liu. Cc thut ton c th c chy trc tip trn cc b d liu a vo hoc gi t ng dng ca ngi dng (vit bng Java). Weka bao gm cc cng c tin x l d liu, phn loi, hi qui, gom nhm, khai thc lut kt hp v mt giao din ha biu din cc kt qu. Ngoi ra, Weka cn h tr cc nh nghin cu mt mi trng thc nghim tin dng pht trin m hnh, thut ton my hc mi.Trong weka: 49 cch tin x l d liu khc nhau, 76 gii thut phn loi/ hi quy, 15 thuc tnh/ tp con, 10 gii thut tm kim, 3 loi hnh biu din kt qu.

D liu nhn vo dng: .arff, .csv, SQL database.

Bc 1: import file d liu c c chng trnh thu thp d liu, giai on tin x l d liu chnh sa thay i cc thng s.

Bc 2: bc hc, tin hnh cho chng trnh hc bng tp d liu va import, trong cc gii thut m weka cung cp, chn gii thut cy quyt nh cho bi ton phn loi (J48), gii thut phn cp i vi gom cm (Hierarchical).

Bc 3: nhp d liu v ly kt qu suy lun cn c t tp d liu hc trn.

Kt qu ca hot ng phn loi s xc nh mc quan tm ca ngi dng ti lnh vc no , y chng em thc hin vi 10 lnh vc: nhp vai, nui trng, tr tu, gii tr, i khng, dn tr . flash game, kinh doanh, xy dng, th thao..Cc vic cn thc hin vi Weka:

c file d liu .

Chun ha d liu.

Chuyn nh dng v dng thch hp Weka nhn bit c kiu thuc tnh l number, catelogy

Hc t d liu .

c file test.

Xc nh phn lp ca cc ngi dng t cn c ca d liu hc. Xc nh tng user thuc cm no da vo tiu ch gom cm xc nh trc.

Giao din ha ca Weka2. Cc ch nng chnhWeka cung cp 4 mi trng chnh thao tc vi cc thut ton my hc. Cc mi trng ny bao gm: console, explorer, exparimenter, knowledage flow. Trong Explorer l mi trng th hin y cc phng tin thao tc vi d liu v cc thut ton my hc c cung cp bi Weka. Cc thnh phn cn li l tng t nh explorer.

a. Explorer

Cung cp mt giao din ngi dng tin dng cho vic thc thi cc thut ton my hc trn b d liu ngi dng a vo.

Giao din ha Explorer

b. Experimenter

Cung cp mi trng thun tin cho vic to v thc thi cc th nghim lin quan n cc thut ton my hc. V d, ngi dng c th to ra 1 th nghim tin hnh chy cc thut ton khc nhau trn cng 1 b d liu u vo, sao kim tra, so snh kt qu t c chn ra thut ton c kt qu tt nht. Thao tc trong mi trng ny hon ton tng t nh thao tc trong mi trng Explorer trnh by trn.

Giao din ha Experimenterc. KnowledgeFlow

L mt thnh phn m rng v nng cao ca Explorer. KnowledgeFlow cung cp mt mi trng l tng xy dng mt chui cc hot ng lin quan n vic thc nghim mt hay nhiu thut ton my hc. Ngi s dng c th chn cc thnh phn t toolbar nh datasource, data loader, classifier, cluster, sau t chng ln mn hnh giao din, v kt ni chng li vi nhau thnh mt Flow thng nht phn tch v x l d liu.

Di y l hnh nh minh ha cho vic xy dng mt chui thao tc hc my trong mi trng knowledage flow ca weka.

Giao din ha KnowledgeFlow

d. Simple CLI

Cung cp kh nng thc thi cc thut ton my hc bng cc thao tc dng lnh command line. Dng lnh c bn thao tc vi cc thut ton my hc l:Java [ j48.txt

Kt qu thc thi c xut ra file j48.txt

Giao din chnh ca mi trng console

III. Thut ton cy quyt nh

1. nh ngha: Cy quyt nh l biu quyt nh pht trin c cu trc dng cy: Gc: Node trn cng cy. Node trong: biu din 1 kim tra hoc 1 thuc tnh n Node l: biu din lp.2. im mnh ca cy quyt nh: Sinh ra cc quy tc hiu c: chuyn i c sang ting Anh hoc SQL. Thc thi trong lnh vc hng quy tc. D dng tnh ton trong khi phn lp. X l vi thuc tnh lin tc v ri rc. Th hin r rng nhng thuc tnh tt nht: phn chia d liu t gc.3. V d

Sn A l qun l ca mt cu lc b nh golf ni ting. Anh ta ang c rc ri chuyn cc thnh vin n hay khng n. C ngy ai cng mun chi golf nhng s nhn vin cu lc b li khng phc v. C hm, khng hiu v l do g m chng ai n chi, v cu lc b li tha nhn vin.

Mc tiu ca Sn A l ti u ha s nhn vin phc v mi ngy bng cch da theo thng tin d bo thi tit on xem khi no ngi ta s n chi golf. thc hin iu , anh cn hiu c ti sao khch hng quyt nh chi v tm hiu xem c cch gii thch no cho vic hay khng.

Vy l trong hai tun, anh ta thu thp thng tin v:Tri (outlook( nng (sunny), nhiu my (clouded) hoc ma (raining)). Nhit (temperature) bng F. m (humidity). C gi mnh (windy) hay khng.

V tt nhin l s ngi n chi golf vo hm . Sn A thu c mt b d liu gm 14 dng v 5 ct.

D liu chi golf

Cc bin c lp

Quang cnhNhit mGiChi

Nng8585khngkhng

Nng8090ckhng

m u8378khngc

Ma7096khngc

Ma6880khngc

Ma6570ckhng

m u6465cc

Nng7295khngkhng

Nng6970khngc

Ma7580khngc

Nng7570cc

m u7290cc

Sau , gii quyt bi ton ca Sn, ngi ta a ra mt m hnh cy quyt nh.

Cy quyt nh l mt m hnh d liu m ha phn b ca nhn lp (cng l y) theo cc thuc tnh dng d on. y l mt th c hng phi chu trnh di dng mt cy. Nt gc (nt nm trn nh) i din cho ton b d liu. Thut ton cy phn loi pht hin ra rng cch tt nht gii thch bin ph thuc, play (chi), l s dng bin Outlook. Phn loi theo cc gi tr ca bin Outlook, ta c ba nhm khc nhau: Nhm ngi chi golf khi tri nng, nhm chi khi tri nhiu my, v nhm chi khi tri ma.

Kt lun th nht: nu tri nhiu my, ngi ta lun lun chi golf. V c mt s ngi ham m n mc chi golf c khi tri ma.

Tip theo, ta li chia nhm tri nng thnh hai nhm con. Ta thy rng khch hng khng mun chi golf nu m ln qu 70%.

Cui cng, ta chia nhm tri ma thnh hai v thy rng khch hng s khng chi golf nu tri nhiu gi.

V y l li gii ngn gn cho bi ton m t bi cy phn loi. Sn cho phn ln nhn vin ngh vo nhng ngy tri nng v m, hoc nhng ngy ma gi. V hu nh s chng c ai chi golf trong nhng ngy . Vo nhng hm khc, khi nhiu ngi s n chi golf, anh ta c th thu thm nhn vin thi v ph gip cng vic.

Kt lun l cy quyt nh gip ta bin mt biu din d liu phc tp thnh mt cu trc n gin hn rt nhiu.Chng 3: Phng php nghin cu( chuong 3 nay thay bang cai khac dc ko chi , no noi ve phan loai tieng viet so minh ko biet, neu ko thi de nguyen cung dc)I. S v hot ng ca h truy hi thng tin ting Vit

xy dng h truy vn thng tin ting Vit, chng ti xut m hnh nh sau v h thng tm gi l vnIR.

M hnh h truy hi thng tin ting Vit vnIR

M hnh thc hin theo 2 giai on nh sau:

Giai on 1: nh ch mc (Indexing)

giai on ny, tp cc ti liu c cng c VietAnalyzer chun ha thnh tp cc ti liu chun ting Vit (tp ti liu chun gm cc cu dng m Unicode v khng cha stopword ting Vit); cc ti liu chun ting Vit l u vo ca giai on nh ch mc (Indexing).

Cc ti liu chun ting Vit c thc hin bng thut ton chun tp tin ting Vit.

Giai on 2: Tm kim v tr kt qu (Searching)

Da trn cu truy vn c ngi dng a vo, cng c VietAnalyzer s thc hin tch t v chuyn sang dng k s. Tip theo, thc hin tm kim cc ti liu lin quan n cu truy vn ca ngi dng dng k s.

Cu truy vn chuyn sang dng k s c thc hin bng thut ton Chuyn cu ting Vit sang dng k s. Kt qu ca giai on ny l mt tp ti liu KQ c vnIR nh gi ng vi cu truy vn.

II. Phn tch cu v t ting Vit

1. Phn tch cu:

Phn tch cu da vo du bo hiu kt thc cu (du hi, chm than) hoc du chm cu. Tuy nhin, do tnh nhp nhng ca du bo hiu kt thc cu nn vic phn tch cu gy khng t kh khn [1]. Hin nay, cc cng trnh phn tch cu s dng cc phng php: TBL, Maxium Entropy,

i vi ting Vit, cc cng trnh phn tch cu c cng b ca cc nhm tc gi Nguyn Cm T,

2. Phn tch t:

Ting Vit l mt trong nhng ngn ng m khong trng khng phi l du hiu nhn dng t m ch mang ngha phn tch cc m tit vi nhau [1]. V d: t t nc c to ra t 2 m tit t v nc, c 2 m tit ny u c ngha ring kho ng c lp, nhng khi ghp li s mang mt ngha khc.

Chnh v th, phn tch t ting Vit l mt trong nhng giai on nh hng rt ln n kt qu cc bi ton x l ngn ng t nhin khc nh phn loi vn bn, tm tt vn bn,Do vy, bi ton ny c rt nhiu nhm nghin cu, cng b v kt qu c chnh xc ngy cng cao nh cng trnh ca nhm tc gi Phan Th Ti, nhm tc gi inh in, nhm tc gi Nguyn Cm T,

III. Xy dng cng c h tr tin x l ( phan nay co can thiet ko vi thay no noi ko hieu va code rac roi)H truy hi thng tin c cc nhim v sau: m hnh biu din vn bn, thut ton tm kim so trng t kha v c ch lc kt qu truy vn [6,8]. Lucene [10] l m ngun m c y cc chc nng ca mt h truy hi thng tin. Do vy, chng ti k tha ton b cc chc nng cng nh cu trc ca Lucene. Tuy nhin, Lucene khng h tr cho ting Vit. Do , cn phi c cng c h tr tin x l ting Vit, cng c ny c chng ti xy dng v tm gi l VietAnalyzer.

Cng c VietAnalyzer gm cc chc nng nh sau:

Chc nng 1: Chun ha cc tp tin sang dng m Unicode.

Chc nng 2: Phn tch ti liu thnh tp cc cu.

Chc nng 3: Phn tch cu thnh tp cc t.

Chc nng 4: Loi b cc t trong danh sch stopword ting Vit.

Chc nng 5: Chuyn cu ting Vit sang cu dng k s.

Chc nng 6: Chuyn cu dng k s sang cu ting Vit.

Cc thut ton h tr cng c VietAnalyzer gm VnFNF, VnS2DS, DS2VnS. Cc thut ton c trnh by chi tit nh sau:

Thut ton VnFNF h tr chc nng 1, 2, 3 v 4 ca cng c VietAnalyzer. Kt qu thut ton l tp cc tp tin chun (tp tin chun gm cc cu dng m Unicode v khng cha stopword ting Vit).

1. Thut ton 1: Chun tp tin ting Vit

(Vietnamese File Normal Formalize VnFNF)

Nhp: Tp ti liu D = {di}, i = 1..n

Xut: Tp ti liu D = {di}, i = 1..n gm cc ti liu chun ting Vit

Phng php

1. Vi mi ti liu di thuc D, i = 1..n

2. Chun ha di;3. Phn tch di thnh tp cu {Sj}, j = 1..m

4. Vi mi cu Sj thuc di5. Sj ( VnS2DS(Sj);

6. Lu Sj vo di7. Quay li 4

8. Quay li 1

9. Return D;

Thut ton VnS2DS h tr vic chun tp tin ting Vit. Kt qu ca thut ton l cu ting Vit dng k s.

2. Thut ton 2: Chuyn cu ting Vit sang dng k s

(Vietnamese Sentence to Digital String VnS2DS)

Nhp: s l cu ting Vit

Xut: s l cu c chuyn t ting Vit sang dng k s

Phng php:

1. Phn tch s thnh tp cc t {wj}, j = 1..m2. s ( ;3. Vi mi t wj thc hin:

4. Nu wj l stopword th loi b;

5. Ngc li: chuyn wj sang dng k s v lu vo wj;

6. s ( s + wj;

7. Quay li 3

8. Tr v s;

Thut ton DS2VnS h tr chc nng 6 ca cng c VietAnalyzer. Kt qu ca thut ton l cu dng ting Vit.

3. Thut ton 3: Chuyn cu dng k s sang ting Vit

(Digital String to Vietnamese Sentence DS2VnS)

Nhp: s l cu dng k s

Xut: s l cu c chuyn t dng k s sang ting Vit

Phng php:

1. AA ( {b, c, f, , x, w, z}; //tp k t khng du

2. s ( ; i = 1;

3. Loi b cc du gch di trong s;

4. Trong khi i nh hn chiu di ca s:

5. Nu k t ci khng thuc AA v khc rng th:

6. Sao chp 3 k t t v tr th i v lu vo t;

7. Chuyn t sang ting Vit v lu vo t;

8. s ( s + t; i+ = 3;

9. Ngc li: s ( s + ci; i+ +;10. Quay li 4;

11. Tr v s;

Trong :

Stopword ting Vit: , , , , a ha, a l, i,

M quy c chuyn t ting Vit sang k s: = a03; = a05, = a07, = a09, = a11, = a13,

Vi cch biu din cc k t c bit nh trn th dung lng tp tin sau khi m ha s gim i mt na so vi cch biu din h Hexa 6 k t nh Lucene biu din cho cc ngn ng nh Hy Lp, Sc,

V d minh ha:

Thut ton VnS2DS: Cho cu ting Vit: b nh my tnh mi

Kt qu phn tch t gm: b_nh; my_tnh; mi

Cc t dng k s: bo23_nho29; ma05y_ti05nh; mo29i

Cc cu dng k s: bo23_nho29 ma05y_ti05nh mo29i

Thut ton DS2VnS:

Cu dng k s: bo23_xu19_ly05_trung_ta25m Intel mo29i

Kt qu cu ting Vit: b x l trung tm Intel mi

IV. Thc nghim

Kho ng liu:

Bng 1: Thng k kho ng liu

K hiuNgun ti liuS lng (tp tin)Dung lng (MB)

D3Wikipedia ting Vit, Th loi: My tnh925,8

D4Khoa hc cng ngh (Lin hip cc hi khoa hc v k thut Tp H Ch Minh), Mc lm bn vi my vi tnh1566,63

D5Thng tin cng ngh, Mc my tnh66046,0

D6Tin nhanh CNTT TT (ICT News) Mc phn cng60934,2

Hin nay, ting Vit vn cha c mt kho ng liu chun nh TREC c cng b. Do , chng ti phi xy dng mt kho ng liu thuc lnh vc my tnh. Kho d liu c ly t cc trang: Wikipedia ting Vit, Khoa hc cng ngh, Thng tin cng ngh v Tin nhanh, Cng ngh thng tin truyn thng. Sau bc tch chng t dng trang web (.html, .htm) sang dng tp tin vn bn (.txt). Kho ng liu c dng nh ch mc phc v tm kim v c thng k theo s lng v dung lng nh bng 1.

Kt qu thc nghim:

Bng 2. Thng k chnh xc, bao ph ca vnIR

K hiuCu truy vnS ti liu tm cS ti liu tm c lin quanTng s ti liu lin quan chnh xc P (%) bao ph R (%)

Q1Card ha mi1452220,14100,00

Q2B nh my tnh147818181,22100,00

Q3Mn hnh tinh th lng485791,4477,78

Q4Card m rng367671,6385,71

Q5Mn hnh LCD49124124549,0898,37

Q6My tnh Dell1471330,20100,00

Q7My ch4646706100,006,52

Q8My tnh14701463147599,5299,19

Q9B x l13201241849,3967,39

Q10B nh86654386762,7062,63

Q11Mn hnh48546499895,6746,49

Q12CPU AMD8349101,0890,00

Q13HDD box115332,61100,00

Q14Mainboard Gigabyte722110,14100,00

Q15AMD38037238497,8996,88

Q16Asus40738841395,3393,95

Q17Samsung12801277128199,7799,69

Q18Chipset26625626696,2496,24

Q19Intel95694996199,2798,75

Q20LAN647646134999,8547,89

Q21Mainboard694616948,798,79

Q22RAM19916046080,4034,78

Q23SATA27625728793,1289,55

Q24Seagate42404295,2495,24

Q25CRT1291275,0075,00

Trung bnh54,6378,83

Mc tiu t ra ca bi bo l pho trin h thng truy hi thng tin ting Vit c tin cy. Do vy, gi tr ca chnh xc (precision) v bao ph (recall) c chn lm tiu ch nh gi trong phn thc nghim. chnh xc v bao ph c xc nh nh sau [8]:

Thng k chnh xc v bao ph theo cu truy vn

chnh xc (P) l t l gia s lng ti liu lin quan tm c so vi s ti liu tm c. bao ph (R) l t l gia s lng ti liu lin quan tm c so vi s ti liu lin quan c trong kho ng liu. Tp cu truy vn l 25 cu truy vn c chn t http://www.pcworld.com.vn/hoi-dap/phan-cung/ lm thc nghim.

Trong phn thc nghim chng ti s dng 14 cu truy vn l ting Anh v t vit tt kim nghim tin cy ca cng c VietAnalyzer nn tin cy ca cng c VietAnalyzer c kim nghim thng qua kt qu thc nghim.

Mt trong nhng kh khn ca bi bo l kho ng liu chun cho ting Vit vn cha c. Do , chnh xc v bao ph ca thc nghim c kt qu khng cao. S d kt qu nh thc nghim l do s ti liu lin quan n thng tin cn tm cn t v cha phong ph. iu ny cho thy kho ng liu cng ng vai tr quan trng trong vic nh gi thc nghim.

V. Kt lun

Bi bo trnh by cch pht trin h truy hi thng tin ting Vit s dng m ngun m Lucene. xy dng h truy hi VnIR cho ting Vit, bi bo s dng m ngun m Lucene (trong vic nh ch mc v tm kim) v xy dng cng c VietAnalyzer h tr Lucene trong vic x l ting Vit. Kt qu thc nghim c chnh xc 54,63%, bao ph 78,83%. Kt qu thc nghim c chnh xc cha cao l do khi ng liu do chng ti xy dng l kho ng liu tnh v ch quan nn s mc ti liu lin quan cn t, cha phong ph. V vy, cng vic tip theo l xy dng kho ng liu ln hn v phong ph hn trong vic nh gi thc nghim chng hn nh sa dng Internet nh l kho ng liu trong vic nh gi thc nghim v tip tc ci tin cc thut ton nhm ci thin chnh xc, bao ph cho h truy hi thng tin ting Vit vnIR.

Chng 4: D kin kt qu t c v ngha

VI. D kin kt qu nghin cu t c

VII. ngha khoa hc ca ti

VIII. ngha thc tin ca ti

Chng 5: K hoch thc hinK HOCH THC HIN CA LUN VN

Thi gianCng vic

8/1/2015 14/2/2015Thc hin tm hiu v l thuyt my hc qua php th ca A. Turing

15/2/2015-22/2/2015Tm hiu cch hot ng v cch s dng phn mm Weka.

23/2/2015- 28/2/2015Tm hiu thut ton cy quyt nh

1/3/2015 15/2/2015Nghin cu vic loi b stopword ting Vit v vit cng

Ti liu tham kho tai lieu tham khao tieng anh rieng , tieng viet rieng[1]inhin,2006.GiotrnhXlNgnngtnhin.NhxutbnihcQuc giaTp.HChMinh.[2]Phcvcngs.PhttrinmththngSearchEnginehtrchotmkimthngtinthuclnhvcCNTT.TrngihcKhoahcTnhinTp.HChMinh.[3]NhnhThun,2011.Vsearch1.0.[4]HunhcVit,VDuyThanhvVTrungHng,2010.NghincungdngmngunmLucenexydngphnmmtmkimthngtintrnvnbn.TpchKhoahcvCngngh,ihcNng,S4(39)2012,trang307-316.[5]ChoochartHaruechaiyasak.InformationRetrievalandSearchEngine.ResearchandDevelopmentonInformation,NationalElectronicsandComputerTechnologyCenter.[6]ChristopherD.Manning,PrabhakarRaghavanandHinrichSchutze,2008.IntroductiontoInformationRetrieval.CambridgeUniversityPress,Chapter9,pp.177-194.[7]NECTEC,2010.SanSarnLookRoadmap.[8]RicardoBaeza-Yates,BerthierRibeiro-Neto,1999.ModernInformationRetrieval.ACMPress,NewYork.[9]NguyenCamTu,PhanXuanHieuandNguyenThuTrang,2010.JVnTextPro:AtooltoprocessVietnamesetexts.version2.0.[10]Lucenedocs,http://lucene.apache.org/core/3Nhn xtThng k d liu theo thuc tnh

Thao tc trn cc Attribute

Chn lc d liu

Thao tc vi d liu

Thc hin: Nhm 8

Trang 20

_1487783963.unknown