A new AntTree-based algorithm for clustering short-text corpora

A new AntTree-based algorithm for clustering short-text corpora

Marcelo Luis Errecalde, Diego Alejandro IngaramoDevelopment and Research Laboratory in Computacional Intelligence (LIDIC)

Universidad Nacional de San LuisSan Luis, Argentina

{merreca,daingara}@unsl.edu.ar

Paolo RossoNatural Language Engineering Lab.,ELiRF,

Departamento de Sistemas Informaticos y ComputacionUniversidad Politecnica de Valencia

Valencia, [email protected]

Abstract

Research work on “short-text clustering” is a veryimportant research area due to the current tendencyfor people to use ‘small-language’, e.g. blogs, text-messaging and others. In some recent works, new bio-inspired clustering algorithms have been proposed todeal with this difficult problem and novel uses of In-ternal Clustering Validity Measures have also beenpresented. In this work, a new AntTree-based ap-proach is proposed for this task. It integrates infor-mation on the Silhouette Coefficient and the conceptof attraction of a cluster in different stages of the clus-tering process. The proposal achieves results compa-rable to the best reported results in this area, show-ing an interesting stability in the quality of the resultsand presenting some interesting capabilities as a gen-eral improvement method for arbitrary clustering ap-proaches.Keywords: Short-text clustering, Bio-inspired algo-rithms, AntTree, Internal Validity Measures, Silhou-ette Coefficient.

1 INTRODUCTION

Automatic document clustering is one of the most im-portant approaches to deal with the information over-load problem caused by the proliferation of docu-ments available on the Web, corporate intranets, newswires, etc. In a nutshell, document clustering is anunsupervised process that assigns documents to un-known categories (or groups) called clusters, whosemembers are similar in some way.

Many of the most interesting potential applicationsof document clustering, involve “short texts”. TheWeb provides us a considerable number of examplesof different types of short documents that are avail-able for automatic analysis such as emails, news, sci-

entific abstracts, blogs, snippets, chats, FAQs and on-line evaluations of commercial products. In all thesecases, clustering methods can play an important roleto analyze and organize this huge number of shortdocuments.

Short-text document clustering is considered a verydifficult problem due to the low frequencies of theterms in the documents. However, some recent re-search works have started studying different aspectsrelated to this problem. These works include the studyof the correlation between internal and external va-lidity measures [8], the estimation of the hardness ofshort-text corpora [11, 5] and the use of bio-inspiredclustering methods [3, 7]. In all these cases, the useof Internal Clustering Validity Measures have playedan important role.

A question that arises from these works is if someInternal Clustering Validity Measures, could also beused in other existing bio-inspired clustering meth-ods, in order to improve their performance. This workaddresses this aspect by proposing a new AntTree-based algorithm named AntSA which integrates intoan unified approach the Silhouette Coefficient [12]and the concept of attraction of a cluster. AntSA isbased on the AntTree algorithm [2] but it incorporatesinformation about both measures in different stages ofthe clustering process.

The remainder of the paper is organized as fol-lows. Section 2 presents some general considerationsabout different uses of Internal Clustering ValidityMeasures in short-text clustering tasks. Section 3 de-scribes the AntSA method proposed in this work. Theexperimental setup and the analysis of the results ob-tained from our empirical study is provided in Sec-tion 4. Finally, some general conclusions are drawnand possible future work is discussed.

JCS&T Vol. 10 No. 1 April 2010

1

2 INTERNAL VALIDITY MEASURES ANDCLUSTERING TASKS

Document clustering is the unsupervised assignmentof documents to unknown categories. This task ismore difficult than supervised document categoriza-tion because the information about categories and cor-rectly categorized documents is not provided in ad-vance. An important consequence of this lack ofinformation is that in realistic document clusteringproblems, results can not usually be evaluated withtypical external measures like F -Measure or the En-tropy, because the correct categorizations specified bya human expert are not available. Therefore, the qual-ity of the resulting groups is evaluated with respectto structural properties expressed in different Inter-nal Clustering Validity Measures (ICVMs). ClassicalICVMs used as cluster validity measures include theDunn and Davies-Bouldin indexes, the Global Silhou-ette (GS) coefficient and new graph-based measuressuch as the Expected Density Measure (EDM) (de-noted ρ) and the λ-Measure (see [8] and [13] for moredetailed descriptions of these ICVMs).

Most of people working on clustering problems arefamiliar with the use of ICVMs as cluster validationtools. However, some recent works have proposedother uses of this kind of measures, specially in thecontext of short-text clustering problems. In [8] forexample, an analysis of the correlation between dis-tinct ICVMs and the well known F -Measure is pre-sented. The evaluation of several ICVMs on the “goldstandard” of different short-text collections is pro-posed in [5] as a method to estimate the hardness ofthose corpora.

ICVMs have also been used as explicit objectivefunctions that the clustering algorithms attempt to op-timize [6, 15]) This idea has recently been used inshort-texts clustering tasks, taking the GS and theEDM ρ measures as objective functions and usingdiscrete and continuous Particle Swarm Optimiza-tion (PSO) algorithms as function optimizers [3, 7].In these works, a discrete PSO algorithm namedCLUDIPSO obtained the best results on differentshort-text corpora when the GS measure was used asobjective function.

The GS measure is an interesting ICVM that com-bines two key aspects to determine the quality of agiven clustering: cohesion and separation. Cohesionmeasures how closely related are objects in a clus-ter whereas separation quantifies how distinct (well-separated) a cluster from other clusters is. The GS co-efficient of a clustering is the average cluster silhou-ette of all the obtained groups. The cluster silhouetteof a cluster C also is an average silhouette coefficientbut, in this case, of all objects belonging to C. There-fore, the fundamental component of this measure isthe formula used to determine the silhouette coeffi-cient of any arbitrary object i, that we will refer as

s(i) and that is defined as s(i) = b(i)−a(i)max(a(i),b(i)) with

−1 ≤ s(i) ≤ 1. The a(i) value denotes the averagedissimilarity of the object i to the remaining objects inits own cluster, and b(i) is the average dissimilarity ofthe object i to all objects in the nearest cluster. Fromthis formula it can be observed that negative valuesfor this measure are undesirable and that we want forthis coefficient values as close to 1 as possible.

Tacking into account that ICVMs like GS or theEDM ρ have played an important role in cluster val-idation, hardness estimation and optimization-basedapproaches to clustering, it would be interesting toinvestigate if these and other ICVMs could be usedin other stages and processes of the clustering algo-rithms in order to improve their performances. In thiswork, a new AntTree-based algorithm named AntSAaims to answer this question by integrating into anunified approach the Silhouette Coefficient [12] andthe concept of attraction of a cluster.

3 THE AntSA ALGORITHM

The AntSA (AntTree-Silhouette-Attraction) algo-rithm is based on the AntTree algorithm [2] but it alsoincorporates information related to the Silhouette Co-efficient and the concept of attraction of a cluster indifferent stages of the clustering process. To under-stand how AntSA works, some preliminary conceptson the AntTree algorithm are neccesary. Therefore,the main ideas on the AntTree algorithm will be firstintroduced in subsection 3.1 before the description ofthe AntSA algorithm in subsection 3.2.

3.1 The AntTree algorithm

The AntTree algorithm [2] is based on the self-assembly behavior observed in certain species of antswhere the living structures are used as bridges or aux-iliary structures to build the nest. The structure is builtby using an incremental process in which ants joint afixed support or another ant for assembling. AntTreebuilds a tree structure representing a hierarchical dataorganization which divides the whole data set. Eachant represents a single datum from the data set andit moves in the structure according to its similarity tothe other ants already connected to the tree under con-struction.

Figure 1: Tree structure generation by self-assembling artificialants (adapted from [2]).

JCS&T Vol. 10 No. 1 April 2010

2

Each node in the tree structure represents a singleant and each ant represents a single datum. The keyaspect in AntTree is the decision about where each antwill be connected, either to the main support (gener-ating a new cluster) or to another ant (refining an ex-isting cluster).

Each ant to be connected to the tree represents adata to be classified. Starting from an artificial sup-port called a0, all the ants will be incrementally con-nected either to that support or to other already con-nected ants. This process continues until all ants areconnected to the structure, i.e., all data are alreadyclustered. Each ant ai has associated the followingterms:

1. I(ai), the ingoing links of ai. A set of links to-ward ai (the ai’s children).

2. O(ai), the outgoing link of ai. A link to its par-ent node (the support or another ant).

3. A datum di represented by ai.

4. Two metrics called respectively similaritythreshold (TSim(ai)) and dissimilarity threshold(TDissim(ai)) which will be locally updatedduring the process of building the tree structure.

Figure 1 shows a general outline of the self-assembling of artificial ants. It can be observed thateach ant ai is either of the two following situations:

1. Moving on the tree: a walking ant ai (gray high-lighted in figure 1) can be either on the support(a0) or on another ant (apos). In both cases, aiis not connected to the structure. Consequently,it will be free of moving to the closest neigh-bors connected to either a0 or apos. In Figure 2is showed the neighborhood corresponding to anarbitrary ant apos.

2. Connected to the tree: in this case ai has al-ready assigned a value for O(ai), therefore, itcan not move anymore. Additionally, an ant isnot able to have more than Lmax ingoing links(|I(ai)| ≤ Lmax). The objective is to boundthe maximum number of incoming links, i.e., themaximum number of clusters.

Figure 2: Neighborhood corresponding to an arbitrary ant apos(adapted from [2]).

Let L be a list (possibly sorted) of ants to be connectedInitialize: Allocate all ants on the support.TSim(aj)← 1 and TDissim(aj)← 0, for all ant ajRepeat

1. Select an ant ai from list L2. If ai is on the support (a0)

then support case (see Figure 4)else ant case (see description in [2])

Until all the ants are connected to the tree

Figure 3: Main loop of the AntTree algorithm.

The main loop implemented in the AntTree algo-rithm is shown in Figure 3. The very first step in-volves the allocation of all ants on the tree supportand their respective thresholds of similarity and dis-similarity are accordingly initialized. In this stage,the whole collection of ants is represented by a (pos-sibly sorted) list L of ants waiting to be connectedin further steps. During the tree generation processeach selected ant ai will be either connected to thesupport (or another ant) or moving on the tree look-ing for an adequate place to connect itself. The sim-ulation process continue until all ants have found themore adequate assembling place; either on the sup-port (the “Support case”, see Figure 4) or on anotherant (the “Ant case”). This last case is not describedin the present work due to space limitations and thefact that our proposal does not affect this componentof the AntTree algorithm. 1

When ai is on the support (Figure 4) and it is thefirst considered ant, it is a simple situation because theant is directly connected to the support. Otherwise,ai is compared against a+, the ant most similar to aiamong all the ants directly connected to the support.If these ants are similar enough, then ai will move tothe subtree corresponding to a+. In case that ai anda+ are dissimilar enough (according to a dissimilar-ity threshold), ai is connected directly to the support.This last action generates a new subtree (i.e., a newcluster) due to the incoming ant is different enoughfrom the other ants connected directly to the support.Finally, if ai is neither similar or dissimilar enough,the respective thresholds (similarity and dissimilar-ity) are updated in the following way: TSim(ai)←−TSim(ai) * 0. 9 and TDissim(ai)←− TDissim(ai) +0. 01. The previous updating rules let ant ai be more“tolerant” when considered in a further iteration, i.e.,the algorithm increases the probability of connectingthis ant in a future time.

It is important to highlight the importance of thearrangement of the ants in the list L (the initial step).Since the algorithm iteratively proceeds taking theants from L, the features of the first ants on this listwill significantly influence the final result. This willbe a fundamental aspect in our proposal of the newAnTSA algorithm described in subsection 3.2.

1A detailed description of the “Ant case” is available in [2].

JCS&T Vol. 10 No. 1 April 2010

3

If no ant is connected to the support then connect ai to a0else

Let a+ be the ant connected to a0 most similar to ai(a) If Sim(ai, a

+) ≥ TSim(ai) then move ai toward a+

(b) elsei. If Sim(ai, a

+) < TDissim(ai) thenconnect ai to a0 (in case there is no morelinks available in a0, then move ai toward a+

and decrease TSim(ai))ii. else decrease TSim(ai) and increaseTDissim(ai)

Figure 4: Support case.

Figure 5: A tree interpreted as a non hierarchical data partition(adapted from [2]).

The resulting tree (see Figure 5) can be interpretedas a data partition (considering each ant connectedto a0 as a different group) as well as a dendrogramwhere the ants in the inner nodes could move to theleaves following the most similar nodes to them.

3.2 The AntSA algorithm

Some steps and processes of the AntTree method havea significative influence during the generation of themain groups. For instance, the initial ordering stepthat determines the order in which ants will be con-sidered to be connected in the support structure (eachone representing a different group) is one of those as-pects. Another important process is the comparison ofan arbitrary ant with the ants connected to the support(Figure 4, step (a)) because it determines the primarycluster assignments of ants, depending on the selectedpath. Our proposal basically attempts to improve theperformance of AntTree by:

1. considering in the initial step of AntTree, addi-tional information about the Silhouette Coeffi-cient of previous clusterings;

2. using a more informative criterium (based on theconcept of attraction) when the ants have to de-cide which path to follow in the support case.

Using silhouette coefficient information in the ini-tial step. The initial ordering step defines the orderin which ants will be connected to the support (each

one representing a different group). Therefore, anylittle modification in this ordering will significativelyimpact the clustering results. Our proposal consists intaking as input the clustering obtained with some ar-bitrary clustering algorithm and using the SilhouetteCoefficient (SC) information of this grouping to de-termine the initial order of ants. The general idea isshown in Figure 6.

1. Use a clustering algorithm to obtain an initial grouping.2. Build k data rows (one for each group obtainedin the previous step) and sort them in decreasing order accordingto the Silhouette Coefficient.3. Connect to the support the first ant of each row.4. Merge the rows by iteratively taking the first ant ofeach non-empty row, until all rows are empty.

Figure 6: A new SC-based ordering for the AntTree’s initial step.

The SC-based ordering of ants carried out in thisstage determines which will be the first ants con-nected to the support structure. The ants with thehighest SC value within each group will be consid-ered more desirable because they are the most repre-sentative ants of their groups.

Support Case: Attraction-based comparison. Akey aspect for an arbitrary ant ai on the support is thedecision about which connected ant a+ should movetoward. In fact, this decision will determine the gen-eral group in which ai will be incorporated. AntTreetakes into account for this decision, the similarity be-tween ai and its most similar ant connected to the sup-port (a+). This is a “local” approach that only consid-ers the ant directly connected to the support structure(a+) but it does not take into account the ants previ-ously connected to a+, that will be denoted asAa+ . Amore global approach that also considers some infor-mation on Aa+ could be useful to improve the clus-tering results. If Ga+ = {a+} ∪ Aa+ is the groupformed by a+ and its descendants, this relationshipbetween the group Ga+ and the ant ai will be referredas the attraction of Ga+ on ai and will be denoted asatt(ai,Ga+).

The idea of having different groups exerting somekind of “attraction” on the objects to be clustered wasalready posed in [14], where it was used as a effi-cient tool to obtain “dense” groups. In the presentwork, we will give a more general sense to the conceptof attraction by considering that att(ai,Ga+) repre-sents any plausible estimation of the quality of thegroup that would result if ai were incorporated to Ga+

(Ga+ ∪{ai}). Thus, the only modification that AntSAwill introduce to the support case of AntTree willbe the replacement of all occurrences of Sim(ai, a

+)by att(ai,Ga+), where a+ now will represent the antwith the highest att(ai,Ga+) value.

JCS&T Vol. 10 No. 1 April 2010

4

To compute att(ai,Ga+) we can use some ICVMthat allows to estimate the quality of individual clus-ters, and to apply this ICVM to Ga+ ∪ {ai}. For in-stance, any cohesion-based ICVM could be used inthis case, but other more elaborated approaches (likethe density-based ones) would also be valid alterna-tives. As an example, an effective attraction measureis the average similarity between ai and all the ants inGa+ as shown in Equation 1.

att(ai,Ga+) =

∑a∈Ga+

Sim(ai, a)

|Ga+ |(1)

4 EXPERIMENTAL SETTING ANDANALYSIS OF RESULTS

For the experimental work, four collections with dif-ferent levels of complexity with respect to the lengthof documents and vocabulary overlapping were se-lected: CICling-2002, EasyAbstracts, Micro4Newsand SEPLN-CICLing. CICling-2002 is a well knownshort-text collection that in different works [9, 1,10, 8, 5, 3, 7] has been recognized as a very dif-ficult collection since its documents are narrow do-main scientific abstracts (short-length documents witha high vocabulary overlapping). Micro4News is a lowcomplexity collection of medium-length documentsabout well-differentiated topics (wide domain). TheEasyAbstracts corpus is composed of short-lengthdocuments (scientific abstracts) on well differentiatedtopics (medium complexity corpus). Finally, SEPLN-CICLing is a corpus that it is supposed to be harderto cluster than the previous corpora since its docu-ments are narrow domain abstracts. SEPLN-CICLingand CICling-2002 have similar characteristics. How-ever, all the SEPLN-CICLing’s abstracts guarantee aminimum quality level with respect to their lengths,an aspect that is not assured by all the CICling-2002’sdocuments.2

The documents were represented with the standard(normalized) tf -idf codification after a stop-word re-moving process. The popular cosine measure wasused to estimate the similarity between two docu-ments. The initial data partitions required by AntSAwere obtained with CLUDIPSO (using GS as objec-tive function). The parameter settings for CLUDIPSOand the remainder algorithms used in the comparisonwith AntSA corresponds to the parameters empiri-cally derived in [7]. The attraction measure (att(·))used in our study corresponds to the formula pre-sented in equation 1. We will refer as AntSA-CLUto this instance of AntSA that takes as input theCLUDIPSO’s results.

2Space limitations prevent us from giving a more detailed de-scription of these corpora but it is possible to obtain in [4, 9, 1, 10,8, 5, 3, 7] more information about the features of these corpora andsome links to access them for research proposes.

4.1 Experimental results

The results of AntSA-CLU were compared with theresults obtained with other five clustering algorithms:K-means, CLUDIPSO [3, 7], Ant-Tree [2], Major-Clust [14] and DBSCAN. K-means is one of the mostpopular clustering algorithms whereas MajorClustand DBSCAN are representative of the density-basedapproach to the clustering problem and have showninteresting results in similar problems. AntTree andCLUDIPSO can be considered as the “basis” ofAntSA-CLU and, therefore, it would be interestingto analyze if AntSA-CLU achieves some improve-ments with respect to these algorithms. The resultsof the different algorithms were evaluated by usingthe classical (external) F -measure on the clusteringsthat each algorithm generated in 50 independent runsper collection. The reported results correspond tothe minimum (Fmin), maximum (Fmax) and average(Favg) F -measure values. The values highlighted inbold in the different rows indicate the best obtainedresults.

Table 1 shows the Fmin, Fmax and Favg val-ues that K-means, MajorClust, DBSCAN, Ant-Tree,CLUDIPSO and AntSA-CLU obtained with the fourcollections. These results confirm the good perfor-mance that CLUDIPSO has already shown in pre-vious works with collections of different complex-ity. However, in this case, AntSA-CLU not only ob-tained the same highest Fmax values that CLUDIPSOachieved in Micro4News, EasyAbstracts and SEPLN-CICLing. It also obtained the highest Fmax value onthe CICling-2002 collection, the most difficult collec-tion analyzed in our experiments. Another interestingaspect of AntSA-CLU, is the fact that its good per-formance was not limited to the Fmax values. It alsooutperformed the remainder algorithms in the Fmin

and Favg values obtained with the four collections.These results show an interesting stability of AntSA-CLU which produced acceptable (or very good) re-sults in most of the experiments. This observationcan be more easily appreciated in Figure 7 where theordered F -measure values obtained in the 50 experi-ments with AntSA-CLU (Black line) and CLUDIPSO(gray line) are displayed.

An interesting aspect to investigate is the analy-sis of the impact that the quality of the initial datapartitioning has on the AntSA’s results. An exhaus-tive study of this problem is beyond the scope of thepresent article. However, in order to get some prelim-inary data about this problem, we also experimentedwith an AntSA version that uses as input the clus-terings obtained with K-means, the algorithm thatreported the worst results in our study. We named

JCS&T Vol. 10 No. 1 April 2010

5

Micro4News EasyAbstracts SEPLN-CICLing CICling-2002

Algorithms Favg Fmin Fmax Favg Fmin Fmax Favg Fmin Fmax Favg Fmin Fmax

K-Means 0.67 0.41 0.96 0.54 0.31 0.71 0.49 0.36 0.69 0.45 0.35 0.6MajorClust 0.90 0.76 0.96 0.69 0.44 0.98 0.59 0.4 0.77 0.43 0.37 0.58DBSCAN 0.82 0.71 0.88 0.66 0.62 0.72 0.63 0.4 0.77 0.47 0.42 0.56AntTree 0.7 0.69 0.82 0.6 0.5 0.67 0.49 0.41 0.64 0.41 0.38 0.48CLUDIPSO 0.93 0.85 1 0.92 0.85 0.98 0.72 0.58 0.85 0.6 0.47 0.73AntSA-CLU 0.96 0.88 1 0.96 0.92 0.98 0.75 0.63 0.85 0.61 0.47 0.75

Table 1: F -measures values.

Micro4News EasyAbstracts SEPLN-CICLing CICling-2002

Algorithms Favg Fmin Fmax Favg Fmin Fmax Favg Fmin Fmax Favg Fmin Fmax

K-Means 0.67 0.41 0.96 0.54 0.31 0.71 0.49 0.36 0.69 0.45 0.35 0.6AntSA-KM 0.84 0.67 1 0.76 0.46 0.96 0.63 0.44 0.83 0.54 0.41 0.7

Table 2: F -measures values.

AntSA-KM this particular version of AntSA. In Ta-ble 2 and Figure 8 the results obtained with K-meansand AntSA-KM are presented. The first observationis that although AntSA-KM is not able to achieve asgood results as AntSA-CLU and CLUDIPSO obtain,it outperformed most of results obtained with DB-SCAN and Ant-Tree and had, in general, a perfor-mance comparable to the MajorClust’s performance.However, the comparison between the performancesof AntSA-KM and K-means deserves special atten-tion. As it can be clearly appreciated in Table 2 andFigure 8, AntSA-KM achieved better F -measure val-ues than K-means on all the considered collections.The previous results provide a strong evidence that al-though other algorithms obtained low quality clusters,AntSA is able to improve them and obtain acceptableresults. Another interesting aspect is that the AntSAalgorithm would seem to be a useful general mecha-nism that allows to refine and improve the results ofvery different clustering algorithms.

Micro4News EasyAbstracts

SEPLN-CICLing CICling-2002

Figure 7: F -measure values: AntSA-CLU (Black Line) vsCLUDIPSO (Gray Line).

Micro4News EasyAbstracts

SEPLN-CICLing CICling-2002

Figure 8: F -measure values: AntSA-KM (Black Line) vs K-Means (Gray Line).

5 CONCLUSIONS AND FUTURE WORK

In this work we presented AntSA, a novel AntTree-based algorithm for clustering short-text corpora.AntSA integrates information on the Silhouette coef-ficient and the concept of attraction in different stagesof the clustering process. AntSA is a general algo-rithm that allows: a) to use different clustering algo-rithms to obtain the initial data partition, b) to definedifferent attraction formulae.

When AntSA worked with the clusterings gener-ated by the CLUDIPSO algorithm (the AntSA-CLUversion), it obtained the best reported results for thefour short-text collections considered in the exper-imental work. When the AntSA-KM version wasused, it improved all the results obtained by K-meansand had results comparable to the remainder algo-rithms. In all these cases, AntSA showed a signi-ficative stability in the quality of its results and pre-sented some interesting capabilities as a general im-provement method for other clustering methods.

Future work includes the use of different attraction

JCS&T Vol. 10 No. 1 April 2010

6

measures and an exhaustive experimental workthat analyzes the potential improvements on otherclustering algorithms. In these experiments, othermore representative document collections will beconsidered in order to determine if the good perfor-mance of AntSA on short-text collections can also beobtained with arbitrary document collections.

Acknowledgments

We thank the TEXT-ENTERPRISE 2.0 TIN2009-13391-C04-03 research project for funding the workof the first and third authors.

References

[1] M. Alexandrov, A. Gelbukh, and P. Rosso. Anapproach to clustering abstracts. In Proc. ofNLDB-05, volume 3513 of LNCS, pages 8–13.Springer-Verlag, 2005.

[2] H. Azzag, N. Monmarche, M. Slimane, G. Ven-turini, and C. Guinot. AntTree: A new modelfor clustering with artificial ants. In Proc. ofthe CEC2003, pages 2642–2647, Canberra, 8-12 December 2003. IEEE Press.

[3] L. Cagnina, M. Errecalde, D. Ingaramo, andP. Rosso. A discrete particle swarm optimizerfor clustering short-text corpora. In BIOMA08,pages 93–103, 2008.

[4] M. Errecalde and D. Ingaramo. Short-text cor-pora for clustering evaluation. Technical report,LIDIC, 2008.

[5] M. Errecalde, D. Ingaramo, and P. Rosso. Prox-imity estimation and hardness of short-text cor-pora. In Proceedings of 5th Int. Workshop onText-based Information Retrieval (TIR-2008),pages 15–19, 2008.

[6] D. Fisher. Knowledge acquisition via incremen-tal conceptual clustering. Machine Learning,2:139–172, 1987.

[7] D. Ingaramo, M. Errecalde, L. Cagnina, andP. Rosso. Computational Intelligence and Bio-engineering, chapter Particle Swarm Optimiza-tion for clustering short-text corpora, pages 3–19. IOS press, 2009.

[8] D. Ingaramo, David Pinto, P. Rosso, and M. Er-recalde. Evaluation of internal validity measuresin short-text corpora. In Proc. of the CICLing2008 Conf., volume 4919 of LNCS, pages 555–567. Springer-Verlag, 2008.

[9] P. Makagonov, M. Alexandrov, and A. Gelbukh.Clustering abstracts instead of full texts. InProc. of TSD-2004, volume 3206 of LNAI, pages129–135, 2004.

[10] D. Pinto, J. M. Benedı, and P. Rosso. Clus-tering narrow-domain short texts by using theKullback-Leibler distance. In Proc. of the CI-CLing 2007 Conf., volume 4394 of LNCS, pages611–622. Springer-Verlag, 2007.

[11] D. Pinto and P. Rosso. On the relative hardnessof clustering corpora. In Proc. of TSD07, vol-ume 4629 of LNAI, pages 155–161. Springer-Verlag, 2007.

[12] Peter Rousseeuw. Silhouettes: a graphical aid tothe interpretation and validation of cluster analy-sis. J. Comput. Appl. Math., 20(1):53–65, 1987.

[13] B. Stein, S. Meyer zu Eissen, and F. Wißbrock.On cluster validity and the information need ofusers. In Proc. of the IASTED03, pages 216–221, 2003.

[14] Benno Stein and Sven Meyer zu Eißen. Doc-ument Categorization with MAJORCLUST. InProc. WITS 02, pages 91–96. Technical Univer-sity of Barcelona, 2002.

[15] Y. Zhao and G. Karypis. Empirical and the-oretical comparison of selected criterion func-tions for document clustering. Machine Learn-ing, 55:311–331, 2004.

JCS&T Vol. 10 No. 1 April 2010

7