High Performance Data Mining Using R on …users.eecs.northwestern.edu/~choudhar/Publications/HighPerformance... · High Performance Data Mining Using R on ... ments of data mining

High Performance Data Mining Using R on Heterogeneous Platforms

Prabhat Kumar, Berkin Ozisikyilmaz, Wei-Keng Liao, Gokhan Memik, Alok ChoudharyDepartment of Electrical Engineering and Computer Science

Northwestern UniversityEvanston, IL, USA

{pku649, boz283, wkliao, memik, choudhar}@ece.northwestern.edu

Abstract—The exponential increase in the generationand collection of data has led us in a new era ofdata analysis and information extraction. Conventionalsystems based on general-purpose processors are unableto keep pace with the heavy computational require-ments of data mining techniques. High performance co-processors like GPUs and FPGAs have the potential tohandle large computational workloads. In this paper,we present a scalable framework aimed at providinga platform for developing and using high performancedata mining applications on heterogeneous platforms.The framework incorporates a software infrastructureand a library of high performance kernels. Furthermore,it includes a variety of optimizations which increase thethroughput of applications. The framework spans mul-tiple technologies including R, GPUs, multi-core CPUs,MPI, and parallel-netCDF harnessing their capabilitiesfor high-performance computations. This paper alsointroduces the concept of interleaving GPU kernels frommultiple applications providing significant performancegain. Thus, in comparison to other tools available fordata mining, our framework provides an easy-to-use andscalable environment both for application developmentand execution. The framework is available as a softwarepackage which can be easily integrated in the R pro-gramming environment.

Keywords-R; GPU; Data Mining; MPI; K-Means;Fuzzy K-Means; PCA; Parallel-netCDF;

I. INTRODUCTION

Knowledge driven decisions are a key to successin today’s world. Business corporations, financialinstitutions, government departments, research anddevelopment organizations collect huge amountsof data with a view to gain a deeper insightin their respective fields. Social networks suchas Facebook and micro-blogging website Twittergenerate enormous amounts of data which can

provide useful information about the latest trendsin the society. Sifting through such vast collectionof data and discovering unknown patterns is not atrivial task, especially when the data sizes are ofthe order of exabytes and petabytes. Data miningpresents a pool of automated analysis techniqueswhich can discover hidden knowledge and predictnew trends and behaviors.

Analyzing large quantities of data requires com-putational resources. Recent times have seen theemergence of many high performance architec-tures like GPGPUs, Cell, Multi-cores, FPGAs,etc., each presenting its own unique benefits. Theparadigm of homogenous computing, where all thenodes have the same architecture, is transformingitself to heterogeneous computing, where eachtask is allocated to the architecture that suits itsproperties best. Since data mining kernels are char-acterized as being computationally intensive, thenew generation of architectures can provide a sig-nificant boost to their performance. Furthermore,storing and retrieving large quantities of data addsto the complexity of data mining applications.

Exploring hidden patterns and trends require acollection of data mining techniques. Tools suchas Clementine[1] and WEKA[2] provide a richcollection of algorithms. However, they lack thecapability to utilize the benefits of co-processorsand do not have scalable I/O capabilities. Thislimits their usability as a high performance dataanalytics tool. This paper describes a scalableframework for developing parallel applications ona heterogeneous computational backbone. It in-corporates a library of compute-intensive kernelsand explores performance optimization techniques

2011 IEEE International Parallel & Distributed Processing Symposium

1530-2075 2011

U.S. Government Work Not Protected by U.S. Copyright

DOI 10.1109/IPDPS.2011.329

1719

2011 IEEE International Parallel & Distributed Processing Symposium

1530-2075 2011

U.S. Government Work Not Protected by U.S. Copyright

DOI 10.1109/IPDPS.2011.329

1715

2011 IEEE International Parallel & Distributed Processing Symposium

1530-2075 2011

U.S. Government Work Not Protected by U.S. Copyright

DOI 10.1109/IPDPS.2011.329

1715

to increase the throughput of applications. In ourframework, an application is written as a scriptwhich is composed of modules (e.g., commonlyused kernels). The framework provides a middle-ware which deploys these modules on a cluster ofheterogeneous hardware platforms. Further, pro-cessing huge amounts of data requires reading andwriting to storage devices, like disk drives, SSDsetc. I/O presents a significant bottleneck in theoverall performance of data mining applications asa poor read/write interface can hinder any benefitobtained from parallel architectures. To alleviatethis problem, our framework incorporates a par-allel I/O interface. Thus, the framework discussedin this paper provides parallelism both for I/O andcomputations while still being simple and flexible.

Besides the above mentioned features, the pro-posed framework outlines a new optimizationtechnique aimed for GPU architectures. This tech-nique involves interleaving kernels from differentapplications to improve their throughput. The opti-mization relies on the domain specific knowledgethat it is not always known apripori, what isthe best algorithm to mine raw data for usefulinformation. In such situations the data is exploredusing multiple algorithms. Since all the algorithmswork on the same dataset, they can run in closecoordination to improve the overall performance.Overall, the major contributions of the paper areas follows:

1) A scalable framework for writing highperformance applications on heterogeneousplatforms.

2) A high performance library of commonlyused kernels for data exploration.

3) An interface to parallel I/O functionality4) Various optimizations to increase the

throughput of applications

The paper is organized as follows. Section IIpresents the related work. Section III presentsthe implementation overview of our framework.Section IV describes how applications can bewritten for the framework. Section V presents adiscussion of the results. We conclude the paperin Section VI with directions to future work.

II. RELATED WORK

R[3] is a widely used programming languagefor statistics and data manipulation. Given thathuge statistical problems have become common-place today, a number of parallel R packages havebeen developed. A few such packages for explicitparallelism are mentioned below.

The Rmpi[4] package provides an interfacefrom R to MPI. The SNOW[5] package runs ontop of Rmpi (or directly via sockets), allowing theprogrammer to express the parallel disposition ofwork more conveniently. Rdsm package[6] givesthe R programmer a shared memory view, but theobjects are not physically shared. Instead, they arestored in a server, and accessed through networksockets, thus enabling a threads-like view. Parallel-R[7] and pR[8] enable the statistical analysisroutines available in R to be deployed on highperformance architecture.

The M0BNI Microarray Lab in University ofMichigan has provided a gputools[9] package forR which constitutes a handful of common statisti-cal algorithms that are used in the biomedical re-search community. Another work in this area is theRGPU package which enables parallel evaluationof linear algebra expressions, as well as access tosome of the function provided in CUDA SDK[10].The magma[11] package provides an interface tothe hybrid Matrix Algebra on GPU and MulticoreArchitectures implementation.

There has been multiple works in using clus-ter of GPUs in parallel. DisMaRC, a distributedGPGPU based MapReduce framework is pre-sented in [12]. In another work by Lawlor[13], theauthor analyses two new communication libraries,cudaMPI and glMPI, that provide an MPI-likemessage passing interface to communicate datastored on the graphics cards of a distributed-memory parallel computer. Then there are nu-merous examples of single applications that areported on cluster of GPUs [14], [15], [16], [17].In the data mining domain GPUs have been usedextensively. Some implementations of K-Meanson GPUs can be found in [18], [19], [20], [21],[22], [23]. Our work focuses on low-level and

172017161716

micro-level performance optimizations which areexplored using a library of customized kernelsincorporated in a scalable framework. In this workwe focus on a bottoms-up approach of developinghigh performance applications using simpler ker-nels. This is in contract to the various works men-tioned above which follow a top-down approach.Furthermore, we stress on the overall throughputof applications as opposed to scaling independentapplications.

III. IMPLEMENTATION OVERVIEW

The framework presented in the paper spansacross different domains to harness their capabil-ities in an attempt to provide a scalable systemfor using data mining applications for knowledgediscovery. This section gives a detailed descriptionof the different components used in the frame-work. The front end of the framework is thewidely used R-statistical tool. MPI is used forcommunication between the cluster nodes andparallel I/O is achieved using MPI-IO and Parallel-netCDF[24] interface. The computation intensivetasks are handled by multi-core CPUs and multi-threaded GPUs [25], [10]. Figure 1 shows thedifferent components of the system.

Figure 2 shows the dataflow in our framework.The framework is launched on all the nodes ina master-slave configuration. The application iswritten as an R script. The script calls the high-performance I/O interface to read the data fromplatform independent netCDF[26] file in parallel.

Figure 1: Overview of the Framework

Figure 2: Dataflow in the framework

Once the data is read, the script invokes high-performance kernels, through an efficient R-Cinterface, to analyze the data. The MPI commu-nication enables the nodes to interact with eachother. The results, which are communicated tothe R environment, can take advantage of therich analysis and/or visualization tools availablein R. The programming model consists of - (1) aprogramming infrastructure and (2) a library ofhigh performance kernels. The former providestools and methodologies for developing scalableapplications while the latter provides a collectionof commonly used data mining kernels to accel-erate application development.

A. Programming InfrastructureThe programming infrastructure provides a soft-

ware platform which presents different methodsfor managing the various components with a viewof scalable implementation and a high perfor-mance scripting interface for an easy-to-use frontend to write various applications.

1) Software Platform: As mentioned above, wehave four major components in our framework:the front end (or the scripting interface), the backend (managed by C/C++/CUDA), communication(which is MPI) and the I/O. Broadly defined, thereare two different implementation methods for glu-ing these components to have a scalable platformwhile keeping it flexible enough to incorporatedifferent kernels. These methods are described inthe following. The first implementation, whichis referred to as C-level parallelism (C-LP), isshown by dotted arrows in Figure 3. In this,the MPI communication is not visible in the Renvironment. Each of the nodes running R call

172117171717

the corresponding C interface functions and all theMPI calls are handled at the C level. The otherway of implementation, called R-level Parallelism(R-LP), in which the MPI communication is visi-ble at the R environment, is shown by solid arrowsin Figure 3. The R nodes call the C interfacedkernels and the communication among the nodesis handled in R. Notice that the C kernels are serialas opposed to MPI-enabled kernels in C-LP.

Both the implementations have pros and cons.R-LP has a higher overhead in sharing data amongthe nodes as the data needs to come to theR environment and then communicated to othernodes before finally filtering down to the C en-vironment, as opposed to C-LP where the datacan be shared at the same level i.e., across theC environment as shown in Figure 3. Secondly,C applications/kernels which are already writtenusing MPI paradigm can be directly interfaced tothe R environment with no or little modifications.On the contrary, R-LP requires application to bewritten as R script. The limitation faced by C-LPis that it requires all the development to be donewithin Rmpi package, i.e., all the code needs to becompiled with the Rmpi code. The reason lays inthe fact that MPI initialization can be done onlyonce for whole system which makes it impossiblefor packages not compiled within Rmi to use

Figure 3: C-level Parallelism (C-LP) - Parallelism embed-ded at the C environment (dotted arrows) and R-level Par-allelism (R-LP) Parallelism exposed to the R environment(solid arrows)

MPI function calls. R-LP is more flexible in thisregard as high performance library packages canbe developed independent of the Rmpi package.Both these approaches have been followed in theframework for different components. As discussedin later sections, parallel I/O interface is built uponC-LP, while the kernel libraries and applicationdevelopment follow the R-LP methodology.

2) High-Performance R: The programming in-frastructure of our framework includes a high-performance scripting language capable of beingused in a distributed computing environment. Thisscripting language interface is based on the widelyused statistical tool R. However, since R is notgood for heavy lifting, an interface to high levellanguages like C/C++/Fortran, known for theircomputational capabilities, is provided. Further-more, since all the accelerator/coprocessors havean interface to high-level languages an efficientR-C interface is necessary for true high perfor-mance scripting capabilities. R serves as a front-end interface to the user. Compiled C functionscan be invoked in the R environment using .C or.Call interface functions. With .C, the R objectsare copied to C data structure before being passedto the C code, and copied again to a R list objectwhen the compiled code returns. On the contrary.Call does not copy arguments. Since data miningalgorithms process huge amounts of data, copyingof arguments can severely hamper the performanceof applications. Our framework uses .Call functionto provide the C interface to R. Also, we haventnoticed any degradation in the execution of theC functions using .Call interface. Besides loweramount of copied data, other advantages of using.Call function include:

• The ability to dimension the answer in C code• Access to the attributes of the vectors• And, access to other types, e.g., expressions

and raw typeThese advantages come at the cost of increased

complexity in writing the interface functions.

B. High-Performance library of KernelsThe second component of our frameworks pro-

gramming model is the library of optimized and

172217181718

Table I: Kernels for large data using CUDA Streams

high performance kernels which can be embeddedin R scripts. The library provides a collection ofcommonly used data mining kernels implementedfor different architectures. Apart from this, intra-node and inter-node optimizations are also in-cluded. To keep the development process simpleand flexible, we follow the R-LP approach (referto Figure 3). Decoupling the high performancekernels from applications gives us the opportunityto develop new applications. Furthermore, kernelsimplemented on different architectures enable usto explore the design space to achieve the bestperformance.

1) Computational Intensive Kernels: We haveimplemented the kernels both for CPU and GPU.CPU kernels are used for hybrid-execution on aheterogeneous cluster comprising of GPUs andCPUs. For CPU, some kernels are already avail-able in R. The implementation is done keeping inview that the kernels can easily scale in a clusterenvironment. For the GPU implementations, theinput data is first shipped to the GPU devicememory and then kernels are launched whichprocess the input data in the device memory. Theresults are subsequently shipped back to the host(CPU) memory. Table I shows a list of kernels andtheir corresponding interface functions for R.

2) Kernel optimization for GPUs: The abovementioned kernels work well when the input data

fits entirely into the GPU device memory. How-ever, since data mining deals with huge amountsof data, typically, the entire data will not fitinto the GPU device memory. Transferring datato and from the GPU device will result in sig-nificant performance degradation. This requiresout-of-core implementation using CUDA Streams.The framework, however, uses the multi-threadedkernels (mentioned above) and schedules them tooverlap with host/device data transfers. This limitsthe need of developing new out-of-core kernels.Figure 4 shows an example where the input datasetis divided into smaller tiles and assigned to twodifferent streams. Each data transfer on Stream 1is overlapped with a kernel execution on Stream2 resulting in reduced overhead.

3) Communication + I/O: In our frameworkthe communication among the nodes of a clusteris handled by MPI though the Rmpi package.However, besides sharing the data during compu-tations, large amounts of data need to be accessedfrom storage devices. Absence of a parallel I/Ointerface will severely affect any performance gainachieved using multi-core multi-threaded kernels.We, therefore, enhace the capabilities of Rmpipackage to provide MPI-IO interface to R forparallel read/write capability. Further, parallel-netCDF is built on top of MPI-IO. We have imple-mented a parallel-netCDF interface for R which

172317191719

Figure 4: Kernels for large data using CUDA Streams

provides the capability of reading/writing netcdffile format to all the nodes in the R-cluster. TableI gives a list of interface functions for parallelread/write.

IV. APPLICATION DEVELOPMENT USING THEFRAMEWORK

Previous section has given a detailed descriptionof the different components that make our frame-work. In this section, we present how applicationscan be written using all these components. We di-vide this section into three subsections discussingabout the implementation of algorithms using thekernels, the optimizations offered by the frame-work, and how to scale the applications to a clusterof nodes. Notice that application development isdone on the front end in R script and the kernelsand I/O functions are called only when necessary.

A. Algorithms

Using the framework, we have developed dif-ferent data mining algorithms. Due to limitedspace we give only a brief description of three ofthem: K-Means[27], Fuzzy K-Means[28], [29] andPCA[30], [31], [32], [33]. K-Means is a widelyused clustering algorithm which attempts to findK partitions of the input dataset by minimizing

the squared error within each partition. The K-Means algorithm can be implemented using theDistance Computation, Cluster Update and His-togram kernels as mentioned in Table I. A vari-ation of K-Means algorithm called Bisection K-Means can also be implemented similarly. FuzzyK-Means is a superset of K-Means algorithm withthe distinction that it allows each record in thedata set to have a degree of membership to eachpartition. Principal component Analysis (PCA)aims at finding the principal components whichare representative of the input dataset and canbe implemented using Eigenvalue and Eigenvectorkernels.

B. Scheduling Optimizations using the Frame-work

Besides the above mentioned kernels, ourframework provides a number of optimizationswhich can help increase the speedup of the ap-plications. We present a couple of optimizationshere. Notice that these optimizations are currentlyspecific to the hardware but as new hardwaredevices are introduced, new optimizations for thatparticular hardware are easy to integrate in thecurrent system.

1) Hybrid Implementation: Hybrid implemen-tation refers to harnessing the capabilities of both

Figure 5: Distribution of tasks in a Heterogeneous environ-ment

172417201720

GPUs and CPUs simultaneously. Consider a situa-tion when we have a GPU and a multi-core CPU inthe system. It would be desirable to distribute thetasks between the GPU and the CPU cores. Sincethe computational power of GPU is significantlyhigher than that of the CPUs, the data need to bedistributed such that the work remains balanced.Our framework provides the functionality to runan application in the hybrid mode. In this mode,the data will be distributed among the nodes andthe corresponding CPU or GPU kernels will belaunched. Figure 5 shows how a hybrid kernel callgets broken down into architecture specific kernelcalls using the framework.

2) Multiple Kernel Optimization: This opti-mization is specific to the CUDA implementation.We notice that data mining kernels process hugeamounts of data and it is not always possible tofit the entire data in the GPU device memory. Asmentioned in Section III-B2, this will require theusage of CUDA Streams to lower the overheadcaused by copying data from host memory tothe device memory. We further notice in ourexperiments that kernel execution time is smaller

Figure 6: Concept of Interleaved Kernel Optimization

than the time it takes to copy smaller tiles of datato the GPU device memory. This presents us aunique opportunity to leverage the time difference.In practical situations, a number of different datamining algorithms are used on a given dataset. Wepropose to run kernels from different applicationson the dataset while it is in the device memoryso as to reduce the overhead of memory copy asmuch as possible. Figure 6 shows the idea behindthis optimization. As an example, three differentapplications App1, App2, and App3 are shownin the figure. For each memory transfer call puton a CUDA Stream, one kernel call from eachof the applications (1.a, 2.a, and 3.a) is allocatedon that particular stream as shown. This can beviewed as a single kernel whose execution time isclose to the combined execution time of the samekernels running separately. The kernel executionand host-device memory copy times can be usedto predict the number of applications which canbe interleaved in the above fashion.

V. EXPERIMENTAL RESULTS

In this section, we evaluate the performance ofthe applications developed using the frameworkas well as various proposed optimizations. On thehardware side, we have a cluster of 4 nodes. Thehost CPU on each node is an Intel Quad Core 2.4GHz processor with 4 GB of main memory. Theco-processor on each node is NVIDIAs GeForce8800GT Graphics Processing Unit with 112 pro-cessing cores and 512 MB of device memory. Onthe GPU, the grid of thread blocks can have amaximum of 65535 blocks on each dimension,with a maximum of 512 threads per block. Eachmultiprocessor has 16 KB of shared memory andcan run 8 thread blocks concurrently. Each nodehas two of these GPUs. The software setup in-cludes R version 2.8.0. MPICH2 version 1.2.1is used for providing the MPI communication.To provide parallel-netCDF functionality Pnetcdflibrary version 1.2 is used. The GPU kernelsare compiled using the CUDA compiler driver,NVCC, release 2.0. The entire software frameworkis compiled using GCC version 4.4.2.

172517211721

A. Performance of Applications

Figure 7 shows the performance of clusteringalgorithms like K-Means and Fuzzy K-Means for20 clusters and different sizes of the input dataset ranging from 10K to 1 million records. Wenotice that as the data size is increased, there is aninitial improvement in the speedup, which even-tually saturates at around 40 for K-Means and 70for Fuzzy K-Means. The performance differencebetween the two algorithms occurs because of thecomputationally intensive membership calculationresulting in higher speedups for larger workloads.Due to limited space we cannot provide perfor-mance charts for other applications. However, forbasic statistical kernels we obtain a speedup of upto 30x and for PCA the speedups achieved are ofthe order of 35x when compared to a single CPUimplementation.

Figure 8 shows the performance results forlarger datasets when the entire dataset does notfit into the GPU device memory. Data is dividedinto smaller tiles of size 768K records and thenumber of CUDA Streams used is 4. We noticethat the speedup in this case is somewhat smalleras compared to K-Means in Figure 7. This canbe attributed to two reasons. First, since the datatransfer time (between CPU and GPU) is notnegligible, copying data for all iterations incursextra overhead lowering the performance gains.Second, since the kernels take less time comparedto the memory transfer, this overhead cannot beeliminated even using CUDA Streams. Hence,the speedup saturates around 25 compared to a

Figure 7: Performance improvement for K-Means andFuzzy K-Means for K=20

Figure 8: K-Means implementation for large dataset

speedup of 40 in Figure 7.

B. Effects of Scheduling Optimizations

In Figure 9, we show the performance gainachieved by interleaving different kernels. As anexample, we consider K-Means with differentnumber of clusters (which is a parameter for K-Means) as different applications. The results showthe speedups obtained for 2 to 7 applications rel-ative to a single application for data sizes rangingfrom 4 million to 20 million records. We noticethat as more kernels are interleaved for the sameamount of data transferred to the device memory,the speedup increases. The speedup increases from1.6x for two applications and saturates around2x as the number of applications are increasedbeyond 4. It should be noted that the speedupof 2x w.r.t. a single application amounts to anoverall performance gain of 50x when comparedagainst a single threaded CPU implementation.The saturation is attributed to the fact that memorycopy time is completely hidden by the kernelexecution time and any further addition of kernelswill not result in a performance gain.

Figure 9: Performance evaluation with interleaving kernels

172617221722

Figure 10: K-Means on heterogeneous platform(GPU+CPU)

C. ScalabilityWe evaluate the performance of the scalability

infrastructure provided by our framework by scal-ing the above applications on a homogeneous andheterogeneous cluster of machines. Our heteroge-neous environment consists of GPUs and CPUs.Figure 10 shows the execution times for differentratios of data distribution between the CPUs andGPUs for K-Means clustering algorithm. The hy-brid middleware invokes CPU kernels and GPUkernels optimized for large datasets for distancecomputation and cluster update. We vary the datadistribution ratio from 20 to 34 and notice thatwe achieve the best performance around the ratioof 29. This heterogeneous implementation resultsin a performance gain of around 9% compared toGPU-only implementation.

Figure 11 shows the scalability of K-Meansalgorithm on a homogeneous cluster of GPUsusing our framework. The framework achieves7.2x speedup when the number of GPUs increasesfrom 1 to 8.

Figure 11: Scalability results for cluster of GPUs

VI. CONCLUSIONS AND FUTURE WORK

In this paper we have presented a scalableframework for developing and using data miningalgorithms. Our framework spans across differenttechnologies to harness their capabilities. Datamining techniques require heavy computations anddeal with huge amounts of data. To provide ascalable environment we have provided an effi-cient interface to handle compute-intensive tasksas well as a parallel I/O interface for optimizedread/write of the data. We have further describedoptimizations which can be easily implementedusing the framework. We introduce the concept ofmultiple kernel optimizations, which runs kernelsfrom different applications for each data transfer.We also present a middleware for heterogeneouscomputations enabling both GPUs and CPUs worktogether. In future, other architectures and moredata mining applications can also be easily inte-grated. Our framework provides the flexibility ofintegrating newer kernels and optimizations easilyin the framework. It provides a library of (highlyoptimized) high performance kernels which arecommonly used in data mining algorithms. The re-sults show that we can achieve significant speedupwith our optimizations. We also present, throughcase studies, how the framework can be used towrite scalable applications.

ACKNOWLEDGMENT

This work is supported in part by NSFaward numbers: CCF-0621443, SDCI OCI-0724599, CNS-0551639, IIS-0536994, andHECURA-0938000. This work is also partiallysupported by DOE grants DE-FC02-07ER25808,DE-SC0005309, DE-SC0005340, and DE-FG02-08ER25848.

REFERENCES

[1] Clementine ver. 12, SPSS Corporation, http://www.spss.com/clementine.

[2] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reute-mann, and I. H. Witten, “The weka data miningsoftware: An update,” SIGKDD Explorations, vol. 11,no. 1, 2009.

[3] An Introduction to R, http://cran.r-project.org/doc/manuals/R-intro.pdf.

172717231723

[4] Rmpi: Wrapper to MPI (Message Passing In-terface), http://cran.r-project.org/web/packages/Rmpi/index.html.

[5] SNOW: Simple Network of Workstatsions, http://cran.r-project.org/web/packages/snow/index.html.

[6] Rdsm: Threads-Like Environment for R, http://cran.r-project.org/web/packages/Rdsm/index.html.

[7] N. F. Samatova, M. Branstetter, A. R. Ganguly, R. Het-tich, A. Shoshani, and S. Yoginath, “High performancestatistical computing with parallel r: Applications tobiology and climate modelling,” Journal of Physics,2006.

[8] X. Ma, J. Li, and N. F. Samatova, “Automatic paral-lelization of scripting languages: Toward transparentdesktop parallel computing,” in IEEE InternationalParallel and Distributed Processing Symposium, LongBeach, CA, March 2007, pp. 1–6.

[9] J. Buckner, J. Wilson, M. Seligman, B. Athey, S. Wat-son, and F. Meng, “The gputools package enables gpucomputing in r,” Bioinformatics, vol. 26, no. 1, pp.134–135, 2010.

[10] NVIDIA CUDA SDK, NVIDIA Corporation,http://developer.download.nvidia.com/compute/cuda/sdk/website/samples.html.

[11] Magma: Matrix Algebra on GPU and MulticoreArchitectures, http://cran.r-project.org/web/packages/magma/index.html.

[12] A. Mooley, K. Murthy, and H. Singh, “Dismarc: Adistributed map reduce framework on cuda,” Universityof Texas, Austin, Tech. Rep.

[13] O. Lawlor, “Message passing for gpgpu clusters: cud-ampi,” in Proceedings of the IEEE Cluster, 2009.

[14] Z. Fan, F. Qiu, A. Kaufman, and S. Yoakum-Stover,“Gpu cluster for high performance computing,” inProceedings of the IEEE Supercomputing Conference,Pittsburgh, PA, November 2004.

[15] B. G. Aaby, K. S. Perumalla, and S. K. Seal, “Ef-ficient simulation of agent-based models on multi-gpu and multi-core clusters,” in Proceedings of theSIMUTools Conference, Torremolinos, Malaga, Spain,March 2010.

[16] D. A. Jacobsen, J. C. Thibault, and I. Senocak, “Anmpi-cuda implementation for massively parallel in-compressible flow computations on multi-gpu clus-ters,” 48th AIAA Aerospace Sciences Meeting andExhibit, January 2010.

[17] M. Fatica, “Accelerating linpack with cuda on het-erogenous clusters,” in Proceedings of the Workshopon General-Purpose Computation on Graphics Pro-cessing Units, Washington, D.C., March 2009.

[18] R. Wu, B. Zhang, and M. Hsu, “Clustering billions ofdata points using gpus,” Unconventional High Perfor-

mance Computing Workshop, pp. 1–6, 2009.[19] R. Farivar, D. Rebolledo, E. Chan, and R. H. Cam-

bell, “A parallel implementation of k-means clusteringon gpus,” International Conference on Parallel andDistributed Processing Techniques and Applications,2008.

[20] B. Hong-tao, H. Li-li, O. Dan-tong, L. Zhan-shan,and L. He, “K-means on commodity gpus with cuda,”in WRI World Congress on Computer Science andInformation Engineering, vol. 3, 2009, pp. 651–655.

[21] S. A. Shalom, M. Dash, and M. Tue, “Efficient k-means clustering using accelerated graphics proces-sors,” International Conference on Data Warehousingand Knowledge Discovery, pp. 166–175, 2008.

[22] R. Wu, B. Zhang, and M. Hsu, “Gpu accelerated largescala analytics,” HP Labs, Tech. Rep. HPL-2009-38,2009.

[23] J. D. Hall and J. C. Hart, “Gpu acceleration of iterativeclustering,” The ACM Workshop on General PurposeComputing on Graphics Processors, August 2004.

[24] J. Li, W. Liao, A. Choudhary, R. Ross, R. Thakur,W. Gropp, and R. Latham, “Parallel netcdf: A scien-tific high-performance i/o interface,” in Processings ofSupercomputing Conference, November 2003.

[25] NVIDIA CUDA Programming Guide, NVIDIACorporation, http://developer.download.nvidia.com/compute/cuda/3 0/toolkit/docs/NVIDIA CUDAProgrammingGuide.pdf.

[26] R. K. Rew and G. P. Davis, “Netcdf: An interface forscientific data access,” IEEE Computer Graphics andApplications, July 1990.

[27] J. B. MacQueen, “Some methods for classification andanalysis of multivariate observations,” in 5th BerkeleySymposium on Mathematical Statistics and Probabil-ity, 1967, pp. 281–297.

[28] J. C. Bezdek, Pattern recognition with fuzzy objectivefunction algorithms. Kluwer Academic Publishers,1981.

[29] J. C. Dunn, “A fuzzy relative of the isodata process andits use in detecting compact well-separated clusters,”Journal of Cybernetics, vol. 3, pp. 32–57, January1974.

[30] I. T. Jolliffe, Principal Component Analysis. Springer-Verlag, 1986.

[31] J. H. Wilkinson, The Algebraic Eigenvalue Problem.London: Oxford University Press, 1965.

[32] C. Lessig, Eigenvalue Computation with CUDA,NVIDIA Corporation, October 2007.

[33] J. E. V. Ness, “Inverse iteration method for findingeigenvectors,” IEEE Tansactions on Automatic Control,pp. 63–66, February 1969.

172817241724