The Role of Agents in Distributed Data Mining: Issues and Benefits

The Role of Agents in Distributed Data Mining:

Issues and Benefits Josenildo Costa da Silva 1, Matthias Klusch 1, Stefano

Lodi 2, Gianluca Moro 2, Claudio Sartori 2

1Deduction and Multiagent Systems,German Research Center for Artificial Intelligence,

Saarbruecken, Germany2Department of Electronics, Computer Science and

Systems,University of Bologna,

Bologna, Italy

22/0

4/23

AgentLink III: TFG1 IIA4WE, Roma2

Distributed Data Mining (DDM)

• Data sets – Massive – Inherently distributed

• Networks– Limited bandwidth– Limited computing resources at nodes

• Privacy and security– Sensitive data– Share goals, not data

22/0

4/23


Centralized solution• Apply traditional DM algorithms to

data retrieved from different sources and stored in a data warehouse

• May be impractical or even impossible for some business settings– Autonomy of data sources– Data privacy – Scalability (~TB/d)

22/0

4/23


Agents and DDM• DDM exploits distributed processing and

problem decomposability• Is there any real added value of using

concepts from agent technology in DDM?– Few DDM algorithms use agents– Evidence that cooperation among distributed

DM processes may allow effective mining even without centralized control

– Autonomy, adaptivity, deliberative reasoning naturally fit into the DDM framework

22/0

4/23


State of the Art• BODHI

– Mobile agent platform/Framework for collective DM on heterogeneous sites

• PADMA– Clustering homogeneous sites– Agent based text classification/visualization

• JAM– Metalearning, classifiers

• Papyrus– Wide area DDM over clusters– Move data/models/results to minimize network load

22/0

4/23


Agents for DDM (pros)• Autonomy of data sources• Scalability of DM to massive

distributed data• Multi-strategy DDM• Collaborative DM

22/0

4/23


Agents for DDM (against)• Need to enforce minimal privileges

at a data source– Unsolicited access to sensitive data– Eavesdropping– Data tampering– Denial of service attacks

22/0

4/23


The Inference Problem• Work in statistical DB (mid 70’s)• Integration/aggregation at the

summary level is inherent in DDM– Infer sensitive data even from partial

integration to a certain extent and with some probability (inference problem)

– Existing DDM systems are not capable of coping with the inference problem

22/0

4/23


Data Clustering• Popular problem

– Statistics (cluster analysis)– Pattern Recognition– Data Mining

• Decompose multivariate data set into groups of objects– Homogeneity within groups– Separation between groups

22/0

4/23


DE-clustering

• Clustering based on non-parametric density estimation– Construct an estimate of the

probability density function from the data set

– Objects “attracted” by a local maximum of the estimate belong to the same cluster

22/0

4/23


Kernel Density Estimation• • The higher the number of data objects

in the neighbourhood of x, the higher density at x

• A data object exerts more influence on the value of the estimate at x than any data object farther from x than xi

• The influence of data objects is radial

22/0

4/23


Formalizing Density Estimators

• The density estimate at a space object x is proportional to a sum of weights

• The sum consists of one weight for every data object

• Weight is a monotonically decreasing function (kernel ) of the distance between x and xi scaled by a factor h (window width )

22/0

4/23


Kernel Functions• Uniform kernel

-3 -2 -1 1 2 3

0.1

0.2

0.3

0.4

0.5

22/0

4/23


Kernel Functions• Triangular kernel

-3 -2 -1 1 2 3

0.2

0.4

0.6

0.8

1

22/0

4/23


Kernel Functions• Epanechnikov’s kernel

-3 -2 -1 1 2 3

0.05

0.1

0.15

0.2

0.25

0.3

22/0

4/23


Kernel Functions• Gaussian kernel

-3 -2 -1 1 2 3

0.1

0.2

0.3

0.4

22/0

4/23


Example (1/2)

• Uniform kernel, h=250

2000 4000 6000 8000 10000

0.0001

0.0002

0.0003

0.0004

0.0005

22/0

4/23


Example (2/2)

• Gaussian kernel, h=250

2000 4000 6000 8000 10000

0.0001

0.0002

0.0003

0.0004

0.0005

22/0

4/23


Distributed Data Clustering (1/2)

• Clustering algorithm A( )• Homogeneous distributed data

clustering problem for A:– Data set S– Sites Lj

– Lj stores data set Dj withM

jj SD

1

22/0

4/23


Distributed Data Clustering (2/2)

• Problem: find clustering Cj in the data space of Lj such that:– Cj agree with A(S) (correctness

requirement):

– Time/communication costs are minimized (efficiency requirement)

– The size of data transferred out of the data space of any Lj is minimized (privacy requirement)

22/0

4/23


Traditional (centralized) solution

• Gather all local data sets into one centralized repository (e.g., a data warehouse)

• Run A( ) on the centralized data set• Unsatisfied privacy requirement• Unsatisfied efficiency requirement

for some A( )

22/0

4/23


Sampling• Goal: given some class of functions of type

represent every member as a sampling series

where:– is a collection of points of – is some set of suitable expansion functions

22/0

4/23


Example• The class of polynomials of degree

1

– Sampling points– Expansion functions

• Finite sum

22/0

4/23


Band-limited Functions• Function f of one real variable• Range of frequencies of a function f

support of the Fourier transform of f

• Any function whose range of frequencies is confined to a bounded set B is called band-limited to B (the band-region)

22/0

4/23


Example: sinc function

-6 -4 -2 2 4 6

-0.4

-0.2

0.2

0.4

0.6

0.8

1

-7.5 -5 -2.5 2.5 5 7.5

0.2

0.4

0.6

0.8

1

22/0

4/23


Sampling Theorem • If f is band-limited with band-

region

then

22/0

4/23


Sampling Theorem (scaled multidimensional version)

• Let– – – where is the -th component of a

vector• If f is band-limited to B then

22/0

4/23


Sampling Density Estimates (1/4)

• Additivity of density estimates of a distributed data set

22/0

4/23



• The sampling series of the density estimate is also additive

where

22/0

4/23



• Truncation errors– The support of a kernel function is not

bounded in general• Aliasing errors

– The support of the Fourier transform of a kernel function is not bounded in general kernel functions are not band-limited

22/0

4/23



• The sampling series of a density estimate can only be approximated

• Trade-off between the number of samples and accuracy– Define a minimal multidimensional

rectangle outside which samples are negligible

– Define a vector of sampling intervals such that the aliasing error is negligible

22/0

4/23


– Applies DE-clustering to Dj and

– Sends the summation back to each Lj

– Orderly sums the samples

– Waits for the samples of local density estimates

– Samples – Sends the samples

to H

– Reconstructs from its samples

The KDEC scheme– Computes a local

density estimate of its data Dj

– Waits for the samples of the global density estimate ][S

][S

][ jD][ jD

][ jD

• Every site Lj:

• Helper H:

22/0

4/23


The KDEC schemeHelper

Site1 Site2

22/0

4/23



0

2

4 x

-4

-2

0

y

00.511.52

0

2

4 x

0

2

4x

-4

-2

0

2

y00.511.52

0

2

4x

22/0

4/23



0

2

4 x

-4

-2

0

y

00.511.52

0

2

4 x

-5-2.5

02.5

5x

-5

0

5

y

-5-2.5

02.5

5x

22/0

4/23



-5-2.5

02.5

5x

-5

0

5

y

-5-2.5

02.5

5x

-5-2.5

02.5

5x

-5

0

5

y

-5-2.5

02.5

5x

22/0

4/23


Helper

-20

24

6

-6-4

-20

2

00.050.10.15-2

02

46

-6-4

-20

2

-20

24

6

-6-4

-20

2

00.050.10.15-2

02

46

-6-4

-20

2

-20

2

4

6

-6

-4

-2

0

2

0

2

4

-20

2

4

6

-20

2

4

6

-6

-4

-2

0

2

0

2

4

-20

2

4

6

-20

2

4

6

-6

-4

-2

0

2

0

2

4

-20

2

4

6

The KDEC scheme

-20

2

4

6

-6

-4

-2

0

2

0

2

4

-20

2

4

6

-20

2

4

6

-6

-4

-2

0

2

0

2

4

-20

2

4

6

-5-2.5

02.5

5x

-5

0

5

y

-5-2.5

02.5

5x

-5-2.5

02.5

5x

-5

0

5

y

-5-2.5

02.5

5x

-1 1 2 3 4

-4

-3

-2

-1

1

2

-1 1 2 3 4 5

-5

-4

-3

-2

-1

1

22/0

4/23


Properties of the approach• Communication complexity depends only

on the number of samples • Data objects are never transmitted over

the network• Local clusters are close to global clusters

which can be obtained using DE-cluster• Time complexity does not exceed the time

complexity of centralized DE-clustering

22/0

4/23


Window width and sampling frequency

• Good estimates when h is not less than a small multiple of the smallest distance between objects

• As , the number of samples rarely exceeds the number of data points

22/0

4/23


Complexity• Site j

– Sampling: O(q(N) Sam) – DE-cluster: O(|Dj|q(Dj))

• Helper– Summation of samples: O(Sam)

• Communication– Time: O(Sam)– Volume: O(M Sam)

22/0

4/23


Complexity(centralized approach)

• Site j– Transmission/Reception of data

objects: O(|Dj|)• Helper

– Global DE-clustering: O(N q(N))• Communication:

– Time: O(N)– Volume: O(N)

22/0

4/23


Stationary agent-based KDEC• The helper engages

site agents to agree on:– Kernel function– Window width– Sampling frequencies– Sampling region

• The global sampled form of the estimate is computed in a single stage

22/0

4/23


Mobile agent-based KDEC• At site Ln the visiting

agent:– Negotiates kernel

function, window width, sampling frequencies, sampling region

– Carries the sum of samples collected at Lm, m<n, in its data space

• The global sampled form of the estimate is returned to the interested agents

22/0

4/23


A Hierarchical Scheme• Additivity allows to

extend the scheme to trees of arbitrary arity

• Local sampled density estimates are propagated upwards in partial sums, until the global sampled DE is computed at the root and returned to the leaves •May provide more

protection against disclosure of DEs

22/0

4/23


Inference and Trustworthiness

• Inference problem for kernel density estimates– Goal of inference attacks: exploit

information contained in a density estimate to infer the data objects

• Trustworthiness of helpers– Trustworthy helper no bit of information

written to memory by a process for the Helper procedure is sent to a system peripheral by a different process

22/0

4/23


• Let

be extensionally equal to a density estimate:

• For example, g is the reconstructed density estimate (sampling series)

Inference Attacks on Kernel Density Estimates

22/0

4/23



• Simple strategy: Search the density estimate or its derivatives for discontinuities

• Example: The kernel is the square pulse– For each pair of projections of objects on an axis

there is a pair of projections of discontinuities on that axis having the same distance as the objects’ projections

– If h is known then the objects can be inferred easily• If the kernel has discontinuous derivatives,

then the same technique applies to the derivatives

22/0

4/23



500 1000 1500 2000

0.0001

0.0002

0.0003

0.0004

0.0005

• If g is not continuous at x an object lies at

h=250

22/0

4/23



• If the kernel is infinitely differentiable the problem is more difficult

• Select space objects and attempt to solve a nonlinear system of equations

22/0

4/23


Attack Scenarios• Single-site attack

– One of the sites attempts to infer the data objects from the global density estimate

– Unable to associate a specific data object to a specific site

• Site coalition attack– A coalition computes the sum of the density

estimates of all the other sites as difference– Special case: the coalition includes all sites

but one the attack potentially reveals the data objects at the site

22/0

4/23


Untrustworthy Helpers• Reputation binary random

variable with probabilities p and 1-p – p is the probability that the helper is

untrustworthy– If the agent community supports

referrals about an agent's reputation as a helper, then an agent might know per agent probabilities

22/0

4/23


Conclusions• Agent technology in DDM

– Preserves the autonomy of data sources and scalability of the DM step

– Privacy protection (inherent in most DDM approaches) may be less effective

• Agent-based distributed density estimate computation & clustering– Scalable– Implementation based on mobile agents– Could be vulnerable to inference attacks on

density estimates perpetrated by coalitions

22/0

4/23


Work in progress (1)• Inference attacks on sampled

density estimates by solving the corresponding system of NL equations– Globally convergent inexact Newton

methods with constraints [Bellavia, Macconi, Morini 2003]

– Gradient method

22/0

4/23


Work in progress (2)• Inference attacks on sampled density

estimates by iteratively reducing the dimensionality of the system of NL equations– pointhunt algorithm– Proceeds iteratively by

• selecting a point x close to the “border” of the density estimate

• guessing an object such that the object is the only contributor to density estimate at x

22/0

4/23


Work in progress (3)• Ontology and protocol• Algorithms for deliberative participation

based on trustworthiness referrals• Probabilities that a local density estimate

may be learned by another agent– Probability that no other agent learns the

density estimate– Probability that at least k other agents learn the

density estimate– Probability that exactly k other agents learn the

density estimate

22/0

4/23


Future Work• Algorithms for the negotiation of

parameters• Formalization of errors

– Bounds on aliasing errors– Clustering errors (e.g., using an index

of partition difference)

22/0

4/23


Thanks!

The Role of Agents in Distributed Data Mining: Issues and Benefits

Documents

tfg1 iia4we

data warehousemay

data set objects

number of data objects

xa data object

ddminfer sensitive data

data sourceunsolicited

ddm algorithms