Top Banner
Linear Separability George M. Georgiou Outline The problem Separating hyperplane Euclidean Geometry Non-Euclidean Geometry Three solutions Conclusion 1 : 29 Linear Separability in Non-Euclidean Geometry George M. Georgiou, Ph.D. Computer Science Department California State University, San Bernardino March 3, 2006 [email protected]
38

Linear Separability in Non-Euclidean Geometry

Sep 12, 2021

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

1 : 29

Linear Separability in Non-EuclideanGeometry

George M. Georgiou, Ph.D.

Computer Science DepartmentCalifornia State University, San Bernardino

March 3, 2006

[email protected]

Page 2: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problemLinear Separability

Geometrically

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

2 : 29

The problem

Find a geodesic that separates two given sets of points onthe Poincare disk non-Euclidean geometry model.

Page 3: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problemLinear Separability

Geometrically

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

2 : 29

The problem

Find a geodesic that separates two given sets of points onthe Poincare disk non-Euclidean geometry model.

Page 4: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problemLinear Separability

Geometrically

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

3 : 29

Linear Separabilityin Machine Learning

Machine Learning is branch of AI, which includes areassuch as pattern recognition and artificial neural networks.

Each point X = (x1, x2, . . . , xn) ∈ Rn (vector or object)belongs to two class C0 and C1 with labels -1 and 1,respectively.

The two classes are linearly separable if there existsW = (w0, w1, w2, . . . , wn) ∈ Rn+1 such that

w0 + w1x1 + w2x2 + . . . + wnxn> 0, if X ∈ C0 (1)

w0 + w1x1 + w2x2 + . . . + wnxn< 0, if X ∈ C1 (2)

Page 5: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problemLinear Separability

Geometrically

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

4 : 29

Geometrically

Page 6: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problemLinear Separability

Geometrically

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

5 : 29

“Linearizing” the circle

(x , y) −→ (x , y , x2 + y2) (3)

Since any circle can be written as

w0 + w1x + w2y + w3(x2 + y2) = 0 (4)

Page 7: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problemLinear Separability

Geometrically

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

6 : 29

Separating hyperplane

w0 + w1x1 + w2x2 + . . . + wnxn= 0n = 2, 2-D: line

n = 3, 3-D: plane

n > 3, higher dimensions: hyperplane

Page 8: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

SeparatinghyperplanePerceptron

Linear inequalities

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

7 : 29

Finding the separating hyperplane

A number of ways

Use the perceptron algorithm

Solve a system of linear inequalities

Use Support Vector Machine (SVM) method. It maximizesseparation between the two classes.

Page 9: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

SeparatinghyperplanePerceptron

Linear inequalities

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

7 : 29

Finding the separating hyperplane

A number of ways

Use the perceptron algorithm

Solve a system of linear inequalities

Use Support Vector Machine (SVM) method. It maximizesseparation between the two classes.

Page 10: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

SeparatinghyperplanePerceptron

Linear inequalities

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

7 : 29

Finding the separating hyperplane

A number of ways

Use the perceptron algorithm

Solve a system of linear inequalities

Use Support Vector Machine (SVM) method. It maximizesseparation between the two classes.

Page 11: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

SeparatinghyperplanePerceptron

Linear inequalities

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

8 : 29

The perceptron (Rosenblatt, 1958)

w0

f(y)

w4

w3

w2

w1

x4

x3

x2

1x1

f (y) =

1, if y > 0

−1, if y < 0(5) y =

i=n∑i=0

wi xi (6)

Perceptron Algorithm1 Start with W = 0.2 Present input vector X and compute error ε = d − f (y).3 Update W using ∆W = α ε X4 Pick another input vector X and goto to step 2.

Page 12: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

SeparatinghyperplanePerceptron

Linear inequalities

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

9 : 29

Solving linear inequalities

Can use linear programming, e.g. simplex method,Karmakar, and variants.

Polynomial complexity

In fact, when dimension is fixed, e.g. d=2,3 as in our case,complexity is linear in the number of input vectors.

Page 13: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

10 : 29

Euclidean Geometry

Axioms of Euclid (ca. 400 BC)

I. Two points determine a unique line.

II. A line is of infinite length.

III. A circle may be described with any center at any distancefrom the center.

IV. All right angles are equal.

V. If a straight line meet two other straight lines, so as tomake the two interior angles on one side of it together lessthan two right angles, the other straight lines will meet ifproduced on that side on which the angles are less thantwo right angles.

(from H.S.M. Coxeter, Non-Euclidean Geometry)

Page 14: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometryPoincare Disk

Three solutions

Conclusion

11 : 29

Non-Euclidean Geometry

Euclid’s Elements considered to be the most successfultextbook of all times.

Nicolai Lobachevsky (1829) and Janos Bolyai (1832) firstto publish geometries without the Euclid’s axiom V.

Gauss feared the “howling of the Boetians.”

Types on Non-Euclidean Geometry

Lobachevskian (hyperbolic geometry) Geometry: from apoint not on a line, an infinite number of parallel lines canbe drawn.

Riemannian (elliptic) geometry: no parallel lines can bedrawn.

Page 15: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometryPoincare Disk

Three solutions

Conclusion

12 : 29

Poincare disk

M

N

b

b

Page 16: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometryPoincare Disk

Three solutions

Conclusion

12 : 29

Poincare disk

M

N

P

b

b

b

Page 17: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometryPoincare Disk

Three solutions

Conclusion

13 : 29

Distance function

The distance between points z1 and z2, taken as complexnumbers, is

D(z1, z2) = | log(z1, z2;µ, λ)|, (7)

where z1 and z2 are points on the unit disk, and µ and λ arethe intersections of the geodesic defined by z1 and z2 with theunit circle. The cross ratio (z1, z2;µ, λ), is defined as

(z1, z2;µ, λ) =z1 − µ

z1 − λ

/z2 − µ

z2 − λ(8)

Page 18: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometryPoincare Disk

Three solutions

Conclusion

14 : 29

Art by M.C. Esher (1878–1972)Limit Circle III

Page 19: Linear Separability in Non-Euclidean Geometry
Page 20: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometryPoincare Disk

Three solutions

Conclusion

16 : 29

The problem

Find a geodesic that separates two given sets of points onthe Poincare disk non-Euclidean geometry model.

Page 21: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometryPoincare Disk

Three solutions

Conclusion

16 : 29

The problem

Find a geodesic that separates two given sets of points onthe Poincare disk non-Euclidean geometry model.

Page 22: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

17 : 29

Three solutions

1 Map points from Poincare disk to the Klein disk, findseparating line, which maps to a desired geodesic on thePoincare disk.

2 Map points from Poincare disk to the Hyperboloid model,find separating plane, which maps to a desired geodesicon the Poincare disk.

3 Map points from Poincare disk to circular paraboloid, findseparating plane, which maps to a desired geodesic onthe Poincare disk. (This is a novel map.)

Page 23: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

17 : 29

Three solutions

1 Map points from Poincare disk to the Klein disk, findseparating line, which maps to a desired geodesic on thePoincare disk.

2 Map points from Poincare disk to the Hyperboloid model,find separating plane, which maps to a desired geodesicon the Poincare disk.

3 Map points from Poincare disk to circular paraboloid, findseparating plane, which maps to a desired geodesic onthe Poincare disk. (This is a novel map.)

Page 24: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

17 : 29

Three solutions

1 Map points from Poincare disk to the Klein disk, findseparating line, which maps to a desired geodesic on thePoincare disk.

2 Map points from Poincare disk to the Hyperboloid model,find separating plane, which maps to a desired geodesicon the Poincare disk.

3 Map points from Poincare disk to circular paraboloid, findseparating plane, which maps to a desired geodesic onthe Poincare disk. (This is a novel map.)

Page 25: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

18 : 29

The Klein disk

Points, like those of the Poincare disk, are the usualEuclidean points.

Lines are the familiar straight Euclidean lines, but ofcourse are bounded but the unit circle.

Angles, unlike those in the Poincare disk, are distorted.

Page 26: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

18 : 29

The Klein disk

Points, like those of the Poincare disk, are the usualEuclidean points.

Lines are the familiar straight Euclidean lines, but ofcourse are bounded but the unit circle.

Angles, unlike those in the Poincare disk, are distorted.

Page 27: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

18 : 29

The Klein disk

Points, like those of the Poincare disk, are the usualEuclidean points.

Lines are the familiar straight Euclidean lines, but ofcourse are bounded but the unit circle.

Angles, unlike those in the Poincare disk, are distorted.

Page 28: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

19 : 29

The two geodesics

B

A

Blue: Poincare

Red: Klein

Page 29: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

20 : 29

From the Poincare to the Klein disk, and back

(0,0,1)

PK

F (x , y) =

(2x

1 + x2 + y2 ,2y

1 + x2 + y2

)(9)

F−1(x , y) =

(x

1 +√

1 − x2 − y2,

y

1 +√

1 − x2 − y2

)(10)

Page 30: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

21 : 29

1st Solution

1 Map all given points from the Poincare disk to the Kleindisk

2 Find a separating line3 Map the two points of the intersection of the line and the

unit circle to the Poincare disk4 Find geodesic that corresponds to the two points

Page 31: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

22 : 29

The Hyperboloid model

H = (x , y ,√

x2 + y2 + 1) : x , y ∈ R (11)

Geodesics are intersections of planes through the origin.

Page 32: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

23 : 29

From Poincare to the Hyperboloid, and back

Poincare Disk

Klein Disk

(x,y,z)

P

P = r + i s (12)

F (P) =

(2r

1 − |P |2,

2s1 − |P |2

,1 + |P |2

1 − |P |2

)(13)

F−1(x , y , z) =x + i y1 + z

(14)

Page 33: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

24 : 29

2nd Solution

1 Map all given points from the Poincare disk to thehyperboloid

2 Find a separating plane that passes through the origin3 Map two points on the intersection of the plane and the

hyperboloid to the Poincare disk4 Find geodesic that corresponds to the two points

Page 34: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

25 : 29

The circular paraboloid I

Ω = (x , y , x2 + y2 + 1) : x , y ∈ R, x2 + y2 < 1 (15)

(x,y,z)

(x,y)

Fact: Circles on the x-y plane are projections of ellipses onf (x , y) = x2 + y2 + 1. (D. Pedoe, Geometry, 1988) showsthis for the surface (x , y , x2 + y2).

Page 35: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

26 : 29

The circular paraboloid II

(x,y,z)

(x,y)

It can be shown: the ellipse on f (x , y) = x2 + y2 + 1 is theintersection of a plane that passes through the origin.

It can be shown: the corresponding circles on the x-yplane are orthogonal to the unit circle.

A geodesic on the Poincare disk maps to an arc of anellipse on Ω that lies on a plane through the origin.

Page 36: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

27 : 29

3rd Solution

1 Map all given points from the Poincare disk to Ω

2 Find a separating plane that passes through the origin3 Map two points on the intersection of the plane and Ω to

the Poincare disk (just discard the z coordinate)4 Find geodesic that corresponds to the two points

Page 37: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutionsKlein disk

Hyperboloid model

Circular Paraboloid

Conclusion

28 : 29

Advantages of the 3rd solution

Mapping to and from is simple.

No division necessary

Unlike 2nd method, no overflow possible in mapping

Page 38: Linear Separability in Non-Euclidean Geometry

LinearSeparability

George M.Georgiou

Outline

The problem

Separatinghyperplane

EuclideanGeometry

Non-EuclideanGeometry

Three solutions

Conclusion

29 : 29

Conclusion

Three ways given two find a geodesic that separates twoclasses of vectors on the Poincare disk.

A new way to visualize the Poincare disk is given.