Μοντέλα Παλινδρόμησης και Επεξεργασία Γνώσης ΧΕΙΜΕΡΙΝΟ ΕΞΑΜΗΝΟ (5 ο ) Regression Models and Knowledge Processing WINTER SEMESTER (5 th ) Τμημα Μαθηματικων Αριστοτελειο Πανεπιστημιο Θεσσαλονικης 54124 School of Mathematics Aristotle University of Thessaloniki 54124 4. Δικτυα και Νευρωνικα Δικτυα Iωαννης Αντωνιου Χαραλαμπος Μπρατσας iantonio @math.auth.gr cbratsas @math.auth.gr Το παρόν εκπαιδευτικό υλικό υπόκειται σε Αδεια Χρήσης Creative Commons
59
Embed
Επεξεργασία Γνώσης Knowledge Processing (5ο WINTER …cosal.auth.gr/iantonio/sites/default/files... · Kandel. 2007, In Search of Memory: The Emergence of a New
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Μοντέλα Παλινδρόμησης καιΕπεξεργασία Γνώσης
ΧΕΙΜΕΡΙΝΟ ΕΞΑΜΗΝΟ (5ο)
Regression Models andKnowledge Processing
WINTER SEMESTER (5th)
Τμημα ΜαθηματικωνΑριστοτελειο Πανεπιστημιο
Θεσσαλονικης54124
School of MathematicsAristotle Universityof Thessaloniki54124
Bilinear Agreggation: g(𝜓𝜈 , 𝑤𝛼𝛽)= 𝜆𝜓𝜆𝑤𝜆𝜅Since the early dates of the study of ΝΝ, the bilinear function has been used to model the aggregated input on each node at time t
Used in PerceptronRosenblatt F. 1958, The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain, Psychological Review 65, 386-408
NN Οutput Activation Maps = Transfer Functions EXAMPLESUnit Step Heaviside Map
φκ (x)= θ(x− ακ) = 1, 𝑖𝑓 𝑥 ≥ 𝑎𝜅0, 𝑖𝑓 𝑥 < 𝑎𝜅
ακ = the Activation Threshold of the Node κ,
We may consider Activation Maps with values in [−1,1]
Constant Weights - No Learning the simplest casew(t) = w , wκλ(t) = wκλ
The structure of the Net does not change with time
The first NN McCulloch and Pitts 1943 , A Logical Calculus of the Ideas Immanent in Nervous Activity,
Bull. Math. Biophysics 5, 115-133
McCulloch and Pitts admitted that this was not the case of the Nervous System
Η Κλασσικη Λογικη ως Αλγεβρα Βοοle
Ορισμος
Eνα Συνολο 𝔏 εφοδιασμενο με τις πραξεις (∨, ∧, ≤) καλειταιL1 Lattice
Πλεγμα
Bounded
Lattice
ΦραγμενοΠλεγμα
Complemented
Lattice
Συμπληρωμενο
Πλεγμα
Orthocomplemented
Lattice
Ορθοσυμπληρωμενο
Πλεγμα
Distributive
Orthocomplemented Lattice
= Boole Αlgebra
Επιμεριστικο
Ορθοσυμπληρωμενο Πλεγμα
= Αλγεβρα Boole
L2
L3
L4
L5
L6
L7
L8
L9
Law-Property Formula L1 Idempotent A ∨ A=A A ∧ A =A L2 Commutative A ∨ B =B ∨A A ∧B =B ∧ A L3 Associative A ∨ (B ∨ Γ)= (A ∨B) ∨ Γ A ∧ (B ∧ Γ) = (A ∧B) ∧ Γ L4 Absorption A ∧ (A ∨ B)= A A ∨ (A ∧B)= A L5 Order A ≤ B ⟺ A = A ∧ B A ≤ B ⟺ B = A ∨ B L6 Bounded O ∨ A = A I ∨ A =I O ∧ A =A I ∧ A =A L7 Complement A ∨ 𝐴𝑐 = I A ∧ 𝐴𝑐 = O L8 Orthocomplement (𝐴𝑐 )𝑐 =A L9 Distributivity A ∨(B ∧ Γ) = (A ∨ B) ∧(A ∨ Γ) A ∧ (B ∨ Γ) = (A ∧ B) ∨ (A ∧ Γ)
Θεωρηματα
1. Τα υποσυνολα ενος συνολου Υ αποτελουν την Αλγεβρα Boole:
(ℬ[𝑌], ∪, ∩, 𝒄 , ⊆, 𝑌 )
2. Θεωρημα Αναπαραστασης Stone:
Καθε Αφηρημενη Αλγεβρα Boole αναπαρισταται ως
Αλγεβρα Boole υποσυνολωνStone M.1936, The theory of representation for Boolean Αlgebras, Trans. Amer. Math. Soc. 40, 37-111
3. Θεωρημα Αναπαραστασης Shannon:
Καθε Αφηρημενη Αλγεβρα Boole αναπαρισταται ως
Αλγεβρα Boole ΔιακοπτωνShannon C. 1938, A Symbolic Analysis of Relay and Swiching Circuits, AMS Transactions 57, 713-723,
MIT Master Thesis
Lee C. 1959, Representation of Switching Circuits by Binary Decision Diagrams, Bell System Techn. J. 38, 985-999
Hebb, D.O. 1949. The Organization of Behavior. New York: John Wiley
The simple Hebbian Learning Rule is Unstable, becausethe synaptic weights will increase or decrease exponentiallyEuliano N. 1999, Neural and Adaptive Systems: Fundamentals Through Simulations. Wiley.
NN Learning by Supervision Widrow-Hoff rule or Delta rule
The adjusting of the weights wακ depends not the actual activation xκ(t) of the node κ but
on the difference between the actual activation xκ(t) and desired activation dκ provided by a teacher: wακ (t+1) = wακ (t) + η xα(t) [dκ(t)−xκ(t)]
wακ (t+1) = wακ (t) + η xα(t) [dκ−xκ(t)]
Learning Hebb Learning: Experimental EvidenceSome synaptic changes observed by Eric Kandel(Nobel in Physiology-Medicine 2000) provide examples of Hebbian learning in the marine gastropod Aplysia californica
Antonov I., Antonova I., Kandel E., Hawkins R. 2003, Activity-Dependent Presynaptic Facilitation and Hebbian LTP are Both Required and Interact during Classical Conditioning in Aplysia, Neuron, 37 (1): 135–147
Kandel. 2007, In Search of Memory: The Emergence of a New Science of Mind, New York: W. W. Norton & Company.
Haykin, S. 1999, Neural Networks: A comprehensive Foundation. Prentice Hall
Boltzmann Machines the activation as the probability of generating an action potential spike, and is determined via a logistic function on the sum of the inputs to a unit.
Ackley D. H., Hinton G. E., Sejnowski T. J. 1985, A Learning Algorithm for Boltzmann Machines,
Hinton G. E., Osindero S.,Teh Y. 2006, A fast learning algorithm for deep belief nets, Neural Computation 18 (7): 1527–1554. DOI:10.1162/neco.2006.18.7.1527. PMID 16764513.
Neural NetsNN are Information Processors able to learn from observed data with or without a supervisor, rather than having to be programmed
NN consist of Interconnected Processing Units called Neurons
NN are defined by:
• the Architecture: the (Directed) Graph of the Connections the Weight (strength) of the Connections
• The Activation Dynamics (Activation Update)• The Learning Dynamics (Weight Update)
NN are useful when the solution of a problem of interest is difficult due to:
• Lack of physical/statistical understanding of the problem
• Statistical variations in the observable data
• Complex (for example Non-Linear) mechanism underlying the data
Neural NetsΜαθηση με Εποπτεια-Επιβλεψη Supervised Learning: 1) a Supervisor provides the target (desired) outputs as
objectives
2) the weights are adjusted according to the difference
between the target (desired) and the actual outputs for a given input
Perceptrons
Supervised Learning Applications: • Function approximation, • regression analysis, • time series prediction • infer a function from observations • infer a model from observations, system identification• estimate and model in/out Response Ops underlying data
Neural NetsΜαθηση ανευ Επιβλεψης Unupervised Learning: 1) the target (desired) outputs are not specified
2) the weights are adjusted to cluster the inputs into groups with similar features
Kohonen Net = SO Maps
AR = Adaptive Resonance
CL = Competitive LearningKohonen T. 1995, Self-Organizing Maps, Springer-Verlag, Berlin
Unsupervised Learning Applications: • estimation problems • source separation • Classification• Filtering, (e-mail spam filtering)• Clustering - Categorization • Signal separation • Compression• Chaos Identification in Time Series
Neural NetsΜαθηση με Υποστηριξη (Διαδραση με Περιβαλλον)Relevance Feedback Reinforcement Learning: External Evaluation Criteria learned by Mutations
Environment as Supervisor (Markov Model)
Genetic Algorithms
Reinforcement Learning Applications:
• pattern recognition (radar systems, face identification, object recognition)
• sequence recognition (gesture, speech, handwritten text recognition)
• Pattern Association
• Novelty detection
• Game Playing, Decision making (backgammon, chess, racing, medical diagnosis, investments)
• Control (vehicle control, process control)
• Optimization
• Data mining (or knowledge discovery in databases, "KDD")
• Search
Pham D., Xing L. 1995, Neural Networks for Identification, Prediction and Control, Springer-Verlag, Berlin
Neural NetsΕπιτροπη Νευρωνικων ΔικτυωνCommittee of NNCollection of different NN that together "vote" on a given example.
This generally gives a much better result compared to other neural network models.
Starting with the same architecture and training but using different initial random weights gives a Variety of vastly different NN.
Committees tends to stabilize the result.
Committee Learning is similar to the general NN learning
Variety is obtained by training the NN Committee Membersfrom different random starting weights rather than from different randomly selected subsets of the training data.
are integrated to form a global impression of the individual
Neuron Model
Information Integration Theory.
Cognitive Algebra
Anderson N. 1971, Integration Theory and Attitude Change. Psychological Review 78, 171–206.
Anderson N. 1981, Cognitive Algebra: Integration Theory Applied to Social Attribution, In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 7, pp. 1–101), Academic Press, New York
Anderson N. 2013, Unified Psychology Based on Three Laws of Information Integration, Review of General Psychology 17, 125–132
Memory
Cognition
Emotion
Imagination
Will
Awareness-Consciousness of
• Sensing: Seeing
Hearing
Smelling
Tasting
Touching, Body
• Cognition
• Emotion
• Imagination
• Errors-Illusions
• Will
• Mind-Self (as Software)
Are supported by the Nervous System
Most probably within the Association Area
Emotion, Imagination, Will, Awareness-Consciousness Not shown (yet, despite the effort) to be
Functionalities of Mathematical NEURAL NETS or other NETS
Boolean Networks = SHANNON GRAPH = Binary Decision Diagram (BDD) propositional directed acyclic graphs (PDAG), as a data structures representing Boolean functionsLogical operations SGMany logical operations on SGs can be implemented by polynomial-time graph algorithms.• conjunction• disjunction• negation• existential abstraction• universal abstractionRepeating these operations several times, may in the worst case result in an exponential time.
Random Boolean Networks (ΡΝΒ) = Kauffman Networks proposed as models of Genetic Regulatory Networks Kauffman, S. A. 1969, Metabolic Stability and Epigenesis in randomly constructed genetic nets.
Journal of Theoretical Biology 22, 437-467.
Kauffman, S. A. 1993, Origins of Order: Self-Organization and Selection in Evolution,
Oxford University Press. ISBN 0-19-507951-5A
RBN is a system of N binary-state nodes (representing genes) with K inputs to each node representing regulatory mechanisms. The two states (on/off) represent respectively, the status of a gene being active or inactive. The state of a network at any point in time is given by the current states of all N genes. Simulation of RBNs is performed in discrete time steps.
Bayesian Networks , Graphical ModelsP[A=α|B=β] = the degree of belief that
the Variable A has the value α based on the fact that B=β
P[Ξ |Η] = P[Ξ∩Η]
P[Η]= P[Ξ]
P[Η]P[Η|Ξ]
P[x1, x2,…, xN ] = 𝑣,𝜅 P[𝑥𝜈|𝑥𝜈𝛼𝜅𝜈] (the parents of each node v)
• Τα δεδομένα περιγράφουν τον εαυτό τους• Αν μια εφαρμογή συναντήσει δεδομένα που περιγράφονται από
άγνωστο λεξιλόγιο, η εφαρμογή μπορεί να κάνει resolve τα URIsπου ταυτοποιούν όρους του λεξιλογίου για να εντοπίσει τον ορισμό τους κατά RDFS ή OWL.
•Ο Ιστός των Δεδομένων είναι ανοικτός• Οι εφαρμογές μπορούν να ανακαλύπτουν νέες πηγές δεδομένων
την ώρα που τρέχουν, ακολουθώντας links.
Τα Ανοικτά Δεδομένα εχουν την μεγαλύτερη «αξία» ως προς την δυνατότητα αξιοποιησής τους από εφαρμογές
Οι Γενεές του WebΕποχή Περιγραφή Πηγή ΥπεραξίαςΠρο-Web 1980 Επιτραπέζιοι ΗΥ Υπολογισμοί
Web1.0: 1990 Αναρτηση Αρχειων: Παροχος⟶ Χρηστης
Περιηγητης (Browser)
Υπολογισμοί
+ Διασύνδεση Αρχείων
(Documents)
Web2.0: 2000 Αναρτηση Αρχειων: Παροχος⇆ Χρηστης
Κοινωνικος Ιστος
Διαδραστικη Επικοινωνια
Συρματοπλεγματα στα Λειβαδια του
Κυβερνοχωρου
Web 1.0
+ Wikipedia
+ Κοινωνία Πληροφορίας
Ανταλλαγή Εμπειρίας
Πληροφορίας, Σφαλμάτων
Web3.0: 2010 Σημασιολογικό Web:
Η Οντολογία (Σημασιολογικό Δίκτυο)
Μηχανική Επεξεργασία Γνώσης
Νέφος Διασυνδεδεμένων Δεδομένων
Διαδίκτυο Πραγμάτων (Internet of Things)
(ΗΥ, Smartphones, Ψυγεία, Φούρνοι,
Αισθητήρες, Ζώα)
Web 2.0
+ Διασύνδεση Εννοιών
+ Συλλογισμοί
+ Μείωση Σφαλμάτων μέσω
Αποσαφήνισης
και Διάκρισης
Οντολογια Αριστοτελη 330 πΧΤην κατά µηδεµίαν συµπλοκή λεγοµένων έκαστον ήτοι ουσίαν σηµαίνει ή ποσόν ή ποιόν τι ή πού ή ποτέ ή κείσθαιή έχειν ή ποιείν ή πάσχειν. 'Εστι δε ουσία µεν ως τύπω ειπείν οίον άνθρωπος ίππος, ποσόν δε οίον δίπηχυ, τρίπηχυ- ποιόν δε οίον λευκόν γραµµατικόνπρος τι δε οίον διπλάσιον, ήµισυ, µείζον' πού δε οίον εν Λυκείω, εν αγορά' ποτέ δε οίον χθές, πέρισυν: κείσθαι δε οίον ανάκειται, κάθηται‘έχειν δε οίον δε υποδέδεται, ώπλισται: ποιείν δε οίον τέµνειν, καίειν πάσχειν δε οίον τέµνεσθαι, καίεσθαι, [Αριστοτελης Κατηγοριες 1,4]
Wolfram S. 2002, A New Kind of Science Wolfram Media,Campaign, ISBN 1-57955-008-8
Net of CellsFinite states (2adic ON, OFF)
An initial state (time t=0) is selected by assigning a state for each cell. A new generation is created (t + 1), according to some fixed rule that determines the new state of each cell in terms of the current state of the cell and the states of the linked cells Typically, the rule for updating the state of cells is the same for each cell and does not change over time, and is applied to the whole grid simultaneously
Example: the cell is "ON" in the next generation if exactly two of the cells in the neighborhood are "ON" in the current generation, otherwise the cell is "OFF" in the next generation
Cellular Automata Von Neumann 4-Neighbourhood
N
W E
S
Cellular Automata Moore 8-Neighbourhood
Probabilistic Cellular Automata
NW N NE
W E
SW S SE
Genetic Algorithms, Evolutionary Strategies, Net Games
Swarms, Agents, Sensors Nets
Collective Intelligence
Immune System Models
Οι Γενεές του WebΕποχή Περιγραφή Πηγή ΥπεραξίαςΠρο-Web 1980 Επιτραπέζιοι ΗΥ Υπολογισμοί