International Journal of Neural Systems, Vol. 9, No. 2 (April, 1999) 129–151 c World Scientific Publishing Company MODULAR NEURAL NETWORKS: A SURVEY GASSER AUDA * and MOHAMED KAMEL † Pattern Analysis And Machine Intelligence Lab, Systems Design Engineering Department, University of Waterloo, ON, N2L 3G1 Canada Received 10 December 1998 Revised 2 February 1999 Accepted 28 May 1999 Modular Neural Networks (MNNs) is a rapidly growing field in artificial Neural Networks (NNs) research. This paper surveys the different motivations for creating MNNs: biological, psychological, hardware, and computational. Then, the general stages of MNN design are outlined and surveyed as well, viz., task decomposition techniques, learning schemes and multi-module decision-making strategies. Advantages and disadvantages of the surveyed methods are pointed out, and an assessment with respect to practical potential is provided. Finally, some general recommendations for future designs are presented. 1. Introduction A Modular Neural Network (MNN) is a Neural Net- work (NN) that consists of several modules, each module carrying out one sub-task of the NN’s global task, and all modules functionally integrated. A module can be a sub-structure or a learning sub- procedure of the whole network. The network’s global task can be any neural network application, e.g., mapping, function approximation, clustering or associative memory application. MNN is a rapidly growing field in NNs research. Researchers from several backgrounds and objectives are contributing to its growth. For example, moti- vated by the “non-neuromorphic” nature of the cur- rent artificial NN generation, some researchers with a biology-background are suggesting modular struc- tures. Their goal is either to model the biological NN itself, i.e., a reverse engineering study, or to try to build artificial NNs which achieve the high ca- pabilities of the biological system. Motivated by the psychology of learning in the human system, some other researchers modularize the NN’s learn- ing in an attempt to achieve clearer representation of information and less amount of internal interference. Another group of researchers develop modular NNs to fulfill the constraints put by the current hardware- implementation technology. Nevertheless, most of the work in the MNN field aims to enhance the com- putational capabilities of the nonmodular alterna- tives, e.g., enhancing the networks’ generalization, scalability, representation, and learning speed. Figure 1 shows the growth of the MNN field in a profile very much similar to the growth in the NN field. Notice that 44% of the MNNs research is done in the last two years. This illustrates the recent high interest in the field. Biologists have studied modularization of the natural brain long time ago (e.g., Ref. 1). How- ever, the first two attempts to build artificial mod- ular neural networks, we are aware of, were in November 1987. E. Micheli-Tzanakou 2 outlined, briefly, a model he built of the vertebrate retina us- ing an artificial MNN. He designed a collection of modules connected in series and parallel and used them to study the effects of lateral connectivity. In the same month, the Third Conference of Artificial * E-mail: [email protected]† To whom requests are to be sent, E-mail: [email protected], Fax (519) 746-4791. 129
24
Embed
MODULAR NEURAL NETWORKS: A SURVEYkovan.ceng.metu.edu.tr/...Neural_Networks_A_Survey.pdf · Modular Neural Networks (MNNs) is a rapidly growing eld in arti cial Neural Networks (NNs)
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Pattern Analysis And Machine Intelligence Lab, Systems Design Engineering Department,University of Waterloo, ON, N2L 3G1 Canada
Received 10 December 1998Revised 2 February 1999Accepted 28 May 1999
Modular Neural Networks (MNNs) is a rapidly growing field in artificial Neural Networks (NNs) research.This paper surveys the different motivations for creating MNNs: biological, psychological, hardware, andcomputational. Then, the general stages of MNN design are outlined and surveyed as well, viz., taskdecomposition techniques, learning schemes and multi-module decision-making strategies. Advantagesand disadvantages of the surveyed methods are pointed out, and an assessment with respect to practicalpotential is provided. Finally, some general recommendations for future designs are presented.
1. Introduction
A Modular Neural Network (MNN) is a Neural Net-
work (NN) that consists of several modules, each
module carrying out one sub-task of the NN’s global
task, and all modules functionally integrated. A
module can be a sub-structure or a learning sub-
procedure of the whole network. The network’s
global task can be any neural network application,
e.g., mapping, function approximation, clustering or
associative memory application.
MNN is a rapidly growing field in NNs research.
Researchers from several backgrounds and objectives
are contributing to its growth. For example, moti-
vated by the “non-neuromorphic” nature of the cur-
rent artificial NN generation, some researchers with
a biology-background are suggesting modular struc-
tures. Their goal is either to model the biological
NN itself, i.e., a reverse engineering study, or to try
to build artificial NNs which achieve the high ca-
pabilities of the biological system. Motivated by
the psychology of learning in the human system,
some other researchers modularize the NN’s learn-
ing in an attempt to achieve clearer representation of
information and less amount of internal interference.
Another group of researchers develop modular NNs
to fulfill the constraints put by the current hardware-
implementation technology. Nevertheless, most of
the work in the MNN field aims to enhance the com-
putational capabilities of the nonmodular alterna-
tives, e.g., enhancing the networks’ generalization,
scalability, representation, and learning speed.
Figure 1 shows the growth of the MNN field in
a profile very much similar to the growth in the NN
field. Notice that 44% of the MNNs research is done
in the last two years. This illustrates the recent high
interest in the field.
Biologists have studied modularization of the
natural brain long time ago (e.g., Ref. 1). How-
ever, the first two attempts to build artificial mod-
ular neural networks, we are aware of, were in
November 1987. E. Micheli-Tzanakou2 outlined,
briefly, a model he built of the vertebrate retina us-
ing an artificial MNN. He designed a collection of
modules connected in series and parallel and used
them to study the effects of lateral connectivity. In
the same month, the Third Conference of Artificial
clustering,140,166–168 control engineering,3,123,169–172
and mathematical modeling.51,136,173,174 Figure 9
shows a comparison between two kinds of MNN re-
search: application-oriented and theory-oriented.
4
5
6
7
8
9
10
11
12
13
90 91 92 93 94 95 96
Com
ulat
ive
perc
enta
ge o
f app
licat
ion-
orie
nted
MN
N p
aper
s
Year
Fig. 9. The percentage of application-oriented MNNs rel-ative to the theory-oriented MNNs.
5. Conclusions
In this paper, modular neural network structures and
algorithms are surveyed. Based on the presented
information, the following recommendations can be
given for future designers.
(a) Other related fields can be utilized in de-
signing MNNs. An example is, using multi-
participant decision-making techniques in
integrating the local decisions of the differ-
ent modules. Another example is studying
the biological discoveries for extracting more
fresh ideas for modularization, e.g., the vi-
sual cortex specialized modules. However, for
building engineering solutions to real applica-
tions, the MNN design does not have to be
completely “biologically plausible.”
(b) The task decomposition technique should put
a constraint on the modules’ sizes. However,
modules should not be forced to be equal-
sized because this will limit the system’s flex-
ibility. Also, it is recommended to design
an automatic task-decomposition technique
rather than depending on human experience.
However, the fixed distance-criterion used in
unsupervised NNs is unsuitable for complex
problems. Changing this criterion according
to different levels of data-complexity may be
more efficient. This can be achieved, for ex-
ample, by using a hierarchy of unsupervised
NNs for task decomposition.
(c) The task-decomposition technique should be
Modular Neural Networks: A Survey 145
able to deal with the wide range of over-
laps in the input space, i.e., to create more
“homogeneous” sub-tasks for modules.
(d) Decoupling the learning equations of the dif-
ferent modules allows them to learn in par-
allel. This gives a faster and less complex
learning.
(e) Multimodule decision-making should effi-
ciently integrate the local “experiences” of all
the modules.
(f) It appears from the survey that MNN-
designers usually do not balance the simpli-
fication of sub-tasks and the efficiency of the
multimodule decision-making strategy. In
other words, the task-decomposition algo-
rithm should produce sub-tasks as simple as
they can be, but meanwhile, modules have
to be able to give the multimodule decision-
making strategy enough information to take
an accurate global decision.
(g) The performance of a new MNN should be
compared with other state-of-the-art MNNs
rather than with the traditional models
which are, usually, less efficient. Also, the
performance criteria should include, in addi-
tion to accuracy and speed, aspects like the
feasibility of hardware implementation, com-
putational “work,” and the memory require-
ments of the model.
References
1. J. Hochberg 1964, Perception (Prentice Hall, NewJersy).
2. E. Micheli-Tzanakou 1987, “A neural networkmodel of the vertebrate retina,” in 9th Ann. Conf.IEEE Engineering in Medicine and Biology Society(Boston, MA, USA), November 13–16.
3. E. Fiesler and A. Choudry 1987, “Neural networks asa possible architecture for the distributed control ofspace systems,” in 3rd Conf. Artificial Intell. SpaceAppl. (Washington D.C., USA), p. 401.
4. D. Hartline 1988, “Models for simulation of real neu-ral networks, in Int. Neural Network Soc. (INNS)First Ann. Meeting (Boston, USA), p. 256.
5. D. Gray, A. Michel and W. Porod 1988, “Appli-cation of neural networks to sorting problems,” inInt. Neural Network Soc. (INNS) First Ann. Meet-ing (Boston, USA), p. 441.
6. Y. Anzai and T. Shimada 1988, “Modular neuralnetworks for shape and/or location recognition,” inInt. Neural Network Soc. (INNS) First Ann. Meeting(Boston, USA), p. 158.
8. D. Rumelhart, G. Hinton and R. Williams 1986“Learning internal representations by error prop-agation,” in Parallel Distributed Processing, eds.D. Rumelhart and J. McClelland (MIT Press),pp. 318–362.
9. S. Regers and M. Kabrisky 1991, An Introductionto Biological and Artificial Neural Networks for Pat-tern Recognition, Vol. TT4. SPIE Optical Engineer-ing Press.
10. G. Auda 1992, “Performance criteria for neural net-work evaluation,” Master’s thesis, Cairo University,Egypt.
11. D. Gardner 1993, “Introduction: Toward neural net-works,” in The Neurobiology of Neural Networks, ed.D. Gardner (MIT Press), pp. 1–20.
12. A. Selverston 1993, “Modeling of neural circuits:What have we learned?” Ann. Rev. Neurosci. 16,531–546.
13. M. Ito 1993, “What can we expect from neural net-work models?” in Int. Joint Conf. Neural Networks,Vol. 1, Nagoya, Japan.
14. T. Bullock 1993, “Integrative systems research inthe brain: Resurgence and new opportunities,” Ann.Rev. Neurosci. 16, 1–15.
15. M. Serna and L. Baird 1996, “Comparison of brainstructure to backpropagation-learned-structure,” inIEEE Int. Conf. Neural Networks (ICNN96), Vol. 2(Washington, D.C., USA), pp. 706–711.
16. P. Baldi and N. Toomarian 1993, “Learning trajecto-ries with a hierarchy of oscillatory modules,” in Int.Conf. Neural Networks (USA), pp. 1172–1176.
17. T. Lee 1991, Structure Level Adaptation for ArtificialNeural Networks (Kluwer Academic Publishers).
18. J. Murre 1992, Learning and Categorization in Mod-ular Neural Networks (Harvester-Wheatcheaf).
19. E. Marder 1988, “Modulating a neuronal network,”Nature 335(22), 296–297.
20. P. Hudson and R. Phaf 1991, “Orders of approxi-mation of neural networks to brain structure: Lev-els, modules and computing power,” in ArtificialNeural Networks, eds. T. Kohonen, K. Makisara,O. Simula and J. Kangas (Elsevier Science Publish-ers), pp. 1025–1028.
21. E. Mingolla, H. Neumann and L. Pessoa 1994, “Amulti-scale network model of brightness perception,”in World Congress on Neural Networks, Vol. 4, (SanDiego, USA), pp. 299–306.
22. B. Graham and D. Willshaw 1994, “Capacity andinformation efficiency of a brain-like associate net,”in Neural Info. Proc. Syst. – Nat. Synth. (Denver,USA).
23. K. Cave, J. Rueckl and S. Kosslyn 1989, “Why arewhat and where processed by separate cortical visualsystems? A computational investigation,” J. Cogni-tive Neuroscience 1(2), 171–186.
146 G. Auda & M. Kamel
24. R. Tsai, B. Sheu and T. Berger 1996, “Vlsi designfor real-time signal processing based on biologicallyrealistic neural models,” in IEEE Int. Conf. Neu-ral Networks (ICNN96), Vol. 2 (Washington, D.C.,USA), pp. 676–679.
25. F. Guyot, F. Alexandre and J. Halton 1991, “Thecortical column: A new processing unit for multilay-ered networks,” Neural Networks 4, 15–25.
26. T. Hrycej 1992, “Modular learning in neural net-works: A modularized approach to classification,”(Wiley).
27. S. Johannes, B. Wieringa, M. Matzke and T. Munte1996, “Hierarchical visual stimuli: Electrophysiolog-ical evidence for separating left hemispheric globaland local processing mechanisms in humans,” Neu-roscience Lett. 210(2), 111–114.
28. A. Chernjavsky and J. Moody 1990, “Note on de-velopment of modularity in simple cortical mod-els,” in Adv. Neural Info. Proc. Syst., Vol. 2, ed.D. Touretzky (Morgan Kaufmann), pp. 133–140.
29. S. Zeki and S. Shipp, 1988, “The functional logic ofcortical connections,” Nature 335(22), 311–317.
30. R. Jacobs 1990, Task Decomposition Through Com-petition in a Modular Connectionist Architecture,PhD thesis, University of Massachusets, Amherst,MA, USA.
31. R. Jacobs, M. Jordan and A. Barto 1991, “Task de-composition through competition in a modular con-nectionist architecture: The what and where visiontasks,” Cognitive Science 15(2), 219–250.
32. R. Jacobs, M. Jordan and A. Barto 1991, “Task de-composition through competition in a modular con-nectionist architecture: The what and where visiontasks,” Neural Computation 3, 79–87.
33. S. Aglioti, A. Beltramello, A. Bonazzi andM. Corbetta 1996, “Thumb-pointing in humans afterdamage to somatic sensory cortex,” Exp. Brain Res.109(1), 92–100.
34. A. Cowey 1981, “Why are there so many visualareas?” in The Organization of the Cerebral Cortex,ed. F. Schmitt (MIT Press), pp. 395–413.
35. W. Lincoln and J. Skrzypek 1990, “Synergy of clus-tering back propagation networks,” in Adv. NeuralInfo. Proc. Syst., Vol. 2, ed. D. Touretzky (MorganKaufmann), pp. 650–657.
36. A. Gupta, L. Mui, A. Agarwal and P. Wang 1994,“An adaptive modular neural network with appli-cation to unconstrained character recognition,” Int.J. Pattern Recognition and Artificial Intell. 8(5),1189–1222.
37. M. Schwarz, B. Hosticka, R. Hauschild, W. Mokwa,M. Scholles and H. Trieu 1996, Hardware archi-tecture of a neural net based retina implant forpatients suffering from retinitis pigmentosa,” inIEEE Int. Conf. Neural Networks (ICNN96), Vol. 2(Washington, D.C., USA), pp. 653–658.
38. G. Edelman 1981, “Group selection as the basisfor higher brain function,” in The Organization ofthe Cerebral Cortex, ed. F. Schmitt (MIT Press),pp. 538–563.
39. G. Auda, M. Kamel and H. Raafat 1994, “A new neu-ral network structure with cooperative modules,” inWorld Cong. Comput. Intell., Vol. 3 (Florida, USA),pp. 1301–1306.
40. W. Tsai, H. Tai and A. Reynolds 1994, “An art2-bp supervised neural net,” in World Cong. NeuralNetworks, Vol. 3 (San Diego, USA), pp. 619–624.
41. N. Franks 1989, “Army ants: A collective intelli-gence,” American Scientist 77, 139–145.
42. G. Auda, M. Kamel and H. Raafat 1995, “Votingschemes for cooperative neural network classifiers,”in IEEE Int. Conf. Neural Networks, ICNN’95,Vol. 3 (Perth, Australia), pp. 1240–1243.
43. C. Stevens 1989, “How cortical interconnectednessvaries with network size,” Neural Computation 1,473–479.
44. H. Ayestaran and R. Prager 1993, “Towards con-tinuously learning neural networks,” in Int. JointConf. Neural Networks, Vol. 3 (Nagoya, Japan),pp. 2280–2283.
45. R. Phaf, J. Murre and G. Wolters 1992, “Calm: Cate-giorizing and learning module,” Neural Networks 5,55–82.
46. M. Moya, L. Hostetler, M. Koch and R. Fogler 1995,“Cueing, feature discovery, and one-class learning forsynthetic aperture radar automatic target recogni-tion,” Neural Networks 8(7/8), 1081–1102.
47. M. Patel 1995, “Constraints on task and search com-plexity in ga+nn models of learning and adaptive be-haviour,” in Evolutionary Computing. AISB Work-shop (Sheffield, UK), pp. 200–223.
48. R. Phaf, J. Murre and G. Wolters 1991, “Calm net-works: A modular approach to supervised and un-supervised learning,” in 1991 IEEE Int. Joint Conf.Neural Networks, Vol. 1 (Singapore), pp. 649–656.
49. S. Abidi 1996, “Neural networks and child devel-opment: A simulation using a modular neural net-work architecture,” in IEEE Int. Conf. Neural Net-works (ICNN96), Vol. 2 (Washington, D.C., USA),pp. 840–845.
50. B. Happel and J. Murre 1994, “Design and evolu-tion of modular neural network architectures,” Neu-ral Networks 7(6/7), 895–1004.
51. Y. Foo and H. Szu 1989, “Solving large-scale opti-mization problems by divide-and-conquer neural net-works,” in Int. Joint Conf. Neural Networks, Vol. 1(New York, USA), pp. 507–511.
52. P. Werbos 1994, “Neural nets, conciousness, ethicsand the soul,” in World Cong. Neural Networks,Vol. 1 (San Diego, USA), pp. 221–228.
53. R. Shadmehr, T. Brashers-Krug and E. Todorov1994, “Catastrophic interference in human motor
Modular Neural Networks: A Survey 147
learning,” in Adv. Neural Info. Proc. Syst., Vol. 7,eds. M. Mozer, D. Touretzky and M. Hasselmo (MITPress).
54. T. Brashers-Krug, Reza Shadmehr and E. Todorov1994, “Catastrophic interference in human motorlearning,” in Neural Info. Proc. Syst. – Nat. Synth.(Denver, USA), pp. 18–26.
55. H. Hackbarth and J. Mantel 1991, “Modular connec-tionist structure for 100-word recognition,” in Int.Joint Conf. Neural Networks, Vol. 2, (Seattle, USA),pp. 845–849.
56. K. Joe, Y. Mori and S. Miyake 1990, “Constructionof a large scale neural network: simulation of hand-written Japanese character recognition on ncube,”Concurrency 2(2), 79–107.
57. R. Webb 1993, “Optoelectronic implementation ofneural networks,” Int. J. Neural Systems 4(4),435–444.
58. J. Ouali, G. Saucier and J. Trilhe 1991, “Distributedlarge neural networks on silicon,” in Silicon architec-tures for neural nets (Elsevier Science Publishers),pp. 11–29.
59. S. Kleynenberg, J. Murre, J. Heemskerk and P. Hud-son 1991, “Integrating the bsp400 neurocomputerwith the metanet network environment,” in 1991IEEE Int. Joint Conf. Neural Networks (Singapore),pp. 1476–1480.
60. J. Murre 1991, “Transputer implementations of neu-ral networks: An analysis,” in Artificial Neural Net-works, eds. T. Kohonen, K. Makisara, O. Simulaand J. Kangas (Elsevier Science Publishers),pp. 1537–1541.
61. R. Means 1994, “High speed parallel hardware per-formance issues for neural network applications,” inWorld Cong. Comput. Intell., Vol. 1, (Florida, USA),pp. 10–16.
62. R. Liu, C. Wu and I. Jou 1995, “Cmos current-modeimplementation of spatiotemporal probabilistic neu-ral networks for speech recognition,” J. VLSI SignalProc. 10, 67–84.
63. M. Costa, E. Fillipi and E. Pasero 1994, “A modu-lar cyclic neural network for character recognition,”in World Cong. Neural Networks, Vol. 3 (San Diego,USA), pp. 204–209.
64. H. Djahanshahi, M. Ahmadi, G. Jullien andW. Miller 1996, “A modular architecture for hy-brid vlsi neural networks and its application in asmart photosensor,” in IEEE Int. Conf. Neural Net-works (ICNN96), Vol. 2 (Washington, D.C., USA),pp. 868–873.
65. Z. Xu and C. Kwong 1996, “A decomposition princi-ple for complexity reduction of artificial neural net-works,” Neural Networks 9(6), 999–1016.
66. A. Blum and R. Rivest 1988, “Training a 3-node neu-ral network is np-complete,” in 1st ACM WorkshopComput. Learning Theory (Cambridge, MA, USA),pp. 9–18.
67. D. Hush and B. Horne 1993, “Progress in supervisedneural networks: What’s new since lippmann?” IEEESignal Proc. Mag. 10(2), 8–39.
68. G. Auda, A. El-Dessouki and A. Darwish 1993,“A comparative study of some supervised neuralnetworks,” The Egyptian Computer Journal 21(1),97–104.
69. M. de Bollivier, P. Gallinari and S. Thiria 1991, “Co-operation of neural nets and task decomposition,” inInt. Joint Conf. Neural Networks, Vol. 2 (Seattle,USA), pp. 573–576.
70. K. Hornik, M. Stinchcombe and H. White 1989,“Multilayer feedforward networks are universal ap-proximators,” Neural Networks 2, 359–366.
71. E. Baum and D. Haussler 1989, “What size net givesvalid generalization?” Neural Computation 1(1),151–160.
72. S. Holden 1994, “How practical are vc dimensionbounds?” in World Cong. Comput. Intell., Vol. 1(Florida, USA), pp. 327–332.
73. V. Vapnik and A. Chervonenkis 1971, “On the uni-form convergence of relative frequencies of eventsto their prababilities,” Theory of Probability and itsApplications 16, 264–280.
74. Y. Abu-Mostafa 1989, “The vapnik-chervonenkis di-mension: Information versus complexity in learning,”Neural Computation 1, pp. 312–317.
75. T. Rognvaldsson 1993, “Pattern discrimination usingfeedforward networks: A benchmark study of scalingbehavior,” Neural Computation 5, 483–491.
76. P. Bartlett 1993, “Vapnik-chervonenkis dimensionbounds for two- and three-layer networks,” NeuralComputation 5, 371–373.
77. S. Ahmad 1988, “A study of scaling and gen-eralization in neural networks,” Technical ReportUIUCDCS-R-88-1454, Department of ComputerScience.
78. Y. Bennani 1995, “A modular and hybrid connec-tionist system for speaker identification,” NeuralComputation 7, 791–798.
79. M. Smith 1993, Neural Networks for Statistical Mod-eling (Van Nostrand Reinhold).
80. G. Tesauro and B. Janssens 1988, “Scaling relation-ships in back-propagation learning,” Complex Sys-tems 2, 39–44.
81. Y. Le Cun, J. Denker and S. Solla 1990, “Op-timal brain damage,” in Adv. Neural Info. Proc.Syst., Vol. 2, ed. D. Touretzky (Morgan Kaufmann),pp. 598–605.
82. P. Werbos 1991, “Links between artificial neural net-works (ann) and statistical pattern recognition,” inArtificial Neural Networks and Statistical PatternRecognition: Old and New Connections (Elsevier Sci-ence Publishers), pp. 11–31.
83. F. Wang and Q. Zhang 1996, “An adaptive and fullysparse training approach for multilayer perceptrons,”in IEEE Int. Conf. Neural Networks (ICNN96),Vol. 3 (Washington, D.C., USA), pp. 102–107.
148 G. Auda & M. Kamel
84. C. Mohan, R. Anand, K. Mehrotra and S. Ranka1995, “Efficient classification for multiclass problemsusing modular neural networks,” IEEE Trans. Neu-ral Networks 6(1), 117–124.
85. D. Zipser 1990, “Subgrouping reduces complexityand speeds up learning in recurrent networks,”in Adv. Neural Info. Proc. Syst., Vol. 2, ed.D. Touretzky (Morgan Kaufmann), pp. 638–641.
86. N. Gagarin, I. Flood and P. Albrecht 1994, “Comput-ing truck attributes with artificial neural networks,”J. Comput. Civil Engi. 8(2), 179–200.
87. S. Nakamura and H. Sawaki 1992, “Performancecomparison of neural network architectures,” Sys-tems and Computer in Japan 23(14), 72–83.
88. W. Baxt 1992, “Improving the accuracy of an artifi-cial neural network using multiple differently trainednetworks,” Neural Computation 4, 772–780.
89. P. Mundkur and U. Desai 1991, “Automatic tar-get recognition using image and network decompo-sition,” in 1991 IEEE Int. Joint Conf. Neural Net-works (Singapore), pp. 281–286.
90. A. Waibel 1989, “Modular construction of time-delay neural networks for speech recognition,” NeuralComputation 1, 39–46.
91. G. Poo and Y. Ou 1994, “A large phonemic time-delay neural network technique for all mandarin con-sonants recognition,” in Proc. 1994 IEEE Region10’s 9th Ann. Int. Conf. (USA), pp. 521–525.
92. P. Murphy and D. Aha 1994, “Uci repository of ma-chine learning databases,” Technical Report, Univer-sity of California, Irvine, USA.
93. S. Haykin 1994, Neural Networks: A ComprehensiveFoundation (IEEE Press).
94. T. Hrycej 1993, “A modular architecture for effi-cient learning,” in Int. Conf. Neural Networks, Vol. 1(USA), pp. 557–562.
95. D. Ballard 1987 “Modular learning in neural net-works,” in 6th Nat. Conf. Artificial Intell. (USA),pp. 279–284.
96. M. de Bollivier, P. Gallinari and S. Thiria 1990, “Co-operation of neural nets for robust classification,” inInt. Joint Conf. Neural Networks, Vol. 1 (San Diego,USA), pp. 113–120.
97. C. Chiang and H. Fu 1994, “A divide-and-conquermethodology for modular supervised neural networkdesign,” in World Cong. Comput. Intell., Vol. 1(Florida, USA), pp. 119–124.
98. G. Auda and M. Kamel 1997, “Modular neuralnetwork classifiers: A comparative study,” in 3rdInt. Conf. Neural Networks Appl. (NEURAP’97)(Marseilles, France), pp. 41–47.
99. C. Lursinsap and V. Coowanitwong 1994, “Analysisof input vector space for speeding up learning in feed-forward neural networks,” in World Cong. on NeuralNetworks, Vol. 3 (San Diego, USA), pp. 414–419.
100. H. Raafat and M. Rashwan 1993, “A tree structuredneural network,” in Int. Conf. Document Analysisand Recognition ICDAR93 (Japan), pp. 939–942.
101. T. Drabe and W. Bressgott 1997, “Task alloca-tion for multiple-network architectures,” in IEEEInt. Conf. Neural Networks (ICNN97), Vol. 1(Houston, Texas, USA), pp. 491–494.
102. Y. Nishikawa 1990, “Nn/i: A new neural networkwhich divides and learns environments,” in Int. JointConf. Neural Networks, Vol. 1 (USA), pp. 684–687.
103. H. Kita, H. Masataki and Y. Nishikawa 1991, “Im-proved version of a network for large-scale patternrecognition tasks,” in 1991 IEEE Int. Joint Conf.Neural Networks (Singapore), pp. 2080–2085.
104. E. Corwin, S. Greni, A. Logar and K. Whitehead1994, “A multi-stage neural network classifier,” inWorld Cong. Neural Networks, Vol. 3 (San Diego,USA), pp. 198–203.
105. G. Bartfai 1994, “Hierarchical clustering with artneural networks,” in World Cong. Comput. Intell.,Vol. 2 (Florida, USA), pp. 940–944.
106. N. Griffith 1994, “Using art2 and bp co-operativelyto classify musical sequences,” in Int. Conf. ArtificialNeural Networks ICANN94, Vol. 1, Italy.
107. L. Burke 1991, “Clustering characterization of adap-tive resonance,” Neural Networks 4, 485–491.
108. Y. Mori and K. Joe 1990, “A large-scale neuralnetwork which recognizes handwritten kanji charac-ters,” in Adv. Neural Info. Proc. Syst., Vol. 2, ed.D. Touretzky (Morgan Kaufmann), pp. 415–422.
109. J. Huang and A. Kuh 1993, “A neural network iso-lated word recognition system for moderate sizeddatabases,” in Int. Conf. Neural Networks, Vol. 1(USA), pp. 387–391.
110. A. Iwata, H. Kawajiri and N. Suzumura 1991, “Clas-sification of hand-written digits by a large scale neu-ral network combnet-ii,” in 1991 IEEE Int. JointConf. Neural Networks (Singapore), pp. 1021–1026.
111. M. Jordan and L. Xu 1993, “Convergence propertiesof the em approach to learning in mixture-of-expertsarchitectures,” Technical Report 9302, Departmentof Brain and Cognitive Science, MIT.
112. R. Battiti and A. Colla 1994, “Democracy in neu-ral nets: Voting schemes for classification,” NeuralNetworks 7(4), 691–707.
113. R. Shadafan and M. Niranjan 1994, “A dynamicneural network architecture by sequential partition-ing of the input space,” Neural Computation 6,1202–1222.
114. T. Salome and H. Bersini 1994, “An algorithmfor self-structuring neural net classifiers,” in Int.Conf. Neural Networks (ICNN94), Vol. 3 (San Diego,USA), pp. 1307–1311.
115. W. Jansen, M. Diepenhorst, J. Nijhuis andL. Spaanenburg 1997, “Assembling engineeringknowledge in a modular multi-layer perceptron neu-ral network,” in IEEE Int. Conf. Neural Net-works (ICNN97), Vol. 1 (Houston, Texas, USA),pp. 232–237.
Modular Neural Networks: A Survey 149
116. L. Wang, N. Nasrabadi and S. Der 1997, “Asymp-totical analysis of a modular neural network,” inIEEE Int. Conf. Neural Networks (ICNN97), Vol. 2(Houston, Texas, USA), pp. 1019–1022.
117. S. Schaal and C. Atkeson 1995, “From isolation tocooperation: An alternative view of a system of ex-perts,” in Adv. Neural Info. Proc. Syst., Vol. 8, eds.M. Mozer, D. Touretzky and M. Hasselmo (MITPress).
118. V. Ramamurti and J. Gosh 1997, “Regularizationand error bars for the mixture of experts network,” inIEEE Int. Conf. Neural Networks (ICNN97), Vol. 1(Houston, Texas, USA), pp. 221–225.
119. V. Tresp and M. Taniguchi 1994, “Combining es-timators using non-constant weighting functions,”in Neural Info. Proc. Syst. – Nat. Synth. (Denver,USA), pp. 419–426.
120. K. Chen and H. Chi 1997, “A modified mixturesof experts architecture for classification with di-verse features,” in IEEE Int. Conf. Neural Net-works (ICNN97), Vol. 1 (Houston, Texas, USA),pp. 215–220.
121. L. Xu, M. Jordan and J. Hinton 1994, “An al-ternative model for mixtures of experts,” in Neu-ral Info. Proc. Syst. – Nat. Synth. (Denver, USA),pp. 633–640.
122. S. Medasani and R. Krishnapuram 1997, “De-termination of the number of components inGaussian mixtures using agglomerative clustering,”in IEEE Int. Conf. Neural Networks (ICNN97),Vol. 3 (Houston, Texas, USA), pp. 1412–1417.
123. B. Lu, H. Kita and Y. Nishikawa 1994, “A multi-sieving neural network architecture that decomposeslearning tasks automatically,” in World Cong. Com-put. Intell., Vol. 3 (Florida, USA), pp. 1319–1324.
124. K. Mohiuddin, J. Mao and T. Fujisaka 1995, “Atwo-stage multi-network ocr system with a soft pre-classifier and a network selector,” in Int. Conf. Docu-ment Analysis and Recognition ICDAR95 (Montreal,Canada), pp. 78–81.
125. R. Venkateswaran and Z. Obradovic 1994, “Efficientlearning through cooperation,” in World Cong. Neu-ral Networks, Vol. 3 (San Diego, USA), pp. 390–395.
126. N. Ueda and R. Nakano 1996, “Generalization errorof ensemble estimators,” in IEEE Int. Conf. Neu-ral Networks (ICNN96), Vol. 3 (Washington, D.C.,USA), pp. 90–95.
127. L. Hansen and P. Salamon 1990, “Neural networkensembles,” IEEE Transactions on Pattern Analysisand Machine Intelligence 12(10), 993–1003.
128. S. Mukherjee and T. Fine 1996, “Ensemble prun-ing algorithms for accelerated training,” in IEEEInt. Conf. Neural Networks (ICNN96), Vol. 3(Washington, D.C., USA), pp. 96–101.
129. J. Ahn, J. Kim and S. Cho 1995, “Ensemblecompetitive learning neural networks with reducedinput dimension,” Int. J. Neural Systems 6(2),132–142.
130. S. Hashem 1997, “Algorithms for optimal linear com-binations of neural networks,” in IEEE Int. Conf.Neural Networks (ICNN97), Vol. 1 (Houston, Texas,USA), pp. 242–247.
131. M. Sonmez, M. Sun, C. Li and R. Sclabassi 1997,“A hierarchical decision module based on multipleneural networks,” in IEEE Int. Conf. Neural Net-works (ICNN97), Vol. 1 (Houston, Texas, USA),pp. 238–241.
132. A. Krogh and J. Vedelsby 1994, “Neural networkensembles, cross validation and active learning,” inNeural Info. Proc. Syst. – Nat. Synth. (Denver,USA), pp. 231–238.
133. H. Elsherif and M. Hambaba 1994, “On modify-ing the weights in a modular recurrent connectionistsystem,” in World Cong. Comput. Intell. (Florida,USA), pp. 535–539.
134. H. Elsherif and M. Hambaba 1993, “A modular neu-ral network architecture for pattern classification,”in IEEE Workshop Neural Networks Signal Proc.(Baltimore, USA), pp. 232–239.
135. S. Zein-Sabatto, W. Hwang and D. Marpaka 1994,“Neural networks sharing knowledge and experi-ence,” in World Cong. Neural Networks, Vol. 3 (SanDiego, USA), pp. 613–618.
136. K. Rohani and M. Manry 1991, “The design of multi-layer perceptrons using building blocks,” in Int.Joint Conf. Neural Networks, Vol. 2 (Seattle, USA),pp. 497–501.
137. A. Maren and R. Pap 1994, “The cooperative-competitive network: A neural network method formaking assignments,” in World Cong. Neural Net-works, Vol. 1 (San Diego, USA), pp. 456–461.
138. G. Rogova 1994, “Combining the results of severalneural network classifiers,” Neural Networks 7(5),777–781.
139. H. Normi 1987, Comparing Voting Systems(D. Reidel Publishing Company).
140. P. Watta, M. Akkal and M. Hassoun 1997,“Decoupled-voting hamming associative memorynetworks,” in IEEE Int. Conf. Neural Net-works (ICNN97), Vol. 2 (Houston, Texas, USA),pp. 1188–1193.
141. E. Alpaydin 1993, “Multiple networks for functionlearning,” in Int. Conf. Neural networks, Vol. 1(California, USA), pp. 9–14.
142. G. Auda 1996, “Cooperative modular neural networkclassifiers,” PhD thesis, University of Waterloo, Sys-tems Design Engineering Department, Canada.
143. G. Auda, M. Kamel and H. Raafat 1996, “Modu-lar neural network architectures for classification,”in Int. Conf. Neural Networks (ICNN96), Vol. 2(Washington D.C., USA), pp 1279–1284.
144. S. Cho 1995, “A neuro-fuzzy architecture for highperformance classification,” in Adv. Fuzzy LogicNeural Networks Genetic Algorithms. IEEE/NagoyaUniv. World Wisepersons Workshop (Japan),pp. 66–71.
150 G. Auda & M. Kamel
145. D. Lee and S. Srihari 1995, “A theory of classifiercombination: The neural network approach,” in Int.Conf. Document Analysis and Recognition ICDAR95(Montreal, Canada), pp. 42–46.
146. C. Ornes and J. Sklansky 1997, “A visual multi-expert neural classifier,” in IEEE Int. Conf. NeuralNetworks (ICNN97), Vol. 3 (Houston, Texas, USA),pp. 1448–1453.
147. I. Otto, E. Guigon and Y. Burnod 1990, “Coopera-tion between temporal and parietal networks for in-variant recognition,” in Proc. Int. Neural NetworkConf., Vol. 1 (Paris, France), pp. 480–483.
148. D. Jimenez 1997, “Locating anatomical land-marks for prosthetics design using ensemble neu-ral networks,” in IEEE Int. Conf. Neural Net-works (ICNN97), Vol. 1 (Houston, Texas, USA),pp. 81–85.
149. V. Petridis and A. Kehagias 1996, “A recurrent net-work implementation of time series classification,”Neural Computation 8, 357–372.
150. P. Blonda, V. la Forgia, G. Pasquariello andG. Stalino 1994, “A modular neural system for theclassification of remote sensing images,” in Int. Conf.Artificial Neural Networks ICANN94, Vol. 2 (Italy),pp. 1173–1176.
151. L. Devillers and C. Dugast 1994, “Hybrid sys-tem combining expert-tdnns and hmms for contin-uous speech recognition,” in Int. Conf. AcousticsSpeech Signal Proc. ICASSP94, Vol. 1 (Australia),pp. 165–168.
152. T. Lefebvre, J. Nicolas and P. Degoul 1990, “Nu-merical to symbolic conversion for acoustic signalclassification using a two-stage neural architecture,”in Proc. Int. Neural Network Conf., Vol. 1 (Paris,France), pp. 119–122.
153. E. Allen, M. Menon and P. Dicaprio 1990, “A mod-ular architecture for object recognition using neuralnetworks,” in Proc. Int. Neural Network Conf., Vol. 1(Paris, France), pp. 35–37.
154. J. Lampinen and E. Oja 1995, “Distortion tolerantpattern recognition based on self-organizing featureextraction,” IEEE Trans. on Neural Networks 6(3),539–547.
155. J. Diaz-Verdejo, J. Segura-Luna, P. Garcia-Teodoroand A. Rubio-Ayuso. Slhmm 1994, “A continu-ousspeech recognition system based on alphanet-hmm,” in Int. Conf. Acoustics Speech Signal Proc.ICASSP94, Vol. 1 (Australia), pp. 213–216.
156. Y. Zhao, R. Schwartz, J. Sroka and J. Makhoul1994, “Hierarchical mixtures of experts methodologyapplied to continuous speech recognition,” in Neu-ral Info. Proc. Syst. – Nat. Synth. (Denver, USA),pp. 859–865.
157. A. de Carvalho, M. Fairhurst and D. Baisset 1994,“A modular boolean architecture for pattern recog-nition,” in Int. Conf. Neural Networks (ICNN94),Vol. 7 (San Diego, USA), pp. 4349–4352.
158. E. Wan 1997, “Combining fossil and sunspot data:Committee predictions,” in IEEE Int. Conf. NeuralNetworks (ICNN97), Vol. 4 (Houston, Texas, USA),pp. 2176–2180.
159. R. Schwaerzel and B. Rosen 1997, “Improving theaccuracy of financial time series prediction usingensemble networks and high order statistics,” inIEEE Int. Conf. Neural Networks (ICNN97), Vol. 4(Houston, Texas, USA), pp. 2045–2050.
160. S. Cho, Y. Cho and S. Yoon 1997, “Reliable rollforce prediction in cold mill using multiple neu-ral networks,” IEEE Trans. Neural Networks 8(4),874–882.
161. C. Tham 1996, “A hierarchical cmac architec-ture for context dependent function approximation,”in Int. Conf. Neural Networks (ICNN96), Vol. 1(Washington D.C., USA), pp. 629–634.
162. H. Lee, K. Mehrotra, C. Mohan and S. Ranka1994, “An incremental network construction algo-rithm for approximating discontinuous functions,” inInt. Conf. Neural Networks (ICNN94), Vol. 4 (SanDiego, USA), pp. 2191–2196.
163. M. Moussa and M. Kamel 1994, “Efficient learn-ing of generic grasping functions using a set of lo-cal experts,” in World Cong. Comput. Intell., Vol. 2(Florida, USA), pp. 183–190.
164. S. Fels and J. Hinton 1994, “Glove-talkii: Mappinghand gestures to speech using neural networks,”in Neural Info. Proc. Syst. – Nat. Synth. (Denver,USA), pp. 843–850.
165. A. Nelson and E. Wan 1997, “Neural speech en-hancement using dual extended kalman filtering,” inIEEE Int. Conf. Neural Networks (ICNN97), Vol. 4(Houston, Texas, USA), pp. 2170–2175.
166. S. Suzuki and H. Ando 1994, “Unsupervised clas-sification of 3d objects from 2d views,” in Neu-ral Info. Proc. Syst. – Nat. Synth. (Denver, USA),pp. 1340–1345.
167. B. Kosko 1989, “Unsupervised learning in noise,” inInt. Joint Conf. Neural Networks IJCNN-89, Vol. 1(Washington D.C., USA), pp. 7–17.
168. S. Ozawa, K. Tsutsumi and N. Baba 1994, “Theestimation of cross-coupled hopfield nets as aninteractive modular neural network,” in Int. Conf.Neural Networks (ICNN94), Vol. 2 (San Diego,USA), pp. 1340–1345.
169. R. Khosla and T. Dillon 1997, “Intelligent hy-brid multi-agent architecture for engineering com-plex systems,” in IEEE Int. Conf. Neural Net-works (ICNN97), Vol. 4 (Houston, Texas, USA),pp. 2449–2454.
170. A. Driss, V. Rodrigues and P. Cohen 1994, “Park-ing a vision-guided automobile vehicle,” in 3rdIEEE Conf. Control Appl., Vol. 1 (Glasgow, UK),pp. 59–64.
171. F. Zanella and M. Corvi 1994, “A neural reflexive
Modular Neural Networks: A Survey 151
navigation system,” in Int. Conf. Artificial NeuralNetworks ICANN94, Vol. 2 (Italy), pp. 1290–1294.
172. M. Copeland 1995, “Modular network control forrobot manipulator,” in Proc. IEEE Southeastcon’95.Visualize the Future, Vol. 1 (NC, USA), pp. 183–187.
173. F. Casilari, A. Jurado, G. Pansard, A. Diaz-Estrella
and F. Sandoval 1996, “Modelling aggregate hetero-geneous atm sources using neural networks,” Elec-tronics Letters 32(4), 363–365.
174. P. Tino and J. Sajda 1995, “Leraning and extract-ing initial mealy automata with a modular neuralnetwork model,” Neural Computation 7, 822–844.