Protocomputing Architecture over a Digital Medium Aiming at …lalsie.ist.hokudai.ac.jp/publication/dlcenter.php?fn=... · ResearchArticle Protocomputing Architecture over a Digital

Research ArticleProtocomputing Architecture over a Digital Medium Aiming atReal-Time Video Processing

Aoi Tanibata ,1 Alexandre Schmid,2 Shinya Takamaeda-Yamazaki,1

Masayuki Ikebe,1 Masato Motomura,1 and Tetsuya Asai1

1Graduate School of Information Science and Technology (IST), Hokkaido University, Kita 14, Nishi 9, Kita-ku,Sapporo, Hokkaido, Japan2Microelectronic Systems Laboratory, Swiss Federal Institute of Technology (EPFL), 1015 Lausanne, Switzerland

Correspondence should be addressed to Aoi Tanibata; [email protected]

Received 26 October 2017; Accepted 9 January 2018; Published 15 February 2018

Academic Editor: Masashi Aono

Copyright © 2018 Aoi Tanibata et al. This is an open access article distributed under the Creative Commons Attribution License,which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

A protocomputational architecture is presented that implements the ancient reaction-diffusion model as a microelectronichardware. A digital medium is selected for the physical mapping of the protoarchitecture as a way to benefit from reliable advancedintegrated circuits fabrication technologies. The extraction of dense motion vector fields from textureless objects in a videosequence is selected as a realistic application. Real-time video processing results at 30 fps are achieved using an FPGA physicalimplementation of the proposed protoarchitecture.

1. Introduction

Modern von Neumann architecture based processors sys-tematically carry over extensive processing tasks governedby the consecutive execution of individual instructions thatimplement algorithms. The fact is widely accepted that thesearchitectures are suitable to efficiently process a massiveamount of simple algebraic operations with limited memoryexchange, while their efficiency dramatically decreases atprocessing tasks that are natural and quick to mammalssuch as face and object recognition, feature, and saliencyextraction. Artificial neural networks [1] and, more recently,machine learning models and algorithms have emerged inresponse to solving such conceptually complex problems, forexample, [2].

Nevertheless, computation has existed in nature priorto the existence of any physical support to engineered andhuman computing, for example, mathematical structuredthinking, geometrical support to calculations, and mechan-ical and electronic computing systems. The operation andbehavior of neurons, the topology of neural networks, andmore generally operations of the brain have been a sourceof inspiration yielding disciplines including computational

neuroscience, neuromorphic computing, and engineering[3]. Protocomputing is emerging as a novel research dis-cipline accepting two fundamental research hypotheses;namely, (a) computing is not limited to excitable tissues andmammals and is also processed by simple organisms aswell asliquids andmaterials in the form of biochemical and physicalprocesses and reactions often forming nonlinear dynamicalsystems with complex behavior and (b) the very fundamentalnature of this processing may hold its properties from theorigin of life and of the universe.

Still in its infancy, the protocomputing discipline ismostly addressed at the level of theoretical developments,modeling, and algorithmic studies [4]. Decision-solvingmethodologies and algorithms have been presented to effi-ciently solve complex optimization problems exploiting spe-cific feature of physical phenomena [5]. Wet-laboratoryexperiments lend themselves particularly well to an experi-mental perspective, specifically considering the chemical andbiology related approaches.The sophisticated spatiotemporaloscillatory dynamics developed by the primitive single-cellamoeba in solving complex environmental adaptation isused as an inspiration to solving NP-hard problems [6].The usage of slime mould to constructing various sensing

HindawiComplexityVolume 2018, Article ID 3618621, 11 pageshttps://doi.org/10.1155/2018/3618621

2 Complexity

and computational building blocks is discussed in detail in[7]. One research direction in synthetic biology consists ofancestral sequence reconstruction to gain understanding inmechanisms of life and its origins [8]. Photonic mediumis applied to experimentally solving the multiarmed banditproblem exploiting the ultrafast chaotic oscillatory dynamicsof lasers [9]. Astronomy and bioastronomy also seek tounderstand computing of the origin of life.

From an electronic viewpoint, protocomputation maybe considered from a classical device-level perspective, thatis, as a discipline aiming at performing unconventionalcomputation from benefiting of specific electrical charac-teristics of novel, post-CMOS devices and thus positioningitself in overlap to advanced switching devices research andmaterial science, for example, single-electron transistors,graphene and quantum devices, and molecular electronics.In general, exploiting such devices requires their circuit-level hybrid usage with CMOS technology. Hence, classicalanalog MOSFET design is used to amplify excessively lowcurrent levels and create functionality that can successfullybe interfaced to other circuits and to the outside world.

The majority of the demonstrated protocomputingdevice-level experiments may be considered as proof-of-concept aiming at understanding the operation and charac-teristics of the computingmedia. Prototypes supporting real-world experiments face scaling issues relating to dynamicallycontrolling and adapting the computing media. In this work,we propose exploiting the stable and well-understood digitaltechnology as the computing media. The protocomputingsystem is implemented as a protoarchitecture that is mappedinto the digital media. The proposed protocomputing systemconsists of a reaction-diffusion process that is observed infundamental chemical and biological condition develop-ments. A cellular automata is used as the method of imple-menting the algorithm [10, 11], which is mapped into a digitalprotoarchitecture. As a merit of the proposed approach, areal application example of video processing is demonstratedin real-time. Abnormal behavior, saliency detection, andattention attraction are part of the reflex cognitive processingof mammals. Motion detection is a prerequisite to thesetasks. Motion detection under conditions where the movingobject has no texture is used as the application exploiting theprotocomputation architecture to achieve real-time.

Section 2.1 details the problem and the selected cellularautomata implementation of the reaction-diffusion algo-rithm. The original RD algorithm is adapted to supportits flexible parallel operation as a solution to enable real-time operation over a wide range or size of the input space.Sections 2.2–2.4 present the fundamental protoarchitectureand its implementation into a digital computation medium.The control of the architecture and its real-time schedulingare presented. Section 3.1 presents implementation resultsand demonstrates the real-time capability of the proposedprotoarchitecture over a real-world application. Finally, amethodology that enables scaling the input space (inputimage size) and that dictates the correct level of parallelismto reach real-time operation is presented in Section 3.2.Section 4 presents the conclusions.

2. Materials and Methods

In the following, a method based on the ancestral reaction-diffusion algorithm is proposed to a real-life video exampleconsisting of creating texture into textureless objects to theaim of enabling the real-time motion vector extraction.A fundamental condition to achieving real-time operationconsists in the specific development of a protoarchitectureand its mapping into a digital medium.

Initially developed within the context of video compres-sion techniques [12], motion vector estimation has foundnew application fields with the advent of modern ubiquitousconsumer electronic products and smart vision sensors, withexamples including target tracking [13], hand gesture userinterface [14], image stabilization [15], surveillance, eventanalysis and automatic anomaly detection from monitoringcameras, depth map estimation, and 3D vision, which reveala dynamic research activity over the recent years [16].Classical motion vector extraction algorithms using blockmatching [17] are known to deliver accurate results in highlytextured regions of images, while they perform poorly inlow textured regions. The recent usage of block matchingin smart machine-vision applications poses increased con-straints on the necessity of a correct motion vector fieldextraction in real-time.The generated field of motion vectorsis used in further algorithmic processing and thus mustsatisfy severe criteria in terms of spatial homogeneity [18].For example, postprocessing tasks such as classification mayrequire an accurate density of the motion vector field inorder to decrease their error rate. Edges tracking and particlefilters have been proposed as a solution to detect and tracktextureless objects, resulting in computationally complexalgorithms, for example, [19], hence with limited practicalusage in energy-constrained portable systems.

2.1. Cellular-Automaton Generating Spatial Patterns Aim-ing at the Motion Vector Estimation of Textureless Objects.Reaction-diffusion (RD) algorithms have initially been intro-duced to model the concentration dynamics of differentchemical species placed in presence of each other within asingle container [20]. Diffusion processes represent a fun-damental natural phenomena underlying the macroscopiceffect observed as a result of numerous irregular microscopicmotion of individual particles that spread out as a result ofthe motion of each. The particles may consist of chemicals,cell or bacteria, or even larger species (animals) [21]. Reactiondescribes the conversion of one involved particle species intoanother, as a result of the diffusion of two or more substan-ces.

In its original expression, the RD model is defined over acontinuous spatial domain. The dynamics of RD consideringthe activators and inhibitors over independent spatial fieldsis adapted following the procedure that is presented in [22].Diffusion of activators and inhibitors on 1D space is describedby the following diffusion equation:

𝜕𝑢 (𝑥, 𝑡)𝜕𝑡 = 𝐷

𝜕2𝑢 (𝑥, 𝑡)𝜕𝑥2 , (1)

Complexity 3

where 𝐷 represents the diffusion coefficient of the activatorsor inhibitors,𝑥 the space, and𝑢(𝑥, 𝑡) the spatial concentrationat time 𝑡. The general solution is given by

𝑢 (𝑥, 𝑡, 𝑛) = 12√𝜋𝐷𝑡 ∫∞

−∞𝑢𝑛 (𝑋) 𝑒−(𝑥−𝑋

2)/4𝐷𝑡𝑑𝑋, (2)

where 𝑢𝑛(𝑥) represents the initial concentration at 𝑡 = 0and 𝑛 represents the cycle index, further used in (4) to (6).In [22], two independent diffusion equations for activatorsand inhibitorswere introduced, where (i) diffusion coefficientof inhibitors 𝐷V was set at much larger value than that ofactivators 𝐷𝑢 and (ii) the same initial concentrations wereset to the two diffusion equations. Here we introduce a novelmethod that uses one diffusion equation only to describethe original model equations. First, diffusion of activators isperformed during time 𝑇𝑢 with initial concentration 𝑢0(𝑥),and the result is obtained as 𝑢(𝑥, 𝑇𝑢, 0). Then the diffusionis further continued for additional time 𝑇V, and the resultis described by 𝑢(𝑥, 𝑇𝑢 + 𝑇V, 0). It should be noticed that𝑢(𝑥, 𝑇𝑢, 0) and𝑢(𝑥, 𝑇𝑢+𝑇V, 0) represent results of diffusions ofactivators and inhibitors, respectively, because 𝑢(𝑥, 𝑇𝑢+𝑇V, 0)is obtained by diffusion of 𝑢(𝑥, 𝑇𝑢, 0) during 𝑇V, which isequivalent to the two-diffusion-equation system under thesame initial concentration with 𝐷V > 𝐷𝑢. Second, as in[22], differential concentration of activators and inhibitors isamplified by the sigmoid function and is set to the subsequentinitial condition 𝑢1(𝑥) as𝑢1 (𝑥) = 𝑓 (𝑢 (𝑥, 𝑇𝑢, 0) − 𝑢 (𝑥, 𝑇𝑢 + 𝑇V, 0) − 𝑐) ,𝑓 (𝑥) = 11 + 𝑒−𝛽𝑥 ,

(3)

where 𝛽 represents the gain of the sigmoid function and 𝑐represents the offset value introduced in [22]. When 𝑇𝑢 ≈0 ≪ 𝑇V, 𝑢(𝑥, 𝑇𝑢, 0) ≈ 𝑢0(𝑥), and hence, by assuming 𝑐 = 0and redefining 𝑇V as 𝑇, (3) is simplified as

𝑢1 (𝑥) = 𝑓 (𝑢0 (𝑥) − 𝑢 (𝑥, 𝑇, 0)) . (4)

By using 𝑢1(𝑥), subsequent 𝑢2(𝑥) is obtained by

𝑢2 (𝑥) = 𝑓 (𝑢1 (𝑥) − 𝑢 (𝑥, 𝑇, 1)) , (5)

and the generalized update equation is

𝑢𝑛+1 (𝑥) = 𝑓 (𝑢𝑛 (𝑥) − 𝑢 (𝑥, 𝑇, 𝑛)) . (6)

Under specific parameters sets, the discrete version ofRD algorithms generates repeatable, stable spatial patterns[23, 24] consisting of stripes or spots froman initial image andapplying an iterative processing. Henceforth, this propertyis used to create texture in a video scene and in particularinto texturelessmoving objects. Assuming that the interframemovement is sufficiently small, then the texture follows themovement of the textureless object, enabling the detection ofthe movement of texture, rather than object edges only, as alimitation of classical algorithms. An algorithm that imple-ments the aforementioned principles has been presented in[25]. The RD process is defined in its fundamental form

over a continuous spatial domain. Following the proceduredescribed in [22], the dynamics of the RD process is adaptedto support the diffusion of activators and inhibitor in inde-pendent discrete fields, which are eventually convoluted intoa 2D array of cells.

Two-dimensional RD over images and video sequencesis applied from a combination of one-dimensional process-ing, following the procedure presented in [25] and whichdelivers patterns that are stable, even in a noisy environment.The theoretical approaches governing the derivation of thereaction-diffusion equation in a continuous time and spatialdomains aremodified to support the circuit-level approach asexpressed in

𝑎𝑖 (𝑡 + 1) = 𝑎𝑖−1 (𝑡) + 2𝑎𝑖 (𝑡) + 𝑎𝑖+1 (𝑡)4 , (7)

where 𝑎 is a natural number representing the pixel intensity,𝑖 indexes the pixel in the row of an image, and 𝑡 representsthe number of diffusion processes, or diffusion steps. Adiscrete update consists of a number of diffusion stepsiteratively applied from the initial row of pixel intensities.Thesubsequent reaction consists of subtraction and amplificationby a nonlinear logistic function. Several updates are requiredto generate stable patterns.

This study extends the earlier theoretical study to theimplementation of the reaction-diffusion pattern generationalgorithms as a protoarchitecture that is mapped into a digitalmedium.Afield-programmable gate array (FPGA) is selectedfor the physical implementation enabling real-time operationof the protocomputation based application.

2.2. One-Dimensional Reaction-Diffusion ProtoarchitectureAiming at a Digital Media Implementation. A data-flow anda block diagram representing the one-dimensional patterngeneration process are shown in Figure 1. The ideal exampleof a step input is presented in Figure 1(a), which is processedby the system until reaching a stable state after updatenumber 10 as a spatial wave (one-dimensional pattern). Theprogress of the step input along the first update is shown,evidencing the edge smoothing obtained by the diffusion,the subtraction of the diffused state from the input step, andthe result of nonlinear amplification. The state-diagram ofthe RD system is presented in Figure 1(b), showing the twofundamental states, namely, diffusion and reaction consistingof subtraction and amplification, as well as the iterativedata passing process that is required and represented asarrows. The state-diagram of the system including the filteris presented in Figure 1(c), where the lower part of the state-diagram represents the RD process that is followed most ofthe time, and the upper part of the state-diagram representsthe Filtering process that is followed repeatedly. Recognizingthat the Filtering process consists of a regular diffusion step,the integrated state-diagram of Figure 1(d) can be derived,which evidences the possible use of identical resources fordiffusion and filtering. Controlling and canceling collisionof spatial waves is obtained by the action of the spatialFiltering process that is applied after the first update, andthen repeatedly after a fixed number of updates, for example,Figure 1(a), in the Filter process (red box). The filter consists

4 Complexity

Update

Input

Di�usion

Ampli�cation

Subtraction

Update: 10

Filter

Update

Subtraction&

ampli�cationDi�usion

Input Output

Input

Update: 10

OutputSubtraction

&ampli�cation

Di�usion

Filter

Filter

Update

Subtraction&

ampli�cationDi�usion

Update

Input

Update: 10

Filter

Output

(a) (b)

(c) (d)

Figure 1: One-dimensional pattern generation concept. (a) Data-flow diagram showing a step input to the Update and Filter processes andthe corresponding sequential signal processing, resulting into a stable spacial intensity wave (1D pattern). (b) State-diagram showing theUpdate process, and (c) state-diagram showing the Update and Filtering processes and evidencing the iterative operation. (d) Integratedstate-diagram, evidencing that the filtering is executed as one diffusion process.

InputOutput

FIFO-A

FIFO-B

MUX

MUX

MUX

MUXD-FF D-FF D-FF

4-inputadder

LUT

Di�usion

Subtraction,amplification

−

≫2

Figure 2: Protoarchitecture of the one-dimensional RD module.

of one step of diffusion (blurring), which is not followedby the reaction process (subtraction and amplification). Inaddition, the spatial filter controls potential effects of noisewhich is diffused such that its effect is not amplified, resultinginto damping out its propagation. In practical terms, thefiltering repetition frequency, the number of diffusion steps,and the maximal gain of the sigmoid function factors aredetermined empirically; for example, in our application case,

one diffusion filtering step is applied every four updates whilethe gain of the sigmoid function is equal to five.

The protoarchitecture of the one-dimensional processingmodule is derived from [26] and consists of four major sub-modules presented in Figure 2.The first diffusion submoduleis in charge of the calculation of diffusion according to (7).The second reaction submodule processes the subtractionand amplification required to complete one update. Finally,

Complexity 5

two line buffers implemented as first-in-first-out (FIFO)memory segments FIFO-A and FIFO-B temporarily storeone row or column, and one processed row or column,respectively. Data is transferred and processed as a word, andthus all operators and transfer lines process or store and,respectively, carry words, which is not drawn to increasevisibility of the figure; 12-bit words are used in the specificdeveloped application further detailed.The entire protoarchi-tecture operates in streaming mode, where data is constantlybeing processed from consecutive memory locations. Hencememory management is reduced to global read/write oper-ation control, and no address needs to be generated for theinternal process.

The input is streamed from a CMOS sensor or a tempo-rary input frame buffer. In both cases, the control of addressesis limited to a row-by-row scanning scheme, which is partof any CMOS imager, while it consists of a simple counter,when the frame buffer memory is used. An input multiplexerserves as a source selection device. The RD process startswith the acquisition of one line (row or column) of a framein a video sequence that is serially streamed to the inputterminal as consecutive words representing the pixel intensityof neighboring, consecutive pixels. The input multiplexer isset to select which input to pass, which is delivered to thediffusion submodule, while it is also stored into FIFO-A inthe first step of diffusion. FIFO-A has the capacity to store anentire row or column and is deactivated upon completion ofstoring.

In parallel to delivering to FIFO-A, consecutive pixels arealso delivered to the diffusion block. A delay line formedof three clocked D flip-flop (D-FF) banks accepts incomingpixels, such that they store three neighboring pixels in a rowafter three clock cycles. Hence, the D-FF bank located in themiddle position stores pixel 𝑎𝑖, the D-FF bank located on therightmost part of the chain stores pixel 𝑎𝑖−1, and the D-FFbank located on the leftmost part of the chain stores pixel 𝑎𝑖+1.This situation is suitable to execute the diffusion computationof pixel 𝑎𝑖 according to (7).The three pixel intensity values arerouted through multiplexers to the four inputs of an adder.Eventually, the adder output is shifted to the right two timeswhich implements the operation of division by four. Thisprocess is repeated at each clock cycle, thus in a streamingmode, and the diffusion of the new consecutive centralpixels 𝑎𝑖 is computed. Each new computed diffusion valueis routed to FIFO-B by a multiplexer. Hence, at completionof the diffusion, one line (row or column) resulting fromone diffusion step is stored in FIFO-B. The two multiplexerslocated at the input of the adder may be reconfigured toproperly handle limit conditions that occur at the boundaryof the image; in general, constant boundary conditions areadmitted, where the limit value, for example, a pixel in row0 or column 0, is used two times in replacement of valuesof 𝑎𝑖−1 and 𝑎𝑖 that do not exist, because they are outsideof the physical array. The central pixel 𝑎𝑖 always connectsto two of the four-input adders, for example, to realize themultiplication coefficient equal to two.

A new step of diffusion can be started immediatelyafter completion of a previous step. The input multiplexer isset to route FIFO-B into the diffusion submodule, and the

process described above repeats. During this new step, noimager input is provided, and FIFO-A is deactivated, storingthe initial nonprocessed frame. Several steps of diffusionmay be processed this way, each implementing one iterationof inhibitor diffusion expressed as 𝑢(𝑥, 𝑇, 𝑛) in (6); theirnumerical count implements parameter 𝑇.

The reaction operation starts while the last step of diffu-sion is streamed through the diffusion submodule. The reac-tion submodule (subtraction and amplification) and FIFO-A are activated as soon as diffusion of the first diffused pixelis completed, that is, delivered at the output of the diffusionsubmodule.The first diffused pixel is subtracted from the firstpixel stored in FIFO-A that pertains to the original nondif-fused line. The result of subtraction is delivered to the finalprocessing consisting of a sigmoid operation implementingthe following function, 𝜁𝑎(𝑥) = (tanh(𝑎𝑥/2) + 1)/2. Function𝜁𝑎(𝑥) and its parameters are tailored to the application byprecomputation of the values. The function is implementedas a lookup table (LUT) storing the precomputed valuesand enabling fast and accurate result delivery. The rightmostmultiplexer routes RD data from the output of the reactionsubmodule into FIFO-B. At the conclusion of the reactioncomputation, one update is completed.

Subsequent updates indexed 𝑛 + 1 in (6) are performeduntil a stable row pattern is generated, yielding 𝑢𝑛+1(𝑥). Noinput is delivered from the image sensor or input framebuffer, and the input data originates from FIFO-B. Hence, anew update starts with delivering the data content of FIFO-B to the diffusion submodule and FIFO-A, in parallel, andproceeds following the procedure described above.

As determined from the algorithmic study, some Filteringprocess is periodically required and is interleaved betweentwo updates.The Filtering process consists of a diffusion step,that is, without reaction. Consequently, the filtering operationis applied strictly using the diffusion submodule, while thereaction and FIFO-A submodules are deactivated.

At completion of the full RD computation of one line,results are delivered to the output. Subsequently, the entireRD process repeats, using the next line until all rowsand columns have been individually processed by the one-dimensional module. At this point, the one-dimensional RDprocess of one frame is completed. Processed data is deliveredfor two-dimensional aggregation, and a new frame can beprocessed by the one-dimensional RD module.

2.3. Two-Dimensional Reaction-Diffusion Protosystem. Two-dimensional RD is applied to acquired images as the appli-cation of the one-dimensional RD process to every rowand column, individually, as depicted in Figure 3(a). Thehardware used to process each row or column is identicaland consists of the circuit earlier presented in Figure 2.Nevertheless, a sequential processing of rows and columnswould not yield real-time operation. As a benefit of theindependent handling of row and columns, the process canbe accelerated by parallel processing. Two columns and tworows are concomitantly processed in a one-dimensional RDcomputation. The subsequent two-dimensional aggregationconsists of a multiplication of the row and column intensity

6 Complexity

Row 1Row 2

Row n

Row n/2Row n/2 + 1Row n/2 + 2

Col 1

Col 2

Col n

Col n

/2Co

l n/2

+ 1

Col n

/2 +

2

...

...

· · ·· · ·

(a)

1D module(y2)

1D module(y1)

1D module(x2)

1D module(x1)

Inputframebu�er(x)

Inputframebu�er(y)

Input

Outputframebu�er(x)

Outputframebu�er(y)

Multiplier Output

WRRD

PR

Initial

Frame

PR

WRRD

1D(y)pattern

InitialFrame

1D(x)pattern

2Dpattern

WR · RD WR · RD

Initial

Frame

PRInitial

FrameInitial

Frame

PR

(b)

Multiplier

(Port B)

1D module(x1)

Frame bu�er(x)

Frame bu�er(x)

1D module(x2)Output

(Port A)(Port B)

Input

(Port A)

Write Read

Process

x:1

x:1

x:1 x:2 x:2x:n

x:1 x:2 x:n

x:1 x:2 x:n

x:n

x:n

x:n/2

x:1 x:2 x:n/2

x:2 x:n/2

x:n/2 + 2

x:n/2 + 2

x:n/2 + 1

x:nx:n/2 + 2x:n/2 + 1

x:n/2 + 1

(c)

Figure 3: Protoarchitecture and operation of the two-dimensional RD system. (a) Independent row and column processing in a one-dimensional algorithm. (b) Block-level system protoarchitecture. (c) Detailed timing chart.

values of a pixel obtained from the RD individual processingof rows and columns.

The two-dimensional real-time RD processing protoar-chitecture is presented in Figure 3(b). RD, WR, and PR,respectively, denote read, write, and processing operations.Data streaming to the system input originates from an imagesensor, in real-time, while the output is to be delivered tofurther processing units, for example, extracting the motionvectors by operating block matching in consecutive resultingRD frames and further exploiting this result.The architecturecomprises two parallel data paths, each processing two rowsand two columns, respectively, in parallel. Two input framebuffers store images of identical horizontal and verticalresolution, for example, 120 × 120 pixels or 250 × 250pixels.The two images are processed by four one-dimensionalRD modules, and the results are stored into two outputframe buffers. The signals that control the four subpaths areidentical which significantly simplifies the controller. In orderto achieve this feature, the image that is stored into input

framebuffer (𝑦) is a rotated andmirrored version of the imagestored in input frame buffer (𝑥). This storage organizationis achieved by appropriate address generation at memorywriting.Hence, in readmode and using identical frame bufferaddresses, input frame buffer (𝑥) delivers two consecutiverows while input frame buffer (𝑦) delivers two consecutivecolumns. Finally, a multiplier aggregates the row and columndata obtained for every pixel from output frame buffers (𝑥)and (𝑦) and delivers the RD image to further processing.

The detailed timing chart of the system is summarizedin Figure 3(c). Here, frame buffer write operations aremarked in blue, read operation is marked in green, andRD module processing is marked in black. The change ofa read/write/process mode is written over the first involveddata, while the vertical wavy line depicts data that is notdisplayed to improve visibility. Only the timing chart of theupper data path handling two rows is shown, for the sakeof clarity. The timing chart of the lower data path handlingtwo columns can be obtained by replicating the timings of the

Complexity 7

Table 1: FPGA synthesis results of the core system.

Characteristic 120 × 120 250 × 250 Characteristic 120 × 120 250 × 250Total logic elements 499 2,221 Total registers 656 3,029Total memory bits 745,472 3,203,072 → Usage rate 13% 57%RAM blocks (⋅/553) 92 396 DSP blocksa 1 15Total RD modules 4 20b Max. frequency 69.12 66.9(Horizontal, vertical) 2, 2 10, 10 (𝐹max)MHzProcessing fps at 𝐹max 34.43 38.89 Processing fps at 50MHz 24.91 29.06aMultiplier used in the RD module. bOperation is pipelined in order to restrain FPGA BRAM utilization.

upper data path, while changing 𝑥 into 𝑦 when input framebuffer (𝑦) is in read model. Here ⟨𝑥 : 𝑛⟩ denotes row 𝑛 of theframe, and ⟨𝑦 : 𝑛⟩ denotes column 𝑛.

Hence, two-dimensional frame processing for texturegeneration is performed by providing the one-dimensionalRD module with data organized in a two-dimensional arrayfollowing a time-division scheme, which reduces the com-plexity of the circuit and system as a whole, while the paralleloperation accelerates the global processing to reach to real-time operation.

2.4. Hardware Implementation and Target Platform. Theprotoarchitecture of the RD system is implemented in ahardware platform with FPGA for real-time processing. ATerasic DE10 Standard board embedding a Cyclone V FPGAand equipped with a TRDB-D5M, 5M pixel CMOS sensorimage acquisition peripheral board are used. Data is acquiredfrom the image sensor in 8-bit RGB format. Results aredisplayed using an external display screen connected througha VGA link. The board operation frequency is 50MHz whilethe core circuits could operate up to 𝐹max (Table 1), and theimage acquisition rate is programmable up to 70 fps (framesper second).

The operational characteristics and synthesis results per-taining to the core of the system that embeds the 1Dmodule ofFigure 2 are presented inTable 1.The entire core is considered,specifically including input frame buffers supporting the120 × 120 and 250 × 250 window processing. In this lattercase, the level of 1D module parallelism is increased from 4to 20 and pipelining is applied as a way to limit FPGA block-RAM usage at constant input frame buffer bit size. However,service circuits such as the camera interface are deliberatelynot included into the synthesis results.

3. Results and Discussion

The operation of the entire two-dimensional RD systemprotoarchitecture is confirmed using numerical simulationand seminatural images. Consecutive video images of size120 × 120 pixels are created from a natural backgroundthat is covered by a textureless moving object. Thoughthe resulting image appears synthetic to human eyes, therecreated scheme in fact conforms an expected real condition,where an object that appears without texture moves into ascene. The vanishing of texture could be resulting from localsaturation of imager pixels due to a high-intensity reflection

in the scene, for example, white object moving under highillumination, or insufficient gain of the imager, potentiallyresulting froman image sensor self-adjusting its gain to a highdynamic range environment.

Figure 4 presents numerical simulations of the aforemen-tioned situation. AVerilog RTLmodel of the hardware is usedto process shown data. In order to obtain visually meaningfulevidence, ten frames separate the original input shown inFigure 4(a) with respect to Figure 4(b); the movement ofthe original object is clearly perceptible. The motion vectorsextracted using the conventional block-matching techniqueare compared to the result of the RD-based proposed tech-nique. The results are presented in Figure 4, where (a) and(c) pertain to the same image, and (b) and (d) pertain toanother same image. The motion vectors obtained from aconventional block-matching algorithm are shown in redin (a) and (b), clearly evidencing that motion can onlybe detected at the edges perpendicular to the motion. Incontrast, (c) and (d) show the vector field obtained usingthe proposed technique and hardware that is simulated usingan RTL model, where a dense array of red motion vectorsis detected inside the object, along edges of the generatedpatterns that have followed the textureless object movement.

3.1. Real-TimeHardware Protosystem. Thehardware platformpresented in Section 2.4 is used to generate real-time resultsof RD pattern generation in naturalmovies acquired from thecamera, in order to further generate motion vector fields.Theimage field is limited to 250×250 pixels by hardware physicalresources. Results are presented in Figure 5 confirming thecorrect operation of the protosystem.

A real indoor environmental situation is used to confirmthe capacity of the algorithm and hardware implementationthat consists of a single dark object of rectangular shapemoving in front of a text background. The processed framesize is equal to 120 × 120 pixels, and the object movementhas been made sufficiently significant, for example, to createqualitatively visible results. The background may be acquiredand processed slightly blurred due to the limited depth of fieldof the unsophisticated optical lens system that is used. Imagesare extracted from the acquired and processed video flowand shown in Figure 5. The dynamic range of images is com-pressed to 3 bits prior to processing, which can be achieved inhardware by straightforward wire connecting. Hence, imagesmay appear dark to human observers. The motion vectorfield that is obtained from software block matching of RD

8 Complexity

(a) (b)

(c) (d)

Figure 4: Simulations of an objectmovement corresponding to one pixel and subsequent detection ofmotion vectors depicted in red. ((a) and(b)) Motion vector results using a conventional block-matching algorithm; ((c) and (d)) corresponding results using the proposed RD-basedmethod and hardware.

(a) (b)

(c)

(d)

Figure 5: Real-time extraction of RD patterns of a textureless object moving over a text background and subsequent estimation of thecorresponding motion vector field. (a) First acquired frame and corresponding RD pattern; (b) second acquired frame and correspondingRD pattern. (c) Close-up of the RD pattern obtained from the second acquired frame with overlaid motion vectors and (d) close-up of thesecond acquired frame with overlaid motion vectors that are acquired using a conventional block-matching algorithm.

Complexity 9

Figure 6: Real-time experimental setup. The processed window is superposed to the full VGA background. The unprocessed and processedwindow are enlarged in the inlets.

frames is presented in Figure 5(c). The generation of a densefield of motion vectors that appear in red color inside themoving textureless object is confirmed. Some unexpectedmotion vectors appear outside the area of the moving object.Those which appear in regions previously covered by partsof the moving object are incorrect, though expected; theyconfirm the necessity to satisfy the hypothesis of a slowmoving object or high acquisition speed. The system is verysensitive to any movement, but also to luminance changeswithin the scene, as a benefit of the iterative RD processthat includes a high amplification nonlinear process. Hence,some vectors may be generated within the scene that do notcorrespond to a movement, but correspond to a parasiticchange of luminance. These vectors are incorrect, but veryfew. A different set of parameters of the algorithms may fixthese incorrectly determined movement vectors, however,to the cost of a decrease in the proportion of the correctlydetected vectors. Consequently, as a real-time system, a trade-off between the speed of operation (number of iterations)and the performance metrics that characterize the systemmust be found. As in any sensor and detection system,these performancemetrics are defined from true positive andnegative, false positive and negative detections, with respectto the total set of expected vectors. These metrics must beextracted from empirical experiments carried over multiplesets of data, that is, various videos acquired with differentconditions, for example, image size, motion distance, andillumination. This systematic study is beyond the scope ofthis paper. In addition, conforming to the theory of reaction-diffusion systems, the proposed system is observed to operatein optimal conditions consisting of a high contrast of intensitybetween the object and its background, and in presence of ahighly textured background where existing features promotelocal and global fixing of the RD patterns. Finally, the motionvector field obtained using classical block matching onlygenerates vectors at the edges, or underneath zones that werecovered by the object in the previous frame. Consequently,

the number of vectors that appear in Figure 5(d) is expected tosignificantly reduce with a smaller movement of the texture-less object. In contrast, the proposed technique results shouldimprove with low amplitude of the movement, where thenumber of incorrect vectors should decrease, while correctvectors should remain stable.

The real-time experimental setup is presented in Figure 6.The optimized parallel implementation of the protosystemenables real-time operation in a window of 250 × 250 pixels.The enlarged view shows the patterns generated inside thecup that is rendered textureless due to its color and a high-intensity illumination of the scene. The window size of 250 ×250 pixels does not represent any limitation of the method orprotoarchitecture but relates to the memory capacity of theFPGA.Hence, window size with respect to hardware resourcescaling is a major concern that is discussed in the following.

3.2. Real-Time Hardware Scaling of the Protoarchitecture. Thecomputational capacity of protocomputational systems isdictated by the nature of the supportingmedium, as well as itsphysical extent. The concept of real-time operation as such isabsent in natural computation. In contrast, the emulation of anatural phenomenon over a substrate of a different type posesissues of accuracy with respect to reflecting natural timings.This issue is well understood in computer science and hasbeen addressed from the concept of real-time that stipulatesthe time elapsed to compute the simulatedmodel conformingto the time required by the real environment. It is common foremulating systems including microelectronic and computersystems to take benefit of virtual, parallel, or redundantimplementations that compute in parallel and support thereal-time capability. Within this context, scaling the extentof the protocomputation system has little or no meaningwith respect to a natural system. However, considering theselected video application, and the parallelization conceptdiscussed above, scaling appears a relevant hardware issue

10 ComplexityRe

sour

ce (m

emor

y bi

ts)

1.E + 10

1.E + 09

1.E + 08

1.E + 07

1.E + 06

1.E + 05

Window size (pixel)

120×12

0

250×25

0

640×480

(VG

A)

1024

×768

(SXG

A)

2048×15

36

(QXG

A)

Figure 7: Window size and hardware resource scaling (log-logscale).

that is fully supported by the proposed protoarchitecture, andis discussed in the following.

The size of the image that is acquired is generally dictatedby application or image sensors standards. The hardwareimplementing the RD algorithm should adapt, for example,to cover the entire image. Considering real-time operationa compulsory specification results into the stringent needto allocate additional hardware resources enabling scalingprocessing to large image sizes.

System scaling may be supported taking three possibleprotoarchitectural adaptations into account. The BRAM-parallel protoarchitecture (BPA) consists of increasing thenumber of pixels in each row while also increasing the levelof parallelism to keep 30 fps, that is, the number of paralleldata paths (1D modules) in Figure 3. Eventually, an imagemay form one single window. A different level of paral-lelism should be selected pertaining to the number of unitsprocessing the rows and the columns in parallel, in orderto accommodate rectangular image formats. The module-parallel protoarchitecture (MPA) dictates parallelizing BPAblocks of small size. Some overlap between neighboringsmall-size windows may be used to guarantee coherency ofthe generatedmotion vectors.The fixed boundary conditionsof the RD process dictate this overlap, whose size depends ontheRDparameters. Finally, a hybrid of the BPAandMPAmaybe considered.

The decision criterion regarding the suitable scaled pro-toarchitecture is based on the analysis of the hardwareresources that are theoretically required in a first-order con-sideration of the window size, as presented in Figure 7. Thetheoretical analysis (blue trace) dictates a linear relationshipbetween window size and hardware resources. The BPA andMPA architectures are equivalent at the location of thistheoretical curve. In practical terms, however, second-ordereffects must be considered, consisting of the necessity toadapt RD parameters to the size of the window, potentiallyto the application. For example, the number of RD updatesthat must be applied to obtain stable patterns may vary withrespect to the window size, which eventually reflects intohardware resources allocation. Hence, a decrease of some key

parameters may result in a real curve proportionally lowerthan theoretically predicted and potentially a tendency tosaturate (red).The BPA architecture is preferable under theseconditions, which takes full benefit of the hardware resourcescaling. In contrast, an increase of some key parameters mayresult in a real curve proportionally higher than theoreticallypredicted and potentially a tendency to an exponentialbehavior (green). The MPA architecture is preferable underthese conditions, which restrains the hardware scaling to astrict theoretical behavior.

4. Conclusions

This paper demonstrates the correctness of an approachconsisting of using a digital physical medium to the purposeof implementing the reaction-diffusion algorithm. A dedi-cated protoarchitecture is developed, which is subsequentlymapped into the implementation media. Real-time opera-tion of a realistic video processing application is demon-strated using the proposed method and protoarchitecture,whereas classical algorithms are known to fail. Specifically,the reaction-diffusion is applied to textureless objectsmovinginto a video frame acquired at 30 fps in order to create textureallowing the generation of dense fields of motion vectorsin real-time. Simulations and real operation results usingwindows of sizes up to 250×250 pixels confirm the suitabilityof the algorithm, the protocomputing architecture, and itshardware implementation.A scalingmethod is presented thatsupports real-time operation.

Conflicts of Interest

The authors declare that there are no conflicts of interestregarding the publication of this article.

Acknowledgments

This study was supported by the JSPS Grants-in-Aid for JSPSFellows and a Grant-in-Aid for Scientific Research on Inno-vative Areas [2511001503] from the Ministry of Education,Culture, Sports, Science and Technology (MEXT) of Japan.

References

[1] S. Haykin, Neural Networks and Learning Machines, Pearson,3rd edition, 2008.

[2] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning,Adaptive Computation and Machine Learning series, MIT Press,Mass, USA, 2016.

[3] A. Schmid, “Neuromorphic microelectronics from devices tohardware systems and applications,” Nonlinear Theory and ItsApplications, IEICE, vol. 7, no. 4, pp. 468–498, 2016.

[4] Y. Suzuki, “Harness the Nature for Computation,” in NaturalComputing and Beyond, vol. 6 of Proceedings in Information andCommunications Technology, pp. 49–70, Springer, Tokyo, Japan,2013.

[5] S.-J. Kim, “Efficient decision making by harnessing the compu-tational power of physical phenomena,” IEICE Tech. Rep, vol.116, no. 514, pp. S2016–45, March 2017.

Complexity 11

[6] M. Aono,M. Naruse, S.-J. Kim et al., “Amoeba-inspired nanoar-chitectonic computing: solving intractable computational prob-lems using nanoscale photoexcitation transfer dynamics,” Lang-muir, vol. 29, no. 24, pp. 7557–7564, 2013.

[7] A. Adamatzky, Advances in Physarum Machines, Sensing andComputing with Slime Mould, vol. 74, Springer, 2016.

[8] B. Kacar and E. A. Gaucher, “Towards the recapitulation ofancient history in the laboratory: combining synthetic biologywith experimental evolution,” in Proceedings of the InternationalConference on the Simulation and Synthesis of Living Systems, pp.11–18.

[9] M. Naruse, Y. Terashima, A. Uchida, and S.-J. Kim, “Ultrafastphotonic reinforcement learning based on laser chaos,” Scien-tific Reports, vol. 7, no. 1, article no. 8772, 2017.

[10] M. Markus and B. Hess, “Isotropic cellular automaton formodelling excitable media,” Nature, vol. 347, no. 6288, pp. 56–58, 1990.

[11] J. R. Weimar and J.-P. Boon, “Class of cellular automatafor reaction-diffusion systems,” Physical Review E: Statistical,Nonlinear, and Soft Matter Physics, vol. 49, no. 2, pp. 1749–1752,1994.

[12] A. J. Tabatabai, R. S. Jasinschi, and T. Naveen, “Motion Estima-tion Methods for Video Compression - A Review,” Journal ofThe Franklin Institute, vol. 335, no. 8, pp. 1411–1441, 1998.

[13] T. Yokoyama, T. Iwasaki, and T. Watanabe, “Motion vectorbased moving object detection and tracking in the MPEGcompressed domain,” in Proceedings of the 7th InternationalWorkshop on Content-Based Multimedia Indexing, CBMI 2009,pp. 201–206, Greece, June 2009.

[14] M. Tanaka, Japan patent Kokai, JP2014-52934A, 2014.[15] Y. Takagi, Japan patent Kokai, JP2012-15959A, 2012.[16] M. P. Vijaykumar, A. Kumar, and S. Bhatia, “Latest trends,

applications and innovations in motion estimation research,”International Journal of Scientific & Engineering Research, vol.2, p. 1.

[17] N. K. Parmar and M. H. Sunwoo, “Recent progress on block-based motion estimation techniques,” IETE Technical Review,vol. 32, no. 5, pp. 356–363, 2015.

[18] M. Mori, T. Itou, M. Ikebe, T. Asai, T. Kuroda, and M.Motomura, “FPGA-based design for motion vector estimationexploiting high-speed imaging and its application to motionclassificationwith neural networks,” Journal of Signal Processing,vol. 18, no. 4, pp. 165–168, 2014.

[19] C. Choi and H. I. Christensen, “3D textureless object detectionand tracking: An edge-based approach,” in Proceedings ofthe 25th IEEE/RSJ International Conference on Robotics andIntelligent Systems, IROS 2012, pp. 3877–3884, Portugal, October2012.

[20] I. R. Epstein and J. A. Pojman, An Introduction to NonlinearChemical Dynamics, Oscillations, Waves, Patterns, and Chaos,Oxford University Press, 1998.

[21] J. D. Murray, in Mathematical Biology I: An Introduction,Springer, New York, NY, USA, 3rd edition, 2002, Chapter 11.

[22] Y. Suzuki, T. Takayama, I. Motoike, and T. Asai, “Stripedand spotted pattern generation on reaction diffusion cellularautomata: theory and LSI implementation,” International Jour-nal of Unconventional Computing, vol. 3, p. 13, 2007.

[23] J. D. Murray, in Mathematical Biology I: An Introduction,Springer, New York, NY, USA, 3rd edition, 2002, Chapter 7.

[24] M. Gerhardt and H. Schuster, “A cellular automaton describingthe formation of spatially ordered structures in chemical sys-tems,” Physica D, vol. 36, pp. 209–221, 1989.

[25] M. Ushida, A. Schmid, T. Asai, K. Ishimura, andM.Motomura,“Motion vector estimation of textureless objects exploitingreaction-diffusion cellular automata,” International Journal ofUnconventional Computing, vol. 12, no. 2-3, pp. 169–187, 2016.

[26] K. Ishimura, K. Komuro, A. Schmid, T. Asai, and M. Moto-mura, “FPGA implementation of hardware-oriented reaction-diffusion cellular automata models,” Nonlinear Theory and ItsApplications, IEICE, vol. 6, no. 2, pp. 252–262, 2015.

Submit your manuscripts athttps://www.hindawi.com

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

MathematicsJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Mathematical Problems in Engineering

Hindawi Publishing Corporationhttp://www.hindawi.com

Differential EquationsInternational Journal of

Volume 2014

Applied MathematicsJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Probability and StatisticsHindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Journal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Mathematical PhysicsAdvances in

Complex AnalysisJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

OptimizationJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

CombinatoricsHindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

International Journal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Operations ResearchAdvances in

Journal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Function Spaces

Abstract and Applied AnalysisHindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

International Journal of Mathematics and Mathematical Sciences

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 201

The Scientific World JournalHindawi Publishing Corporation http://www.hindawi.com Volume 2014

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Algebra

Discrete Dynamics in Nature and Society

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Decision SciencesAdvances in

Journal of

Hindawi Publishing Corporationhttp://www.hindawi.com

Volume 2014 Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Stochastic AnalysisInternational Journal of