INFOTECH OULU Annual Report 2016 1 BIOMIMETICS AND INTELLIGENT SYSTEMS GROUP (BISG) Professor Juha Röning and Dr. Heli Koskimäki, Biomimetics and Intelligent Systems Research Unit, Faculty of Information Technology and Electrical Engineering, and Professor Seppo Vainio, Oulu Center for Cell- Matrix Research, Faculty of Biochemistry and Molecular Medicine, University of Oulu juha.roning(at)oulu.fi, heli.koskimaki(at)oulu.fi, seppo.vainio(at)oulu.fi http://www.oulu.fi/bisg Background and Mission Biomimetics and Intelligent Systems Group (BISG) is a fusion of expertise from the fields of computer sci- ence and biology. In BISG, our basis are intelligent systems and our research areas include data mining, machine learning, robotics, and information security. More precise research topics vary from data mining algorithm development and optimization of industrial manufacturing processes all the way to environmental monitoring with mobile robots. Bringing expertise from ICT and Biotech together, we will reach the skills to make use of the mechanisms common in information processing and the biological data processing system and extrapolate this to intelli- gent solution making in ICT. One important goal of this program is to be able to physically link living cells via identified signaling systems to establish learning complex that involves Bio and ICT in a unified bifunc- tional interactive machine. The group consists of four sub-groups: Data Analysis and Inference Group, Organ BioEngineering Biology, Robotics and Secure Programming We have conducted basic research in intelligent sys- tems and tissue engineering for over ten years as indi- vidual groups. Now we have joint our efforts. Our team consists of 2 professors, 10 post-doctoral researchers and 15 doctoral students. The annual external funding of the group is more than two million Euros, in addi- tion to our basic university funding. In the reported year, there have been 23 completed doctoral degree from the group. From the research of the group, 11 spin-off companies have been established so far: Codenomicon, Clarified Networks, Hearth Signal, Nose Laboratory, Nelilab, Atomia, Indalgo Probot, Aquamarine Robots, Radai and IndoorAtlas. We co-operate with many international and domestic partners. In applied research, we are active in European projects. In addition, several joint projects are funded by the Finnish Funding Agency for Technology and Innovation (Tekes) and industry. We were a research partner in the SIMP and CyberTrust SHOKs. Prof. Juha Röning was selected as ACO (Academic coordi- nator) of the Cyber Trust program. We are active in the scientific community. For exam- ple, Prof. Juha Röning is acting as visiting professor of Tianjin University of Technology and as the Robot Science Adviser of Tianjin Science and Technology Center for Juveniles. He served as a member of the Board of Directors in euRobotics and as a member of the SAFECode International Board of Advisors. He chaired the euRathlon / TRADR Summer School 2016 in Oulu, Finland, 22nd to 26th of August. It was a five- day course to provide participants with a full overview and hands-on experience with multi-domain real robot- ic systems. He also chaired with prof. Othmane the First International Workshop on Agile Development of Secure Software (ASSD’16) in Salzburg 1st of Sep- tember. With robotics group, he participated NORDRUM project where aim was to collect radiation data from the environment using an unmanned aerial vehicle (UAV). The testing area was located in Nor- way, Hauerseter Leir military campsite (5th to 7th of September). During the reporting year, the group organized the 9th International Crisis Management Workshop and Winter School (CrIM’16) and NordSec conference which brought together both Finnish and international infor- mation security experts. The group also organized Summer School 2016 of Data mining, big data and open data together with Exactus DP and Aurora DP 15th to 19th of August. Representing Finland as a Partner for Peace (PfP) na- tion, BISG / Prof. Röning participated the Specialists’ Meeting on Intelligence and Autonomy in Robotics, held in Wachtberg, Bonn, Germany on 25 – 27 October 2016. Celentano and Röning are co-editors, together with collaborating partners, of an IEEE Access special sec- tion on Recent Advances in Socially-aware Mobile Networking. Prof. Seppo Vainio has been the chair in the Minisym- posium ”Omics in Biomedicine” (2016). He is also part of a European nanotechnology ”HyNanoDend” net- work. Scientific Progress Intelligent Systems Incorporating Security Within the Biomimetics and Intelligent Systems Group, the Oulu University Secure Programming Group (OUSPG) has continued research on security and safety in intelligent systems. Security and safety challenges in
23
Embed
BIOMIMETICS AND INTELLIGENT SYSTEMS GROUP (BISG)Biomimetics and Intelligent Systems Group (BISG) is a fusion of expertise from the fields of computer sci-ence and biology. In BISG,
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
INFOTECH OULU Annual Report 2016 1
BIOMIMETICS AND INTELLIGENT SYSTEMS GROUP (BISG)
Professor Juha Röning and Dr. Heli Koskimäki, Biomimetics and Intelligent Systems Research Unit, Faculty
of Information Technology and Electrical Engineering, and Professor Seppo Vainio, Oulu Center for Cell-
Matrix Research, Faculty of Biochemistry and Molecular Medicine, University of Oulu
Figure 11. In autumn 2013, 250 conscription aged men were recruited from call-ups to test the wellness coach-ing service developed in the MOPO project.
Foundations of knowledge discovery and data mining:
Knowledge discovery in data (KDD) was defined in
1996 by Fayyad et al. as “the nontrivial process of
identifying valid, novel, potentially useful, and ulti-
mately understandable patterns in data”. Although this
definition still has its merits, it represents a rather nar-
row interpretation of the concept of knowledge that
may prove a hindrance to the development of more
advanced KDD tools. Meanwhile, the seminal process
model proposed by Fayyad et al., which depicts the
KDD process as a sequence of five major steps, is still
embedded in most KDD process models, including the
standard model CRISP-DM. This established model,
while essentially correct, represents a limited perspec-
tive on the KDD process that is likely to prove inade-
quate in the long run.
In its research on the foundations of KDD and data
mining, BISG has sought to expand this traditional
view of the nature of KDD. The resulting model, like
the established one, accounts for the data transfor-
mations required in order to get from raw data to
knowledge, but also for the actors of the process and
the interactions among them that need to take place for
the process to move forward. Furthermore, the model
explicitly considers the contributions of non-expert
actors, as well as the possibility of technology taking
on a more autonomous role in the process, which is
likely to be realized in the near future as KDD software
grows more intelligent and becomes capable of han-
dling tasks that currently require a human actor. Hav-
ing a model that provides a more complete account of
the KDD process is essential in unlocking the full po-
tential of KDD technology, which in turn is crucial in
making sense of the deluge of digital data that seems to
have become a permanent feature of high-technology
societies. Figure 12 illustrates the process actors and
how different interactions among them lead to different
types of KDD processes.
Intelligent Systems Incorporating Robot-ics and Cybernetics
euRathlon Summer School
The ERL Emergency/TRADR summer school 2016
was organized from the 22nd to 26th of August by the
robotics group members and the staff from TRADR
(Long-Term Human-Robot Teaming for Disaster Re-
sponse). The summer school was attended by 55 stu-
dents, mostly doctoral, originating from 17 different
countries (Figure 13). Also, 6 invited lecturers from
TRADR held lectures during the summer school. Ac-
cording to the satisfaction survey, the participants were
very pleased with the summer school as all of those
who answered the survey would recommend it to oth-
ers.
Figure 12. The actors of the KDD process can be illustrated as the vertices of a triangle, with technology in the cen-ter, being both an actor in its own right and a mediator of interactions among human actors (a). The process can take on a number of different forms, characterized by which of the actors are present and how they interact: the standard KDD process (b), KDD using personal data (c), KDD using volunteer computing (d), and KDD driven by a non-expert actor (e). A good example of the latter is the so-called Quantified Self movement.
INFOTECH OULU Annual Report 2016 9
Figure 13. Attendees and organizers in the ERL Emer-gency/TRADR summer school 2016 held at the Univer-sity of Oulu.
This year, the ERL Emergency summer school focused
on developing algorithm for controlling land robots
with a strong focus on SLAM and multi-source persis-
tent data integration.
In total, the summer school lasted for four and a half
days consisting roughly 35% of lectures and 65% of
practical exercises in which the students developed
control and SLAM algorithms. These practical sessions
were held indoors in the University of Oulu’s facilities
and outdoors in a nearby (less than 1km of walking)
botanical garden where electricity, shelter and internet
access were also provided. The hands-on practices
culminated to a challenge scenario that each team per-
formed on the last day at the botanical garden.
The students were provided with two of TRADR’s
UGVs and one UAV (Figure 14) provided by Ascend-
ing Technologies. The students could modify and de-
velop software for the two UGVs but the UAV was
flown only by the trained representative of Ascending
Technologies. Before the summer school, these UGVs
and UAV were also used to gather preliminary data for
software development during the exercises. The raw
data was used to form initial maps that were given to
the students to work on for testing and performing
simulations. This was done to reserve the students’
time for more meaningful tasks as the generation of
maps from raw data takes many hours of processing on
a desktop computer.
During the registration process the students were asked
to provide a brief description of their programming
experience. This info was used on the first day to form
eight balanced teams of six or seven persons. The bal-
ancing was done mainly in regard of C, Python and
ROS experience as at least one person in each group
had to have at least basic understanding of these to
ensure that the practices would proceed in a timely
fashion.
Figure 14. TOP; One of the two identical unmanned ground robots (UGV) provided by TRADR. BOTTOM; The UAV, AscTec Falcon 8, provided by TRADR part-ner Ascending Technologies.
On the first day the student teams were presented with
a challenge scenario they would perform and compete
on the fifth, and last, day of the summer school. The
scenario consisted of a simulated toxin leak at the bo-
tanical garden and the teams’ task would be to find and
localize the toxic materials using the UGV, UAV and
the pre-recorded maps of the area.
To fulfil this task, the teams needed to fulfil the sub-
tasks:
Map the area using one UGV and one UAV.
Update and refine the map based on new data.
Develop strategies to safely navigate the UGVs in
the danger zone.
Navigate the UGV to designated points of interest
with the highest degree of autonomy possible.
Detect objects automatically if possible.
Have the UGV perform automatic collision avoid-
ance if possible.
The practical sessions focused on developing ways to
fulfil these tasks and mapping the scenario area (see
Figures 15-18).
INFOTECH OULU Annual Report 2016 10
Figure 15. The used waypoint planner and simulator.
Figure 16. Outdoor testing at the botanical garden.
The challenge scenario was run on the final day of the
summer school. The toxic leaks were simulated with
bright balloons and rough estimates of their locations
were given to the teams. Compared to previous days,
the scenario environment was somewhat changed by
added obstacles (chairs, tables, etc.). Each team had 30
minutes of time to complete the mission, during which
they had full access to the UGV. The teams also had a
limited 5 minutes access to the UAV, flown by the
trained operator, to get a rough overview of the envi-
ronment.
Figure 17. Scenario briefing and composed map of the area.
The students were also given a demonstration of the
Aquamarine Robots Dolphin marine robot.
Figure 18. Aquamarine Robots Dolphin robot shown during the summer school.
INFOTECH OULU Annual Report 2016 11
Robotics Research
In 2016 BISG had a wide range of research in the area
robotics, including industrial safety, aerial data gather-
ing, battery life management and control of complex
wheeled land robots.
ReBorn
The EU funded ReBorn project has ended. The project
had participants from 17 industrial and academic insti-
tutions from 10 different countries. During the project,
the sufficiency of current standards related to robot
development and reusability in industrial environments
was investigated by a paper review and by surveys sent
to the project partners. The current standards (Figure
19) for designing user safe robots were deemed suffi-
cient for fulfilling the requirements to implement safe
robots for traditional industrial applications. However,
in some applications shortcomings in the currently
available standards were found.
Figure 19. Main standards applicable in implementing user safety in industrial robotics.
One of the commonly mentioned issues was that there
is a lack of standardized commonly applicable perfor-
mance descriptions for the existing line of robots. This
especially hinders the flexible use of heterogeneous
and modular robotics. Also, reuse and repurposing old
robots for new applications is more difficult without a
common form of performance descriptions and capa-
bilities of the robots. It was also found out there cur-
rently are no dedicated ISO/EN/DIN standards specifi-
cally related to safety and design assisting in imple-
mentation of reconfigurable manufacturing cells. Other
area that was mentioned in the survey responses from
the project partners was the lack of LCC (Life Cycle
Cost) standards, similar to what are already in use in
the building construction industry.
One area of interest in the project was the requirements
for implementing CWS (Collaborative Work Spaces).
In this field, some standards already exists (Figure 20)
that can be utilized to implement the minimum safety
features required to avoid serious injuries. However,
from the applications side, the standards are currently
vague on how the software should be implemented and
how the user should be taken in to account as an agent
acting in the control loop when performing tasks co-
operatively with a robot. The actions of the human in
the loop needs quite a lot of prediction and behavior
observation to implement co-operation efficiently and
safely in flexible manufacturing units. This is an area
that is currently under a lot of research and appropriate
standards should also be developed on how the user
monitoring and behavior prediction should be per-
formed on hardware and algorithm level.
Cloud computing is an area that could be better utilized
in industrial environments as a channel for data pro-
cessing, learning and teaching of industrial robots in
the future. Cloud computing could also be used for
openly collecting and sharing data about robot reliabil-
ity for evaluation of the reusability, safety and costs of
running specific types of robots in certain tasks.
Figure 20. Standards applicable for collaborative work spaces.
NORDUM Exercise
NORDUM (Intercomparison of Nordic unmanned
aerial monitoring platforms) exercise was organized in
the Hauerseter Leir military campsite, Gardermoen,
Norway. The NKS-B activity NORDUM is the first
joint Nordic exercise for unmanned systems. All in all,
five teams participated in this event coming from dif-
ferent universities and radiation safety related institu-
tions located in Norway, Sweden and Finland.
In the NORDUM exercise, the objective was to locate
and identify potential radioactive materials from the
arranged scenario areas. The scenario areas varied from
cluttered areas containing large shipping containers and
various metal structures to open field and forest scenar-
ios. For the measurements, a stand-alone sensor pack-
age was constructed containing a RTK (Real Time
Kinematic) capable GPS (u-blox C94-M8P-3), meas-
urement computer (Raspberry Pi 3 Model B), a 433
MHz (3DR) radio, a 3.7V Li-ion battery and a gamma
radiation spectrometer (Kromek GR1-A).
In the arranged scenarios, the teams needed to localize
hidden radiation sources and visualize their location
utilizing GPS. In the scenarios, the constructed stand-
alone sensor package performed well in most scenari-
os, although some radio link related issues were en-
INFOTECH OULU Annual Report 2016 12
countered especially near large metal structures. The
utilized Kromek GR1-A was sensitive enough that
radiation sources could also be identified from local
spectrum histograms when enough flybys near the
radiation source was made.
The stand-alone sensor package was carried with a DJI
the vehicle. This made possible manoeuvring the sen-
sor very close to the objects being measured. The
quadcopter had a flight time of 10 minutes with a 5.7
Ah 22.2V Li-ion flight battery and the sensor package.
The 4k resolution camera was utilized during flight to
observe the sensor position and to manoeuver it to
wanted positions. Although the conditions were rather
windy, flying with the sensor package was manageable.
The quadcopter with the sensor package is shown in
Figure 21. The measurement results from one of the
three scenarios is shown in Figure 22 and Figure 23.
Figure 21. The DJI Inspire 1 carrying the constructed stand-alone sensor package containing a GPS, a radio transceiver and a gamma radiation detector. On the right is a still image captured by the onboard 4k resolu-tion camera.
Figure 22. Gamma radiation measurements made with the stand-alone sensor package carried by the quad-copter in one of the testing scenarios. The brightness of the green color indicates the intensity of the detected gamma radiation activity.
Figure 23. Local histogram collected from area en-closed by the red circle 3 in the scenario image. The detected spike corresponds with Cs-137 (Cesium with a theoretical gamma radiation energy emissions of 661.64 keV).
Robots
The Mörri robot has been equipped with more easily
maintainable and more powerful electronics in its rein-
carnation. The main drive electronics are now mostly
off-the-shelf components controlled with an Ardupilot
APM2 based controller that is connected to an onboard
computer handling the overall robot control and com-
munications with a remote control station. The Mörri
platform is also used in testing the test batch of intelli-
gent battery modules that have been constructed. The
functional diagram of the new drive system is shown in
Figure 24 and the Mörri mobile platform in Figure 25.
Figure 24. The overview of the renewed fundamental electrical system required to drive the Mörri robot.
Figure 25. Mörri with Microsoft’s Kinect 2 sensor driving on a field and on snow with tracks put on.
In anticipation of performing joint missions simultane-
ously with multiple UAVs and UGVs, also custom
quadcopter platforms are being constructed. The basic
platform, shown below in Figure 26, is low-cost and is
constructed from off-the-shelf components for better
maintainability. The quadcopter platform is built
around the open-source ArduPilot PX4 flight control-
ler, allowing more freedom for customization and test-
ing our own implementations required for autonomous
operation, which is not as easy to do with most prebuilt
and significantly more expensive quadrotors. Com-
bined with LIDAR (Light Detection And Ranging), the
copters will be used for SLAM and environment classi-
fication efforts in joint missions with UGVs, such as
Mörri. Because both Mörri and the quadcopter utilize
ArduPilot based controllers, the development of both
platforms is simpler due to having very similar proto-
cols for using the controller responsible for inertial
measurements and platform control.
INFOTECH OULU Annual Report 2016 13
Figure 26. A semi-ready customizable low-cost quad-copter platform.
Intelligent battery modules
Intelligent battery modules (Figure 27) have been de-
veloped in collaboration with Probot Ltd. and the test
batch is being tested with our robot platforms. The
initial tests of the test batch have showed that the de-
signed battery electronics are functioning as was in-
tended. The battery module has an integrated heater for
winter operation and a charger module allowing energy
transfer from one module to another in any parallel
connected energy bus. With a developed charge control
module, the bus can also be potentially used to recover
energy from multiple power sources, such as solar
panels.
Figure 27. Assembled intelligent battery module for general use in modular robotics.
Control of Complex Wheeled Robots
Pseudo-omnidirectional robots with individually steer-
able wheels offer a good balance between payload,
robustness and mobility. However, the non-holonomic
nature of the regular wheels and the often redundantly
actuated structure of these robots make their control a
complex issue. This complexity of control is further
exacerbated when the wheels are not rigidly connected
to the robot body but are instead connected via actuated
chains which allow the wheels move relative to the
body. BISG has developed control algorithms for such
Articulated Wheeled Vehicles (AMW). The control
algorithms are mathematically simple closed-form
analytical functions and are thus computationally light
but are currently limited to planar cases. The computa-
tional load is only linearly dependant on the number of
wheels making the developed control algorithm suita-
ble for multi-wheel configurations and/or low-powered
embedded MCUs. The control algorithms synchronize
the rolling and steering velocities of complex planar
robots (plausible simulated example in Figure 28) with
freely located wheels forming fixed or variable foot-
prints. The rolling and steering velocities remain syn-
chronized even with very complex motions of the robot
(Figure 29). With the developed control algorithms, the
traversable path, robot’s heading on different points of
the path and the path velocity can be controlled sepa-
rately, thus offering great freedom on how to control
the robot on a given practical task. The control algo-
rithms do not in practice suffer from representation
singularities which are a common problem in wheeled
control. The control algorithms also compensate for the
proximity of mechanical singularities by adjusting the
robot’s path velocity according to the maximum capa-
bilities of its wheels’ steering and rolling actuators. In
fact the developed control algorithms are time optimal
in a sense that at any given moment the robot is either
traversing with maximum allowed path velocity or at
least one of its steering or rolling actuators is turning at
its maximum velocity (Figure 30), i.e. the robot
traverses the given path in the given way with the giv-
en velocity restrictions as fast as it possibly can.
Figure 28. Example of complex wheeled planar robot.
Figure 29. Simulation run of Figure 26’s robot traversing a given yellow path while keeping its front directed at all times to a point of interest (green larger dot). Note the smooth convergence of the robot (black line) and the target path.
INFOTECH OULU Annual Report 2016 14
Figure 30. (Top) wheel rolling speeds, (Middle) wheel
steering speeds and (Bottom) robot path velocity for
the first 30 seconds of a simulation run.
In summary, the developed control algorithms can be
used in a wide range of robot configurations and sce-
narios with low computational cost. The control algo-
rithms are currently limited to planar surfaces and can
cause sudden and large changes in velocity and the
control algorithms are being extended to work also
with uneven surfaces and limited motor torques.
In year 2016 the control algorithms have been en-
hanced to allow the wheels to have non-zero lateral and
longitudinal offsets, making the algorithm suitable for
practically any configuration of a wheeled planar robot.
In addition, a path tracking algorithm was developed.
The algorithm is very simple yet provides smooth and
robust path convergence in simulated environments
(Figure 31).
Figure 31. Smooth path convergence in cluttered envi-ronment.
Two ERDF project started; Labrobot and OuluZone+
projects. Labrobot focuses on Food industry, and
OuluZone+ for autonomous vehicles in harsh condi-
tions.
Labrobot-project focuses on boosting regional Food
industry by technology transfer demonstrations, build-
ing up test facility and network of stakeholders. By
surveying challenges in factories, combined with
knowledge of robotics, big data, machine vision and
biotechnologies; new kind of solutions are searched for
base of new business possibilities. This project is done
in cooperation with Center of Machine Vision and
signal processing, Biocenter Oulu and Luke. Project is
partly funded by City of Oulu, Yaskawa, Probot,
Maustaja, Antel, Kinnusen mylly, mekitech, and SR-
Intruments.
In the OuluZone+ project the focus is on automatic
road building machines and smaller mobile robots
(UGV and UAV) for supporting operation on the field.
In the project are studied how the capabilities of auton-
omous cars could be formally verfied, and tested from
perspectives of operting in all weather conditions and
all situations. Project is partly funded by City of Oulu,
OSEKK, Ouluzone Operointi Oy and industrial part-
ners.
The Evolutionary Active Materials
The Evolutionary Active Materials (EAM) project,
which is funded by the Academy of Finland, is a joint
effort between the Computer Science and Engineering
laboratory (CSE) and the Microelectronics and Materi-
als Physics laboratories. The aim of the EAM project is
to develop novel, evolutionary computation (EC) based
design methods for active and versatile materials and
structures. The first components are being developed
through a novel holistic design process utilizing con-
stantly increasing computation power, the development
of multi-physics simulators, and EC techniques, such
as genetic algorithms (GA).
During 2016, the height and the top diameter of Cym-
bal type piezoelectric actuator were optimized by ge-
INFOTECH OULU Annual Report 2016 15
netic algorithm and FEM modelling. From the opti-
mized results, maps of electromechanical capabilities
of different structures were generated. The blocking
force of the actuator was maximized for different val-
ues of displacement by optimizing the height of the cap
and the length flat region of the end cap profile. By
using values obtained from a genetic algorithm optimi-
zation process, a function was formulated for design
parameters. Using the function, a map of displacement,
the steel thickness and the height of the end cap the
optimized length of flat region was constructed (Figure
32). A similar map with the length of the flat region for
the optimized height of end cap was created. The re-
sults will be published at 2017.
Figure 32. The top diameter of the steel cap as a func-tion of steel thickness and displacement for Cymbal.
New type of actuator called Mikbal (Figure 33) was
invented, optimized with genetic algorithm and
realized. Mikbal was developed from Cymbal by
adding additional steel structures around the steel cap
to increase displacement and save the amount of used
piezoelectric material. The best displacement to
amount of used piezo material ratio was achieved with
25 mm piezo material diameter in the case of 40 mm
steel structures, and lower height and top diameter of
the cap increased the displacement. The results will be
published during 2017.
Figure 33. The von Mises stresses in Mikbal actuator under 500 V voltage.
Also optimization of the end cap structure of the Cym-
bal type energy harvester was done with genetic algo-
rithm and FEM modeling software Comsol Multiphys-
ics. The aim was to improve harvested power levels
from human walking (Figure 34). The power produced
by the energy harvester was increased by allowing the
algorithm to modify thickness in certain regions as
grooves in the end cap. By evolution of the structure,
power produced by the harvester increased by 38 %
compared to traditional linear type Cymbal harvester
which was also optimized by the algorithm. Increase in
power was obtained by change of mode in mechanics
of the harvester by grooves.
Figure 34. Cymbal type energy harvester in a shoe and an optimised profile for the harvester. In the profile piezoceramic disc is depicted in yellow and steel cap in grey. The grooves shown in the left side of the profile have been found by the genetic algorithm.
New grooved Cymbal energy harvester (Figure 35)
gave promising results in physical measurements pro-
ducing same power with less force than uniform shape.
The model was invented based on results given by
genetic algorithm optimization process with spline
shapes. Grooved Cymbal is easy to produce compared
to spline shape. Depth and place of grooves were opti-
mized by genetic algorithm. The parameters of the
algorithm itself were optimized also with GA, called
metaGA. Results of the metaGA will be published
during 2017.
Figure 35. Grooved cymbals.
INFOTECH OULU Annual Report 2016 16
Intelligent Systems Incorporating Bio-IT solutions
We have taken part in the Ruby/Diamond HILLA pro-
ject. This was based on collaboration between the poly-
techniques, VTT and BISG. This and a previous Tekes
project lead to establishment of four strategies that
should offer openings in the aims to establish minimal-
ly invasive of non-invasive wellness and health param-
eter monitoring technologies. Via a collaborative net-
work, we acquired novel nanomaterials offering ways
to couple electronics to biomonitoring behaviour of
live cells.
Developing novel real-time biosensors for glucose
monitoring. For developing “second generation biosen-
sors”, we have taken use of our skills to purify and
culture the skin derived progenitor cells that are re-
sponsible in skin renewal and regeneration. We ob-
tained for the project a Tekes strategic opening fund-
ing. With this support, we have advanced the work to
develop of a novel biosensor strategy (Figure 36).
ing cells are set to culture and a specific responsive component is engineered to target a tag to the 3´end of the coding sequence in the genome. Such a cell is then implanted to the donor to serve as a measure for a given physiological parameter. These serve to offer novel ways to biomonitor in real time physiologically relevant factors with and external electronic reader that is coupled wirelessly to the cloud to data analysis of multiple sensors at the end.
By now, we have been able to conduct the proof of
principle set up in the sensor construction. These indi-
cate that the skin is indeed responsive to the changes in
certain serum constituents. The data also indicated that
the cells with in the skin can also be engineered and be
converted genetically to serve as biosensors, thus to
report changes in the physiological parameters such as
glucose. We have screened in selected biological phe-
nomena with the proteomics and transcriptomics the
respective mediators in the glucose response in the
skin. We also generated experimental diabetic models
to identity diabetes associated and insulin independent
responders. The approach has turned a successful one.
First of all the skin appears responsive for physiologi-
cal levels of glucose. Due to this reason we also were
able to identify candidate factors whose genes and
encoded are currently being engineered to convert the
respective protein into an isoform whose activity can
read with an external electronic device.
We have also tested the capacity to culture of FACS
purified cells of the skin and if such cells can be trans-
planted with a fluorescent tagged vital sensor cells to
the donor so that the cells indeed become incorporated.
We assayed the stability of the sensor cells as trans-
plants. The data suggest that a syngeneic host suggest-
ing that the aimed biosensor strategy is feasible accepts
the skin progenitor graft.
In collaboration with VTT we have also developed the
electronic unit, a tunable spectral camera. This has the
capacity to measure the changes in the skin basal pro-
genitor cell integrated sensor. We have filed a patent of
these biotechnological avenues with VTT.
Developing an ex vivo supernatural personal mobile
biosensor device. To advance the goal to develop novel
wearable sensory devises we started to assemble first
via a HILLA funded project a micro fluidistic set up
that will be converted to a bio recognition tool. During
the research period, several micro fluidistic prints were
planned, made and tested. Out of these a configuration
was obtained that collected successfully, the skin asso-
ciated fluids as depicted by the presence of color dye in
the fluidistic chamber (Figure 37). A patent search of
the strategy has been conducted.
Figure 37. A micro filudistic print design is able to col-
lect the skin-associated fluids as depicted by the accu-mulation of a blue indicator dye in the chamber.
During 2016, we developed capacity to the micro fluid-
istic set up to monitor specific biomolecules present in
the skin fluids. This work lead to an opening via identi-
fication of novel types of biological nanomaterial’s
from the skin. These components are generated nor-
mally by the cells, they cargo wealth of physiologically
relevant biomolecules and they can cross the biological
barriers. Given the numerous amounts, small size of
the nano scale components, the opening has stimulated
a need to establish both bio and databanks. This is
INFOTECH OULU Annual Report 2016 17
currently being conducted with via deep sequencing
and proteomics to diagnose the samples that are de-
rived from cohorts.
During 2016, wealth of medical technical developmen-
tal lines with VTT and companies have been initiated
and also a new Tekes project grant filed. We obtained a
new Academy of Finland funded grant from the Bio
Future 2025 program to advance the nanobioelectronic
analysis strategies, one of the Infotech Oulu research
program targets.
To advance the biosensor openings we have started to
develop at the same time more complex diagnostic
platforms as the fluidic champers. To be able to read
the fluorescence that is revealed by specific antibodies
bound to the diagnostic components reagents against
these factors are being developed during 2017 with our
collaborator. Our partners in the HILLA project were
able to develop a mobile phone based micro fluidistic
reader capacity. Together with the developed biochips,
such printable materials are likely to set the stage for
the point of care diagnostics in the field of personalized
medicine during 2017.
Screening of electromagnetic and opto/chemo/electro
genetic responses in organs generated from stem cells. The genetic engineering offers opportunities to devel-
oped technologies where the cellular in or output sig-
nals can also be regulated by certain wavelengths in the
electromagnetic spectrum. Alternatively the cellular
actions can be genetically constructed so that a signal
will be transmitted to a biosensor that will convert it to
a form readable by an electric device. To advance these
tasks we have initiated with private funding screens
that aim to identify cellular channels that are regulated
by specific spectral frequencies such as the RF ones.
Such diagnostics use a paradigm shift where the cellu-
lar responses to given stimuli will be screened primari-
ly via vital “biosensors” with live cellular tags. Thus
the approach in the bioelectronics analytics have be-
come possible via the crisp Cas9 genome-editing tech-
nologies where libraries of gene edited diagnostic cells
can be generated.
During 2016, we developed novel tissue engineering
technologies that do enable introduction of specific
gene expression constructs to individual cells of the
model organ such as the mammalian kidney. Here the
organ primordia is dissociated to single cells, the genet-
ic construct encoding the protein of interest such as the
opto, chemo or radiogenetic responsive component is
transduced to such a cell with a reporter for the read
out screens. There after the organ is let to self-assemble
and placed for a long-term culture (Figure 38).
Figure. 38. An organ primordia can be dissociated to single cells, the constitute cells transduced with a ge-netic construct to acquire opto-, chemo- and radio ge-netic guidance capacity to the morphogenetic cells ex vivo.
With the developed model systems we have taken use
of the image analysis technologies to visualize how the
morphogenetically active cells behave in three dimen-
sion in the 4D conditions that offer a whole organ pri-
mordia to be cultured ex vivo. To achieve this we
applied defined pressure to the assembled organ pri-
mordia in ex vivo setting depicted in Figure 39.
Figure 39. The 3D kidney organ primordia that is rela-tively thick being composed of multiple cell layers de-velops also under a mild pressurize in ex vivo. Here the mechanical pressure converts the 3D development more towards a 2D configuration. The developed setup will offer ways to identify pressure sensors in the cells and also to develop novel organ pressure monitoring tools. The power of this novel “organoid” culture set up is that it enables for the first time is complex organs image analysis and follow up of the behavior of the individual constituent cell while the complex 3D anatom-ical structure of the organ become laid down. It is im-portant that the quality of the data good enough to offer segmentation and “computer vision” analysis. With such “Fixed Z-Dimension” (FZD) culture we are in a process of illustrating the fine details how biological shapes, namely the organ structure in 3D becomes constructed from the cellular building blocks. These data serves also as the digital 3D landscape for developing 3D bio printing when advancing a European Union FET
INFOTECH OULU Annual Report 2016 18
FLAGSHIP representing a regenerative medicine and nanotechnology initiative.
We found that under a defined pressure the organ flat-
tens towards two dimension (2D) but yet morphogene-
sis progressed (Figure 40). This novel set up has made
it possible follow the fate of individual cells is the cells
are constricting a detailed manner while the natural
form.
Figure 40. Operetta confocal workstation coupled to a robotic set up and an incubator was assembled. A) A
holder for plates and transported by the robotic arm (B) and the cells with in will be transported to an incubator (C). The whole set up is inside a hood (D) and the ro-botic arm transports the plates to the Operetta confocal semi-high throughout microscope fluorescent reader. The data is analyzed by wealth of machine vision/image analysis programs present with in the assembled bio robotic set up. The bio robotic core facility will be used to screen with a library of live indicators cellular re-sponse to specific frequencies in the electromagnetic spectra.
To target the detailed dynamics by which the form is
assembled in a model organ we took use of the genet-
ically engineered Wnt4CreGFP knock in mouse model.
This was crossed to the floxed Rosa26 Yellow Fluores-
cent Protein (YFP) transgenic mice. In this genetic
crossing the stem cells that generate whole of the neph-
ron will become labeled with the YFP.
With the fixed Z-dimension culture we have captured
3D movies from the developing kidney with the confo-
cal microscope in a time-lapse setting. We are in a
process of analyzing the detailed cell behavior via the
machine learning/computer based image analysis with
Prof. Janne Heikkilä. With Dr. Jari Juuti we aim to
construct a specific device that allows detailed measure
of the pressure forced encountered by the tissue under-
going morphogenesis. These novel capabilities now
allow analysis in great detail the mode by which the
spatial and temporal organization of the cells go on to
construct natural form that is open at present in any
developing organ system. We will use models to identi-
fy the pressure sensors from the cells with the OMICS
technologies.
Developing high throughput robotic aided platforms to
screen complex cellular responses to magnetic/electric
fields via signaling pathway reporters. To advance the
strategies to measure in a high through put manner the
cellular responses to stimuli we have assembled a bio
robotic workstation. Here an Operetta confocal micro-
scope was obtained and this was coupled to a hood that
contains an automated plate-cargo arm, a rack for the
plates with a bar code reader and incubator for long
term exposure of the cells to compounds such as drugs
or specific electromagnetic spectral radiation (Figure
40). The Operetta confocal microscope has machine
learning/image analysis capacity for wealth of meas-
urements to be conducted from the cells.
To take use of the set up a yeast cell library was ob-
tained and three replica clones from it was generated
and stored for later use. The library is composed of cell
where each of the 3´end of each of the yeast gene was
targeted by a green fluorescent protein (GFP) tag. The
next goal is to obtain capacity to start to use the set up
to define the oscillating properties of the cellular genes
and to use it as live measures for screening responses
to stimuli such as those mediated by the opsins for the
visible light frequencies. Such genome wide screens
vital bio indicator based scan be expected to lead to
identification of novel biosensor pathways for certain
spectral frequencies. When the strategy will be subject-
ed to patient derived gene edited human induced plu-
ripotent (iPS) cells and those whose fate has been engi-
neered to defined directions this technology should
offer avenues for the era personalized medicine diag-
nostic developmental aims.
Intelligent Systems with cohort data sets: Cohort data
set is a special data set from the medical domain, which
has not been studied with a machine learning approach
before. The data set, Northern Finland Birth Cohort
1966 (NFBC 1966), is a unique data set with over 14
000 original variables in various yet heterogeneous
formats (numerical, ordinal, categorical, images, text
etc.) from a population of over 12 000 mothers and
their children without any complete data points. The
amount of variables rises to millions if genetics and
epigenetics are considered (p >> n).
There are two extremely important aspects of modeling
this type of data: confidence of the predictions made
with the model and model interpretability. Steps to-
wards instance level confidence estimates have been
made in our previous work (see above) and we will
continue to pursue this goal, along with keeping model
interpretability in focus also, when we start digging
into this fascinating data set. Our goal is to use a ma-
chine learning approach to make novel discoveries
from the data that traditional data analysis approach
has not yet uncovered.
Elders are an increasingly large fraction of the popula-
tion in developed countries. From one hand people
INFOTECH OULU Annual Report 2016 19
expect an independent life also in presence of more or
less important diseases. On the other hand the treat-
ments to care those diseases, often together with co-
morbidities, imply larger costs. To respond to both
these goals, the disease progress should be kept as low
as possible (see Figure 41), which means early disease
detection, deinstitutionalisation and personalised medi-
cine, striving to allow a better quality of life, a more
cost-efficient healthcare system and a more inclusive
access to healthcare both in developing countries and
in remote areas in developed countries.
Novel Bio-ICT technologies are needed to achieve
these targets and BISG is active in this area in many
fronts summarised below.
By tracking health status of large groups and including
in the analysis a wealth of metrics and parameters,
large amounts of data are generated. On the other hand,
by downscaling biology-based technologies down to
the nanoscale including sensing biological parameters
directly from living cells, potential security threats are
correspondingly moving into human bodies, but prom-
ising tools are offered for personalised medicine and
treatments, including tight biological interaction, pros-
theses and their control (Celentano and Röning 2015).
BISG is strong in all these areas (data analysis, security
and robotics) and it is therefore pushing itself among
the world leaders in this growingly important area.
0
Healthy state
1
Degeneration starts,
no noticeable impact
on everyday life
2
Mild impacts on
everyday life
3
Disturbs appear
evident or important
4
Severe degeneration
D
Death (complications,
accidents, suicide)
C
Daily care needed
B
Assistance needed
A
Normal life (almost)
preserved
D
Medical
Doctor
H
Hospital
Active
H
osp
ita
lise
d
Cost-
eff
ective
E
xp
ensiv
e
Figure 41 Progress of a disease (left), outcome (right) and access to healthcare (Celentano and Röning 2015).
Towards a Holistic Self-awareness in Humans and AI
Artificial entities like robots and unmanned or autono-
mous vehicles are more and more present in the human
environment. Social interaction among all the players
in such a heterogeneous scenario (Figure 42) calls for a
number of research issues to be addressed and its study
offers interesting potentialities.
Figure 42. Interwork among heterogeneous agents and within them. From Celentano & Röning (2016b).
Self agency. Self-awareness in humans plays a role in a
number of brain functions and disturbances. On the
other hand, self-awareness improves the efficiency in
robotic systems (Celentano and Röning 2016a).
Awareness of the self is achieved through analysis of
observations, or measurements, of various entities
involved. This interwork in a heterogeneous multi-
agent system (Figure 42) may occur with different
topologies: sensing the actuation of other entities, as in
Figure 43a; acquiring information shared by others, as
in Figure 43b; exploiting different functions for
self/nonself discrimination. In short, through perrcep-
tion, action, and sharing information (Celentano &
Röning 2016a).
Figure 43. a) Left: An entity (bottom) sensing the actua-tion of two entities (top). b) Right: Entities (bottom) acquiring instructions shared by another entity (top). From Celentano & Röning (2016a).
Embodied agent. As psychologist James Gibson ob-
served, there is an interdependency of perception and
action (“perceive to act, act to perceive”). We study the
social intelligent entity as embodied (Celentano &
Röning 2016b), where are brought to evidence not only
the interaction among entities but also the interwork
within them (cf. Figure 42 and Figure 44).
Considering that an instantiation of the agent may
possess only part of its functions, the same generic
model can be used at different scales, applied to enti-
ties, at brain, body and world domains (the latter possi-
bly including other entities or agents), benefitting
modularity and scalability (Celentano & Röning
2016b).
INFOTECH OULU Annual Report 2016 20
Figure 44 The embodied agent and its environment. From Celentano & Röning (2016b).
Structured information representation and instruction
logic. Interwork among modular agents include as seen
perception and action but also the exchange of infor-
mation or commands, both referred to as instructions as
in Celentano & Röning (2016a). These communica-
tions may be subject to noise, as it is the case in an
operating room or in air traffic control (Figure 45, top).
Whereas machines are subject to environmental noise
different steps of the information communication pro-
cess (Figure 45, bottom).
decodingencoding
interpretation mapping transfer mapping interpretation
cognitive noise
environmental noise
Figure 45. Top: Interaction among heterogeneous agents in a noisy environment. Bottom: Information communication between remote source and destination entities in noisy conditions. From Celentano & Röning (2016c).
For reliable information exchange among heterogene-
ous agents is needed a formal representation of the
exchanged instructions, usable by both humans and
machines (Figure 46).
Unambiguous
Language
Formal
Representation
Human
Robot
ImplementationMappingInterpretation
Atomi
ROSMSDL
higher level lower level
Human
Robot
Elements
Definitions
BML
Figure 46 Interaction through specified processes (lan-guages and representation). From Celentano & Röning (2016c).
Using the instruction logic in Celentano & Röning
(2016c), the example situation in which mobile m0 at
x0 orders mobile m3 to be in x1 at t1 to search a book b
and bring it immediately to m0 can be represented by
«report_to,m3,m0,x0,t,1»
«move_to,m3,-,x1,t<t1,1»
«search,m3,b,-,t,1»
«bring,m3,b,x0,0,1».
Exploitation of Results
BISG continued co-operation with the SpAtial, Motor