DESIGN AND DEVELOPMENT OF PRODUCT -SERVICE SYSTEM …

UNIVERSITY OF PATRAS

DEPARTMENT OF MECHANICAL ENGINEERING &

AERONAUTICS

DIVISION OF DESIGN AND MANUFACTURING

LABORATORY FOR MANUFACTURING SYSTEMS AND

AUTOMATION

DIPLOMA THESIS

DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE

ANGELOPOULOS IOANNIS

5931

Supervisor PROFESSOR MOURTZIS DIMITRIS

Diploma thesis submitted at Department of Mechanical Engineering & Aeronautics at University

of Patras

PATRAS, July 2021

DESIGN AND DEVELOPMENT OF PRODUCT-SERVICE SYSTEM

BASED ON AUGMENTED REALITY FOR MACHINE MAINTENANCE ANGELOPOULOS Ioannis

Department of Mechanical Engineering & Aeronautics ii

Division of Design & Manufacturing

University of Patras, Department of Mechanical Engineering & Aeronautics ANGELOPOULOS IOANNIS © 2018 – All rights reserved

UNIVERSITY OF PATRAS

DEPARTMENT OF MECHANICAL ENGINEERING &

AERONAUTICS

DIVISION OF DESIGN & MANUFACTURING

LABORATORY FOR MANUFACTURING SYSTEMS &

AUTOMATION



Department of Mechanical Engineering & Aeronautics iii


Η παρούσα διπλωματική εργασία παρουσιάστηκε

από τον

ANGELOPOULOS IOANNIS

5931

την 15η Ιουλίου 2021

Η έγκριση της διπλωματικής εργασίας δεν υποδηλοί την αποδοχή των γνωμών του συγγραφέα.

Κατά τη συγγραφή τηρήθηκαν οι αρχές της ακαδημαϊκής δεοντολογίας.

ΣΧΕΔΙΑΣΜΟΣ & ΑΝΑΠΤΥΞΗ ΣΥΣΤΗΜΑΤΟΣ ΠΡΟΪΟΝΤΟΣ-ΥΠΗΡΕΣΙΑΣ ΒΑΣΙΣΜΕΝΟ ΣΤΗΝ ΕΠΑΥΞΗΜΕΝΗ

ΠΡΑΓΜΑΤΙΚΟΤΗΤΑ ΓΙΑ ΤΗΝ ΣΥΝΤΗΡΗΣΗ ΒΙΟΜΗΧΑΝΙΚΩΝ ΚΑΛΟΥΠΙΩΝ



Department of Mechanical Engineering & Aeronautics iv


ΠΕΡΙΛΗΨΗ

Η συντήρηση είναι ένα απαιτητικό σύνολο εργασιών που εκτελούνται σε βιομηχανικό εξοπλισμό.

Οι τεχνικοί υποχρεούνται να φέρουν έντυπα εγχειρίδια, προκειμένου να εκτελέσουν όλες τις

απαραίτητες διορθωτικές ενέργειες. Επιπλέον, υπάρχουν περιπτώσεις στις οποίες ο τεχνικός

πρέπει να επικοινωνήσει με έναν μηχανικό ή ακόμη και να καλέσει εξειδικευμένο προσωπικό για

την εκτέλεση πολύπλοκων εργασιών συντήρησης. Οι πρόσφατες εξελίξεις στις τεχνολογίες

κινητής τηλεφωνίας και στη Μικτή Πραγματικότητα (Mixed Reality), επιτρέπουν στους

μηχανικούς σε όλο τον κόσμο να σχεδιάζουν και να υλοποιούν χρήσιμες εφαρμογές για την

παροχή όλων των απαιτούμενων οδηγιών και την υποστήριξη επικοινωνίας, με τη χρήση

οποιασδήποτε έξυπνης συσκευής, βασισμένη στην προβολή τρισδιάστατου ψηφιακού

περιεχομένου στο πραγματικό περιβάλλον. Η Επαυξημένη Πραγματικότητα (Augmented Reality)

είναι μια πρωτοποριακή ψηφιακή τεχνολογία που διευκολύνει τους μηχανικούς να μειώσουν το

γνωστικό φορτίο των τεχνικών. Επιπλέον, με τη χρήση του AR μειώνεται επίσης το κατώτατο

όριο για ελάχιστες ειδικές δεξιότητες ή / και εκπαίδευση. Λαμβάνοντας υπόψη τις εξελίξεις στις

Τεχνολογίες Πληροφορίας και Επικοινωνιών (ΤΠΕ), οι ειδικοί μηχανικοί είναι σε θέση να

δημιουργήσουν τους απαιτούμενους διαύλους επικοινωνίας με τους τεχνικούς του καταστήματος,

σε μια προσπάθεια να παρέχουν εξ αποστάσεως καθοδήγηση. Ως εκ τούτου, ο συνολικός χρόνος

καθώς και το κόστος συντήρησης του βιομηχανικού εξοπλισμού μπορούν να μειωθούν περαιτέρω

καθώς και να απλοποιηθεί η διαδικασία της συντήρησης. Ο σκοπός αυτής της ερευνητικής

εργασίας είναι ο σχεδιασμός και η ανάπτυξη ενός πλαισίου απομακρυσμένης και έξυπνης

συντήρησης βασισμένης σε AR για εξοπλισμό Engineered to Order (ETO). Το framework

υλοποιήθηκε ως μια κινητή εφαρμογή AR, η οποία παρέχεται στους πελάτες από τον

κατασκευαστή σαν υπηρεσία, δημιουργώντας έτσι ένα Σύστημα Υπηρεσίας Προϊόντων (PSS).

Λέξεις κλειδιά

Επαυξημένη Πραγματικότητα, Συντήρηση, Καλούπι, Βιομηχανία, Συστήματα προϊόντων -

υπηρεσιών



Department of Mechanical Engineering & Aeronautics v


ABSTRACT

Maintenance is a demanding set of tasks performed on industrial equipment. Technicians are

obliged to carry printed manuals, in order to perform all the necessary corrective actions.

Moreover, there are cases where the technician needs to communicate with an engineer or even

call specialized personnel in order to accomplish complex maintenance tasks. Technological

advances in mobile technologies and mixed reality, have enabled engineers over the world to

produce useful applications for providing all the needed instructions and communication with the

use of any smart device, just by registering 3D content on the real environment. Augmented Reality

is a cutting-edge digital technology facilitating engineers in reducing the cognitive load of

technicians. Further to that, with the utilization of AR the threshold for minimum special skills

and/or training is also reduced. Taking into consideration the advances in Information and

Communication Technologies (ICT), expert engineers are capable of establishing sufficient

communication channels with the shopfloor technicians, in an attempt to provide to provide remote

guidance. As a result, the total maintenance time of industrial equipment can be further reduced

and simplified. The purpose of this research work is the design and development of an AR based

remote and smart maintenance framework for Engineered to Order (ETO) equipment. The

framework can be realized as a mobile AR application, which is provided to the clients from the

OEM as a Service, thus creating a Product Service System (PSS).

Keywords:

Augmented reality, Product-Service System (PSS), maintenance, Engineered-to-Order (ETO)



Department of Mechanical Engineering & Aeronautics vi


LIST OF TABLES

Table 1 Table for AR IDEs comparison ....................................................................................... 11

Table 2 Types of PSS offered and their percentages .................................................................... 17

Table 3 List of common server protection techniques .................................................................. 24

Table 4 Involved actors and their rights ....................................................................................... 30

Table 5 List of additional functionalities ...................................................................................... 44

Table 6 List of common faults for injection molds....................................................................... 57

Table 7 KPIs used for the test case ............................................................................................... 65

Table 8 Mold maintenance process steps...................................................................................... 67



Department of Mechanical Engineering & Aeronautics vii


LIST OF FIGURES

Figure 1 Industrial revolutions combined with Machine Tool and Operator Evolutions ............... 4

Figure 2 Nine (9) pillar technologies used in Industry 4.0 ............................................................. 5

Figure 3 Virtuality-Reality continuum and its variations ............................................................... 6

Figure 4 Oculus rift, a modern HMD [29] ...................................................................................... 8

Figure 5 Vuzix Augmented Reality glasses [31] ............................................................................ 9

Figure 6 Tablet used for AR application [32] ................................................................................. 9

Figure 7 AR tracking workflow .................................................................................................... 13

Figure 8 i) Typical example of Image Target used in AR applications; ii) yellow points indicate

the recognized point array for the position and pose estimation .................................................. 14

Figure 9 PSS categorization .......................................................................................................... 16

Figure 10 Percentage of PSS adoption from global firms ............................................................ 18

Figure 11 PSS system flowchart ................................................................................................... 26

Figure 12 First step in the sequence of maintenance .................................................................... 27

Figure 13 Second step in the sequence of maintenance ................................................................ 28

Figure 14 Third step in the sequence of maintenance................................................................... 28

Figure 15 Final step in the sequence of maintenance ................................................................... 29

Figure 16 System architecture - the numbers present the sequence of interactions ..................... 31

Figure 18 UML Class Diagram supporting the data modelling of the proposed framework ....... 33

Figure 19 Introductory Graphical User Interface (GUI), MMT® ................................................. 34

Figure 20 GUI for the registration of new users in the platform, MMT® ..................................... 35

Figure 21 GUI for the login of registered users, MMT® .............................................................. 36

Figure 22 Client's start page GUI, MMT® .................................................................................... 36

Figure 23 PSS customizer interface, MMT® ................................................................................ 37

Figure 24 Date picker interface, MMT® ....................................................................................... 37

Figure 25 Customer information sheet, MMT® ............................................................................ 38

Figure 26 Engineer's start page, MMT® ....................................................................................... 39

Figure 27 Download file interface, MMT®................................................................................... 39

Figure 28 BOP creation GUI, MMT® ........................................................................................... 40



Department of Mechanical Engineering & Aeronautics viii


Figure 29 XML file structure for the generation of BOP, and generation of AR instructions ..... 40

Figure 30 Example XML file based on the structure presented previously ................................. 41

Figure 31 Mold-list GUI ............................................................................................................... 42

Figure 32 Shop Inspection GUI, MMT®....................................................................................... 43

Figure 33 Technician's Start Page, MMT® ................................................................................... 43

Figure 34 3D presentation interface, MMT® ................................................................................ 44

Figure 35 Explode functionality in 3D presentation, MMT® ....................................................... 45

Figure 36 Annotation functionality in 3D presentation, MMT®................................................... 46

Figure 37 Annotation on 3D component, MMT® ......................................................................... 46

Figure 38 GUI for Report creation & submission to the Cloud platform, MMT® ....................... 47

Figure 39 Reports, saved on the Cloud FTP server ...................................................................... 48

Figure 40 AR visualization of the exploded assembly ................................................................. 49

Figure 41 AR mold projection, health monitoring........................................................................ 50

Figure 42 Cloud database components ......................................................................................... 51

Figure 43 Typical paradigm of PHP script ................................................................................... 52

Figure 44 Administrator interface, MMT® ................................................................................... 53

Figure 45 Administrative tools interface, MMT® ......................................................................... 53

Figure 46 The mold manufacturing company ............................................................................... 55

Figure 47 Injection mold cut-out .................................................................................................. 56

Figure 48 Injection mold cycle with the corresponding times ...................................................... 57

Figure 49 Injection mold common failures ................................................................................... 58

Figure 50 Application of the developed framework ..................................................................... 62

Figure 51 Workflow of the validation scenario of first prototype of the tool. .............................. 68

Figure 52 Tool evaluation results ................................................................................................. 69



Department of Mechanical Engineering & Aeronautics ix


ABBREVIATIONS

3D 3 (Three) Dimensional

AEC Architecture, Engineering & Construction

AR Augmented Reality

BOM Bill of Materials

BOP Bill of Processes

CAD Computer Aided Design

CBM Condition Based Maintenance

CNC Computer Numerical Control

ETO Engineered to Order

FTP File Transfer Protocol

GUI Graphical User Interface

HMD Head Mounted Display

ICT Information and Communication Technology

ID Identification

IDE Integrated Development Environment

IM Injection Mold

IPS2 Industrial Product Service Systems

LAN Local Area Network

MR Mixed Reality

MRI Magnetic Resonance Imaging

OEM Original Equipment Manufacturer

OHMD Optical Head Mounted Display

PBH Pay By the Hour

PSS Product-Service System

QR code Quick Response code

R&D Research and Development

SDK Software Development Kit

VE Virtual Environments

VR Virtual Reality

XML Extensible Markup Language



Department of Mechanical Engineering & Aeronautics x




Department of Mechanical Engineering & Aeronautics xi


CONTENTS

ΠΕΡΙΛΗΨΗ................................................................................................................................. IV

ABSTRACT .................................................................................................................................. V

LIST OF TABLES ...................................................................................................................... VI

LIST OF FIGURES .................................................................................................................. VII

ABBREVIATIONS ..................................................................................................................... IX

CONTENTS................................................................................................................................. XI

CHAPTER 1. INTRODUCTION ........................................................................................... 1

CHAPTER 2. LITERATURE REVIEW ............................................................................... 3

2.1 4TH INDUSTRIAL REVOLUTION ................................................................. 3

2.2 VIRTUAL ENVIRONMENTS (VE).................................................................. 5

2.3 FIELDS OF APPLICATION ............................................................................. 6

2.4 HARDWARE USED IN AUGMENTED, MIXED, AND VIRTUAL

REALITY .............................................................................................................................. 8

2.5 SOFTWARE USED IN AR ................................................................................. 9

2.6 USER TRACKING MODEL ............................................................................ 12

2.7 PRODUCT-SERVICE SYSTEMS (PSS) ........................................................ 15

2.8 DATA PROTECTION AND DATA ENCRYPTION .................................... 23

CHAPTER 3. PROPOSED METHODOLOGY AND SYSTEM ARCHITECTURE25

3.1 PSS MODEL ...................................................................................................... 25

3.2 SYSTEM ARCHITECTURE ........................................................................... 29

3.3 DATA FORMALIZATION .............................................................................. 32

3.4 APPLICATION FUNCTIONALITIES ........................................................... 34

CHAPTER 4. INDUSTRIAL TEST CASE ......................................................................... 54

4.1 INJECTION MOLD OVERVIEW .................................................................. 56

4.2 CASE STUDY DESCRIPTION ....................................................................... 60

CHAPTER 5. RESULTS – KEY PERFORMANCE INDICATORS ............................... 63

5.1 KEY PERFORMANCE INDICATORS.......................................................... 63

5.2 TEST CASE CURRENT SITUATION DESCRIPTION .............................. 63



Department of Mechanical Engineering & Aeronautics xii


5.3 EXPECTATIONS OF ADOPTING THE PROPOSED METHODOLOGY

64

5.4 VALIDATION METHODOLOGY ................................................................. 66

5.5 RESULTS ........................................................................................................... 68

CHAPTER 6. CONCLUDING REMARKS AND OUTLOOK ......................................... 70

ACKNOWLEDGEMENT .......................................................................................................... 72

PUBLICATIONS RELEVANT TO THE THESIS ................................................................. 72

REFERENCES ............................................................................................................................ 74



Department of Mechanical Engineering & Aeronautics 1


CHAPTER 1. INTRODUCTION

Manufacturing is a system in which product design is the initial stage and the delivery of finished

products to the market is the final output. Manufacturing can also be described as a whole of

decisions through the processes. Generally, there are four (4) classes of manufacturing attributes

to be considered during decision making: cost, time, quality, and flexibility. Bearing also in mind

that manufacturing has moved from the mass production era, to the mass customization era,

engineers around the world have to constantly innovate, in order to keep the competitive edge in

their field of operation [1]. Augmented Reality is a powerful tool, that can help minimizing

production costs and times while ensuring the quality of the products [2].

Augmented Reality is an emerging technology which allows the user to implement computer-

generated content on the physical, real-world environment using suitable equipment. The content

displayed can be of any type: images, 3D models, animations, videos, and sound. The key feature

of AR is that the user is not obliged to fully immerse on a virtual environment. Although AR is

considered to be on a premature level, is a quite old technology. The first idea of a virtual

environment was made by L. Frank Baum (character maker). The first application of such

technology was developed by Morton Heilig with a simulator called Sensorama [3]. However, MR

is subject to constant development as users’ needs constantly increase by the time. At this point, it

is commonly accepted that equipment used in such applications is at its second generation.

In the past two decades, Augmented Reality (AR) has received a growing amount of attention by

researchers in the manufacturing technology community, because AR can be applied to address a

wide range of problems throughout the assembly phase in the lifecycle of a product, namely

planning, design, ergonomics assessment, operation guidance and training [4].

The major question to all the MR developers is what is the purpose of building and using such

systems? The answer is simple. MR systems in AEC aim to help users overcome difficulties while

performing certain tasks (e.g. help on-sight worker identifying a system failure with the help of a

master engineer in real-time), reduce the required cognitive level (e.g. help workers perform





maintenance tasks on complex systems without the need of special knowledge or even carrying

printed instruction manuals).

AR as mentioned is an immature technology although it exists (even as an idea) for many years.

The last two decades has met great advance and still goes on with the technological advances

contributing to its advance. AR applications cover a wide variety of user needs. The most common

applications are: (a) industrial / manufacturing, (b) military, (c) entertainment and (d) education.

The purpose of this thesis is to address some issues related to an industrial use case. More

specifically, AR is nice-to-have tool in maintenance industry. Maintenance personnel often need

to dismantle complex mechanisms and/or in some cases there are hidden components playing a

dramatic role in the maintenance process. The traditional solution is to refer to paper-printed

manuals, communicated via telephone with a master engineer or in the worst case, report the failure

to the manufacturer, and then a highly trained worker is sent to the sight. The above situations

present a whole of disadvantages, for both clients and manufacturers. First of all, maintenance

costs can rise very high, a matter which is amplified by the risk included (mainly during the fault

diagnosis and during the repair tasks). Secondly it is time consuming, as inexperienced workers

try to sort out the fault and explain it to the master engineer, providing limited info. Especially in

the case of sending a highly trained worker from the manufacturer, machinery down-times rise in

catastrophic rate for the production line [5].

The framework proposed in this thesis can be realized as an AR application prototype that aiming

to enable workers perform maintenance tasks without the need of carrying printed manual

instructions. Concretely, for the utilization of the AR application, technicians carry a smart device,

such as a smart phone or tablet, which contains all the steps and precautions in vivid 3D/AR

presentations accompanied by text instructions. Moreover, if communicating with a master

engineer is inevitable, the features of such a smart device (camera, phone calls, connection to

worldwide web, etc.), help worker share the proper amount of information with the engineer, who

in the best-case scenario will be able to produce real-time instructions via video streaming.





CHAPTER 2. LITERATURE REVIEW

Modern industrial equipment and machinery are composed of complex systems and delicate

components. Those systems and components often break down from excessive use or other

unpredictable factors. Thus, the need for a cluster of maintenance tasks is mandatory. Moreover,

the aim on modern maintenance scheduling is reducing or even eliminating, if possible,

downtimes. A solution to this problem lies on Condition Based Maintenance (CBM). CBM is a

maintenance strategy that monitors the actual condition of the asset to decide what maintenance

needs to be done. CBM dictates that maintenance should only be performed when certain

indicators show signs of decreasing performance or upcoming failure. Checking a machine for

these indicators may include non-invasive measurements, visual inspection, performance data and

scheduled tests. Condition data can then be gathered at certain intervals, or continuously (as is

done when a machine has internal sensors). The goal of CBM is to spot upcoming equipment

failure so maintenance can be proactively scheduled when it is needed. Asset conditions need to

trigger maintenance within a long enough time period before failure, so work can be finished

before the asset fails or performance falls below the optimal level. An advanced condition

monitoring solution can be necessary to keep product quality high and scrap low in the injection

molding process.

2.1 4TH INDUSTRIAL REVOLUTION

The last three centuries, the Industrial landscape has undergone a series of changes which have

improved the efficiency of manufacturing processes and manufacturing systems, have enabled

manufacturers to integrate customers in the development and production of new products,

companies have improved their business strategies and finally, through the collaboration of

different parties the creation of advanced manufacturing networks has become reality. The above-





mentioned are enabled/facilitated by the integration of a plethora of technological advances, which

can be clustered and summarized in the so-called Industrial Revolutions. Up to today, a total of

four Industrial Revolutions have occurred. In Figure 1, the Industrial Revolutions are illustrated

based on their chronological appearance [6-9]. What is interesting in this figure, is the evolution

of manufacturing personnel as a result of the technological advances, which are indicated as

Operator 1.0-5.0. Therefore, it becomes evident, that maturity, knowledge, and experience of

technicians has been upscaled, enabling humans collaborate with machines and by extension

improved productivity rates, achieve better quality, minimize errors, etc.. The current situation in

the manufacturing domain can be described as the fourth industrial revolution, better known as

Industry 4.0. Furthermore, this term refers to the wave of change the industrial world is facing, the

last five years. It is originated by Germany has proven to be at the top competitors in the fields of

R&D, production of manufacturing systems and management of industrial processes [10].

Figure 1 Industrial revolutions combined with Machine Tool and Operator Evolutions

Following the recent developments regarding new digital industrial technologies, under the

framework of Industry 4.0, the collection and data analysis across machines is enabled / facilitated,

thus allowing for the complete digitalization of modern manufacturing systems. Consequently, the

production of higher quality at reduced costs in faster, more flexible, and more efficient processes.

As a result it is expected that the above-mentioned advances will boost productivity rates, improve

the economic efficiency of manufacturing processes, promote industrial growth, as well as they





will also change the key characteristics of human workforce. By extension the competitiveness of

manufacturing companies can be improved. Taking into account recently published research

works, it can be concluded that cutting-edge digital technologies are increasingly being integrated

in the manufacturing domain, providing a strong indication that the manufacturing domain is

transformed [11-16]. Moreover, it will lead to higher efficiencies and alter traditional supplier,

manufacturer, and customer production relationships, as well as human and machine relationships.

The building blocks of Industry 4.0 can be summarized into the nine technology trends presented

in Figure 2 [17].

Figure 2 Nine (9) pillar technologies used in Industry 4.0

2.2 VIRTUAL ENVIRONMENTS (VE)

Virtual environments are one of the latest technological advances. Combined with the increasing

commercial availability of such equipment, virtuality has become the latest trend. However, these

technologies can be used in modern industry in order to facilitate the automation of processes and

speed up production. For this purpose, a number of different implementations can be integrated in

modern industrial business models, in order to provide more developed product services to





customers, which in turn create added value for the manufacturers, thus increasing their revenue

and extend product lifecycle. The most common variations of VE found in modern industry, are

virtual reality, augmented reality, and holograph based implementations. In this thesis, the

proposed application is based on the implementation of Augmented Reality. Augmented Reality

is the technology, which enables the projection of 3D virtual objects onto the physical

environment. In contrast with virtual reality, the user of AR is not fully immersed in a virtual

environment, he/she maintains his/her contact with the real world, which is where the virtual

objects are superimposed [18].

Figure 3 Virtuality-Reality continuum and its variations

Augmented Reality is used extensively in maintenance applications. Not only that, but AR

applications can also be found in projects where visualization of warehouse or production line

status is needed [10,19,20]. Following the same concept augmented reality can also be used in

conjunction with data obtained from sensors attached to the equipment in order to monitor

condition. Such functionality can be proved to be useful in predictive maintenance.

2.3 FIELDS OF APPLICATION

Despite being around for a short period of time, augmented reality has invaded many scientific

fields. In some fields, it is used as a mean of impressing and promoting. Some other fields use it

for prototype visualization in a cheaper, portable, intriguing, and flexible way while others use it

as a tool for continuous information flow to an operator for performing complex tasks.





Military: Military research groups, have developed AR applications for troop support on the field

or for remote maintenance. It is fact that military conducts many scientific researches, while most

of them are of great interest, due to the higher quality militaristic needs and due to great

investments. Military seeks, new technologies that could be implemented on weapon systems, in

order to increase their efficiency. In 1993, Rockwell International created video map overlays of

satellite and orbital debris tracks to aid in space observations at Air Force Maui Optical System.

Such systems are also used in telescopes, in order to visualize satellite and space debris trajectories

[21].

Medical: The use of new technologies in medical applications is limited, mainly due to the fact

that both doctors and patients are reluctant to try innovative methods. Despite the rapid

development, especially in the usage of robot for higher accuracy in surgeries, lack of knowledge

and increased cost has diminished its growth. Augmented Reality can be used in this field as a tool

by the doctor for data flow and disease diagnosis with relatively low cost. More specifically, a

wide variety of applications has been developed so as to allow the doctor-operator to have a

continuous image derive from an ultrasound, a Computed Tomography, or a Magnetic resonance

imaging (MRI) scan [22].

AEC: For at least a decade, several AEC researchers have promoted AR technology adoption

because of its potential as a visualization aid. Roberts et al. [23] developed an AR system to render

images of underground structures onto a view of the site. Hammad et al. [24] and Thomas et al.

[25] found the potential benefits of AR in infrastructure field tasks and architectural assembly

guidance, respectively. Some researchers demonstrated AR systems for planning or design process

such as design detailing [26], outdoor architectural designs, and urban planning [27]. Behzadan

and Kamat [28] proposed an AR system for AR rendering of construction processes onto the

construction site.

Advertising/ Promotion: Advertisers, always try new things and technologies, in order to get

shoppers’ attention. The technological era world is going through, is strictly connected to digital

information. Moreover, smart phones have become a necessity to modern peoples’ life. Thus, AR

is a brilliant tool to advertise and promote new products, without the need to spend huge amounts

on tangible commercial material (e.g. flyers) or promote new products using outdated





technologies. A great example is the use of QR coding system, which in fact is a two-dimensional

barcoding system (matrix barcode).

2.4 HARDWARE USED IN AUGMENTED, MIXED, AND VIRTUAL REALITY

Head Mounted Displays (HMD): A head mounted display, is a display device, worn on the user’s

head or as part of a helmet, that has a small display optic in front of on (monocular HMD) or each

eye (binocular HMD). There is also an optical head mounted display (OHMD), which is a wearable

display that has the capability of reflecting projected images as well as allowing user to see through

it.

Figure 4 Oculus rift, a modern HMD [29]

AR glasses: Vuzix is one of the leading companies in the AR glasses industry. The lab is equipped

with a pair of Vuzix Star 1200 XL glasses [30], which have performed satisfactorily under various

circumstances. After many hours of testing and different projects they are used in, two major

drawbacks have emerged, which have to be taken into consideration for an industrial AR use case,

where those glasses would have a primary role. The first drawback is the limited mobility due to

the cables which allow the glasses to be connected to the computer. Cables, limit worker’s

maneuverability, thus working in tight spaces or working around machinery may require worker

to remove glasses, or even worse carry the computer (laptop) in a backpack. The second drawback

is the weight of the glasses themselves. These glasses are too heavy creating fatigue if used for

long hours. Especially, if working in hot sites, sweat makes the glasses slip on the worker’s face

although they have an adjustable strap. However, Vuzix seems to have encountered those details

and below follows a list with their latest models and comparison of their key features.





Figure 5 Vuzix Augmented Reality glasses [31]

Smart Devices: A smart device is an electronic device, capable of connecting to other devices or

networks via different wireless protocols (e.g. Bluetooth, Wi-Fi, 3G etc.) and operating to some

extent interactively and autonomously. Some types of smart devices are smartphones, tablets,

smartwatches, and smart bands. Such devices can be complementary used in MR applications,

where additional user input is needed or can be used to solely implement an AR application on

them. This type of devices is currently preferred among the others, due to their compactness, their

mobility and the fact that are not purpose oriented. The latter, meaning that one can use a smart

device as a general tool with which he/she can implement a MR application, without the need of

any dedicated equipment.

Figure 6 Tablet used for AR application [32]

2.5 SOFTWARE USED IN AR

Besides the hardware described in the previous section, specific software is used in order to

develop augmented reality applications. In this section, all the aspects of an AR application,

regarding software are discussed. An AR application consists of scenes. These scenes consist of

3D models, text etc. and scripts.

The 3D models are produced from 3D CAD software or other similar software. If needed dedicated

software is used to convert the 3D geometries into a convenient filetype (.fbx, .obj, .dae, .3ds, .dxf

and .skp) [33-35].





For the development of AR applications there are several development kits on the market available.

Having a list with the predominant SDKs available in the market, the most suited has to be chosen.

The criteria based on which the choice will be made, are overall performance, cost, and support.

In Table 2, which has been adopted from reference [36], the most common AR SDKs are listed

and analyzed based on the above-mentioned criteria. Taking into consideration these criteria the

Vuforia SDK has been chosen [37]. The reasoning supporting this decision is provided briefly in

the following paragraph.

Although, from the comparison presented in Table 1, Wikitude gets the highest score, Vuforia is

the chosen SDK, which comes 2nd, having better features (maximum distance capturing,

recognition stability, minimum angle recognition), features that are thought to be more crucial

when developing an industrial AR based application. Another reason that led to choosing Vuforia

over the other SDKs, is support. Vuforia, is mating with Unity 3D™ [38] seamlessly and there is

plenty support from online users. Moreover, working in similar projects in the past, has gained a

considerable amount of knowledge concerning the use of Vuforia in Unity 3D™.

As stated above, the production of AR scenes requires a series of scripts. Scripts are written

instructions on a programming language, which instruct computer how to act in all situations. The

programming language chosen is C# (C-sharp). The choice of programming language was made

based on internet support and material, on the fact that it can be used for PC and Android

development and finally because it is one of the three supported programming languages, Unity

3D supports. In order to write scripts, one needs a suitable Integrated Development Environment

(IDE). Unity 3D is supported by the MonoDevelop IDE. However, in order to extend the

functionality of the proposed framework and the AR application, the Visual Studio IDE [39] has

been selected. Further to that, the Visual Studio IDE has been selected due to the fact that it is

more stable and offers advanced debugging capabilities.





Table 1 Table for AR IDEs comparison

SDK

Feature

description

Vuforia EasyAR Wikitude ARToolKit Kudan MaxST Xzimg NyARToolKit

Maximum

distance capturing

/ holding marker

(m)

1.2 / 3.7 0.9 / 2.7 0.8 / 3 3 / 3 0.8 / 3 0.5 / 0.9 0.7 / 5 0.7 / 1

Recognition

stability of

immovable marker

10 7 6 8 10 7 8 5

Recognition

stability of

movable marker

6 3 4 6 6 2 7 3

Minimum angle recognition

30o 35o 40o 10o 30o 50o 35o 45o

Minimum

visibility for

recognition

overlapped marker

20% 10% 30% 100% 25% 50% 10% 75%

2D Recognition ✓ ✓ ✓ ✓(bordered) ✓ ✓ ✓ ✓

3D Recognition ✓ - ✓(beta) - ✓ ✓ - -

Geo-Location - - ✓ - - - - -

Cloud Recognition ✓ - ✓ - - - - -

SLAM* - - ✓ - ✓ ✓ - -

Total (rating) 7.1 4.4 7.5 2.8 6.9 5.2 4.7 3.1

*SLAM stands for Simultaneous Localization and Mapping and is a technology which enables the

understanding of the physical world through a set of feature points [37].





2.6 USER TRACKING MODEL

Tracking is a crucial part in AR as it positions the virtual objects in the real environment of the

user [40,41]. To achieve tracking, it uses the camera sensor integrated to the device for the

recognition of predefined frame markers which are placed within the field of view of the user. The

transformation T between a camera and a marker is:

𝑥𝑥𝑐𝑐 = 𝑇𝑇 ∗ 𝑋𝑋 (1)

Where: X is a point in world coordinates, xc is its projection in ideal image coordinates and T is

the pose matrix. Transformation T consists of translation vector t and 3 x 3 rotation matrix R:

𝑥𝑥𝑐𝑐 = [𝑅𝑅|𝑡𝑡] ∗ 𝑋𝑋 (2)

which, using homogeneous coordinates becomes:

�𝑥𝑥𝑦𝑦𝑧𝑧� = �

𝑟𝑟1 𝑟𝑟2 𝑟𝑟3 𝑡𝑡𝑥𝑥𝑟𝑟4 𝑟𝑟5 𝑟𝑟6 𝑡𝑡𝑦𝑦𝑟𝑟7 𝑟𝑟8 𝑟𝑟9 𝑡𝑡𝑧𝑧

� �

𝑋𝑋𝑌𝑌𝑍𝑍1

� (3)

In addition to that, in order to map between frame marker ideal image (xc) and observed pixel

coordinates (xpix) a camera calibration matrix K is used:

𝑥𝑥𝑝𝑝𝑝𝑝𝑥𝑥 = 𝐾𝐾 ∗ 𝑥𝑥𝑐𝑐 → �𝑥𝑥𝑝𝑝𝑝𝑝𝑥𝑥𝑦𝑦𝑝𝑝𝑝𝑝𝑥𝑥

1� = �

𝑓𝑓 0 𝑝𝑝𝑥𝑥 00 𝑓𝑓 𝑝𝑝𝑦𝑦 00 0 1 0

� �𝑥𝑥𝑐𝑐𝑦𝑦𝑐𝑐𝑧𝑧𝑐𝑐� (4)

As a result, when the camera detects the frame marker,

𝑥𝑥𝑝𝑝 = 𝐾𝐾 ∗ 𝑇𝑇 ∗ 𝑋𝑋𝑝𝑝 (5)

Where xi are the positions of the four corners (i=1,2,3,4) of the marker in the camera image and Xi

are their corresponding world coordinates. Thus,

�𝑥𝑥𝑝𝑝𝑦𝑦𝑝𝑝1� = �

𝑓𝑓 0 𝑝𝑝𝑥𝑥 00 𝑓𝑓 𝑝𝑝𝑦𝑦 00 0 1 0

� �

𝑟𝑟1 𝑟𝑟2𝑟𝑟4 𝑟𝑟5

𝑟𝑟3 𝑡𝑡𝑥𝑥𝑟𝑟6 𝑡𝑡𝑦𝑦

𝑟𝑟7 𝑟𝑟80 0

𝑟𝑟9 𝑡𝑡𝑧𝑧0 1

� �

𝑋𝑋𝑝𝑝𝑌𝑌𝑝𝑝𝑍𝑍𝑝𝑝1

� (6)





The system then calculates M = K * T,

�𝑥𝑥𝑝𝑝𝑦𝑦𝑝𝑝1� = �

𝑚𝑚1 𝑚𝑚2 𝑚𝑚3 𝑚𝑚4𝑚𝑚5𝑚𝑚9

𝑚𝑚6𝑚𝑚10

𝑚𝑚7 𝑚𝑚8𝑚𝑚11 𝑚𝑚12

� �

𝑋𝑋𝑝𝑝𝑌𝑌𝑝𝑝𝑍𝑍𝑝𝑝1

� (7)

which is the final transformation matrix and applies it to all visualized geometries, in order to

estimate the position of the digital visualizations on the virtual environment. With the aim of

creating a system whose tracking is accurate and robust, PTC Vuforia SDK was selected. It is a

commercial product that provides an extension for Unity 3D™, for creating AR applications. An

illustrative example of the AR tracking algorithm based on the model discussed above is presented

in the following figure (Figure 7).

Figure 7 AR tracking workflow

At this point, it is stressed out that the above mentioned mathematical model for the position and

pose estimation of the user within an Augmented Environment can be adjusted by the addition of

the following models in order to be implemented on Mixed Reality equipment, such as the

Microsoft Hololens (version 1, and 2). More specifically, the model has to take into account the

feedback from the four (4) integrated depth sensors / cameras. Consequently, the following

equation for the depth calculation is implemented:





𝐷𝐷 =𝑅𝑅

√𝑈𝑈2 + 𝑉𝑉2 + 1 (8)

Where:

• D is the calculated depth

• R denotes the range, as measured from the HMD ToF (Time-of-Flight) camera

• U,V are the measured values for distance of a specific, user-defined point in the physical

environment

As a result, the interpretation of the depth value in real-world coordinates is derived from Equation

9:

�𝑥𝑥𝑦𝑦𝑧𝑧� = 𝐷𝐷�

𝑈𝑈𝑉𝑉1� (9)

Next, in Figure 8, a common Image Target used for the development and implementation of AR

applications is presented. Concretely, on the left side of the figure, the Image Target is illustrated

in its physical form, whereas in the right side, the points recognized by the sensing system of the

AR device have been highlighted for reader-reference [42].

Figure 8 i) Typical example of Image Target used in AR applications; ii) yellow points indicate the

recognized point array for the position and pose estimation





2.7 PRODUCT-SERVICE SYSTEMS (PSS)

Product Service Systems (PSS) is a term which was introduced in the early 60s by Bristol Siddeley

as a system of engines followed by support services (e.g. maintenance). However, the initial idea

was named differently (Pay by the hour, PBH), it is the predecessor of PSS. Goedkoop M. in [43],

defines PSS as the combination of product(s) and service(s) in a system to deliver the required user

functionality in a way that reduces the impact on the environment and moreover on states and

governments.

A PSS values asset performance or utilization rather than ownership and achieves differentiation

through the integration of product and services that provide value in use to the customer. PSS and

IPS2, can be categorized. The three main categories in which all product service systems are

included are, Product-Oriented Services, Use-Oriented Services and Result-Oriented Services.

Product-Oriented services is the kind of services in which customers get full ownership of the

tangible product while they benefit from the integration of a series of services (e.g. maintenance

contracts).

Use-Oriented services is a kind of services in which the service provider retains ownership of the

tangible product. In this type of services customer benefits from the function of the products.

Examples of these services are leasing, pooling, and sharing.

Result-Oriented services is the type of services in which, although, the customer and the supplier

agree on a result there is no product pre-determined [44].

Many different categorizations for PSSs have been introduced. The approach given above is about

the main categories and it must be mentioned that they are globally accepted. The other approaches

are about further categorization, like the categorization shown below [44].

Over the years the global market changes, as a result demands to manufacturers have increased

leading to the development and implementation of product service systems (PSS). PSS refers to

services (usually intangible) that support tangible products. They offer strategic innovation which

companies use in order to separate resource consumption from its link to profit and standard of

living improvements, to find new profit centers, to compete and generate value and social quality

while decreasing total resource consumption. Generally, PSS can be considered as a win – win

solution for both suppliers and customers. A special category of PSS is those that refer to business

products (Business to Business or B2B). That kind of services is called Industrial Product Service





Systems (IPS2). Also, IPS2 provide strong opportunities for business innovation and sustainability

improvement. In many researches, we see that IPS2, are aiming in higher sustainability of the

products they support. The relation between IPS2 and sustainability is a two dependency. As stated

in [45] modern markets shift their focus to sustainability and ecological efficiency of products.

From the former statements, one can easily understand the importance of sustainability in IPS2 and

generally in PSS. Sustainability can be implemented by establishing closed loop recycling

management with reuse of services. Another aspect of sustainability is prolonged life cycles by

preserving the usability of the IPS2 offers.

Technological evolution has a great impact on IPS2. Having more complex machines on the

market, the need for IPS2, and moreover for advanced IPS2, is mandatory. On the other hand,

advances in technological aspects, such as wireless data transmission, mobile devices, information,

and communication technology (ICT) [46], give opportunities for more advanced and satisfactory

IPS2, that meet the demands of the modern market.

The aim of integrating IPS2 is to increase economic competitiveness by selling functionality

instead of selling products. However, IPS2 is a suitable solution for companies that have positioned

themselves as niche players (e.g. agricultural engineering, packing). In order to provide more

sophisticated IPS2, suppliers have to take into consideration that by implementing IPS2 to their

tangible products, their relationship with the customer changes to an integrative cooperation.

Figure 9 PSS categorization





In [47] we can extract useful information about the impact of IPS2 in the global market, as well as

clarify how beneficial their implementation is for both suppliers and customers.

Below follows a table which shows the most common types of IPS2 and also analyses the

percentages and number of companies that provides them, from a sample of 12521 firms. The

sample consists of 29.52% combined manufacturing enterprises and 68.70% pure manufacturing

enterprises. Table 2 Types of PSS offered and their percentages

Which services are offered % of firms offering service Number of firms offering

service

Consulting Services 2.69% 291

Design & Development

Services

21.92% 2723

Financial Services 3.89% 421

Installation &

Implementation Services

5.10% 552

Leasing Services 1.07% 116

Maintenance & Support

Services

11.94% 1293

Outsourcing & Operating

Services

1.68% 182

Procurement Services 1.15% 125

Property & Real Estate 3.83% 415





Retail & Distribution

Services

12.18% 1319

Systems & Solutions 15.70% 1700

Transportation & Trucking

Services

0.20% 22

By the interpretation of the information presented in Table 3, it can be safely concluded that among

the top three (3) IPS2 are the Design & Development services, the Systems & Solutions services,

and the Retail & Distribution services.

Figure 10 Percentage of PSS adoption from global firms

The above chart is featured in [3]. It shows the percentage of enterprises from various markets

around the world that have implemented IPS2 on their products. It is clear that western markets

have shifted their focus on IPS2, while eastern markets have not yet. This can be useful as engineers

have a clearer view on the global market needs. However, in order to equalize the bars on the





former chart, engineers have to examine each market (what manufacturers produce and the target

groups) individually, in order to provide optimum solutions.

More and more companies around the world provide PSS packages. Some of the firms are solely

providing PSS while others manufacture tangible products and integrate services on them.

The idea of a company that provides only services for equipment bought from another firm is very

beneficial. Firstly, customer can retain their management model, or in some cases adapt it keeping

costs at low level. Secondly, service companies, because of their nature can provide services of

higher quality as it is their one and only field of expertise. Thirdly, with the creation of such

companies, new positions for jobs are created. However, the only downside in the whole idea is

the level of integration in the customer’s company.

The most common IPS2, found in today’s markets are:

• Maintenance

• Repairing

• Training

• Retrofitting/Customizing/Upgrading

• OEM replacement parts distribution

• Machine tool calibration

• Emergency services

• Plant installation/relocation

• Customer requested services/Design

• Safety Services

• Leasing/renting equipment/machinery

• Data acquisition and analysis

Machine tool calibration (metrology/laser calibration): Modern machines are usually CNC

which means that they incorporate many measuring components, which are electronically

controlled. Over time these machines tend to “lose” their calibration/ mapping, resulting in faulty

products, or in most severe cases damage in machine components (e.g. if a mill head gets out of

control may exceed its movement limits). Companies like Mori Seiki, API Services and MTT

Machine Tool Technologies, provide such services utilizing experienced personnel. We have to

note, that Mori Seiki is a manufacturer and has implemented that type of service to their machines.





The other two firms (API Services and MTT Machine Tool Technologies) are exclusively service

companies.

Training: Manufacturing is composed of many demanding processes, so is the machinery used.

Taking into consideration the technological advances, it is easily understandable, that operating

machinery is a very difficult process, and moreover getting the most of it requires the best

knowledge of its usage, otherwise the workflow is not optimal and the safety of the on-sites

workers is put under great risk. In these terms, mostly manufacturers and less service companies,

have integrated their products with training services, in order to help customers, organize their

plant optimally, increase their productivity and reduce emissions and resource usage by operating

the machinery in the correct way. Such companies are Machine Tool Technologies (MTT), API

Services, Mori Seiki, Tetra Pak, Sitec Industrietechnologie GmbH.

OEM replacement parts: Modern machines consist of a large number of components. Some of

them are complicated and others are simple and are treated as consumables. Either way, a business

willing to keep its competitive edge, needs to maintain their machinery in the best possible

condition. Also, if an unexpected damage occurs, minimum down time and correct fitment parts

are highly required. The best way to satisfy all the former conditions is to use OEM consumables

and other replacement parts. In this type of service, we meet only manufacturers, as they tend to

keep a large stock of all the components and consumables used in their products. Ideally immediate

support is required (within 24 hours the components are sent to the customer). Firms like Niles

Simmons, Mori Seiki, Tetra Pak, and Sitec Industrietechnologie GmbH keep a sufficient stock of

their components and they state that they ship them to the customer within 24 hours.

Maintenance/Repairing – Retrofitting*/Upgrading/Customizing: As stated above keeping

machinery to its best condition, maximum results can be achieved in terms of productivity,

energy/resources consumption and profit. Although using OEM equipment is highly suggested,

servicing, diagnosing, overhauling, and maintaining machinery is equally demanding process to

operating them. Thus, manufacturers and service companies provide their expertise to their

customers in order to ensure that the jobs done on customers’ machinery are done the best way

possible. From the customers’ side, such services are useful as they eliminate the risk of a wrong

job (resulting in further damage, longer downtime, safety issues etc.) and relief them from the cost

of training personnel specifically for that kind of tasks.





*Retrofitting is the addition of new technology/features to older systems and power plants,

focusing on improving power plant efficiency, increase output and reduce emissions [48].

Emergency services: It is widely known that industries, in order to maximize their productivity,

they work 24/7/365. As a result, equipment wears out much faster and sometimes unexpectedly.

Thus, customers need immediate support, to diagnose the problem, if it is a simple fix to repair it

and/or if possible to have constant communication with the supplier in order to get instructions on

how to repair the broken machine. Manufacturers and service companies for that reason, maintain

hot lines 24/7/365, in order to help their customers in every possible way, such firms are MTT

Machine Tool Technologies, EMCOR Group [49], Sitec Industrietechnologie GmbH [50].

Plant installation/relocation: Achieving the best factors for productivity, product life cycle,

emissions and resources control, workflow, and profit, can be done using various techniques,

models, and services. Here stating the obvious, we present plant installation or even relocation

after a new purchase/customization. Customers are not supposed to know the best way to install

their equipment, in the case of a new machine special treatment may be needed or if the product

produced may need special environmental conditions (e.g. food industries). Thus, companies like

MTT Machine Tool Technologies, Williams Industrial Services Group LLC, EMCOR Group,

Sitec Industrietechnologie GmbH, and Tetra Pak [51], provide their expertise at the service of their

customers. Expanding the term of installation and relocation, service companies like Williams

Industrial Services Group LLC, EMCOR Group and ISG Inc. offer extended services on building

the whole plant, the piping, the vessels and generally do all the tasks to utilize the customer’s

premises optimally.

Customer requested services: From the previous paragraphs it can be safely concluded that both

manufacturers and service companies, have implemented a wide range of services, covering almost

every customer. However, if customers discover a need that is not covered by any available service,

they can refer to EMCOR Group, Williams Industrial Services Group LLC, ISG (Industrial Service

Group Inc.) and other firms similar to them, in order to cooperate and find a viable solution/design

a new service (or series of services) in order to cover the void.

Safety services: Industrial environments because of their nature are very dangerous places to work

in. Especially in cases where resources are harmful not only for the environment but also for the

on-site workers. Although protective meters are of great importance, there are some services that





ensure safety under all situations (e.g. special insulating of components & machines, thorough

cleaning of walk-on metal grates etc.). Such companies are ISG (Industrial Service Group Inc.),

Williams Industrial Services Group LLC, EMCOR Group.

Concluding, from the results of this research it is unveiled that although there are firms that provide

a wide variety of services, the global market needs more integration. From all the examined

companies, Williams Industrial Services Group LLC seems to provide the most services, API

Services and ISG (Industrial Service Group) focus on specific sectors, Tetra Pak combines

manufacturing and servitizing in a very intriguing way. Also, of great importance is that Williams

Industrial Services Group LLC specializes also in energy production plants. They provide services

specifically designed for Nuclear, Fossil, and Hydropower Power Generating Facilities. In the list

provided below, a list of the most commonly offered PSS is presented. Namely, the following PSS

cover a wide variety of major industrial activities, including resources management, asset

management, production planning and scheduling, energy management and distribution etc..

• Skilled Craft Labor and Supervision

• Craft Labor Management

• Refueling Outage Support

• Capital Project Support

• Plant Modifications

• Work Planning & Scheduling

• Quality Assurance / Quality Control Programs

• Systems Modifications / Upgrades

• Security Screening / Background Investigation Services

• Tools & Equipment Management

• Reactor Head Maintenance

• Steam Generator Replacement Support

• Vessel and Heat Exchanger Replacement

• Condenser Tubing and Cleaning

• In processing, Security Clearances & Background Investigations

The above list is used as a reference to what extent their services expand.





Leasing/renting equipment/machinery: If a new enterprise is to be established and the budget is

low, or if specific machinery or equipment is needed for a known time period, then there are

solutions to keep costs low while acquiring the needed equipment. Byline Financial Group, API

Services and CUSHMAN & WAKEFIELD are two companies that provide leasing/rental services.

Data acquisition and analysis: In global market, all companies are striving for maximum profit

and sales, while maintaining customers’ interest for their products. Thus, the need for market

analysis, finding new trends, defining challenges, and seizing opportunities, is obvious. Not only

that but a company also has to ensure its steps (e.g. expanding to new markets, launching new

products etc.). the solution to this problem is given by firms, such as Hoovers, Euromatic

International and CMIE (Industrial Analysis Service). These companies have databases (which are

updated on a regular basis) and upon customers’ request they collect all the needed data, analyze

them, and send back reports according with the customers’ needs.

Another point that should be taken into consideration is that ISG (Industrial Service Group Inc.)

provides services relative to plastics injection molding industries. In the official website it is stated

that the company provides precision cleaning of hot runners. They also service plastic processing

components (e.g. molds, dies, tooling).

2.8 DATA PROTECTION AND DATA ENCRYPTION

With the use of Cloud computing, a big amount of data is exchanged at any moment. Cloud

computing offers a variety of benefits to its users, while containing a factor or risk. Data being

transferred need to be secured and encrypted in order to prevent attacks and leaks from hackers.

In fact, although the technological advances in IT technologies shift to the use of Cloud services,

many organizations hesitate to migrate their data to Cloud due to security issues [52].

In order to cope with the rise of cybercrime, data encryption techniques are used. According to

Ayers [53], there are many reasons that dictate the use of encryption in all kind of data that are

being transferred. First of all, if any kind of data is being transferred, then it is considered as

vulnerable as it can be stolen during the transport. Second, in some cases of cybercrime, targeted

data are not to be stolen, but to be altered in a way to perform acts of fraud. Moreover, a serious

amount of data is considered as sensitive data. Sensitive data have to be encrypted in order to





ensure users’ privacy and anonymity. Finally, a big amount of data is transferred through devices

(e.g. through mobile phones, computers etc.), thus creating a need for protection of these data, not

only while being stored but also while being transferred between devices. Following, is a table

with some of the most common security techniques used in server protection.

Table 3 List of common server protection techniques

A/I Security Measure

1 SSH Keys

2 Firewalls

3 VPNs and Private Networks

4 Public Key Infrastructure and SSL/TLS encryption

5 Service Auditing

6 File Auditing and Intrusion Detection Systems

7 Isolated Execution Environments





CHAPTER 3. PROPOSED METHODOLOGY AND SYSTEM ARCHITECTURE

The tool proposed in this document is targeted for used with Android tablet or other similar

devices. Thus, a considerable amount of development has been done in order to utilize as much as

possible of the Android functionalities. Android platform is popular among worldwide technology

users, so it is wise that used against other platforms. The tool alone is not enough for the proposed

method. At least a computer has to be utilized to backup tool’s operations as a Cloud server. The

server side is developed in parallel to the application proposed, in order to communicate with the

application. The first section of this chapter, focuses on the presentation of the PSS model adopted,

based on the requirements for the developed tool. In the second section, the system architecture is

explained in detailed and the remainder of the chapter, focuses on the discussion of the developed

tool functionalities.

3.1 PSS MODEL

The developed system composes a product-oriented PSS solution that supports the product with

the important service of maintenance throughout its lifecycle. The PSS proposed can be acquired

either when buying a new mold or as an aftersales service, in order to support older equipment and

equipment sold by third party companies. The function of this PSS is dependent to the product

from its design phase, where the engineers make the 3D CAD components and assemblies of the

equipment. The CAD files are needed, in order to be converted to suitable 3D geometries based

on which the creation of 3D and AR scenes is done. Moreover, with the use of smart algorithm,

running on the CAD program, a datasheet with the assembly and disassembly steps can be

automatically generated. The proposed framework includes all the needed methods that combine

the 3D geometries and the steps created in order to create a series of 3D/AR scenes that guide

workers through the steps of assembly and disassembly of the equipment. At this point, it is





stressed out that the algorithm can run in real-time (e.g. in case of new equipment) or near real-

time (in case of previously maintained equipment) where the scenes have already been created.

The procedure followed by the framework is depicted in Figure 8. In the figure, the components

of the flowchart are divided into steps, according to the actions taken during the maintenance

procedure of a machine, using the proposed approach.

Figure 11 PSS system flowchart





More specifically, the first step, which is presented in Figure 9, integrates all the needed

steps/actions that take place, during the inspection phase of the machine, starting from the premises

of the client when they request maintenance of their equipment by sending a report to the

engineering department of the PSS supplier.

Figure 12 First step in the sequence of maintenance

Continuing in the procedure of maintenance, is the report exchange phase between the engineer

and the client, where the time and cost estimation is performed. This is an important step, as it is

the time when the maintenance/repair tasks are discussed and agreed. When, the tasks are agreed

from both sides then the final time and cost estimation are reported to the customer. The

importance of this step is boosted by the fact, that the proposed approach, based on historical data

(which are constantly updated on the Cloud database), helps the maintenance engineer, produce

more accurate estimations. The following figure (Figure 10), presents the aforementioned

components.





Figure 13 Second step in the sequence of maintenance

The third step in the maintenance procedure is the actual undertaking of the maintenance tasks

agreed in the previous step. This set of tasks has not been affected from the implementation of the

proposed approach. The only advancement is the task assignment to each technician. In Figure 11,

the included components of the flowchart are presented.

Figure 14 Third step in the sequence of maintenance





Following the repair/maintenance of the mold is the validation of the jobs done (). During this step,

the technicians, test the machine based on the machine and production line specifications, which

are retrieved from the Cloud database. If the validation procedure is completed successfully, then

the maintenance can be considered successful, thus the machine is sent back to the client’s

premises.

Figure 15 Final step in the sequence of maintenance

3.2 SYSTEM ARCHITECTURE

The developed framework includes a series of methods that facilitate the development of AR

assembly/disassembly instructions and inspection of equipment based on predictions from a

formerly created knowledge database and data from sensors, integrated to the mold (e.g. cycle

counters). During maintenance, three different parties participate: (i) the customer, which owns





the equipment and uses the service, (ii) the engineer who has the expertise and is responsible for

scheduling and overseeing maintenance and finally (iii) the technicians, who are responsible for

the maintenance tasks. As it is presented in Table 5, in the designed application there are 3 major

user-groups and one group with administrative rights. The division of the users into the presented

groups is done so that each user can view only the data required to accomplish their tasks, whilst

maintaining their cognitive level and information feed to a minimum level. The provision of

administrator users has been given so that changes can be made from authorized personnel from

any point in the world just by accessing server’s domain through the corresponding graphical user

interface (GUI).

Table 4 Involved actors and their rights

Actor Responsibility

Client Sends and receives information about the mold characteristics and

maintenance tasks.

Engineer Supervises the workers, send reports, inspects annotations, and creates the

Bill of Processes, makes the fault diagnosis, has access to the AR tool and the

database, builds the scenes for the AR/3D tool.

Technician Makes the fault diagnosis, identifies the code of the mold, makes annotations,

has access to the AR tool and the database.

Admin Has access to all interfaces and has rights to modify the database.





Figure 16 System architecture - the numbers present the sequence of interactions





3.3 DATA FORMALIZATION

The Unified Modeling Language (UML) is used in order to model the proposed system. UML

stands for a standardized model to describe an object-oriented programming approach. Hence, it

was used in order to design the system architecture for the proposed methodology in an

understandable and structured way. UML consists of classes. Each class contains several data and

objects. While this is not enough to fully describe the proposed framework, UML, also provides

all the required tools, in order to describe dependencies and interactions between classes. Taking

into consideration all of the above, the author, created a UML diagram of the proposed framework

in the very beginning of the development described in chapters 2 through 4.

In Figure 51, the final version of the UML diagram, for the proposed framework is presented. In

order to conclude to this diagram, initially, the test case requirements, were discussed. Following,

the sets of data to be involved in the framework were determined. Following the determination of

the datasets, is their categorization to the corresponding classes. Having the basic infrastructure

ready, the interactions between the classes are modeled.

At this point, it is stressed out that every step in the design of the UML diagram is posed a challenge

for the author. Each dataset had to be determined carefully, in order to define the correct type of

data either to be expected or to be exported from each class so that confusion is avoided. It is

important, also, that these steps are designed carefully, as the UML diagram serves as the guide

for the actual development of the tool. However, during the development of the tool, as mentioned

in chapter 4, the developed tool, was reviewed in webinars and workshop meetings. These

intermediate evaluation procedures assisted the author to refine the UML presented in Figure 51,

and moreover several other aspects of the test case emerged, which led to the implementation of

more features to the tool, thus creating new dependencies between the initial classes.





Figure 17 UML Class Diagram supporting the data modelling of the proposed framework





3.4 APPLICATION FUNCTIONALITIES

The GUIs shown in this section are developed in Unity 3D™. Unity 3D™ is a powerful game

engine, which supports augmented reality application development with the use of the Vuforia

add-on. The CAD files used, are requested from the equipment manufacturer directly in a universal

form (such as stl, step, iges) which are then imported in Blender, a free and open source 3D creation

suite, in order to convert them to a suitable filetype for the Android application (such as dae,

Collada or obj files).

The colors used in this application where chosen based on the colors Microsoft uses. The main

criteria for color choosing are:

Vividness: Textual and 3D content must be clearly visible under all circumstances, especially

while working on harsh environments – such as a machinery shop. For this purpose, a black

background combined with green textual content is used.

Eye relaxing: this application is meant to be used for long hours; thus the user must not feel

anxiety, tiredness or any other feeling related to smart-device use and/or augmented reality use.

Figure 18 Introductory Graphical User Interface (GUI), MMT®





Registering: Every user of the service must be registered in order to use it. It is a simple process,

with a minimum of 4 pieces of information needed:

• Username or email address.

• Personal code.

• The user group in which user belongs.

• Email address

When user fills all the information, the service automatically saves his data in a remote server. The

online server is used to handle all users’ data and recall them whenever needed. It must be noted

that no sensitive data are required for registration, thus users can feel free to use the tool without

risking getting stolen valuable personal information. Finally, when a new users registers to the

tool, an email is sent automatically from the tool, containing the credentials and user-group which

was selected during the registration phase.

Figure 19 GUI for the registration of new users in the platform, MMT®

Login: After initial registering, user can start using the service just by logging in using their

information. It must be noted, that after registration, user is redirected to the login page,

automatically. Then the service recognizes in which user-group the user belongs, so that the correct

features are loaded. Each user-group has access to specific features of the appication or different





configurations of the same features (e.g. the AR tool is common for all users but client cannot use

AR as an assembly/disassembly manual).

Figure 20 GUI for the login of registered users, MMT®

Client’s Start Page: In Figure 19, the clients’ start page is presented. As it can been seen in the

above-mentioned figure, a wide variety of functionalities have also been developed for the

clients/customers, in an attempt to facilitate the data gathering from the production environment.

By extension, this will improve the completeness / integrity of the cloud repository, for future

reference.

Figure 21 Client's start page GUI, MMT®





More specifically, from this GUI, clients can inspect their equipment in AR mode (health

monitoring, explained in later section), request design of a new PSS according to their needs. After

completing all the required data, the expert engineer, designs the PSS and informs the client.

Figure 22 PSS customizer interface, MMT®

Also, from this GUI the clients, can view a date picker applet (calendar) which is presented in

Figure 21.

Figure 23 Date picker interface, MMT®

When using this functionality, clients can request and schedule a maintenance program, taking into

consideration the availability of the maintenance-contractor’s production schedule. Using the





screenshot button, users can take a picture of their equipment in High Definition. The screenshots

are automatically saved on the Cloud database, and then they can be sent supplementary to the

malfunction report, in order to describe the problem more adequately. Finally, in the customer

profile, each customer, can view and edit their personal data, as illustrated in Figure 22. It is

stressed that during the first login, i.e. following the user registration, the clients are redirected to

this GUI in order to complete their profile.

Figure 24 Customer information sheet, MMT®

These data are edited from each customer and saved to the Cloud database, so that the maintenance

contractor can keep track of their customer list. In the corresponding following section, the

maintenance engineer, can view a list of all the customers serving.

Finally, the client’s start page, has also available a “Report” button, which leads the corresponding

customer to the report creator GUI. The report creator is presented later on this chapter, as it is

common for all the user-groups. What is worth noticing, is that whenever a customer, submits a

new report to the system, the users, that are registered as “engineers” to the Cloud database, are

notified via email, to the email address, they used for registration.

Engineer’s Start Page:





Figure 25 Engineer's start page, MMT®

With the mold inspection button, the AR mode opens, and the engineer can inspect the arrived

piece of equipment in AR mode.

With the download file button, the engineer, gains access at the file repository of the Cloud

database. The download interface opens (Figure 25), and the user can select through the available

files. Also, the user has to select a folder (from the provided list of folders), in order to save the

corresponding file. When the download is complete, user gets notified from the tool that the action

was completed successfully.

Figure 26 Download file interface, MMT®





With the “BOP” button, the engineer is directed to the BOP creation GUI, where a set of dropdown

lists is presented. Using these dropdown lists, the engineer, selects through the available list of

machines and the corresponding tasks. After finishing the composition of the BOP list, the

engineer can export the list to the Cloud so that the tasks can be assigned to the workers. At the

right lower part of the GUI, a progress bar can be viewed (Figure 25), which informs the user for

the progress of the undergoing task.

Figure 27 BOP creation GUI, MMT®

At this point it must be noted that the BOP can be exported as an XML file. The XML file is

structured as presented in Figure 26.

Figure 28 XML file structure for the generation of BOP, and generation of AR instructions





Therefore, it becomes obvious from Figure 26, that elements are used in the XML structure. The

elements were used in order to add more flexibility to the produced XML file.

In Figure 27, an example XML exported directly from the tool is presented, featuring a set of three

tasks, along with their specifications, such as the sequence number, the task name, the task

description and the product ID.

Figure 29 Example XML file based on the structure presented previously

The engineer, however, from the BOP GUI, can use another functionality, hitting the “History”

button. In the history GUI, the engineer, inserts the desired date, the desired mold model, or both,

and the tool makes an inquiry on the Cloud database, in order to fetch, if any available, historical

data.





Returning to the start page of engineers there is another crucial functionality, described as the

alarming tool. Using that tool, the engineer, gains access to all available machines, and views all

the available info, as well the expected time to arrive for maintenance. This functionality is useful,

as it assists the engineering department of the maintenance contractor, to schedule their production

line, based on the machines expected to arrive. In order to inform the maintenance engineer about

the incoming mold, the corresponding listing in the mold list is colored red, indicating an alarm.

In Figure 31 the mold-list GUI is presented. Furthermore, this functionality, exports a variable, the

availability of the maintenance contractor, which is used in the date picker functionality, presented

previously, in which the client, views the available dates for maintenance.

Figure 30 Mold-list GUI

Finally, using the “Shop Inspection” functionality, the corresponding GUI opens, enabling the

engineer to get an overview of the availability of the machines at the shop floor, without the need

to get to the production line, physically. For the machine availability, three colors are used. Red is

used for machines NOT available; green is used for available machines and finally yellow is used

for machines that will be soon available. This functionality, aims to assist the engineer, organize

the maintenance contractor’s production line more efficiently. In Figure 30 the “Shop Inspection”

GUI is presented.





Figure 31 Shop Inspection GUI, MMT®

Technician’s Start Page: In Figure 31, the starting GUI for the experienced technicians at the

shopfloor is presented. What is worth noticing, is the access to two modes of AR, on the one hand

being the inspection mode and on the other hand being the assembly/disassembly mode. Moreover,

the technician can view a 3D presentation of the mold he/she is working on. Hitting each button

from the ones listed above, leads the technician to the corresponding GUI. These GUIs are

presented in the remainder of this section.

Figure 32 Technician's Start Page, MMT®





3D View interface: A GUI similar to the AR GUI. User sees a 3D representation of the mold. The

buttons add functionality to this interface. After hitting the “Menu” button at the upper right corner,

the user is able to select through different functionalities, see the following table.

Table 5 List of additional functionalities

A/I Functionality name

1 Exploded view

2 Annotation

3 Step selection

4 Object rotation

While working on that interface, user can select any component he wants to interact with or even

select the whole assembly. In order to clarify if a component is selected, when the user taps on it

(thus selecting it) it becomes green and returns to its default color (white) when deselected.

Figure 33 3D presentation interface, MMT®

Exploded View: Exploded view offers the user, the opportunity to view all the components of a

mold moved away from each other. Hidden objects, such as internal components and

subassemblies are now clearly viewed. This functionality is based on the assembly instructions





created during design phase of the mold. The service’s algorithm analyses assembly steps and

component tier and accordingly translates them in a way that everything is visible and interactable

through the smart-device touchscreen. This functionality is accompanied by a slider, which

translates its movement to distance between the parts. When slider is returned to its initial position,

the components are put to their positions as when assembled.

Figure 34 Explode functionality in 3D presentation, MMT®

Annotation Button: When selected, user has the ability to select a specific component or even

select the whole assembly and make useful annotations. These annotations may contain

information about maintenance tasks performed or information about any unusual measurements

taken throughout the inspection phase. The annotations are saved to the server so that when the

next person in the production line/machine shop gets the components of interest, using the tool can

retrieve from the server all the annotations made so far. The tool has been developed, so that the

user can easily select through the components of the machine. In order to select the component of

interest, user has to tap it. The selected component, then turns green, in order to notify the user of

his selection. In order to deselect any component a double tap is required. By default, if the user

has not selected any of the available components, and yet he submits a new annotation, the

annotation is registered to the whole assembly. In Figure 33, the selected component can be seen,

as well as the input field for the annotation is visible on the upper left corner of the GUI.





Figure 35 Annotation functionality in 3D presentation, MMT®

The same sequence if followed when the user wants to retrieve annotations made in the past. If

there are any available annotations saved on the Cloud database, then they will be displayed on

the user’s GUI. In Figure 34, once again the selected component can be recognized. What is more

on the top center portion of the GUI, a previously made annotation on the selected component is

presented (circled component).

Figure 36 Annotation on 3D component, MMT®

In order to boost functionality of this GUI, several other components have been implemented, such

as the “Explode” button, which is an option to explode the machine’s components so that the inner,





yet not visible, parts can be manipulated. What is more, the 3D model can be manipulated, in terms

of rotation and scale, by using the “Rotate” button and “pinch” gesture (common gesture in

Android devices) accordingly.

Select Step Button: During the designing phase and after completing the whole assembly of the

equipment, the engineer using an add-on attached to the CAD program can produce a sheet with

the assembly/disassembly steps of the equipment automatically. This sheet is then used by the

presented tool in order to import the sequence of all assembly/disassembly steps that need to be

followed. Then a dropdown menu is created automatically. The available options of the

beforementioned dropdown menu are created automatically from the tool after reading the sheet.

What is left, is the user to select the step which he is interested in to view guidance. After step

selection, user guidance begins, either in 3D mode or AR mode, using animations wherever

possible accompanied by 3D textual content. User at any point of the assembly/disassembly

sequence can pause, stop, and use any of the described functionalities provided with the tool.

Report tool: Report is similar to the annotation tool, but they serve different purposes. Reports can

be created from two user-groups, the clients, and the engineers. The report creator GUI is presented

in Figure 35.

Figure 37 GUI for Report creation & submission to the Cloud platform, MMT®

Clients use this functionality in order to write maintenance requests to the equipment manufacturer

and to report any unusual behavior from the machinery. Engineers use this functionality, in order





to write time and cost quotations. In the report GUI, a text editor appears where the users can write

the content they wish. Then there is the “Submit” button which, when pressed, sends the report to

the server. The file sent to the server can be saved in multiple types. The formal type is exported

as PDF, the file aimed for internal use, is saved as TXT file and finally, in the server there is a

table where all the components of the report are categorized and saved separately. The report

repository can be viewed in Figure 36.

Figure 38 Reports, saved on the Cloud FTP server

Rotate Button: As the name implies, hitting that button, the user can rotate the whole mold

assembly on his screen in order to view hidden spots, or help view from a more convenient point

the mold. At this point it must be clarified that once hitting “Rotate” button the user can rotate 3D

objects. Hitting again the button and the functionality is disabled.





Figure 39 AR visualization of the exploded assembly

Mold Inspection: As discussed earlier, it is crucial to ensure that an injection mold works under

the specified parameters, in terms of pressure, vibration, and temperature. The service provides a

tool so that a customer who has retrofitted the necessary sensors to their mold can view in real

time the condition of the components of interest. The working principle of this functionality is

simple. The data collected from the individual sensors are stored in suitable databases. The service

has access to this data, so that it can compare them to the ideal working values. For this purpose,

it is supposed that every measured variable has a min and max value. Thus, a working range is

created. If the measured value is somewhere between that range, then the component is “OK” and

the components are colored in green color. If the measured value lies on either of the verges of

the working range, then it might require client’s attention. Thus, the component(s) of interest are

colored in yellow color. Finally, if the measured value falls out of the working range, that means

immediate action is required, so the component(s) are colored in red color.

Finally, the components of interest are augmented on the tangible machine, in true scale and shown

on the user’s screen. The user can take a snapshot, using the corresponding functionality, of the

faulty component and save it on Cloud. When the machine is sent to the maintenance contractors’





premises for repair/maintenance, the snapshot will be retrieved from the Cloud, in order to assess

the machine’s condition. The measured value is saved on a report sheet.

Alternatively, if the client has not yet installed sensors on their machine, data can be estimated

from the working cycles, based on the measuring from the already installed cycle meter.

Figure 40 AR mold projection, health monitoring

Server side: As discussed in the introduction of this section, the tool is supported by a server. The

development of these two components (application and server) is done in parallel. Actually, server

acts as the backbone of the tool presented. These two components are dependent to each other.

Server holds a database in which data sets from the tool are saved. Also, the server supports FTP

suitable for file exchange between users. The functionality lying here, is the creation of a cloud-

based CAD exchange, between the client, the manufacturer and the company that undertakes

maintenance tasks, in case it is different from manufacturer. In case of customizer, client can

request a new product or modify an existing one. In either case the tool supports FTP transfer

protocol so that the client provides the required CAD files. Also, in customization, another feature





is the addition of sensors in points/components of interest so that the health of the product can be

monitored in real-time. In this case the CAD files, are used in order to produce AR scenes

presenting the position in which the selected sensor(s) must be placed on the product.

Server is also needed for data handling. Data, acquired from sensors or extracted as input from the

users, are saved automatically to the server. Timing and cost estimation can be a risky task,

especially if the estimation is about new tasks. In order to encounter that problem, the proposed

method, consists of a table which holds data about all the tasks performed during maintenance of

equipment and time and cost. With this approach, when the customer requests a maintenance plan,

the assigned engineer can accurately with the help of the tool to estimate time and cost for each

maintenance task according to the data-tables held on the server. Moreover, the data acquired from

the sensors of a machine (in case a machine has integrated sensors), are used from the application,

in order to inform the customer (the end user of the equipment), at all times about the health of his

equipment. The way, machinery health, is monitored is described in previous section.

Figure 41 Cloud database components

Furthermore, in order to accomplish communication between the tool and the Cloud, a set of

special scripts were written. The language used for that scripts, is PHP. The aforementioned

programming language was selected among others (e.g. HTML), as it is free, it is user-friendly, it





is compatible with HTML and there is plenty support on the internet. Due to the needs of the tool

for constant communication with the Cloud database, several PHP scripts were written and

uploaded to the Cloud database. Each time the tool needs to communicate with the Cloud, the local

script (running in the background of the tool) calls the corresponding script and the communication

is established. Thus, the tool requires constant access to the worldwide web. For that purpose, each

time the application starts, checks internet connection, and notifies users accordingly. In Figure

41, a typical example of a PHP script developed is presented.

Figure 42 Typical paradigm of PHP script

Administrator user-group: A special user-group, has also been implemented to the developed

tool, in order to offer extra functionality to the users, while ensuring the application’s good

operation. For that purpose, the administrator has access to all the application’s interfaces, as

presented in Figure 42.





Figure 43 Administrator interface, MMT®

Additionally, the administrator’s GUI has been enriched with the provision of administrative tools

(Figure 42). These tools are used to process the Cloud database (e.g. create, edit, and delete tables

in the Cloud database).

Figure 44 Administrative tools interface, MMT®





CHAPTER 4. INDUSTRIAL TEST CASE

In order for the utilization of the proposed platform and its functionalities to be demonstrated, a

case study is presented below.

A mold manufacturing company, responsible for creating and maintaining molds, is selected for

the case. The mold company selected for the test case, is a high-end SME located in Greece, with

a total count 50 people personnel, manufacturing state-of-the-art, highly accurate Engineered-to-

Order (ETO) tools and equipment. The mold company is specialized in producing highly accurate

precision parts, progressive cutting and forming dies, injection molds and undertakes demanding

product development and industrialization projects, in collaboration with OEMs.

The selection of this manufacturer, as the test case to implement the proposed framework poses a

great challenge. That is, because of the nature of the tasks this company has to accomplish. The

company, as mentioned before specializes in the manufacturing of ETO IM, thus there is low to

no repeatability in the maintenance tasks of each IM. Moreover, taking into consideration the

delicacy of the components consisting an IM, its extended life cycle is highly dependable on the

maintenance and repair services. Currently, the company offers such services only in the context

of the after-sales plan, which creates an opportunity for the enterprise to adopt servitization in its

business model.

Regarding other aspects of the IM maintenance, it is often impossible to plan maintenance

operations. Thus, the design and development of a product-based PSS oriented in preventative

maintenance, could possibly offer a robust solution for the PSS-supplier, who also gains their

customers’ loyalty. In the remainder of this chapter, a possible solution to this problem is proposed

with the use of sensors, installed on the IM. In conjunction with the proposed framework from

Chapter 1, the provision of an all-around, customizable PSS solution is presented. The customer

has the opportunity, when buying a new mold to buy also, a PSS which will automatically extend

his IM life cycle, thus creating more added value for the physical assets of their company.

However, the provision of such solutions, is not available, only for new customers. The proposed

PSS is designed, so that with the addition of specific features, and the acquirement of the bare





minimum equipment (which may already exist in the company), customer can benefit even though

they do not purchase new equipment.

The molds for maintenance may be produced by the company itself or from other manufacturers.

Nowadays, the maintenance in this industry is done non-automatically. The customer sends the

mold in the company’s premises and workers after thorough examination of the mold, identify the

jobs that need to be done in order to restore the mold. A report is sent back to the customer and if

he approves the maintenance activities, he sends a document with some measurements and

characteristics of the molds so as to proceed with the maintenance. The maintenance tasks start,

and the workers are responsible for them. Then the workers monitor the duration of the

maintenance and send it to a supervisor engineer to fill in some documents for the duration of the

tasks.

The mold manufacturer receives the mold that needs to be maintained and through a smart device

the worker inspects the parts. The maintenance history is also checked so as to extract information

about the time and cost of previous maintenance activities. During mold maintenance, the service

provides a smart tool able to monitor the maintenance and another AR tool able to simulate the

assembly and disassembly of the mold. Although there are tools fully automated in the service, the

role of the user is not neglected. The case study consists of three actors shown in the Table 4.

Figure 45 The mold manufacturing company





4.1 INJECTION MOLD OVERVIEW

The process of injection molding is fairly simple, yet there are areas, which if are not inspected

can destroy the end product and the production flow. Injection molding consists of three tasks, (1)

melting the media to be injected (plastic), (2) injection of the melted plastic and (3) ejecting the

finished product/component [54].

Injection molding is a repetitive process. Initially the plastic is heated up to its melting temperature,

so that moisture escapes and plastic can be easily injected. Then the hot plastic fluid is injected

through the injection unit to the mold cavities.

Molds consist of plates (stationary and moving) which when are sealed together, form the cavities

in which the hot plastic is injected. If production rate demands the injection of multiple pieces at

a time, then plates have more than one cavities. Cavity shapes may differ to each other. In order

for the hot plastic to run through all cavities, channels are grooved on the plates. These channels

are called runners (or hot runners). All the above components can be viewed in the Figure 44,

which presents a cut-out of a typical injection mold (IM).

Figure 46 Injection mold cut-out





During and after injection, the plastic stays pressurized in the cavities until it solidifies. After

some seconds, the product has cooled to a point where it can be ejected from the mold. Injection

mold cycle has completed and its ready to start over. The duration of each cycle is lasting some

seconds, which depends from the product’s characteristics as well technical specifications of the

mold machine itself. In Figure 46, a complete injection cycle is presented with the corresponding

times of each individual task. The maximum time of a complete injection cycle, lasts up to 24

seconds, depending on the characteristics of the final product.

Figure 47 Injection mold cycle with the corresponding times

There are several points throughout the injection molding process where changes in an asset’s

temperature, pressure, or humidity can compromise product quality. If left unchecked, various

imperfections in moldings and in production line can happen. A list with with the most common

faults is provided in Table 6.

Table 6 List of common faults for injection molds

A/I Fault Name / Description

1 Part deformity

2 Stress cracks

3 Extra flashing

4 Excess scrap

5 Increased recall costs





A/I Fault Name / Description

6 Trapped air/moisture

7 Shrinkage

8 Voids in moldings

9 Overall poor end result

10 Unmet specifications, like especially tight tolerances

In Table 6, a list with the common failures of IM is presented. However, for the development

of the proposed PSS, that information is not enough. To encounter that probem Figure 46, is used

instead, as it lists the failures and the corresponding actions needed, to properly repair and

maintenance an IM.

Figure 48 Injection mold common failures

In order to prevent the beforementioned damages and ensure top quality of the molded

products, sensors can be installed on key points of the injection mold and monitor vital

components’ health, a crucial step for preventive maintenance. These sensors are:





Accelerometers: Accelerometers are used for vibration measurement (excessive vibration is

most likely to cause of wear and tear, as well as of damages to the mold).

Thermocouples: Thermocouples, are used for mold temperature monitoring.

Cavity Pressure Sensors: Cavity pressure sensors, are used for analyzing the quality of the

injection process.

Consistent and accurate condition monitoring is critical to ensure the molding process is not

costing the company thousands annually in additional operational costs.

Temperature monitoring: If equipment is running too hot or too cold, the two seconds it takes

for the initial mold process can be enough to jeopardize product quality. From the resin storage

area to the mold itself, there are varying temperatures that must be kept consistent. (e.g. clogged

water channels in the injection mold will not show an increase in water temperature, but cavity

temps will rise, hindering the cooling process and leading to product deformations)

Pressure monitoring: From the injection screw that pushes material into the mold to the clamp

that holds the mold shut, insufficient pressure can let air and moisture seep into the material,

leading to batch defects:

Short shot: A void of material in a portion of the part due to insufficient pressure.

Flash: Excess pressure forces mold open just enough to allow material seepage, resulting in

defective parts and potentially damaging the mold.

Dimensional variation: Packed mold is fully compressed, but cavity pressure changes at a

critical point.

Warp: Variations in packing pressure (as well as cooling rate/time) that leads to random

distortions in part. Maintaining proper working pressure is a vital part of the injection process.

Humidity monitoring: Plastic resin is subject to changes in ambient humidity conditions in

storage containers, as well as the process lines. Too much humidity can add moisture to the resin,

changing its properties and preventing it from molding like it’s supposed to.

However, measuring ambient humidity is not enough. Humidity in the plastic itself, as well as

the moisture content of the air being blown from resin dryers has to be monitored in order to avoid

the creation of voids within the internal structure of the molded component/product.





4.2 CASE STUDY DESCRIPTION

The first steps of the mold maintenance service are the same with the non-automated version

of the maintenance. Customer sends the mold and the technician in mold manufacturer premises

identifies the code of the mold that needs to be maintained by using a smart device and an AR tool.

The smart device will be a tablet and the scanning is realized with the camera. The identification

is based on a stamp that the mold has on it and by using this tool it is also possible to explode the

parts of the mold, see inside and retrieve information of all the mold components (ID, Name,

condition etc.). In Chapter 4, which deals with the future development, alternative solutions for

the tracking are presented. For the visualization of the mold in a smart device the CAD files are

needed. These files are provided by the customer or the mold manufacturing company retrieve

them from a database in case that he is the manufacturer. The service handles the communication

between the customer and the engineer and therefore the CAD files are provided on time and only

when needed. The communication feature of the service has three different dimensions, the

receiving of the CAD file, the mold characteristics (measurements, and production cycle

specifications) that the customer sends for the fault analysis and finally the report is sent to the

customer. The engineer inspects the mold together with the workers so as to conduct the fault

diagnosis. To this end, measurements are made in the mold and they are compared to the

measurements that the customer sent. In case of deviations, the engineer together with the worker

come to a conclusion related to the problems detected in the mold. By the end of the inspection

phase, the engineering department, composes the BOP and BOM lists required for the mold to be

repaired. Along with those two lists, the engineering department also composes a quotation which

reports the time and cost estimation for the malfunctions found or reported on the mold. By using

the mold maintenance tool, mold manufacturing company aims to create a database with mold

characteristics for every customer. In the current maintenance processes, each time that a

maintenance needs to take place the industry asks for additional information from the customer.

The existence of a database will support time consuming processes and the communication

between the industry and the client. A file with all the customers of the manufacturing company is

also already stored in the database. In case that the mold has been produced in the company, worker

gains access to the Bill of Processes and Bill of Materials of the mold. On the other hand, if





maintenance has been already done in the company in the past, information about the cost and time

are retrieved from the database. An estimation is made for both of these values and is integrated

in the report that the engineer sends to the customer. When the mold is delivered for the first time

in the industry, the cost and time estimation is made by the engineer. As long as the customer

approves the report that the industrial partner sent, annotations should be made by the worker based

on the 3D model visualized by the AR tool. These annotations should be relevant to the

maintenance activities that are going to take place. In the mold manufacturing company, may be

more than one workers that are able to make annotations. Every worker has a smart device and is

able to read the annotations of the others and add new ones. This is also useful during the

maintenance tasks. The responsible workers see the comments of the others so as the processes to

be done properly. Each annotation may refer to a different component of the mold and they should

be legible and in a structured way. The engineer checks the annotations made and creates the Bill

of Processes (BOP) of the maintenance. The BOP consists of a list with specific maintenance tasks,

the corresponding machine, and the related worker. This will make the maintenance tasks to be

easily transformed in real tasks in the mold manufacturing company. During the maintenance

phase in the company, the mold maintenance tool records the real processes. The time used for the

tasks is stored in the database so as to be used in the future. One of the innovations offered is that

the worker is able to pause the time if a machine breaks down or something else occurs. Therefore,

the real maintenance time is monitored and not just the starting and end time of the processes. The

AR tool that the service provides is able to visualize the mold, or the component of the mold that

need to be maintained. The position of this component in the mold is also visualized. Before and

after the mold maintenance disassembly and assembly activities take place respectively.

Instructions about these activities are also generated by the AR tool. The order of the parts, the

tools used for the assembly/disassembly are information provided by the tool. The tools refer to

simple tools such as screwdrivers or pincers in order to be easier for the worker to proceed with

these processes.





Figure 49 Application of the developed framework





CHAPTER 5. RESULTS – KEY PERFORMANCE INDICATORS

5.1 KEY PERFORMANCE INDICATORS

Key Performance Indicators could be considered as any kind of metric to quantitatively measure

the performance of a system or the processes within an organization [55].

High repetitiveness and the generation of vast amount of data characterize manufacturing

activities [56]. Manufacturing systems support the production products. Thus, the focus is on

meeting customer requirements. Through the stage of manufacturing, feedback in the form of data

is gathered. These data, if processed correctly can produce physical assets’ results. Manufacturing

performance can be directly linked to quality, availability, customer service, operating costs,

safety, and environmental integrity [57]. Maintenance and reliability business metrics provide a

clean indication of compliance to the maintenance business process.

5.2 TEST CASE CURRENT SITUATION DESCRIPTION

The mold company selected as the test case is a SME located in Greece, focusing on manufacturing

and maintenance of injection molds. In the present, mold maintenance is not documented

adequately thus creating gaps in matters such as maintenance tracking, maintenance history of sold

molds, historical data about mold assemblies and the scheduling is performed by manually by the

mold maker engineering department. More specifically, if a customer requests mold maintenance,

the physical product is sent to company’s premises along with a document about the mold





specification. Upon mold arrival, the company’s engineering department, receives the mold and in

conjunction with the technicians, the mold inspection is performed. Continuing, the engineering

department after collecting all the needed data, composes the BOP, along with a time and cost

estimation. The time and cost quotation is sent to the customer for approval. If customer’s approval

is granted, then the actual maintenance procedures take place. Finally, the corrected mold is tested

(if such an action is needed according to the company’s engineering dept. instructions) and is

returned back to the customer. Another identified gap is about following the maintenance plan

agreed during purchase contract. When the mold company sells a new mold to their customer, in

the contract a maintenance schedule is also dealt. Unfortunately, both mold company and the

customer cannot follow the dealt maintenance plan due to their production requirements. The

proposed framework has implemented an alarming tool, accessible from both stakeholders. The

system’s purpose is to alarm the customer that their mold needs to be corrected (preventive

maintenance) soon enough to reschedule their production. Finally, the last identified gap is about

unscheduled maintenance and lack of an alarming system for the mold company’s engineering

department in order to continuously keep track of all customers’ mold health. Currently, the

company, has no input from their customers, about mold health during their operation. Thus, mold

company’s production cannot be scheduled effectively, creating the possibility of mold stack in

their premises. The alarming system will serve as a tool to predict mold arrival for each customer

thus enabling company’s production planning team to prepare a long-term schedule for their

production.

5.3 EXPECTATIONS OF ADOPTING THE PROPOSED METHODOLOGY

In this section, the validation metrics for the usability testing of the tool are presented. The

validation should have two parts, the first one regarding the usability of the tools and the second

one regarding the impact on the business process under study. In order to facilitate the processes

of information gathering and validation, a questionnaire is setup regarding the pilot case.





The validation scenario for the Mold Manufacturing Use Case focuses on the case of the

configuration, design, manufacturing, and usage of a Mold PSS. This scenario will also

demonstrate the value that will be provided to the mold user in later stages of the Mold Product

Service lifecycle by demonstrating the use of the provided PS for the maintenance of the mold.

The validation scenario includes all the steps for the configuration, design, and usage of a Mold

Product Service for the maintenance of the mold, later inside the Mold’s lifecycle.

This process is described in the following subsections in the form of a validation scenario that

is provided in a sequence of discrete validation steps.

The business process will be assessed not only subjectively based on the questionnaires but

also objectively by getting some measure. Towards that end firstly a qualitative validation through

questions to the experts (company’s engineers, workers, Customers) is presented and then specific

Validation metrics for this case are presented in Table 7. Table 7 KPIs used for the test case

KPI Definition As-Is

Value

To-Be

Value

How is it measured?

Time for

Inspection

Actual time (in days) from

the announcement of the

repair request until all

information is collected

and the team is ready to

inspect the mold.

2 days 1 day Date that the inspection takes

place - Date of the first

customer contact (E-mail or

phone).

Monitored by company’s

project manager.

Time for

inspection

documentation

Actual time (in working

hours) for the inspection

findings to be documented

1.5

hours

0.5

hours

The time spent to prepare the

hand-written service sheet

and digitally transfer the

notes in a quotation. Timed

by company’s project

manager.

Maintenance

History Time

Retrieval

Actual time (in working

hours) for examining past

maintenance cases.

3 hours 0.15

hours

The time spent to retrieve

information from past





KPI Definition As-Is

Value

To-Be

Value

How is it measured?

maintenance cases. Timed by

company’s project manager.

On time

delivery time

Percentage of

maintenance jobs that

were delivered on time.

70% As-Is information available in

the legacy system (excel

based)

Total

Maintenance

Jobs

The total amount of

maintenance jobs in one

year.

64

(2016)

As-Is information available in

the legacy system (excel

based)

5.4 VALIDATION METHODOLOGY

The test case discussed above will undergo a validation process which is planned to be twofold.

The software usability, functionality and user friendliness will be evaluated with the use of

qualitative questionnaires. Additionally, the business impact of using the proposed solution is

going to be measured and quantified as objectively as possible. In order to do so, as there is no

current business process that can be directly compared to the PSS oriented process that the

proposed platform introduces to the company, the current work flow needs to be analyzed in steps

that can accommodate the use of objective metrics. Although this is mainly possible for test cases

3 and 4, there is currently no structured process of designing and managing PSSs to compare test

case 1 impact during project duration, nor an official PSS offering to the customer to compare with

test case 2. However, some metrics that can be used for assessing test cases 1 and 2 are defined,

yet the to-be figures cannot be estimated realistically.

The current service business process subject to validation is the mold maintenance process,

which is analyzed below in Table 8:





Table 8 Mold maintenance process steps

Step 1. Request for repair

The customer identifies a problem with a mold and commences the process by communicating via e-mail or phone to the mold maker the identified issues, the mold serial number or description and sends the mold to the mold making company premises.

Step 2. Inspection

The mold is delivered to the mold making company. The mold serial number is cross checked for validation by the project manager, previous cases of maintenance are retrieved, and a hard copy service sheet is prepared, including the issues identified by the customer. A team of a project manager, an engineer and an expert mold technician are gathered to inspect the mold. Issues are identified and noted on the service sheet, solutions to these issues – basically machining operations - are also proposed and noted in detail. A rough estimation of cost and lead time is also being made.

Step 3. Quoting and confirmation

The project manager cross checks similar past cases of maintenance for consistency, discusses the specific case with the production planning team with respect to the current workload and the team estimates an actual delivery time. A formal quotation is then prepared and sent to the customer, which is confirmed in due time.

Step 4. Machining Operations

Following the confirmation from the customer, machining, procurement, and other necessary operations are executed and the mold is repaired/restored to its initial state.

Step 5. Test sample production and mold delivery

The repaired mold is put in operation in the mold making company premises, in order to validate the repair and that the tool runs at the desired – agreed condition. The mold is then prepared for pick up along with the test samples, and the customer is notified that the repair is completed.

In the following figure (Figure 51), a flowchart is illustrated, representing the sequence of

actions for the validation of the proposed framework. It must be noted that the procedure is divided

into three tiers. In the first tier the test case steps are described. In the second tier the developed

tool impact is presented through its main functionalities. Finally, in tier three the outcome, thus

the measured values are recorded. Based on these results the validation was performed.





Figure 50 Workflow of the validation scenario of first prototype of the tool.

5.5 RESULTS

With the adoption of the proposed methodology derived from this thesis, the mold company,

although it is a SME, offers services to their clients, in order to maintain its competitiveness and

their customers’ loyalty. Moreover, any financial risk is eliminated due to the use of equipment

already existing in an industrial environment (tablets, smart phones etc.). However, if not the

required equipment is not existent its cost is really low to pose any kind of threat for a SME.

Also, in the test case presented in previous chapter, there was identified a gap of digital

organization for all the data required to perform maintenance. Thus, in many cases, the

maintenance procedure has been delayed in order for the maintenance contractor to acquire all the

needed data. With the implementation of the designed tool, the maintenance contractor, obtained





access to these data more quickly and more easily. The results from the tool evaluation phase,

indicated a reduction of approximately 70% for the inspection documentation.

The proposed framework also serves as a communication tool between the maintenance

contractor and the client, thus enabling them to communicate more efficiently. Based on that, and

on the use of historical data, which are constantly saved on the Cloud database, the maintenance

contractor was able to accomplish more maintenance tasks, thus making their production line even

more efficient. The quantitative results gained from the evaluation phase indicated a 56% of

increase for the tasks successfully completed.

In Figure 52 the quantitative results are presented compared to the original values given by the

test-case company.

Figure 51 Tool evaluation results





CHAPTER 6. CONCLUDING REMARKS AND OUTLOOK

The tool presented is constantly updated, so that bugs are encountered as well adding extra

functionalities which in turn provide better experience to the end user. Moreover, feedback from

actual users is needed so that weak spots of the tool can be easily spotted, and new ideas come to

the surface. Also, it must be taken in consideration that current technological progress in the field

of AR and generally in mixed reality environments, will provide in the near future capabilities that

will boost the performance of the proposed tool. The current set of ideas is discussed further in the

following section.

The first goal for AR developers is getting rid of trackers and image targets (markerless AR).

This task is currently feasible in some cases, but the results are not satisfactory enough due to

insufficient technological equipment and low feature recognition. Thus, further research and

development has to be conducted in order to accomplish a seamless connection of the virtual

content with the tangible objects. A convenient solution is the use of object recognition. Currently

based on the Microsoft Hololens sensing system the formation of a relatively accurate point cloud

is feasible. By extension, the augmentation/super imposition of the digital content on the real

environment is performed. Object recognition relies on the camera feedback, get a point cloud and

depending on what features it recognizes, a set of virtual contents is registered on the physical

environment.

The second goal is to support real time integration of CAD files to the tool. As FTP exchange

protocol is currently supported, meaning that CAD files can be exchanged between users while

communicating through the tool, is not enough, the next step is to convert them online and load

them without the need of a dedicated engineer to create the according scenes in Unity™ editor.

This task requires the integration of a file conversion tool, that will convert CAD filetypes to 3D

game engine filetypes (the kind of filetypes Unity™ and Android applications need to work with).

By accomplishing that task, instantly the application becomes lighter, as there is no longer needed

to package all the 3D geometries during releasing a version of the tool. The user would easily

select the model of equipment he wishes to work with and then the application should perform all





the required actions to get the geometries from the server and present them to the user’s device.

All is needed is access to the server, implying the necessity of an internet connection or LAN

connection existence if the server is local.

As discussed in the previous paragraphs, client uses the customizer tool to order new pieces of

equipment or order changes to the existing equipment. For this purpose, the client is required to

provide at least the engineering drawings or complete 3D designs. In either case, a sandbox

functionality would provide much help in creating or modifying an existing piece of equipment in

real-time. For example, in the test case scenario, of the mold industry, described in the previous

section, it would be very helpful if the client could modify the shape of a mold’s cavity in such a

way that the produced molding could transform in the desired form, instead of providing a set of

drawings or even invest time to design the new cavity from the scratch.

Another goal is to use QR code system for equipment recognition. QR code system also known

a matrix barcode [59]. This system offers fast readability and great storage capacity, making it

suitable for industrial applications [60].

Finally, the application, the server and all of its components are controlled by an administrator.

Thus, the need for the suitable administrative tools rises. Currently, a set of administrative tools

has been developed, but over time as the application obtains more added value, expansion and

adaptation of the existing administrative tools might be required. It must be also taken in

consideration that a huge amount of raw data is accumulated over time, making necessary the

development of a series of sub-tools, for efficiently managing these data.

Finally, a topic of great importance is data security. The proposed framework has to manage a

big amount of data, which could be considered as sensitive. As a result, encryption techniques

have to be implemented. In the current state, of the developed framework, the data exchanged

between the smart device and the server, are encoded using the Base64 encoding system [61],

which is provides the least amount of protection.





ACKNOWLEDGEMENT

This work has been partially supported by the H2020 EC funded project “An Integrated

Collaborative Platform for Managing the Product-Service Engineering Lifecycle – ICP4Life” (GA

No: 636862). The author would like to thank the industrial partner involved in this research work.

PUBLICATIONS RELEVANT TO THE THESIS

Part of the research work presented in this thesis has been published as a research paper in the

proceedings of the 16th IFAC Symposium on Information Control Problems in Manufacturing

(INCOM 2018), which was held in Bergamo, Italy, 11–13 June 2018. The conference proceedings

have been published by Elsevier as a special issue edited by Marco Macchi, László Monostori,

Roberto Pinto. The full citation of the published manuscript is the following:

• Mourtzis, D., Angelopoulos, J., & Boli, N. (2018). Maintenance assistance application of

Engineering to Order manufacturing equipment: A Product Service System (PSS)

approach. IFAC-PapersOnLine, 51(11), 217-222. DOI:

https://doi.org/10.1016/j.ifacol.2018.08.263

Below is a list of other research works published in scientific journals with high impact factor, and

proceedings of conferences indexed by Scopus.

• Mourtzis, D., Siatras, V., & Angelopoulos, J. (2020). Real-time remote maintenance

support based on augmented reality (AR). Applied Sciences, 10(5), 1855. DOI:

https://doi.org/10.3390/app10051855

• Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2020). Intelligent Predictive

Maintenance and Remote Monitoring Framework for Industrial Equipment based on

https://doi.org/10.1016/j.ifacol.2018.08.263

https://doi.org/10.3390/app10051855





Mixed Reality. Frontiers in Mechanical Engineering, 6, 99. DOI:

https://doi.org/10.3389/fmech.2020.578379

• Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2020). A framework for automatic

generation of augmented reality maintenance & repair instructions based on convolutional

neural networks. Procedia CIRP, 93, 977-982. DOI:

https://doi.org/10.1016/j.procir.2020.04.130

• Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2021). Collaborative manufacturing

design: a mixed reality and cloud-based framework for part design. Procedia CIRP, 100,

97-102. DOI: https://doi.org/10.1016/j.procir.2021.05.016

• Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2020). Recycling and retrofitting for

industrial equipment based on augmented reality. Procedia CIRP, 90, 606-610. DOI:










REFERENCES

1 Mourtzis, D., & Doukas, M. (2014). The evolution of manufacturing systems: From

craftsmanship to the era of customisation, Design and Management of Lean Production

Systems, V. Modrak, P. Semanco. Pennsylvania: IGI Global. DOI: https://doi.org/10.4018/978-

1-4666-5039-8.ch001

2 G. Chryssolouris and F. Frederick, Manufacturing Systems : Theory and Practice Sprin ger.

1991.

3 Johnson, J. (2012). "The Master Key": L.Frank Baum envisions augmented reality glasses in

1901. Mote & Beam.

4 Wang, X., Ong, S. K., & Nee, A. Y. (2016). A comprehensive survey of augmented reality

assembly research. Advances in Manufacturing, 4(1), 1-22. DOI:

https://doi.org/10.1007/s40436-015-0131-4

5 Mourtzis, D., Zogopoulos, V., & Vlachou, E. (2017). Augmented reality application to support

remote maintenance as a service in the robotics industry. Procedia CIRP, 63, 46-51. DOI:


6 Mourtzis, D. (2020). Machine Tool 4.0 in the Era of Digital Manufacturing. 17th International

Multidisciplinary Modeling & Simulation Multiconference. DOI:

https://doi.org/10.46354/i3m.2020.emss.060

7 Romero, D., Bernus, P., Noran, O., Stahre, J. and FastBerglund, Å. (2016). The operator 4.0:

human cyber-physical systems & adaptive automation towards human-automation symbiosis

work systems. IFIP international conference on advances in production management systems.

Springer, Cham. DOI: https://doi.org/10.1007/978-3-319-51133-7_80

8 Liu, C. and Xu X. (2017). Cyber-physical Machine Tool – The Era of Machine Tool 4.0,

Procedia CIRP, 63, 70-75, DOI: https://doi.org/10.1016/j.procir.2017.03.078

9 Benotsmane, R., Kovács, G. and Dudás, L. (2019). Economic, Social Impacts and Operation of

Smart Factories in Industry 4.0 Focusing on Simulation and Artificial Intelligence of

Collaborating Robots. Social Sciences, 8(5), 143. DOI: https://doi.org/10.3390/socsci8050143

https://doi.org/10.4018/978-1-4666-5039-8.ch001

https://doi.org/10.4018/978-1-4666-5039-8.ch001

https://doi.org/10.1007/s40436-015-0131-4


https://doi.org/10.46354/i3m.2020.emss.060

https://doi.org/10.1007/978-3-319-51133-7_80


https://doi.org/10.3390/socsci8050143





10 Kollatsch, C., Schumann, M., Klimant, P., Wittstock, V., & Putz, M. (2014). Mobile augmented

reality based monitoring of assembly lines. Procedia Cirp, 23, 246-251. DOI:


11 Kiritsis, D., Bufardi, A., & Xirouchakis, P. (2003). Research issues on product lifecycle

management and information tracking using smart embedded systems. Advanced Engineering

Informatics, 17(3–4), 189–202. https://doi.org/10.1016/j.aei.2004.09.005

12 Mourtzis, D., Vlachou, E., Milas, N., & Xanthopoulos, N. (2016). A Cloud-based Approach for

Maintenance of Machine Tools and Equipment Based on Shop-floor Monitoring. Procedia

CIRP, 41, 655–660. https://doi.org/10.1016/j.procir.2015.12.069

13 Mourtzis D. (2020). Simulation in the design and operation of manufacturing systems: state of

the art and new trends, International Journal of Production Research, 58:7, 1927-1949, DOI:

https://doi/org/10.1080/00207543.2019.1636321

14 Zhu Z. and Xu X. (2020). User-centered information provision of Cyber-Physical Machine

Tools, Procedia CIRP, 93, 2020, 1546-1551, DOI: https://doi.org/10.1016/j.procir.2020.04.091

15 Mourtzis, D., Milas, N., Vlachou, E. and Liaromatis, J. (2018). Digital transformation of

structural steel Mourtzis D. | 429 manufacturing enabled by IoT-based monitoring and

knowledge reuse. International Conference on Control, Decision and Information Technologies

CoDIT’18. 295-301, DOI: https://doi.org/10.1109/CoDIT.2018.8394874

16 Liu, C. and Xu X. (2017). Cyber-physical Machine Tool – The Era of Machine Tool 4.0,

Procedia CIRP, 63, 70-75, DOI: https://doi.org/10.1016/j.procir.2017.03.078

17 BCG Analysis (2020). Embracing Industry 4.0 and Rediscovering Growth, Available online:

https://www.bcg.com/capabilities/operations/embracing-industry-4.0-rediscovering-

growth.aspx (Accessed 10/06/2021)

18 Azuma, R. T. (1997). A survey of augmented reality. Presence: teleoperators & virtual

environments, 6(4), 355-385.

19 Baines, T. S., Lightfoot, H. W., Evans, S., Neely, A., Greenough, R., Peppard, J., ... & Wilson,

H. (2007). State-of-the-art in product-service systems. Proceedings of the Institution of

Mechanical Engineers, Part B: journal of engineering manufacture, 221(10), 1543-1552.

20 Meier H., Roy R., & Seliger, G. (2010). Industrial Product-Service Systems-IPS2. CIRP Annals

- Manufacturing Technology, 607-627.


https://doi.org/10.1016/j.aei.2004.09.005


https://doi/org/10.1080/00207543.2019.1636321


https://doi.org/10.1109/CoDIT.2018.8394874


https://www.bcg.com/capabilities/operations/embracing-industry-4.0-rediscovering-growth.aspx

https://www.bcg.com/capabilities/operations/embracing-industry-4.0-rediscovering-growth.aspx





21 Abernathy, M., Houchard, J., Puccetti, M., & Lambert, J. (1993). Debris Correlation Using the

Rockwell WorldView System. In Proceedings of 1993 Space Surveillance Workshop (Vol. 30,

pp. 189-195).

22 Lamata, P., Ali, W., Cano, A., Cornella, J., Declerck, J., Elle, O. J., ... & Gómez, E. J. (2010).

Augmented reality for minimally invasive surgery: overview and some recent advances.

23 Roberts, G. W., Evans, A., Dodson, A., Denby, B., Cooper, S., & Hollands, R. (2002, April).

The use of augmented reality, GPS and INS for subsurface data visualization. In FIG XXII

International Congress (Vol. 4, pp. 1-12).

24 Hammad, A., Garrett, Jr, J. H., & Karimi, H. A. (2002). Potential of mobile augmented reality

for infrastructure field tasks. In Applications of Advanced Technologies in Transportation

(2002) (pp. 425-432).

25 Thomas, B., Piekarski, W., & Gunther, B. (1999). Using augmented reality to visualise

architecture designs in an outdoor environment. International Journal of Design Computing

Special Issue on Design Computing on the Net (DCNet), 1(4.2).

26 Dunston, P., Wang, X., Billinghurst, M., & Hampson, B. (2003). Mixed reality benefits for

design perception. Nist Special Publication SP, 191-196.

27 Shen, J., Wu, Y., & Liu, H. (2001). Urban planning using augmented reality. Journal of urban

planning and development, 127(3), 118-125.

28 Behzadan, A. H., & Kamat, V. R. (2005, December). Visualization of construction graphics in

outdoor augmented reality. In Proceedings of the Winter Simulation Conference, 2005. (pp. 7-

pp). IEEE.

29 Oculus Rift, Available online: https://www.oculus.com/ (Accessed 10/06/2021)

30 Vuzix Star 1200 XL, Available online: https://www.slashgear.com/vuzix-star-1200-xl-see-

through-ar-headset-gets-even-more-immersive-18248074/ (Accessed 10/06/2021)

31 Vuzix Blade-Enterprize, Available online: https://www.vuzix.com/Products/Blade-Enterprise

32 Woodward, C., Kuula, T., Honkamaa, P., Hakkarainen, M., & Kemppi, P. (2014).

Implementation and evaluation of a mobile augmented reality system for building maintenance.

In Proceedings of the 14th International Conference on Construction Applications of Virtual

Reality, CONVR 2014 (pp. 306-315)

https://www.oculus.com/

https://www.slashgear.com/vuzix-star-1200-xl-see-through-ar-headset-gets-even-more-immersive-18248074/

https://www.slashgear.com/vuzix-star-1200-xl-see-through-ar-headset-gets-even-more-immersive-18248074/

https://www.vuzix.com/Products/Blade-Enterprise





33 Pintzos, G., Rentzos, L., Papakostas, N., & Chryssolouris, G. (2014). A novel approach for the

combined use of AR goggles and mobile devices as communication tools on the shopfloor.

Procedia CIRP, 25, 132-137. DOI: https://doi.org/10.1016/j.procir.2014.10.021

34 Mourtzis, D., Vlachou, E., & Zogopoulos, V. (2018). Mobile apps for providing Product-

Service Systems and retrieving feedback throughout their lifecycle: A robotics use case.

International Journal of Product Lifecycle Management, 11(2), 116-130. DOI:

https://dx.doi.org/10.1504/IJPLM.2018.092821

35 Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2020). A framework for automatic

generation of augmented reality maintenance & repair instructions based on convolutional

neural networks. Procedia CIRP, 93, 977-982. DOI:


36 8 Best augmented reality SDK for AR development for iOS and Android in 2017. (2017,

January). (ThinkMobiles) Retrieved September 2017, from https://thinkmobiles.com/blog/best-

ar-sdk-review/

37 Vuforia PTC. (2017). (Vuforia) Available online: https://www.ptc.com/en/products/vuforia

(Accessed 10/06/2021)

38 Wikitude SLAM Technology (2021), Available online: https://www.wikitude.com/wikitude-

slam/#:~:text=SLAM%20(simultaneous%20localization%20%26%20mapping),and%20Mark

er%2Dbased%20AR%20Projects (Accessed 10/06/2021)

39 Unity - Game Engine. (2021). Available online: https://unity3d.com/ (Accessed 10/06/2021)

40 Visual Studio IDE, Available online: https://visualstudio.microsoft.com/ (Accessed

10/06/2021)

41 Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1995, December). Augmented reality:

A class of displays on the reality-virtuality continuum. In Telemanipulator and telepresence

technologies (Vol. 2351, pp. 282-292). International Society for Optics and Photonics.

42 Siltanen, S. (2012). Theory and applications of marker-based augmented reality: Licentiate

thesis.

43 Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2020). Intelligent Predictive Maintenance

and Remote Monitoring Framework for Industrial Equipment based on Mixed Reality. Frontiers

in Mechanical Engineering, 6, 99. DOI: https://doi.org/10.3389/fmech.2020.578379


https://dx.doi.org/10.1504/IJPLM.2018.092821


https://thinkmobiles.com/blog/best-ar-sdk-review/

https://thinkmobiles.com/blog/best-ar-sdk-review/

https://www.ptc.com/en/products/vuforia

https://www.wikitude.com/wikitude-slam/#:%7E:text=SLAM%20(simultaneous%20localization%20%26%20mapping),and%20Marker%2Dbased%20AR%20Projects



https://unity3d.com/

https://visualstudio.microsoft.com/






44 Goedkoop, M. (1999). Product service systems. Ecological and economic basis.

45 Tukker, A. (2004). Eight types of product–service system: eight ways to sustainability?

Experiences from SusProNet. Business strategy and the environment, 13(4), 246-260.

46 Shimomura, & Akasaka. (2013). Toward Product-Service System Engineering: New System

Engineering for PSS Utilization. In H. Meier, Product- Service Integration for Sustainable

Solutions (pp. 27-40). Springer.

47 Meier, H. V. (2011). Industrial Product-Service Systems (IPS2). The International Journal of

Advanced Manufacturing Technology, 52(9), 1175-1191.

48 Rese, M., Karger, M., & Strotmann, W. C. (2009). The dynamics of Industrial Product Service

Systems (IPS2) - using the Net Present Value Approach and Real Options Approach to improve

life cycle management. CIRP Journal of Manufacturing Science and Technology, 1(4), 279–

286. https://doi.org/10.1016/j.cirpj.2009.05.001

49 Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2020). Recycling and retrofitting for

industrial equipment based on augmented reality. Procedia CIRP, 90, 606-610. DOI:


50 EMCOR Group, Available online: https://www.emcorgroup.com/ (Accessed 10/06/2021)

51 SITEC, Available online: https://www.sitec-technology.de/home (Accessed 10/06/2021)

52 Pak, T. (n.d.). Tetra Pak ® Maintenance Services Proactive care to secure performance.

Available online: https://www.tetrapak.com/solutions/services/service-portfolio/maintenance-

services (Accessed 10/06/2021)

53 T. Mahboob, M. Z. (2016). Adopting Information Security Techniques for Cloud Computing -

A Survey. Proceedings - 2016 1st International Conference on Information Technology,

Information systems and Electrical Engineering, ICITISEE, (pp. 7-11).

54 Ayers, R. (2016, September 12). 5 Advantages of Using Encryption Technology for Data

Protection. (SmartDataCollective) Retrieved September 11, 2017, from

https://www.smartdatacollective.com/5-advantages-using-encryption-technology-data-

protection/

55 Rosato, D. V., & Rosato, M. G. (2012). Injection molding handbook. Springer Science &

Business Media.

https://doi.org/10.1016/j.cirpj.2009.05.001


https://www.emcorgroup.com/

https://www.sitec-technology.de/home

https://www.tetrapak.com/solutions/services/service-portfolio/maintenance-services

https://www.tetrapak.com/solutions/services/service-portfolio/maintenance-services

https://www.smartdatacollective.com/5-advantages-using-encryption-technology-data-protection/

https://www.smartdatacollective.com/5-advantages-using-encryption-technology-data-protection/





56 Meier, H., Lagemann, H., Morlock, F., & Rathmann, C. (2013). Key performance indicators

for assessing the planning and delivery of industrial services. Procedia CIRP, 11, 99-104. DOI:


57 Mourtzis, D., Boli, N., & Fotia, S. (2017). Knowledge-based estimation of maintenance time

for complex engineered-to-order products based on KPIs Monitoring: a PSS Approach.

Procedia CIRP, 63, 236-241. DOI: https://doi.org/10.1016/j.procir.2017.03.317

58 Mourtzis, D., Fotia, S., & Vlachou, E. (2017). Lean rules extraction methodology for lean PSS

design via key performance indicators monitoring. Journal of manufacturing systems, 42, 233-

243. DOI: https://doi.org/10.1016/j.jmsy.2016.12.014

59 Karrach, L., Pivarčiová, E., & Božek, P. (2020). Identification of QR Code Perspective

Distortion Based on Edge Directions and Edge Projections Analysis. Journal of Imaging, 6(7),

67. DOI: https://doi.org/10.3390/jimaging6070067

60 Wada, T., Kishimoto, Y., Nakazawa, E., Imai, T., Suzuki, T., Arai, K., & Kobayashi, T. (2020,

October). Multimodal User Interface for QR Code based Indoor Navigation System. In 2020

IEEE 9th Global Conference on Consumer Electronics (GCCE) (pp. 343-344). IEEE. DOI:

https://doi.org/10.1109/GCCE50665.2020.9292026

61 The Base16, Base32, and Base64 Data Encodings. (2006). (The Internet society) Available

online: https://www.rfc-editor.org/rfc/pdfrfc/rfc4648.txt.pdf (Accessed 10/06/2021)



https://doi.org/10.1016/j.jmsy.2016.12.014

https://doi.org/10.3390/jimaging6070067

https://doi.org/10.1109/GCCE50665.2020.9292026

https://www.rfc-editor.org/rfc/pdfrfc/rfc4648.txt.pdf

DESIGN AND DEVELOPMENT OF PRODUCT -SERVICE SYSTEM …

Documents