Summary - unclezan.comunclezan.com/downloads/pathfast/Report.docx  · Web viewPATHFASTTM (from MITSUBISHI CHEMICAL EUROPE) is a fully automatic immunoassay analyzer, which combines

3D Programming CI7500Assignment 2 Game/App Development

Hand in date: Dec 11, 2017

Author: Gabriele Santolini

Dropbox link:

Targets link: ARManual\Assets\Editor\Vuforia\ImageTargetTextures\target_images

ContentsSummary................................................................................................................................3Introduction...........................................................................................................................4Application design...............................................................................................................5

Concept...............................................................................................................................5Interpretation....................................................................................................................5Usage...................................................................................................................................7

Application Development...............................................................................................10Code analysis..................................................................................................................11

Competitive analysis........................................................................................................17Features not implemented.........................................................................................17Bug reports......................................................................................................................18

Bibliography.........................................................................................................................19

SummaryPATHFASTTM (from MITSUBISHI CHEMICAL EUROPE) is a fully automatic immunoassay analyzer, which combines the progressive chemiluminescence technology with the patented Magtration® technology.The aim of this project is to help the users in the process of calibration of this equipment to ensure its maximum accuracy and functionality provide the hospital staff with a more efficient virtual artefact.

Report contentThe document contains two main chapter: Design and Development

IntroductionThe usage of PATHFAST machines, within Italian hospitals, is increasing steadily; it is ideal at the point of care or in emergency labs, where is crucial to reduce the delay between the operations and to ensure the best and most accurate care.PATHFAST needs to be calibrated regularly. The first calibration is demonstrated by a specialist; the hospital staff also receive assistance and an instruction manual.To operate and maintain this machine no special skills are required; however, the process may not be as obvious for everyone and the users frequently need to contact the company, causing inconveniences bot for the client and the seller.

I was recently contacted by a company that presented me with this issue. They showed me a video currently being used as a demonstration of the machine usage: it was a 3D animated model showing all the key steps of the process. However, I considered the video inconsistent:

It lacked important information during key stages of the process. It was a silent video, thus not engaging for the users. The video was long and dispersive.

I believe that nowadays there are quicker, more immersive and effective solution: therefore, I offered the company an interactive application in AR.

The original video covered every process of the usage and each one will be reworked separately.This project will focus on the development of an augmented reality application for the calibration process.

Application design

Concept

“AR manual” is a virtual tutor to enhance the instruction manual. It uses a marker based augmentation to provide the users with intuitive information about the calibration process through an immersive experience. It is a mobile application which makes use of the device camera and simple gestures (swiping, touches, etc.).

Interpretation

I have designed this application through my interpretation of the machine’s manual of instructions, as I was not given a brief to refer to. I was not familiar with the machine, I had no knowledge about the calibration process and I received unclear information about it. This brought some conflicts between the design and the final product. Also, when the project was finished, I realized the design could be improved for better performance, functionality and code reusability.

Within the manual, the calibration process is divided into two major stages: “data insert”(to be performed only for new reagent batches) and “parameters calibration”; therefore, the process was divided in two scenes within the application.The two stages are then divided into many minor steps for the user to perform. Each step was categorized by its relevance and designed accordingly: the users are asked for interactive input for the steps they need to memorise (like what button they need to press on the machine screen), whereas other steps make use of automatic animations avoiding the users to be distracted by the interactions.

User

Get reagent kit

Prepare the calibration solution

«include»

Register new batch«extend»

Calibrate tool

«include»

Load the cartridges rack

«include»

Empty/clean the bin for the used tips«extend»

Fill the designed container with the solution

«include»

Load the tips rack

«include»

Select the test to calibrate

«include»

Adjust the cartridges rack«extend»

Wait for printed results

«include»

Stop the assay (EMERGENCY)«extend»

UserInterface Mobile

Start application

Wait for application to load

Scan first mark

Wait for scene to load

Select mode

Select PATHFAST menu option

Start the camera

Display next instruction

Continue input

Play animation

Wait for animation

Give next instruction

Scan second mark

Wait for scene to load

Continue input

Play animation

Wait for animation

Select object

Give information aout the object

Continue input

Continue input

Play long animation

Wait for animation

Control animation flow

Select test type

Give next instruction

Close application

Register new batch

Load cartridge

Loop

Calibration use case Calibration sequence diagram

Usage

On the starting screen of the application the users can choose between two modes: enhanced manual or explore.

Starting screen

Enhanced manual: in the instruction manual there are marks on each key stage; pointing the camera to the mark give access to the corresponding scene. Each scene is an interactive demonstration of that specific stage. The users can access each scene in any order they need, however, should they skip any of the scene they will be given information on how to access the previous one. In the same way, once the scene is completed, they are given instructions to find the mark for the next scene.

Go to previous stage Go to next stage

In every scene, there is a dialog box to guide the user through the steps. The size of the text can be increased by pulling the zoom slider on the side.

Dialog box

Some of the steps need the users’ input to be completed; as additional support to the navigation, there are gesture tips to indicate which object the users can interact with.

Gesture tip

Other steps are an automatic demonstration of the process: the users can manage the flow of the animations by using the controls, allowing them to review the training and to decide the pace of the animation.

Animation controls

Application DevelopmentThe application was fully developed using Unity 3D with Vuforia SDK. All the assets are original and developed using Autodesk Maya and Adobe Photoshop. The references for the models were either provided from the client or retrieved from the PATHFAST website. The fbx files were imported into Unity through the Maya/Unity new integration.

The main difficulties encountered during the development of the project were:

- Finding a good naming convention.- The reproduction of a procedure of which I had no knowledge about.- Working in an Augmented Reality environment.- A memory leak issue.

The development was mainly slowed down by a big memory leak issue that caused the graphic memory to increase indefinitely when using any UI element (even a simple panel with a content size fitter). Initially, I thought I was not handling the UI elements properly when changing from one screen to another (PATHFAST screen). Then, I realized that the problem was mainly caused by the dialog box on the top of the device screen (the instructions).

In Unite Europe 2017 talk, Ian Dundore explains brilliantly some of the operation that Unity does not handle properly, causing major performance decreases. The way UI elements are handled in Unity is quite performance heavy: every time any element on any “Canvas” gets “dirty” (enabled/disabled, moved, resized, etc.); as a result, Unity looks for every component in any element until it reaches the root of the “Canvas” using “foreach” loops and that causes heavy drops of performance in big UI hierarchies. However, following the good practices explained in the talk, did not change the performance slightly. The issue could not be ignored, as it caused the test to be ineffective: every time the application needed to be tested, it could not be tested through the Unity editor, because that would cause the application to cause after 3 or 4 times. Making a new build of the application for every test was not a solution either and using “Unity remote” did not help. What was discovered from the experience is that the issue was caused by Unity’s profiler window itself and running the application without monitoring the resources would not cause the memory leak.

Code analysisThis section contains the explanation to the main parts of the code. It is ordered coherently to the runtime execution order, to give the reader a more efficient way to read the code. For each class, it is showed and explained only the most important methods and parameters, except for small classes.

Starting screenpublic class LanguageOption : MonoSingleton<LanguageOption> {

private void Start() { Vuforia.VuforiaBehaviour.Instance.enabled = false; } public void SelectLanguage(Dropdown target) { Utility.language = (Language)(target.value - 1); }

public void StartApplication() { SceneManager.LoadScene(1); }}

The starting screen contains a drop-down menu and a button that correspondingly access the two methods above: SelectLanguage and StartApplication. The class above takes care of disabling Vuforia playmode as soon as the application starts, sets the language of the dialog box, and start the play scene.

This class was only created because of the development delay explained before and implementing these features in an existing class would have costed more time. I preferred to save some time to fix other bugs and improve the overall experience.

Main Classpublic class Main : MonoSingleton<Main> {

// Use this for initialization void Start () { Vuforia.VuforiaBehaviour.Instance.enabled = true; OnGestureTipChangeEvent += OnGestureTipChangeHandler;

rsm = ResourceManager.Instance; animMng = AnimationManager.Instance;

Utility.StringLanguageSetter(); //SwipingUp = true; }

// Update is called once per frame void Update () { if (Input.GetMouseButton(0) && touchesEnabled) GetPhysicsTouches(); }

void GetPhysicsTouches() { Ray ray = arCamera.ScreenPointToRay(Input.mousePosition); RaycastHit hit; //Tap behaviour if (Input.GetMouseButtonDown(0)) { if (Physics.Raycast(ray, out hit)) { #region SELECTABLE_ON_CLICK/TAP } } //Swipe behaviour else if (Input.GetMouseButton(0)) { if (Physics.Raycast(ray, out hit)) { #region SWIPEABLE_ON_SWIPE } } }

void OnGestureTipChangeHandler(GestureType newValue) { if (newValue == GestureType.SWIPE_UP) { StartCoroutine(animMng.SwipeUp()); } else if (newValue == GestureType.TAP) { StartCoroutine(animMng.Tap()); } else if (newValue == GestureType.SWIPE_RIGHT) { StartCoroutine(animMng.SwipeRight()); } }

public void SetExtendedTracking(Text target) { int result = TrackableManager.Instance.SetExtendedTracking();

if (result == 0) { target.text = "ExtTracking OFF"; } else if (result == 1) { target.text = "ExtTracking ON"; } }}

This is the class responsible for managing the main interactions in the scene, it calls the initialization for the resources needed in the scene and it works as a director between the other classes.

At the start, it switches the Vuforia playmode on again, enabling the users to track the marks defined in the database. It registers a “handle method” for the event ”OnGestureTipChangeEvent” and initializes the resources as well as the text in the selected language.

Every frame, the Update method checks if the user interacted with any of the interactive objects available in the scene. Firstly, it checks for click/tap events if there is any occurrence, it makes sure that the interactions are enabled in the scene. Interactions are disabled during the long animation of the second stage to avoid the users triggering events that should not be triggering at that specific time. Finally, if the interactions are not disabled, it makes uses of Physics.Raycast to search for interactive objects. If the user hits any of them, their behaviour get triggered based on their type: "Swipeable" and "Selectable".

The OnGestureTipChangeHandler takes care of playing, through a coroutine, the animation of the gesture tips in the scene every time the OnGestureTipChangeEvent event gets triggered.

The SetExtendedTracking method gets called by a UI button on the bottom left corner of the screen. It toggles the extended tracking feature of the

Vuforia “ImageTargetBehaviour” class. It is used for debugging, to give the user the ability to have more control over the stability of the models (keep it on for playing the stage, turn it off when scanning another target).

public class PATHFASTScreen : MonoSingleton<PATHFASTScreen>

{ public Transform menu; //Menu content Transform subMenu1; public Transform inserimentoMC; public Transform datiMC; public Transform analisi; public Transform analisiHome; public Transform qalQcCampione;

//Menu content handlers public static Transform previousContent; public static Transform currentContent; public Transform CurrentContent { get { return currentContent; } set { if (currentContent == value) return; previousContent = currentContent; currentContent = value; if (OnContentChangeEvent != null) OnContentChangeHandler(currentContent); } }

void Start() { OnContentChangeEvent += OnContentChangeHandler; OnStepChangeEvent += OnStepChangeHandler; subMenu1 = menu.Find("SubMenu1"); SetUpButtons(); dialogBox = ResourceManager.Instance.dialogBox; dialogText = dialogBox.GetComponentInChildren<Text>();

animator = AnimationManager.Instance.scanAnimator; continueBtn = ResourceManager.Instance.continueBtn;

currentContent = analisi; }

This class contains the behaviour of PATHFAST’s interface. The menu contains the whole hierarchy of PATHFAST’s menu developed within Unity’s editor. The //Menu content fields contain the actual Transform of each content page contained in the menu and they are managed through the event change handler OnContentChangeHandler. Whenever the value of CurrentContent gets set, if the new value is different from the old value, the value of the old content is stored in the variable previousContent, the new content is updated, and the event gets triggered.

The different steps of the stage are handled in the same way, using OnStepChangeEvent, based on the current step number, the scene gets updated appropriately (see code below).

void OnStepChangeHandler(int stepN) { //TODO find a better position for these animator.ResetTrigger("BarcodeTrigger"); animator.ResetTrigger("ScannerTrigger"); switch (stepN) { case 1: CurrentContent = analisiHome; dialogText.text = Utility.SCAN_DIALOG_1; dialogBox.GetComponent<Animator>().SetTrigger("FadeIn"); break; case 2: CurrentContent = analisi; dialogText.text = Utility.SCAN_DIALOG_2; break; case 3: CurrentContent = inserimentoMC; dialogText.text = Utility.SCAN_DIALOG_3_1; Main.Instance.GestureTip = GestureType.SWIPE_UP; break; case 4: dialogText.text = Utility.SCAN_DIALOG_4; CurrentContent = datiMC; break; case 5: dialogText.text = Utility.SCAN_DIALOG_5; StartCoroutine(FadeOut()); break; case 6: dialogText.text = "Insert valid text here"; AnimationManager.Instance.dBoxAnimator.SetTrigger("FadeIn"); Main.Instance.GestureTip = GestureType.SWIPE_RIGHT; CurrentContent = analisi; break; case 7: CurrentContent = qalQcCampione; break; default: Debug.LogError("Entered invalid step!"); break; } }

Competitive analysis

Having a background in game development, this was the first time I developed an application based mainly on UI and animations. On top of that, AR was a completely new technology to explore for me. I can not say I am happy with the result, but I am happy with knowledge gained from the project.

Finding an effective naming convention was not easy because the application was designed for an Italian company while the project was developed in English, the PATHFAST’s manual provided me by the client was also in Italian. Swapping continuously from one language to another was confusing considering I was not familiar with the content and the hospital terminology in general. Also, most of the time was difficult to find a word that would make sense and would keep the project readable for both me and the reader.

Features not implemented

Some of the features mentioned in the design or not, that during the development process were set apart to give priority to more important tasks:

For the instructions, it was originally intended to use a humanoid avatar with a voiceover, however, in the end it got replaced by the transparent dialog box. The solution was successful as it did not take too much space in the viewport being transparent and resizable. But after running some usability tests I realized this would only work with right background (for example a white surface). Also, sometimes the users were too concerned about scanning the mark and position the mobile correctly to see the augmented models that they would not even notice the instructions.

More tips (gesture, arrows, glowing effects, etc.) are needed to ensure a flawless navigation.

Indication for the focus of the animation currently being played. Animation controls, although during the main animation in the

second stage, the animation can be paused by pressing “P” on PC or by tapping with two fingers on mobile.

Model free exploration mode. Something like Tomb Raider’s inventory, where the user could select the model to augment and then explore it.

Mixing Vuforia with device motion tracking it could be created an AR experience that uses mixed augmentation techniques to allow

the users more control over the AR world and more stability after augmenting the models.

Instructions on the second stage. Full working PATHFAST’s screen for the users to explore. Feedback on completing the procedure

Bug reports

The gesture tips have unwanted behaviours, if you lose focus of the scene they might disappear.

The last tap gesture in the second stage does not disappear because of the conflict between the animation and the coroutine.

Some values do not reset correctly, so it might be better to restart the app after finishing the stages. Especially when changing from stage 2 to stage 1.

BibliographyMemory leak issue: https://issuetracker.unity3d.com/issues/possible-memory-leak-for-development-mode-standaloneUnite Europe 2017: https://www.youtube.com/watch?v=_wxitgdx-UI&t=1426sPATHFAST: http://www.pathfast.eu/pathfast-immunoanalyzer