GOAL-SEEKING DECISION SUPPORT SYSTEM TO EMPOWER PERSONAL WELLNESS MANAGEMENT A Dissertation Presented to The Graduate Faculty of The University of Akron In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy Mukesh Kumar Chippa December, 2016
135
Embed
GOAL-SEEKING DECISION SUPPORT SYSTEM TO EMPOWER PERSONAL WELLNESS
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
GOAL-SEEKING DECISION SUPPORT SYSTEM TO EMPOWER PERSONAL
WELLNESS MANAGEMENT
A Dissertation
Presented to
The Graduate Faculty of The University of Akron
In Partial Fulfillment
of the Requirements for the Degree
Doctor of Philosophy
Mukesh Kumar Chippa
December, 2016
GOAL-SEEKING DECISION SUPPORT SYSTEM TO EMPOWER PERSONAL
WELLNESS MANAGEMENT
Mukesh Kumar Chippa
Dissertation
Approved:
AdvisorDr. Shivakumar Sastry
Committee MemberDr. Nghi H. Tran
Committee MemberDr. Igor Tsukerman
Committee MemberDr. William H. Schneider IV.
Committee MemberDr. Victor Pinheiro
Accepted:
Interim Department ChairDr. Joan Carletta
Interim Dean of the CollegeDr. Donald P. Visco
Dean of the Graduate SchoolDr. Chand K. Midha
Date
ii
ABSTRACT
Obesity has reached epidemic proportions globally, with more than 1 billion adults
overweight - at least 300 million of them clinically obese and is a major contributor
to the global burden of chronic disease and disability. This can also be associated
with the rising health care costs with in USA alone accounting for more than 75% of
health care costs dedicated to Diabetes and Hypertension. While there are various
technological advancements in building various fitness tracking devices such as fitbit,
etc, and more and more corporations offering wellness programs, they have not been
able to create a long term change in the life style of its users. One of the primary
reasons, such devices fail to create an impact is that these devices are not personalized.
The challenge in keeping healthy people healthy and making them intrin-
sically motivated to manage their own health is at the center of Personal Wellness
Management. In this dissertation, this problem is presented as a decision making
under uncertainty where the participant takes an action at each discrete time steps
and the outcome of the action is uncertain. In this setting, under reasonable set of
assumptions the problem is formulated as a Completely Observable Markov Decision
process and a Partially Observable Markov Decision Process. The results presented
in this highlights the advantages and disadvantages of using each of these frameworks.
One of the major challenges in formulating the wellness problem in the above
mentioned frameworks is the need for clinically validated data. Also, the solutions
proposed for solving the formulated problem, solve for an optimal solution. While it
iii
may be unrealistic to find such experimentally validated data, it is also known that
in solving complex problems such as the PWM, good enough solutions are sufficient.
In this dissertation, Goal-Seeking framework is presented as an alternative to the
above frameworks. Bulk of the thesis is dedicated to document how the Goal-Seeking
framework is different from other frameworks.
This dissertation identifies each of the artifacts in formulating the problem
of Personal Wellness Management. A software system architecture is laid out and
many of the existing software technologies such as Object Oriented design, MySQL,
RESTful API’s etc are leveraged in the implementation of this framework.
iv
ACKNOWLEDGMENTS
I express deep gratitude to my research advisor Dr. Shivakumar Sastry for his solid
mentorship, regular guidance in both technical and non-technical aspects, and inspi-
rational discussions during my course of doctoral studies at the University of Akron. I
am extremely grateful to my dear parents, my wife for their live, courage and endless
support.
I sincerely acknowledge the support from Dr. Victor Pinheiro and Dr. Ju-
dith A. Juvancic-Heltzel in helping me understand various human behavior models
and also in providing resources and coordinating participants during the exercise
performance data collection phase of the project.
I am thankful to all my colleagues at the Complex Engineering Systems
Laboratory(CESL) at the University of Akron, especially Arijit Ghosh, for helping me
understand Bayesian update methods, Hemanth Pidaparthy, Prakash Gaddam and
Sriharsha Vankamamidi, for their continuous support and useful discussions during
various phases of this project. My sincere gratitude also goes to the faculty members
of the Electrical and Computer Engineering department, University of Akron, for
strengthening my knowledge and skills via graduate level courses and workshops.
Without the solid foundation laid by my past teachers and mentors at the
high school level and undergraduate studies, this doctoral dissertation would not have
been a success. Finally, I sincerely acknowledge financial support from the University
v
of Akron, US National Science Foundation (NSF) under the grant #IIS-1237069 that
immensely helped to financially support my doctoral studies.
3.5 State Space for Illustrative Example . . . . . . . . . . . . . . . . . . . . 36
3.6 Three of the Nine Optimal Policies computed for MDP with re-stricted action sets shown in Figure 3.3. . . . . . . . . . . . . . . . . . . 40
3.7 State Transition Matrix used for Low Intensity Actions . . . . . . . . . 46
3.8 State Transition Matrix used for Medium Intensity Actions . . . . . . . 46
3.9 State Transition Matrix used for High Intensity Actions . . . . . . . . . 46
3.10 Observation Matrix used for Low Intensity Actions . . . . . . . . . . . 47
3.11 Observation Matrix used for Medium Intensity Actions . . . . . . . . . 47
3.12 Observation Matrix used for High Intensity Actions . . . . . . . . . . . 47
3.13 Snapshot of the POMDP policy. The policy had over 250 vectorsto represent the states and only a few are shown in this table as anillustration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
ix
4.1 The motivation of the participant affects the variance of the nutritionand exercise actions as shown below. . . . . . . . . . . . . . . . . . . . 55
5.1 The Goal Seeking Artifacts and the corresponding classes implemented 77
6.1 An illustration of the first 18 values of the 150-dimension featurevector for the Jumping Jacks exercise. . . . . . . . . . . . . . . . . . . . 85
2.1 Joint locations obtained from the Microsoft Kinect 2.0 cameras afterexecuting the skeletal extraction algorithm. . . . . . . . . . . . . . . . . 14
2.2 Overview of MDP. Decisions are made in discrete time epochs basedon the observed state (weight). At each epoch, the weight of theindividual is measured and a corresponding action is recommended.The implementation of the action results in a reward to the indi-vidual and the time advanced to the next epoch. . . . . . . . . . . . . . 17
2.3 Overview of POMDP. Decisions are made in discrete time epochsbased on an estimate of the state of the system. In each state ofthe system the system is said to emit an observation stochastically.The observation and the estimated state is mapped to an actionand recommended. The implementation of the action results in areward to the individual and the time advanced to the next epoch. . . . 19
2.4 Overview of the Goal-Seeking Paradigm. Decisions are made indiscrete time epochs. Since there is no attempt to model the statesof the system, the individual provides a set of feasible actions tothe decision-maker. The decision maker selects one of the feasibleactions based on the Reflection Mapper and establishes a limit forperformance that can be achieved in the interval of time to the nextepoch. At the next epoch, the actual performance is evaluated.If the difference between the expected performance and the actualperformance is within the tolerance bound for that individual, thedecision-maker continues to recommend actions. . . . . . . . . . . . . . 21
3.1 Quantized Action Space used for the MDP. Each action represents aspecific choice of caloric intake (nutrition) and caloric expenditure(physical activity level). . . . . . . . . . . . . . . . . . . . . . . . . . . 27
xi
3.2 An example state transition matrix computed with h = 0.8. . . . . . . . 34
3.4 Block diagram on how POMDP and MDP frameworks are used toselect an action from the action set. After constructing the pomdppolicy, at each time step, depending on the observation, selects thebest intensity level of the action. This intensity level is used tofilter the action set for the MDP framework. Using the humanweight dynamics model and user preferences, the MDP frameworkselects one action among many actions of the same intensity level. . . . 39
3.5 Shows all possible states from Initial to Target. For each state, thebars represent the action recommended. The exercise scale (red) ison the right (PAL) and nutrition (blue) scale is on left (cals/day).This policy was computed using a Linear Reward function. Therecommendation is to eat less and work out more. . . . . . . . . . . . . 42
3.6 The expected weight loss trajectory for a participant who initiallyweighs 120 Kg and follows the policy in Figure 3.5. . . . . . . . . . . . 43
3.7 Actions computed from a policy that rewarded a target rate of weightloss. Higher or lower loss is penalized. The target loss can beachieved via multiple actions. The figure illustrates one stochasticchoice that helps the participant achieve target weight. . . . . . . . . . 43
3.8 Weight loss trajectory and uncertainty regions when executed thepolicy computed for recommended weight loss. . . . . . . . . . . . . . 44
3.9 MDP policy obtained using a reward function that rewarded actionsthat were within 30% of baseline exercise activity level. . . . . . . . . . 45
3.10 Weight loss trajectory for participant not willing to perform highintensity exercises but willing to go on a calorie-restricted diet. . . . . . 45
4.3 The Reward function used in the goal seeking framework to assigncost (negative rewards) to consequences of each action that is esti-mated through reflection mapper. . . . . . . . . . . . . . . . . . . . . 60
4.6 Expected weight trajectory and its error bars for a participant withNutrition Adherence Level : 0.1 and Exercise Adherence Level: 0.1.Since the adherence level is too small, the participant does not ad-here to the recommended actions and therefore is expected to in-crease weight and never reach the target weight. . . . . . . . . . . . . . 66
4.7 Expected weight trajectory and its error bars for a participant withNutrition Adherence Level : 0.5 and Exercise Adherence Level: 0.5.Since the participant adheres to the recommended action 50% of thetime, the participant neither increases weight nor decreases weightas is evident by the large error bars. . . . . . . . . . . . . . . . . . . . 66
4.8 Expected weight trajectory and its error bars for a participant withNutrition Adherence Level : 1.0 and Exercise Adherence Level: 1.0.In this scenario, the participant adheres to each recommended ac-tion and therefore after few time steps, the goal seeking frameworkrecommends high intensity actions. Since the adherence levels arehigh, the error bars in estimating the wight trajectory to the rec-ommended action is small. . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.1 The Personal Wellness Management System Architecture Design . . . . 69
5.6 RESTful API for Scalable Deployment of PWM. The web servicesprovided for Goal Seeking resides in the server along with otherservices in the PWM wellness software suite. The client and serverinteract with each other using the JSON data format. . . . . . . . . . . 78
xiii
5.7 An example of client software initiating with a HTTP request touse the reflection mapper web service hosted on the server modulethrough RESTful api. The data required for the server is formattedinto a JSON object and is inserted in post request. The serverafter processing the request, formats its response into another JSONobject and places it into the HTTP Response that is then sent tothe client. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.1 Four Exercises that were selected from the HICT Suite for this study. . 82
6.2 Sample time-series data captured from the Kinect camera for thex-coordinate of a few joints is as shown on the left. The figure onthe right shows the trajectories of the (x, y) coordinates for eachjoint when the Jumping Jacks exercise is performed. . . . . . . . . . . . 83
6.3 Projection of all the joints on the Y Z-Plane for the Jumping Jacksexercise. This was obtained by drawing a line between the aver-age starting position and the average finish position of each jointillustrated in Figure 6.2. . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6.4 Confusion Matrix achieved using the projections-based feature vector . 84
6.5 Comparison of the Standing shoulder adduction exercise performedby an expert participant(on the left) and a beginner participant(onthe right). Observe lot of variations in the wrist position of thebeginner participant. This shows that the beginner participant isnot in control of the exercise and is prone to injury. . . . . . . . . . . 86
6.6 The 3D Joint coordinates from each camera are provided by consid-ering the location of the camera as the origin. When data for thesame exercise are collected using multiple cameras, it is necessaryto translate and/or rotate the coordinates from one camera to theframe of reference of the other camera. . . . . . . . . . . . . . . . . . . 88
6.7 Demonstrating the need of multiple Kinect camera. While a par-ticipant performing Lunges exercise, the Kinect camera 1 has un-tracked frames of ankle joint during brief moments when the par-ticipant bent forward. During this period, Kinect camera 2 has noproblem tracking the joint as its positioned with an angle to theparticipant. Therefore Kinect camera 2 frames can be used in placeof Kinect camera 1, however they need to be transformed to Kinectcamera 1 frame of reference. . . . . . . . . . . . . . . . . . . . . . . . 89
xiv
6.8 Superimposed frames from Kinect camera 1 frames(blue data points)and transformed Kinect camera 2(green data points) data frames forthe ankle joint during a Lunges exercise. Observe the green datapoints cluster highlighted in the circle appear in the absence of bluedata points. The combined data can now be used to analyze if theparticipant made an error during the exercise activity. . . . . . . . . . . 90
6.9 Plot of spine base over a single jump. . . . . . . . . . . . . . . . . . . . 104
6.10 Impulse calculated using force plate, vicon and kinect data. Thegraphs shows that the impulse calculated using force plate data isapproximately equal to impulse calculated using kinect and vicondata. This shows that the change in momentum is conserved duringjumping phase. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
xv
CHAPTER I
INTRODUCTION
Personalized Wellness Management (PWM) is a critical national priority that has the
potential to significantly reduce healthcare costs that are currently greater than 16%
of the nation’s GDP [1, 2]. Approximately 75% of total health care costs are associ-
ated with management of chronic illnesses such as Diabetes and Hypertension [3]. In
addition to the costs, there is loss of productivity and morale, and families are con-
fronting worsening Quality of Life [2]. To address these growing societal challenges,
the current reactive approach to medical practice , which is focused on disease man-
agement must be fundamentally transformed into a proactive approach that empowers
individuals to manage their personal wellness.
PWM is a complex multi-disciplinary problem that lies at the intersection of
areas such as psychology, exercise science, medicine, behavioral management, nutri-
tion and emerging computing/communications technologies [4, 5, 6, 7, 8, 9]. Although
there are several devices that can assist, monitor, track, or support individuals to
monitor their wellness, such devices have only offered small or incremental advances
in wellness management. As is evidenced by the escalating medical costs and obesity,
such technologies have not been able to create a paradigm shift in the state-of-the-
practice. To address this need, we propose a comprehensive decision-maker for PWM
that accounts for the diversity of individual choices, disease risks, socio-economic sta-
tus, ethnicity, genetic predisposition, and uncertainties in the operating environment
of individuals.
1
Figure 1.1: Personalized Wellness Management
Since PWM affects the health and well-being of individuals, it is very closely
related to the medical profession. In addition, wellness is a complex enterprise that
involves the confluence of many disciplines. For these reasons, it is necessary to
manage wellness in a rigorous and predictable framework. However, in order to
account for the diversity of the individuals and the disease risks, it is necessary to
personalize the wellness care. Ultimately, the objective for the PWM enterprise is
to encourage participants to become intrinsically motivated in the management and
sustenance of their own wellness. The goal-seeking framework we present achieves
this by relying on the principle of bounded-rationality [10], i.e., humans will make
rational decisions if they are presented with the right information that affects their
own wellness at the right time.
Several new approaches have emerged to assist participants in improving
their wellness. For example, the quantified self movement [11] has resulted in several
wearable devices and support systems to track physical activity. Similar advances
2
are being investigated to track and monitor nutrition activity. Classical models for
weight dynamics such as [12] have also been developed in the recent years. Our focus
is on investigating a goal-seeking approach [13, 14] in which there is no attempt to
optimize the exercise and nutrition actions.
1.1 Goal-Seeking Approach
We use a Goal-Seeking formulation as a basis to design a new decision-support to em-
power individuals to manage their personal wellness. The Goal-Seeking formulation
is grounded in a systems paradigm proposed by Mesarovic and Takahara [15] that
has been used extensively to study large-scale and complex systems [16, 17, 18]. In
recent years, this paradigm has been used as a basis to coordinate the management of
a collection of batteries [19, 13]. The highlights of this paradigm are that there is no
attempt to optimize the selection of actions and that uncertainty can be represented
and accounted for explicitly. This paradigm is described in more detail in Chapter 4.
Managing human wellness is viewed as an instance of sequential decision-
making. In this view, the system, i.e., the human, evolves by making a sequence of
actions. For this investigation, we assumed that the weight of a person, which can be
directly measured, is a strong indicator of the person’s wellness 1. Thus, in accordance
with the human weight dynamics model developed at the Diabetes Research Center
of the National Institute of Health [12], the two actions that critically affect an
individual’s weight are the nutrition intake (calories input) and the physical activity
(calories output). When these actions are executed, they result in consequences. The
uncertainties that affect the consequences are the internal metabolic constraints of
the individual and the motivation of the individual to adhere to the selected actions.
1In the future, this assumption can be relaxed by assuming other factors such as metabolism;however, one needs to have validated models for the weight dynamics before such assumptions canbe integrated in a formal decision-support framework such as the one proposed here.
3
Thus, this approach to wellness management fits well with the constructs of the goal-
seeking paradigm as is explained in more detail in Chapter 4. A complete formulation
of the wellness management problem in the Goal-Seeking framework required the
design of a reflection mapper that is described in Section 4.1.4 and a cost mapper ;
the selection of a cost mapper function is also described in the same section.
In order to understand how the goal-seeking approach compares with the
classical approaches to sequential-decision making reported in the literature, alter-
native formulations for wellness management were developed using the framework
of Markov Decision Processes (MDP). In contrast to the goal-seeking approach, the
MDP approach required the creation of a state-transition model for each individual.
An extended MDP framework called the Partially Observable Markov Decision Pro-
cess (POMDP) was used to represent the uncertainties in the formulation. The hu-
man weight dynamic models were incorporated into these formulations and “optimal”
policies were computed. These policies served as a basis for the selection of nutrition
and exercise actions in the MDP and POMDP frameworks. The selected actions and
their consequences were compared with the actions/consequences obtained using the
goal-seeking approach. Chapter 3 presents the MDP formulations and the results
of the comparison. The evaluation mappers that were required were collaboratively
developed with Master’s students in the team and are described in [20, 21, 22, 23];
these works are briefly described in chapter 6 for completeness.
1.2 Software System Design
Another important dimension of the goal-seeking formulation of the wellness man-
agement problem was that the formulation served as the basis for a scalable and
flexible systems architecture for the decision-support system. This robust architec-
ture provided the foundation for several investigations focused on personal wellness
4
management. The systems architecture is described in detail in Chapter 5. Many
elements of this comprehensive software system are being implemented and field trials
based on large populations of participants are being planned based on the decision-
support system that is described in this thesis.
1.3 Contributions
In summary, the contributions of this investigation are
1. The first Goal-Seeking Formulation for empowering Personalized Wellness Man-
agement (PWM).
2. Comparing Goal-Seeking approach and the classical approaches for sequen-
tial decision-making, i.e., Markov-Decision Processes and Partially Observable
Markov Decision Processes.
3. Design of Reflection Mapper for the Goal-Seeking framework for PWM. De-
signed and implemented a reference model for human weight dynamics in Java.
4. Selection of Cost function based on healthy weight loss rate for the Goal-Seeking
framework for PWM.
5. Design of Evaluation Mappers for evaluating and guiding exercise behaviors
6. Goal-Seeking system architecture, storage and software design.
7. Integration and implementation of Goal-Seeking software system ecosystem.
5
CHAPTER II
BACKGROUND
This chapter presents background that is necessary to describe the outcomes of
this investigation. Section 2.1 briefly describes trends in personalized medicine and
motivates the need for personalized wellness management. In order to make wellness
interventions more quantified and rigorous, we need a basis for these interventions.
Literature in the area of human weight dynamics is described in Section 2.2 and
the specific energy-balance model that was used in this investigation is described in
Section 2.3. The goal-seeking decision-support system must operate in the complex
domain of human behaviors. For this reason, well-known theories for human behavior
that have originated in psychology and social sciences are described in Section 2.4.
In order to evaluate the exercise recommendations, an evaluation tool was designed
based on the Microsoft Kinect Camera. Section 2.5 presents an overview of the
capabilities enabled by this device and a review of literature that has used this camera
for capturing and analyzing human motion. The goal-seeking approach that is the
basis for this investigation must be eventually compared to classical decision-making
frameworks. Two classical approaches for sequential decision-making are described
in Section 2.6. A more precise formulation of wellness management using these two
approaches is presented in Chapter 3. The goal-seeking paradigm is briefly presented
in Section 2.7 and a more precise formulation of wellness management based on this
paradigm is described in Chapter 4.
6
2.1 Personalized Medicine
Personalized wellness management that is envisaged by the proposed decision-support
tools is grounded in the idea of personalized medicine which dates back many hun-
dred of years. It is well-known that rapid developments in computational biology
and medical imaging have created new avenues for researchers to create and analyze
personalized diagnosis and treatments [24, 25, 26]. Despite such advances, there is
a long way to go in understanding why different individuals experience disease or
respond to treatment differently. This lack of knowledge has led clinicians prescribe
drugs to a patient based on general information about what might actually work for
that particular patient. If the medication does not work after a few weeks, the pa-
tient might be switched to another medication. This ”trial and error” approach can
lead to patient dissatisfaction, adverse drug interaction and side effects let alone the
money spent. Such a process, based on trial and error, is unfortunately the state-of-
the-practice today in the domain of exercise and nutrition. While there are general
guidelines and principles prescribed [27, 28], there are few quantitative methods that
can conclusively determine how much exercise an individual must perform and what
diet an individual must consume to remain healthy. Thus, the principal objective of
the decision-support system proposed in this thesis is to keep healthy people healthy
by designing personalized wellness prescriptions.
2.2 Human Weight Dynamics
Many factors that affect the wellness of an individual such as inherent disease risk
or metabolic issues cannot be precisely quantified or observed. However, the weight
of a person can be observed via simple measurement. For this reason, weight and
other metrics that can be derived from the weight such as Body Mass Index (BMI)
7
are often used as proxies for the wellness of a person. Using such metrics, recent
reports indicate that about two thirds of the US population is overweight [29]. Such
weight is correlated with chronic conditions such as diabetes [30], hypertension [31]
and heart disease [32]. For this reason, there has been considerable interest to develop
mechanisms and mathematical models that accurately predict outcomes of life style
changes and weight loss intervention strategies [33, 34]. These models offer opportu-
nities to understand how individuals lose and gain weight and can be used to help
patients manage their weight change and improve adherence to life style changes. In
this regard, many researchers have attempted to model human body weight change
dynamics as described in the following paragraphs.
A model proposed by Hill et. al [35] aims to provide a measure of average
daily energy imbalance. The authors assumed that a pound of weight represented
3500 kcal and that excess energy is converted to storage at 50% efficiency. Based on
historical trends, they estimated that the median daily energy imbalance gap between
intake and expenditure needed to explain a population weight gain of 1.8-2.0 lb per
year was about 30 kcal d−1. Another model developed by Swinburn et al. [34] used a
labeling technique to estimate the total energy expenditure (TEE) which the authors
assumed equals total energy intake (TEI) when the weight is stable. The authors
used TEE to estimate the difference in energy flux between states of weight balance.
From this model, Swinburn et al. [36] and Hall and Chow [37] concluded that the
increase in US food supply over the last 30 years was correlated with the increase
in weight in the US population. From such studies it was concluded that a TEI of
about 94KJd−1 would result in a weight increase of about 1 Kg for adults [38].
Weight change does not happen instantaneously. To calculate the time taken
for body weight change, dynamical system models have been developed that use food
8
intake and physical activity levels as model inputs, and calculate changes in energy
expenditure and body weight over time.
The first reference to a two-compartment body composition model that con-
siderd fat mass and lean mass tissue dates back to 1942 [39]. Here, the body mass
was considered to be either fat mass, which had a density of 0.9007 kg/liters, or
lean mass with a density of 1.1 kg/liters. This idea was further used by Forbes [40],
who analyzed body composition data collected across different studies and suggested
that the weight loss is exponential over time. This contribution was later refined
by Antonetti [41] to obtain a model for calculating weight change as a function of
time and caloric intake values. However, these models used constant values for the
Basal Metabolic Rate (BMR) 1 and the activity level; these values resulted in inaccu-
racies in the weight change predictions. Antonetti used an activity coefficient ka to
describe the physical activity performed by the participant [41]. More recently, this
idea is formalized by defining the Physical Activity Level (PAL), which is defined as
PAL = TEE ÷ BMR. Physical activity can also be measured using the Metabolic
Equivalent intensity levels(METs) [44] which expresses the energy expenditure of
physical activity as multiples of Resting Metabolic Rate (RMR) 2. Westerterp [45]
described another two-compartment model, where the changes in fat mass and fat-
free mass are driven by a system of two ordinary differential equations. The model
helped to predict how exercise affects weight gain, and how changes in caloric intake
factors in to the system. The model also accounted for how the body composition
changed over time affects weight change. Other models for human weight dynamics
include [46, 47, 48, 49, 50, 33]. A recent model incorporates both weight and height
dynamics for a wide range of ages and accounts for gender and race [51]. In [52], the
1The minimum rate of energy expenditure that is needed to support nominal body functions [42,43].
2RMR describes the energy needed to maintain the basic physiological processes [42].
9
authors develop a dynamic model for weight dynamics by incorporating an algebraic
relationship between FM and LM that is based on data collected by the Center for
Disease Control. This model is useful for estimating the change in weight as a result
of adherence to recommended nutrition intake.
The main idea in the two-compartment model is that the energy store of the
human body is either fat mass, FM , or lean mass LM . The daily energy balance,
EB(t), i.e., the difference between the calories consumed and the calories expended,
affects these stores as
d FM(t)
dt=
(1− p(t))EB(t)
ρFM, (2.1)
d LM(t)
dt=
p(t)EB(t)
ρLM, (2.2)
where ρFM and ρLM are energy densities and p(t) is the p-ratio that represents the per-
centage of the imbalance denoted by EB to the two compartments, respectively [53].
Hall [54] defined p using the Forbes formula [55] as p = CC+FM
, where C = 10.4 ρLM
ρFM.
2.3 Energy Balance Model
More recently, a three-compartment model was developed by Hall at the National
Institute of Health, Diabetes Research Center [56]. This model is particularly at-
tractive because it was calibrated using the data from the well-known Minnesota
Semi-Starvation Experiment [57] that used 32 healthy men. These men were placed
on a control diet for 12 weeks. They were then on a semi-starvation diet for 24 weeks.
They were then placed on a restricted rehabilitation diet for 12 weeks and on a unre-
stricted rehabilitation diet and 8 weeks. The models proposed by Antonetti [41] and
Hall [56] show a good agreement with the data collected from the Minnesota semi-
10
starvation study. Hall extended the two-compartment model to a more accurate
three-compartment model [37, 58, 59].
The key ideas in the three-compartment model are as follows. The normal
daily energy balance EB(t) is defined as
EB(t) = EI(t)− EE(t), (2.3)
where EI(t) is the energy consumed and EE(t) is the energy expended each day.
The daily energy intake, measured in kilo-calories (kcal), is calculated from the car-
bohydrate intake (CI), fat intake (FI) and protein intake (PI) for the day as
EI(t) = a1CI(t) + a2FI(t) + a3PI(t), (2.4)
where a1 = 4 kcal/gram, a2 = 9 kcal/gram and a3 = 4 kcal/grams. CI, FI and PI
are measured in units of grams/day. The energy expenditure expressed in kcal/day
is calculated as
EE(t) = TEF (t) + PA(t) +RMR(t) (2.5)
where the thermic effect of feeding TEF (t) denotes the energy expended in processing
food and PA(t) is the energy expended in physical activity. The value of TEF is
usually approximated to 10% of the energy intake. The energy expended per day,
PA(t), expressed in kcals, represents the energy expended as a result of conducting
work activities, household tasks, and physical exercise [60]. RMR can be estimated
using the Harris-Benedict prediction equation [61] as
Markov Decision Processes (MDPs) and Partially-Observable Markov Decision Pro-
cesses (POMDPs) are two classical frameworks for sequential decision-making. In
a sequential decision making setting, the individual must make multiple decisions
15
at different times to achieve an objective. In the context of wellness interventions,
these decisions would correspond to nutrition and exercise actions that must be fol-
lowed at different times. As is well-known, there is considerably uncertainty when
a human being attempts to implement the recommended actions. The MDP and
POMDP frameworks have been designed for modeling decision-making under uncer-
tainty. Such frameworks provide a formal basis for the design of a decision-support
system. These ideas will be introduced in the next two sections and more precisely
defined in Chapter 3.
2.6.1 Markov Decision Processes
MDPs offer a rigorous framework for modeling decision-making under uncertainty [95,
96, 97, 98, 99] when the state of the system is completely observable. MDPs have
been used extensively in the variety of problem settings and applications including
robotics [100], automated control, economics and manufacturing [101, 102], health-
care [103, 104] An MDP formulation for wellness management was presented in [73].
Figure 2.2 illustrates the key ideas of how MDP is embodied in a decision-
support system. In this view, the individual engages in wellness activities by mak-
ing decisions at discrete time epochs and implementing these decisions between the
epochs. Consistent with the energy balance model discussed in Section 2.3, these
decisions involve the selection of an action for nutrition and an action for exercise.
Since the underlying models focus on energy, we use calories as the units for these
actions. Thus, every decision involves selecting an action for nutrition, i.e., number
of calories that must be consumed per day, and an action for exercises, i.e., number
of calories that must be expended per day through exercise. As illustrated in the
figure, the MDP formulation is embodied in the decision support subsystem as will
be explained in more detail in Chapter 3. The MDP formulation assumes that the
state of the system is completely observable. For this reason, we selected the weight
16
of the individual as the state. Thus, in this formulation, at each epoch, the weight
of the individual is measured and the decision-support system selects an action, i.e.,
nutrition-exercise choice, depending on the weight. The MDP formulation, however,
allows for uncertainty in the implementation of the actions. Thus, the weight of the
individual at the next time epoch is not necessarily known until that time to the next
epoch as elapsed.
DecisionSupport
Whatshould Idonow?
N EH
si aj
t
rijk
DecisionSupport
N EH
sk
t+1
Figure 2.2: Overview of MDP. Decisions are made in discrete time epochs based onthe observed state (weight). At each epoch, the weight of the individual is measuredand a corresponding action is recommended. The implementation of the action resultsin a reward to the individual and the time advanced to the next epoch.
A more precise formulation of the MDP for wellness management will be
presented in Chapter 3. The quantized action space and the support necessary to
enable transitions will be described more precisely. The actions that are recommended
in each time epoch are based on an optimal policy that can be obtained by solving
the MDP. An overview of the well-known solution methods and examples of optimal
In many applications, the state of the system cannot be observed. For example, a
POMDP is reported in [105] to monitor a production process that cannot be observed
directly and the status must be inferred by observing the quality of the output. A
POMDP is used to maintain a distribution of future energy prices and use this fore-
casted information to schedule the operation of appliances in homes [106]. There
are many such examples in a broad spectrum of applications that range from health-
care [107] to aircraft collision [108]. In this specific context of wellness management,
it is easy to note that the individual’s motivation affects the implementation of both
nutrition and exercise actions. In fact, the motivation of an individual to implement
exercise actions may be different than the motivation to implement nutrition actions.
An extension of MDP in which the state is only partially visible is called a Partially
Observable Markov Decision Process (POMDP).
Figure 2.3 illustrates the key ideas of POMDP. As in the case of MDP, the
POMDP framework would serve as the basis for designing the decision-support sys-
tem that helps the individual in each time epoch. The state is not fully observable
and in each state, the system is expected to emit an observation stochastically. In
this context of wellness management, we used the constructs embodied in the theo-
ries of human behavior described in Section 2.4 as states. Specifically, we used the
Social Cognitive Theory (SCT) as a representative example and used the constructs
supported in this theory as the states. The states used were precontemplation, con-
templation, preparation, action, and maintenance. An underlying state-transition
probability matrix determines how the states change stochastically in response to
an action recommended. In a POMDP model for a system, there is a observation
probability matrix, one for each action. These probabilities govern what observation
is emitted from which state after a specific action is performed. We viewed the ad-
18
herence or otherwise of the individual to a recommended action as two observations
— adhered and not adhered.
Since the state is not fully observable, the POMDP maintains a belief dis-
tribution over the states of the system. Based on its current estimated state and
the observation, it selects an action that is recommended to the individual. The
implementation of the action results in a reward to the individual.
DecisionSupport
Whatshould Idonow?
Estimate(si)aj
t
N EH
M
t+1
DecisionSupport
Estimate(sk)
N EH
M
rijk
O1
Figure 2.3: Overview of POMDP. Decisions are made in discrete time epochs basedon an estimate of the state of the system. In each state of the system the system issaid to emit an observation stochastically. The observation and the estimated stateis mapped to an action and recommended. The implementation of the action resultsin a reward to the individual and the time advanced to the next epoch.
A more precise specification of the POMDP for wellness management that is
based on SCT is presented in detail in Chapter 3. As in the case of MDP, we can
obtain an optimal policy for selecting actions in a POMDP. Examples of such policies
and a comparison of the POMDP policy with that obtained from an MDP is also
discussed.
19
2.7 Goal-Seeking Paradigm
The objective of this investigation was to design a Goal-Seeking decision-support
system.
The goal-seeking paradigm [15] is an approach to modeling and describing
systems that differs from the MDP and POMDP approaches briefly described above.
Both the MDP and POMDP frameworks assumed that states of a system are pre-
cisely describable; and the dynamics of the system are described by a state-transition
function. In the POMDP framework, the observations emitted depended on the state
of the system.
In contrast, there is no attempt to describe the system states in the goal-
seeking paradigm and, hence, the system model is simpler than what one may expect
when using a state-transition paradigm. Instead, the decision-making process is for-
mulated using the following constructs. There is a set of Alternate Actions, Π, from
which the decision-maker can select actions. In the wellness management context,
these actions are nutrition and exercise recommendations as already described in the
MDP and POMDP frameworks. Anticipated system perturbations and disturbances
are represented as a set of Uncertainties, ∆. If a given perturbation δi ∈ ∆ occurs,
it would impact the success of a selected action. Consequences are outputs that are
produced by the system; the set of Consequences, Ψ, includes all outcomes that may
result from the implementation of some action. The decision-maker uses a function
called Reflection, Ξ : Π ⊗ ∆ → Ψ, as its view of the environment. Suppose that
the decision-maker selects an action π1 ∈ Π; the decision-maker uses Ξ to estimate
the consequence, ψ1 ∈ Ψ, that π1 would produce if a given perturbation occurs. An
Evaluation Set, Λ, represents a Performance Scale that is used to compare the results
of alternate actions according to an Evaluation Mapping, Ω : Ψ ⊗ Π → Λ. That
is, if the decision-maker has the option to select one of two actions π1, π2 ∈ Π, and
20
these actions are expected to result in consequences ψ1, ψ2 ∈ Ψ, respectively, then the
decision-maker uses values of Λ as the metric to determine whether one of the two
actions is preferred over the other. Ω is also used to evaluate the actual measured
output of the system. A Tolerance Function, Γ : Π ⊗ Ψ → Λ provides a bound on
how much the performance can vary before being considered as unsatisfactory.
Figure 2.4: Overview of the Goal-Seeking Paradigm. Decisions are made in discretetime epochs. Since there is no attempt to model the states of the system, the indi-vidual provides a set of feasible actions to the decision-maker. The decision makerselects one of the feasible actions based on the Reflection Mapper and establishes alimit for performance that can be achieved in the interval of time to the next epoch.At the next epoch, the actual performance is evaluated. If the difference between theexpected performance and the actual performance is within the tolerance bound forthat individual, the decision-maker continues to recommend actions.
Using these artifacts and transformations, the task of the decision-maker may
be stated as
21
Continue to select an action π ∈ Π as long as the outcome is within
tolerance limits, i.e., Ω(π, ψ) > Γ(π, ψ), for any possible perturbation
δ ∈ ∆.
Rather than finding an optimal policy like the MDP or the POMDP, the
goal-seeking approach aims to find a satisfying solution that is within an acceptable
tolerance limit. Such an approach is useful when it is not possible, or desirable, to
construct a precise model of a system. Consequently, in this paradigm, the control of
a complex system does not require decision-maker that is based on complex models.
It is feasible to design algorithmic modules that offer systematic integration of multi-
disciplinary constraints to address the needs of complex issues such as personalized
wellness management.
In the next chapter, we present more precise MDP and POMDP formulations
for wellness management as envisaged in this investigation. A detailed goal-seeking
formulation is presented in Chapter 4.
2.8 Notation
The notations used throughout this thesis are presented in this section. Table 2.1
shows the notation that is used throughout the thesis.
Table 2.2 shows the notation used in the MDP formulation for wellness man-
agement that is described in Chapter 3.
Table 2.3 shows the notation used in the POMDP formulation for wellness
management that is described in Chapter 3.
Table 2.4 shows the notation used in the Goal-Seeking formulation for well-
ness management that is described in Chapter 4.
22
Table 2.1: Notation used throughout the thesis.
Symbol Descriptionci Nutrition action i. Each action identifies a specific (quan-
tized) caloric intake per day.lj Exercise action j. Each action identifies a specific (quan-
tized) physical activity level for exercise activities. Thislevel can be uniquely mapped to caloric expenditure perday.
< ci, lj > A wellness recommendation; identifies a specific nutritionaction and an exercise action.
t = 1, 2, · · ·NDecision epoch or ”tick” during which a decision is made.U [cmin, cmax] The Uniform distribution of dietary calories/day, that the
participant eats.U [lmin, lmax] The Uniform distribution of physical activity level, that the
participant exercises.
Table 2.2: Notation used in the MDP Formulation in Chapter 3.
Symbol DescriptionSM Set of states.AM Set of actions.pijk The probability of transition from si ∈ SM to sk ∈ SM using
aj ∈ AM .rijk The reward received for transitioning from state si ∈ SM to
sk ∈ SM using aj ∈ AM .Vt(s) The value function, i.e., sum of expected rewards accumu-
lated, when starting from state s ∈ SM and acting optimallyfor a horizon of t steps.
23
Table 2.3: Notation used in POMDP Formulation in Chapter 3.
Symbol DescriptionSP Set of states.AP Set of actions.pijk The probability of transition from si ∈ SP to sk ∈ SP using
aj ∈ AP .rijk The reward received for transitioning from state si ∈ SP to
sk ∈ SP using aj ∈ AP .DP Belief Distribution over POMDP states.O1 Observation that participant adhered to recommended ac-
tion in the POMDP Formulation.O2 Observation that participant did not adhere to recom-
mended action in the POMDP Formulation.poijk Probability of emitting an observation Oi after performing
action aj when the system is in state sk in the POMDPFormulation.
Table 2.4: Notation used in Goal-Seeking Formulation in Chapter 4.
Symbol DescriptionΠ Set of actions. Each πi ∈ Π corresponds to < ci, lj >.∆ Set of uncertainties. The occurrence of δi ∈ ∆ impacts the
consequence of a selected action.Ψ Set of consequences.Ξ Reflection Mapper. Ξ : Π⊗∆→ Ψ.Λ Performance Scale [0 . . . 100].Ω Evaluation Mapper. Ω : Ψ⊗ Π→ Λ.Γ Tolerance Function.DM Motivation Distribution that is used in the Goal-Seeking
Formulation.
24
CHAPTER III
WELLNESS MANAGEMENT FORMULATIONS IN CLASSICAL
FRAMEWORKS FOR SEQUENTIAL DECISION-MAKING
This chapter presents wellness management formulations in the two classi-
cal frameworks for sequential decision-making, namely Markov Decision Processes
(MDP) and Partially-Observable Markov Decision Processes (POMDP). Both these
frameworks require an underlying state-representation to support the decision-making.
In MDP, the state must be fully observable and in POMDP the state is not fully ob-
servable. Uncertainty is captured in both these frameworks by the stochastic transi-
tions between the states. This formulations in this chapter demonstrate how wellness
actions can be selected in these frameworks. The advantages and disadvantages of
these frameworks are discussed.
Section 3.1 presents the MDP formulation. Here, since the state of the system
must be observable, the weight of the individual was selected as a representation of the
state. In Section 3.2, the hidden states are based on the theories of human behavior
discussed in Section 2.4. As detailed in the discussion, the POMDP model maintains
a belief distribution over these states and actions are stochastically selected.
3.1 Markov Decision Processes
An MDP is defined by the four-tuple (SM ,AM , T, R), where SM denotes the states
of the MDP, AM is the Action Space, T is the state transition probability matrix and
25
R is the reward matrix. The construction of each of these artifacts, the computation
of an optimal policy and the use of this policy to select actions for a participant are
described in this section.
3.1.1 State Space
Since the state of an MDP must be completely observable, we used the weight of
the participant as the state. A range of [75 . . . 120] was selected as a representative
example and the weight of the individual was rounded to the closest integer in this
range.
3.1.2 Action Space
The Action Space, AM , was formulated by quantizing the nutrition actions and the
exercise activities. The caloric intake was assumed to be in the range [500, 5000]
calories, in steps of 250 calories. Exercise activity was assumed to be a physical
activity level in the range [1 . . . 3] in steps of 0.1. Recall, this is a dimensionless value
and can be mapped to caloric expenditure as illustrated in Equation 4.4.
Thus, as illustrated in Figure 3.1, the action space AM comprises the set of
quantized nutrition-exercise pairs. As a representative example for this investigation,
using the choice of ranges described above, we had 360 choices in the action space.
3.1.3 State Transition Probability Matrix
Uncertainty in the system was captured by designing stochastic transitions between
the states as described here. The probability of transitioning from state si ∈ SM to
state sk ∈ SM using action aj ∈ AM is denoted as pijk. The probability of transition
between si and sk were represented as a state transition probability matrix, Tj, one
for each action. The construction of Tj is described will be described in more detail.
Since MDP represents how the states of the individual changes, the three-
compartment weight dynamics model described in Section 2.2 was used to construct
26
Figure 3.1: Quantized Action Space used for the MDP. Each action represents aspecific choice of caloric intake (nutrition) and caloric expenditure (physical activitylevel).
T . The duration of time between two epochs was selected as 3 weeks (21 days). To
account for metabolic variability in individuals, we arbitrarily selected a threshold
value h = 0.8. We then assumed that this value represented the chance that the
individual would achieve the weight predicted by the weight dynamics model if the
corresponding action was selected. Since the weights were quantized in steps of 1 kg,
it was easy to identify neighboring states for an action as follows.
Starting in state si, we used the weight as input to the weight dynamics
model; the caloric intake and expenditure corresponding to action aj were also used
as inputs to the model. The model provided a prediction of the state sk that would
27
be attained if action aj was implemented for 3 weeks. We assigned
pijk = h;
the two neighboring states, i.e., sk+1 and sk−1 excluding boundary conditions, were
assigned probabilities as
pijk+1 =1− h
2,
and
pijk−1 =1− h
2.
3.1.4 Reward Matrix
In order to compute an optimal policy using the MDP, it was necessary to design
a reward for each transition. The reward obtained when transitioning from state
si ∈ SM to state sk ∈ SM using action aj ∈ AM was denoted as rijk. The reward is
assumed to be non-negative and bounded, i.e. 0 ≤ rijk ≤ rmax, ∀i, k ∈ SM , aj ∈ AM .
The reward matrix, one for each action aj ∈ AM is denoted as Rj.
Instead of computing a large number (360) of reward matrices, we designed
a two reward functions that returned a real value between 0 (undesirable) and 100
(desirable).
1. Linear Reward: Using the initial and target weights, wi and wt, for the
individual, the slope was selected by assuming that (wi, 0) and the (wt, 100)
were the end points of a line. Given si, sk ∈ SM , the linear reward function was
evaluated at sk.
2. Safe-Loss Reward: This reward function was designed to make sure that an
individual does NOT loose more than 1 kg per week; this is an example of a safe
weight loss policy to ensure that there are no adverse effects on the individual.
28
For this purpose, the reward function was
100− |sk − si − 3| × 10.
Note the reward is maximum if the weight loss in the time epoch is close to 3,
i.e., 1 kg per week.
3.1.5 Computing Optimal Policies
A policy is a function that maps the current state of the MDP to an action. An
optimal policy is one that maximizes the total reward that the individual can earn,
in an expected sense. The purpose of formulating a problem, such as the wellness
management problem, as an MDP is to obtain an optimal policy. The objective of this
investigation was to understand how the goal-seeking approach described in Chapter 4
compares with classical frameworks for sequential decision-making, we assumed that
the policies are stationary, i.e., the policy will not change with time. This means that
whenever the MDP is in a specific state, the action recommended and the next state
that is selected stochastically will not change.
Since the MDP is used make decisions at different (sequential) times, it is
important to specify how the current reward must be traded off with a potential
future reward that can be obtained by selecting the current action. This is captured
by using a discount factor, γ. Thus, given an MDP, the goal is to find a policy P
that maximizes the expected total discounted reward.
Suppose the MDP is allowed to evolve for infinite time, the value accrued by
using a policy P assuming that the MDP starts in a particular state s0 ∈ SM is
V π(s0) = E
[∞∑t=0
γtrijk
],
where 0 < γ ≤ 1.
29
Among all possible policies for an MDP, the policy that maximizes the ex-
pected total discounted reward is called the optimal policy, p∗, and accrued reward
by starting in state s0 is called the optimal value and is denoted as
V ∗(s0) = maxp∈P
V p(s0).
In this application of MDP for wellness management, since the policies are
assumed to be stationary, the optimal policies can be computed using two classical
algorithms, namely Value Iteration and Policy Iteration. Such a policy can be used
to select actions in each time epoch as will be explained.
3.1.5.1 Value Iteration
The key idea in value iteration is to compute the value accrued in each of the states
at time epoch t; for state si, the value of each possible action aj is examined and the
highest value is recorded
Vt+1(si) = maxaj∈AM
∑sk∈SM
pijkrijk + γ∑sk∈SM
pijkVt(sk)
, (3.1)
This expression can be understood by thinking “backwards”; Irrespective of
the value accrued in the preceding t steps, the best action in the step t+ 1 is the one
that provides the highest value. Since the classical value iteration algorithm assumes
the infinite horizon setting, the γ team appears in the above expression to discount
future rewards.
After computing the value for each state, the action that provides maximum
value is selected to be the action for that state in the current policy, i.e.,
π(si) = arg maxaj∈AM
Vt(si). (3.2)
30
The value iteration terminates when the maximum difference between two
successive value functions, called the Bellman Error Magnitude is less than a pre-
specified threshold value, ε). It is known [109] that there exists a t∗, which is poly-
nomial in the size of the state space, |SM | × |AM |, when the magnitude of the largest
value of the largest reward, rijk and 11−γ , are such that the action that maximizes V ∗t
is the optimal action.
Instead of calculating the bound on t in advance and carrying out value
iteration for that period, the iteration is terminated when
|Vt(s)− Vt−1(s)| < ε,∀si ∈ SM .
When the above bound is valid, it is known that [110]:
maxsi∈SM
|Vt(s)− V ∗(s)| < 2εγ
1− γ. (3.3)
3.1.5.2 Policy Iteration
In value iteration, the value of each state is updated in each iteration. In contrast,
policy iteration assumes a random policy for the initial step. Assuming that each
state si ∈ SM is the starting state, compute the value obtained by implementing the
current policy π until the value of each step converges as described the preceding
paragraph. For each state, the value is now
Vt(si) =∑sk∈SM
pijk[rijk + γVt(sk)] ∀si ∈ SM , (3.4)
where the action aj is specified by the current policy π.
31
The current policy is improved by selecting the action, for each state, that
maximized the value in Vt(.), i.e.,
π(si) = arg maxaj∈AM
Vt(si). (3.5)
The value function of a policy is the expected discounted reward that will
be gained at each state by executing that policy. Once the value of each state is
known under the current policy, the option of improving this value by changing the
first action taken is considered. If it can, the policy is changed to take the new action
whenever that state is encountered. This step is guaranteed to strictly improve the
performance of the policy. When no improvements are possible, then the policy is
guaranteed to be optimal. Since there are at most |A||SM | distinct policies, and the
sequence of policies improves at each step, the policy iteration algorithm terminates
in at most an exponential number of iterations. It is known that the running time
is pseudo polynomial and that for any fixed discount factor, there is a polynomial
bound in the total size of the MDP [111].
3.1.6 Illustrative Example
To illustrate the MDP formulation in this section, a specific example is now pre-
sented. In this example, the state space was selected have seven states as illustrated
in Table 3.1. This represents an individual whose weight is in the range 145 kg to
150 kg. The objective of this example is to demonstrate all the artifacts of the MDP
formulation and is not intended to be realistic.
The action space for this example is shown in Table 3.2. Note there are only
three possible actions as indicated in this table.
32
Table 3.1: State Space for Illustrative Example
State Weight (kg)s1 145s2 146s3 147s4 148s5 149s6 150
Table 3.2: Example Actions for Illustrative Example
The MDP formulation assumed that the state is completely observable. For this
reason, a measurable attribute such as weight was selected to represent the state in
the preceding section. In reality, the state is not fully observable. For example, in the
domain of wellness management, there are several hidden factors such as metabolic
limitations, motivation and disease conditions that may cause the result of an action
to be different from that anticipated by the weight dynamics model.
A Partially-Observable Markov Decision Process (POMDP) is a tuple
(SP ,AP , T, R,Ψ,O),
where
• SP ,AP , T, R describe a Markov Decision Process,
35
• Ψ is a finite set of observations the agent can emit, and
• O : SP ⊗AP → Ψ is an observation function.
A POMDP is an MDP in which the agent is unable to observe the current
state. Instead, in the POMDP framework, an observation can be emitted in each
state. This observation provides some indication of the state of the POMDP. As
in the MDP, there is a reward associated with selecting an action in each state; an
optimal policy for a POMDP maximizes the total expected discounted reward and
the purpose of formulating a POMDP problem is to compute an optimal policy that
can guide action selection.
In addition to constructing the state-transition probability matrix and a re-
ward matrix for each action, it is necessary to design the observation probability
matrices for each action aj ∈ AP . Because this is a large space, the action space was
further abstracted as illustrated in Figure 3.3.
In order to formulate the wellness management problem in the POMDP
framework, a precise set of state are necessary. For this purpose, we selected the
states from the theory of planned behavior as illustrated in Table 3.5.
Table 3.5: State Space for Illustrative Example
State Descriptions1 Precontemplations2 Contemplations3 Preparations4 Actions5 Maintenance
Since these states cannot be directly observed, we maintain a belief distribu-
tionDP over these states. The key idea is that based on the current belief distribution,
the optimal policy helps to select an action. This action is implemented. Depending
36
Figure 3.3: POMDP Action Space
on the next state of the POMDP, an observation is emitted. This observation, the
recommended action and the current belief distribution is used to update the belief
distribution.
Let p(si) denote the probability assigned to si ∈ SP in DP . We know that
0 ≤ p(si) ≤ 1 ∀si ∈ SP and that∑
si∈SP p(si) = 1. To compute a new belief
state p′(si), given an old belief state p(si), an action aj, and an observation ψk, The
new degree of belief in some state s′, d′(s′) can be obtained from probability theory
37
as follows:
p′(si) = p(si|ψ, aj, DP )
=p(ψ|si, aj, DP )p(si|aj, DP )
Pr(ψ|aj, DP )
=p(ψ|si, aj)
∑sk∈SP Pr(si|aj, DP , sk)p(sk|aj, DP )
p(ψ|aj, DP )
=O(si, aj, ψ)
∑sk∈SP pkjip(sk)
p(ψ|aj, DP )
The denominator, p(ψ|aj, DP ), can be treated as a normalized factor, independent
of si, that causes si to sum to 1. The state estimation function SE(DP , aj, ψ) has
as its output the new belief state p′(si) for each state. Thus, the state-estimation
component of a POMDP controller can be constructed from a given model.
The POMDP uses its current belief distribution and a policy that is obtained
by solving the POMDP, to select actions. Each time the action is implemented, the
POMDP transitions to a new state and emits an observation. The belief distribution
is updated and the process continues.
Figure 3.4 presents an overview of how the POMDP framework is useful in
wellness management. As illustrated, once the problem is formulated in the POMDP
framework, one can compute an optimal policy. Since the states are not observable,
we maintain a belief distribution. Based on the belief distribution, the POMDP policy
recommends one of the actions shown in Figure 3.3. Since each of these high-level
actions can correspond to several choices of nutrition and physical activity level as
illustrated in Figure 3.1, we use an MDP to select the action based on the weight of
the individual.
38
Figure 3.4: Block diagram on how POMDP and MDP frameworks are used to selectan action from the action set. After constructing the pomdp policy, at each timestep, depending on the observation, selects the best intensity level of the action. Thisintensity level is used to filter the action set for the MDP framework. Using thehuman weight dynamics model and user preferences, the MDP framework selects oneaction among many actions of the same intensity level.
Using the restricted action set in each of the nine categories shown in Fig-
ure 3.3, we computed nine MDP policies. Three of these policies are illustrated in
Table 3.6. Note, the missing entries in the second column correspond to actions with
low caloric intake and low physical activity levels. The entries are missing because
the weight dynamic model could not find suitable actions in this range.
3.3 Examples of Wellness Interventions
This section presents results for a hypothetical participant with an initial weight of
120 kg. This person aspires a target weight of 80 kg. His height is 1.7 meters and age
is 29 years. The current physical activity level is 1.2. The state space for the MDP
39
Table 3.6: Three of the Nine Optimal Policies computed for MDP with restrictedaction sets shown in Figure 3.3.
was discretized in steps of 1 kg. The minimum and maximum number of calories
consumed by the participant per day was in the range [500, . . . , 5000] in steps of 250
calories. Similarly, the physical activity level for the participant was restricted in the
range [1, . . . , 3] in steps of 0.1. All the possible combinations of nutrition and physical
activity in these ranges constituted the action space.
3.3.1 Results obtained using MDP Formulation
The state transition probability matrix was calculated as described earlier using hu-
man weight dynamics model. Given wt, the weight of the participant at time t, and an
action aj, the weight wt+1 was obtained using the weight dynamics model described
in Section 2.2. Suppose w′ represents the predicted weight, then The transition prob-
ability from state wt to state w′ was set as
P (w′∣∣w, a) =
0.8, if w′ = wt+1
0.1, if w′ = wt+1 − 2
0.1, if w′ = wt+1 + 2.
(3.6)
This means that the transition to the next state followed the weight dynamics model
80% of the time, but was off by ±2 Kg 20% of the time.
The Brown-UMBC Reinforcement Learning and Planning (BURLAP) Java
library [112] was used to obtain the optimal policies using different reward functions.
Figure 3.5 illustrates the policy obtained by solving the MDP using a linear
reward function; this reward encourages a participant to reach the target weight in
the fastest possible manner. This can be accomplished with high physical activity
levels and low caloric intake.
The weight trajectory is shown in Figure 3.6. Notice that the figure shows
that for a given number of days, there is a variability in the weight. These error
41
Figure 3.5: Shows all possible states from Initial to Target. For each state, the barsrepresent the action recommended. The exercise scale (red) is on the right (PAL) andnutrition (blue) scale is on left (cals/day). This policy was computed using a LinearReward function. The recommendation is to eat less and work out more.
bars show the minimum and maximum weights obtained for 50 different (stochastic)
executions of the MDP policy for the same participant.
Figure 3.7 illustrates the policy obtained by solving the MDP using the Safe-
Loss Reward. Higher reward is assigned to state transitions for actions that result in
weight change not more than healthy weight loss rate. It is to be noted that, in this
case we do not differentiate between two actions that result in same weight loss rate.
For example, low-calorie diet and low-intensity exercise may result in same weight
loss rate as high-calorie diet and high-intensity exercises.
The weight loss trajectory using the above policy is shown in Figure 3.8.
To illustrate how to capture participant preferences, we now show how to
compute a policy for a participant who does not wish to have physical activity level
that is more than 30% of the baseline activity level. This can be accomplished by
42
0 20 40 60
80
90
100
110
120
Time(Days)
Body
Wei
ght(
Kg)
grid
Figure 3.6: The expected weight loss trajectory for a participant who initially weighs120 Kg and follows the policy in Figure 3.5.
Figure 3.7: Actions computed from a policy that rewarded a target rate of weightloss. Higher or lower loss is penalized. The target loss can be achieved via multipleactions. The figure illustrates one stochastic choice that helps the participant achievetarget weight.
43
0 50 100 150 200 250 300
80
90
100
110
120
Time(Days)
Body
Wei
ght(
Kg)
Figure 3.8: Weight loss trajectory and uncertainty regions when executed the policycomputed for recommended weight loss.
designing a reward function that increases reward accordingly. Figure 3.9 and Fig-
ure 3.10 illustrate the policy and the weight loss trajectory obtained using such a
reward function, respectively. Notice that the caloric intake is higher in high weight
states and gradually decreases as the participant weight decreases. This is in ac-
cordance with the weight loss dynamics presented in section 4.2 that baseline calorie
intake is proportional to the total body mass, and therefore the participant is required
to consume more calories to support his weight. As the weight starts to decrease, the
baseline calories also decrease and so does the caloric intake. The expected weight loss
trajectory in this scenario is shown in figure 3.10 and can be seen that the expected
time to reach the target weight is now pushed further than the time in Figure 3.8;
this delay can be attributed to the fact that participant preference is now considered
in the policy.
44
Figure 3.9: MDP policy obtained using a reward function that rewarded actions thatwere within 30% of baseline exercise activity level.
0 100 200 300 400 500
80
90
100
110
120
Time(Days)
Body
Wei
ght(
Kg)
Figure 3.10: Weight loss trajectory for participant not willing to perform high inten-sity exercises but willing to go on a calorie-restricted diet.
45
Table 3.7: State Transition Matrix used for Low Intensity Actions
Table 3.10: Observation Matrix used for Low Intensity Actions
Adhered O1 Not Adhered O2
s1 0.1 0.9s2 0.8 0.2s3 1.0 0s4 1.0 0s5 1.0 0
Table 3.11: Observation Matrix used for Medium Intensity Actions
Adhered O1 Not Adhered O2
s1 0.1 0.9s2 0.3 0.7s3 0.5 0.5s4 1.0 0s5 1.0 0
In addition to the state-transition matrices, it was necessary to construct
observation probability matrices corresponding to each action. Table 3.10, Table 3.11
and Table 3.12 show the corresponding observation probability matrices.
The optimal policy computed maps the belief distribution over the states to
actions. An example of a part of this map is illustrated in Figure 3.13. As in the
case of the MDP, an optimal policy for the POMDP is computed offline. During the
implementation of the policy, a belief distribution is maintained over the states of
Table 3.12: Observation Matrix used for High Intensity Actions
Adhered O1 Not Adhered O2
s1 0 1.0s2 0 1.0s3 0 1.0s4 0.6 0.4s5 0.8 0.2
47
the POMDP. At each step, the a dot product is computed using the current belief
distribution and a policy vector shown in Table 3.13. The action corresponding to
the policy vector that yields the highest value after the dot product is selected.
Table 3.13: Snapshot of the POMDP policy. The policy had over 250 vectors torepresent the states and only a few are shown in this table as an illustration.
and ηLM = 230 kcal/kg and K are coefficients and constants for the calculation of
RMR. The constant K accounts for the initial energy balance conditions and is
determined by solving Equation 2.5 at initial steady state conditions, i.e.,
EI − EE = 0, (4.5)
58
Figure 4.2: The Energy Balance Model
d FM
dt=d LM
dt= 0, (4.6)
and
K = −γLM LM − γFM FM − δ BM + EI (1− β). (4.7)
Finally, the body mass at time t is obtained as
BM(t) = FM(t) + LM(t) + ECF (t). (4.8)
In summary, this three-compartment energy balance model accepts as input
EI(t) comprising CI(t), FI(t) and PI(t), change in sodium intake ∆diet, and the
physical activity level δ. The model produces FM(t), LM(t) and ECF (t) as outputs
from which we can compute the body mass, BM(t), of the individual as illustrated
in figure 4.2.
The implementation of the actions was designed to account for uncertainties
as illustrated in Table 4.1.
The reflection mapper selects the best action from among a set of feasible
actions so as to get the best consequence even when uncertainties come to pass. For
this purpose, a cost-mapper is required to map the anticipated consequence to the
59
performance scale. This function ranks the consequences on a common scale so that
the action with the best consequence is selected. The reflection mapper estimates the
consequences of each action in the action set, and the cost mapper assigns a cost (or
negative reward). In this thesis, a simple cost mapper is selected that operates on
the estimated weight lost by the participant using the reflection mapper discussed in
Section 4.2. One such function is shown in Figure 4.3.
Figure 4.3: The Reward function used in the goal seeking framework to assign cost(negative rewards) to consequences of each action that is estimated through reflectionmapper.
4.3 Bayesian Update of Human Motivation Distribution
In POMDP modeling, the human motivation distribution was discretized into 5 states,
according to the definition of Social Cognitive Theory. POMDP modeling restricts
60
the problem size as the algorithms that are used to solve the problem, does not scale
well as the number of states increase. However, in practice, the human motivation
level can be quantized anywhere between 0 - 100%, where 0% can be classified as
amotivated individual and 100% can be classified as highly motivated individual. In
real world, the participant may be somewhere in between that cannot be observed
directly. Here we describe the Bayesian update technique that is employed in the
goal-seeking to update the motivation distributionDM , that is not restricted to the
number of states in the distribution.
DM is the random variable, that represents the motivation level of the partici-
pant. Hypothesis : The hypothesis H that is being tested, i.e. whether the participant
is motivated or amotivated. O : the observation data set O=Adhered, Not-Adhered
is the new information, that is inputted to the model. This information is generated
by the decision-maker upon evaluating the participant at each time step, by analyzing
the performance of the participant with respect to the action recommended in the pre-
vious time-step. Prior-Distribution: The prior motivation distribution p(DM = m),
such that|DM |∑m=1
p(DM = m) = 1
Likelihood : The likelihood function ie p(O|H) i.e. the probability of the data being
generated assuming that the hypothesis is true. For example, p(O = ”adhered”|H =
”50%motivated”) is the probability that the participant will adhere to the recom-
mendation given that the participant is 50% motivated. After gathering the new
information, the Bayes theorem is used to compute the compute the probability of
each hypothesis given the data Posterior Distribution. The Bayes theorem is given
61
by the equation
P (DM |O) = P (O|DM )P (DM )P (O)
(4.9)
= likelihood∗priorNormalization
where, normalization is calculated as follows
P (O) =
|DM |∑m=1
P (O|DM = m)P (DM = m)
and the likelihood function that is used is given as follows
P (O|DM = m) =
m/100, if O=”Adhered”
1−m/100, if O=”Not-Adhered”
(4.10)
4.4 Simulation Results
This section presents the simulation results for the goal seeking architecture described
in the previous section. To be consistent with the results obtained using the Com-
pletely Observable MDP and Partially Observable MDP, the same user scenarios are
used in the simulations i.e. a Male participant initial weight of 120 kg aspiring a tar-
get weight of 80 kg. His height is 1.7 meters and age is 29 years. The current physical
activity level is 1.2. In order to simulate the uncertainty in the action execution, an
adherence metric for nutrition and exercise is used. An adherence level of 1 represents
that the participant adheres to the recommended action in each time step, while an
adherence level of 0 represents the participant does not adhere to the recommended
action. Figure 4.5 shows how the motivation distribution DM at each various time
steps along the simulation, gets updated for a nutrition and exercise adherence level
62
of 0.5 each. Observe that the motivation distribution at the end of 12 time steps,
converged to approx 45− 55% meaning that the participant is neither motivated nor
a-motivated. Figure 4.4 shows a similar scenario however for a nutrition and exercise
adherence level of 08 and 0.4 respectively. Observe that after few iterations, the nu-
trition motivation distribution updates shifts towards 100% motivated and exercise
motivation distribution shifts towards 0% motivated; meaning that the participant is
inclined towards going on a calorie restricted diet rather than doing exercise activities.
Also observe that the action recommended at each iteration follows the motivation
level at each stage. For example, in Figure 4.4 the action recommended in the first
stage is (2750, 1.3). Since in this stage no information about the motivation level is
known to the goal seeking agent, it starts out by selecting a low intensity action. As
more observations are received and motivation distribution is updated, the selected
action intensity level reflects that of the motivation distribution. Thus the action
(1500, 1.8) is selected to reflect high motivation towards nutrition and low motivation
towards exercise activities.
Figures 4.6, 4.7 and 4.8 simulate the expected weight trajectories of the par-
ticipant for different levels of adherence levels of exercise and nutrition. As from the
above descriptions, as the participant moves towards high levels of motivation, high
intensity actions are recommended at each time step. If the adherence level of the
participant is also high, then the time taken for for the participant to reach the target
weight gets smaller.
63
(a) Selected Action is (2750,1.3) (b) Selected Action is (2750,1.3)
(c) Selected Action is (2250,1.2) (d) Selected Action is (2250,1.2)
(e) Selected Action is (1500,1.8) (f) Selected Action is (1500,1.8)
Figure 4.6: Expected weight trajectory and its error bars for a participant with Nu-trition Adherence Level : 0.1 and Exercise Adherence Level: 0.1. Since the adherencelevel is too small, the participant does not adhere to the recommended actions andtherefore is expected to increase weight and never reach the target weight.
0 100 200 300 400 500 600
80
100
120
Time(Days)
Weight(KG
)
Figure 4.7: Expected weight trajectory and its error bars for a participant with Nutri-tion Adherence Level : 0.5 and Exercise Adherence Level: 0.5. Since the participantadheres to the recommended action 50% of the time, the participant neither increasesweight nor decreases weight as is evident by the large error bars.
66
0 100 200 300 400 500 60070
80
90
100
110
120
Time(Days)
Weight(KG
)
Figure 4.8: Expected weight trajectory and its error bars for a participant withNutrition Adherence Level : 1.0 and Exercise Adherence Level: 1.0. In this scenario,the participant adheres to each recommended action and therefore after few timesteps, the goal seeking framework recommends high intensity actions. Since theadherence levels are high, the error bars in estimating the wight trajectory to therecommended action is small.
67
CHAPTER V
SYSTEMS ARCHITECTURE AND SOFTWARE DESIGN
This chapter presents the system architecture and design that was implemented to
operationalize the personalized wellness management system. In its current form, this
chapter is only a preliminary description that is intended to provide an overview of
the software system. A more detailed design will be specified in the final dissertation.
5.1 Overview of the PWM Software Architecture
The Personal Wellness Management system architecture along with the user & expert
interaction is depicted in Figure 5.1. The System consists of the following modules.
5.1.1 Participant Dialog Module(PDM)
PWM is a personalized action recommendation system and, consequently, interaction
with participants is important for the proper functioning of the system. The objective
of the Participant Dialog Module is to gather user information, present potential
recommendations to the participant, gather participant preferences and present the
final recommendations. The user interacts with the software using this module that
starts with a login window. The user is first required to register, where in they are
required to enter the anthropometric data that is securely saved in the database.
Once the registration process is finished, the user can use this login information to
use the PWM services anytime later. For a returning user, after logging into the
software, a performance visualizer window is presented wherein the user can enter his
68
User Details Collector
Action Set Generator
USER
Action Recommender
<Update>
<Accept>
Reflection Mapper
Weight Dynamics Model
Estimator
Reward Mapper
GSF
GUI
DB
Expert
Act
ion
Se
lect
ion
A
gen
t
Performance Analyzer
Perf
orm
ance
M
app
er
User Measurements
<Participate>
Human Behavioral Model
< Profile>
Performance Visualizer
MEAN$ Estimation
<Feedb
ack>
Dat
abas
e C
on
nec
tio
n
Action Manager
<Statistics>
Motivational Questionnaire
Figure 5.1: The Personal Wellness Management System Architecture Design
actual nutrition and exercise activities on a per day basis. The data flow model for
the user login screen is presented in Figure 5.2
The participant initially is presented with a login screen, upon requesting to
use the PWM software services. Since no assumptions can be made on the platform,
or technical capabilities of the participant, the software design should be platform
independent and easy to be used. The decision-maker and the participant interacts
with the software using the GUI, which is responsible to show the requested infor-
mation in an ambiguous way and also to securely store the information inputed by
the user into the database. Any service that is requested the user, will then have to
operate on this data stored in the database.
5.1.2 Performance Analyzer
This module can be used by the user to estimate how their weight change trajectories
looks like and accordingly set performance objectives with the help of decision-maker.
69
Figure 5.2: Activity Flow in the Participant Dialog Module
Through this module, the user can retrieve historic data and other body metric
measurements that are taken by the decision-maker during each visit. The current
implementation of human weight dyanmics is the three compartment model of the
human weight dynamics by the NIDDK as discussed in Section 4.2. The weight
dynamic models in its current form, do not take into account the motivation of the
participant or any other human behavior models. Moreover, based on investigations
reported in [73] by Mahamadi, there is ambiguity in the accuracy of these models.
Therefore to allow for a future integration of these models into the weight dynamics
models and also allow a scalable deployment through RESTful APIs and to seamlessly
integrate the model into the PWM software architecture, the decision was made
70
to implement the three compartment model from scratch in Java using the Object
Oriented software design principles shown in the Figure 5.4.
The corresponding class diagram is shown in Figure 5.3. The three compart-
ment model is derived from its parent class, Human Weight Dynamic Model. This
class, makes use of the Action object to retrieve the daily nutrition input and exercise
expenditure calories. The anthropometric information retrieved from the database is
then used to integrate the differential equations using the Rangekutta method.
Figure 5.3: The class diagram of the implementation of Three compartment modelof the human body weight dynamics.
5.1.3 Goal Seeking Framework
The module implements all the artifacts of the Goal-Seeking Paradigm as described
in Section 4. A list of these artifacts along with the description of all implemented
Java Classes is presented in Table 5.1.
71
Figure 5.4: Implementation of the three compartment human weight dynamic modelsin JAVA. Simulating a hypothetical Participant aged 27 years, 100KG and 1.7 mheight, sedentary life style, eating 2000 Calories/day and doing moderate intensityexercises will weigh approximately 88.5 Kgs in 180 Days.
5.1.4 Database Architecture
The following section presents the detailed design of the database architecture, as
it plays a central role in the operationalization of the PWM software system. The
PWM Store was designed to provide access to participant data in an Object Oriented
framework. This store was designed as a layer above a traditional relational database.
To accommodate the different update rates for different data in the system, the data
tables were classified as three groups:
1. Meta Data: These data defined the parameters that were used to specify
exercises and participant profiles. These data remain constant after system
setup. There are no additions or deletions throughout the lifetime of the system.
72
2. Static Data: These data do not change after being entered into the PWM Store.
New records can be added to these data as new exercises and participants are
incorporated.
3. Dynamic Data: These data change frequently, perhaps, with each new inter-
action with the participants. The data reflect participant inputs and the results
of computations in the system.
A hierarchical design of the database based on the above considerations is
shown in Figure 5.5.
5.2 Versatile and Scalable Deployment Architecture
The objective of the personal wellness management software suite, is to provide var-
ious web services to the participant such as nutrition and exercise data logging, nu-
trition and exercise preferences, estimate weight trajectory for a certain nutrition
73
Meta Data
exercise types exercise levels exercise goals
participant occupations
Static Data
exercises
participants
participant exercise health
Dynamic Data
participant exercise preferences
participant general health
administrators
participant activity history
participant exercise schedules
exercise categories
activity levels
Figure 5.5: Multitier Structure of Database Storage in PWM.
and exercising behaviors. This software suite is designed to be used by exercise and
nutrition experts to monitor each participant progress and also to interact with the
decision support tool in recommending actions to them. Such services are typically
computationally expensive and require Giga Bytes of memory to operate on. Further-
more, the software suite should allow multiple clients and exercise professionals to
login at the same time. To improve the end user satisfaction, the software tool should
be able to run on different platforms and operating systems. Therefore a versatile
and scalable deployment architecture is to be designed. In this thesis, a RESTful web
service architecture is employed.
74
REST is an architecture style for designing networked applications. The idea
is that, rather than using complex mechanisms such as CORBA, RPC or SOAP to
connect between machines, simple HTTP is used to make calls between machines.
RESTful applications use HTTP requests to post data (create and/or update), read
data (e.g., make queries), and delete data. This architecture allows to decouple server
and client software and makes the implementation independent of each other. This
makes it easy to add or upgrade server web services with out changing the client
implementation. Figure 5.6 illustrates the design of deployment architecture pursued
in this investigation.
The Goal Seeking framework, along with its artifacts such as Reflection Map-
per (RM), Cost Mapper (CM) and Evaluation Mapper(EM) are all deployed in a
remote server that hosts several other web services such as recommender systems1.
Each web services has its own Uniform Resource Identifier (URI) that the client
software uses to explicitly mention to the server which service it wants to use. For
example, if the end user participant would like to get a weight trajectory for a specific
nutrition and exercise behavior, the client software would send a HTTP request to
the following URI
POST http://cesl.uakron.edu/pwm/reflectionMapper/weightTrajectory
This design makes no assumptions as to which device or operating system
that the participant or expert uses to use these services. Any data transfer that
happens between the client and the server is using HTTP requests and responses.
The data that is exchanged between client and server during these communications
is represented using standard JSON format. JSON is a lightweight data-interchange
format that makes it easy for for machines to parse and generate. It is based on a
subset of the JavaScript Programming Language, Standard ECMA-262 3rd Edition.
1Not discussed in this thesis
75
JSON is a text format that is completely language independent but uses conventions
that are familiar to programmers of the C-family of languages, including C, C++,
C#, Java, JavaScript, Perl, Python, and many others. These properties make JSON
an ideal data-interchange language.
Figure 5.7 illustrates an example of how RESTful API operates. When the
participant wants to use the service offered from Reflection mapper, the client software
initiates a HTTP request with all the data required for the service to operate on, in a
JSON object. This object is placed in the HTTP POST request and sent to the server.
The server upon serving the request, creates another JSON object and populates it
with the result of the computation requested by the client. This object is placed
inside the HTTP response sent by the server to the client. Through this method of
communication, the sever implementation of the web service can be upgraded with
out changing the client’s implementation.
76
Goal Seeking Artifact Class Name DescriptionAction PWMAction The Nutrition and Exercise tuple.
ActionOutcome An enumerator class to keep trackof adherence levels.(Adhered orNot-Adhered)
Consequence The estimated consequence ofeach action. In the current im-plementation, this class estimatesthe amount of weight change bypursuing a given action.
IntensityLevel The intensity level of an exerciseand nutrition activity to be se-lected by the participant at eachtime epoch.
Cost Mapper CostMapper The parent class for the CostMapper artifact of Goal Seeking.
UtilityCurve The utility curve parameters thatcaptures the decision-maker pref-erences of the consequences ofeach action.
Reflection Mapper ReflectionMapper The parent class that estimatesthe consequences of each action inthe action set
BodyWeightDynamicsModel The implementation of the threecompartment model
Evaluation Mapper EvaluationMapper The parent class for the evalua-tion mapping artifact for the goalseeking paradigm
PerformanceEvaluator The child class that evaluateswhether the action recommendedin the previous time step has beenadhered or not.
Tolerance Level ToleranceLevel A class that captures the toler-ance level for the action recom-mended. In the current imple-mentation, a tolerance level of1KG is used.
Table 5.1: The Goal Seeking Artifacts and the corresponding classes implemented
77
JSONInterfaceforParticipantDialogModule(PDM)
Internet<HTTP>
PWMStorage
MobileAppsWebApps
PDM
ExerciseRecommenderSystemsBackendLibrary
IM DKM EM
RESTfulWebService
GoalSeekingFrameworkBackendLibrary
RM CM EM
Figure 5.6: RESTful API for Scalable Deployment of PWM. The web services pro-vided for Goal Seeking resides in the server along with other services in the PWMwellness software suite. The client and server interact with each other using the JSONdata format.
78
79
Figure 5.7: An example of client software initiating with a HTTP request to use the reflection mapper web service hosted on the server module through RESTful api. The data required for the server is formatted into a JSON object and is inserted in post request. The server after processing the request, formats its response into another JSON object and places it into the HTTP Response that is then sent to the client.
CHAPTER VI
EVALUATING AND GUIDING EXERCISE BEHAVIORS
Recognizing and analyzing the activity when the participants perform exercises is an
important problem for improving Personalized Wellness Management (PWM). Such
recognition and analysis of human motion is important in many applications including
and assistive technologies [118] to improve quality of life [119, 120, 121, 122].
This chapter presents two applications of the Kinect Camera to improve ex-
ercise performance and adherence. The first application focuses on detecting errors
in exercises that could result in injuries and, hence, a lack of adherence. The second
application focuses on estimating the ground reaction forces while performing exer-
cises by using only the Kinect camera. These forces were later used in estimating
calories burnt during the physical activity in real-time [21]
6.1 Supporting Exercise Performance using Kinect Camera
In the recent times, it is not uncommon that participant’s perform basic exercises at
home in the absence of physical trainer. In these home settings, wearable sensors are
not the performers first choice as it hinders participant’s motion and performance.
In such scenarios, technology can play an important role in monitoring the exercise
performance and provide real time feedback to the participant to minimize injury
risk and improve home exercise adherence. Such technologies should be reliable in
monitoring the activity and fast enough to provide real time feedback. The sensors
80
that provide data to these software should be cost efficient and flexible to mount such
that they do not hinder the participant’s range of motion [123].
Toward this end, the scope of the analysis on time-series data obtained from
a non-invasive sensor, such as the Microsoft Kinect 2.0, was explored. Specifically, we
selected a set of 4 body weight exercises [124] as shown in Figure 6.1, and used the
data captured by the Kinect camera to identify the errors during exercise performance
of the participant1.
The data was collected from 46 participants. After completion of the IRB
process, each participant performed all the four exercises. Hence, a total of 184
datasets were collected. Each dataset contained 75 time series, three for each joint,
that was captured from the Kinect 2.0 camera at 30 frames per second for the duration
of the exercise. Participants were required to perform 10 repetitions of each exercise
with a short break between exercises. This data was separated into a training set
with 64 data sets and a testing set with the remaining 120 data sets. The training
set was used to train the SVM classifier. The testing set was used to evaluate the
goodness of the classifier and the results are presented in the confusion matrices.
6.1.1 Sagittal, Frontal and Transverse Plane Projections
The first feature vector considered was derived by projecting the time-series data on
the Sagittal, Frontal and Transverse planes. These three planes are important as
exercise science experts and medical professionals evaluate the exercises using these
three planes. Recall that the data obtained from the Kinect camera is the 3D, i.e.,
(x, y, z), coordinates for each joint, where the z coordinate corresponds to the depth
1The results reported in this chapter were achieved in collaboration with Master’s students [20,21, 23] in the Complex Engineered Systems lab, in the Department of Electrical and ComputerEngineering, University of Akron. The overall design and strategy was developed as a part of thisinvestigation and the Master’s students completed the detailed design, implementation and collecteddata to validate the approach. There are additional results reported in this chapter that were not apart of the Master’s thesis efforts.
81
(a) Jumping Jacks (b) lunges
(c) squat (d) high knees
Figure 6.1: Four Exercises that were selected from the HICT Suite for this study.
or distance from the camera. The XY -pane, Y Z-plane and ZX-plane corresponds
to Frontal plane, Sagittal plane and Tranverse plane respectively. Figure 6.2 presents
sample data captured from the Kinect camera. This figure also shows the trajectories
of the (x, y) positions of the data as an illustration.
Figure 6.3 shows the projections of the time-series data of the joints in the
Y Z-plane. These were used to compute the feature vectors. For each joint, using the
average initial position, the average final position, the extent of movement (`yz) in
82
Figure 6.2: Sample time-series data captured from the Kinect camera for the x-coordinate of a few joints is as shown on the left. The figure on the right shows thetrajectories of the (x, y) coordinates for each joint when the Jumping Jacks exerciseis performed.
83
Figure 6.3: Projection of all the joints on the Y Z-Plane for the Jumping Jacksexercise. This was obtained by drawing a line between the average starting positionand the average finish position of each joint illustrated in Figure 6.2.
the plane was measured. The slope (θyz) of each line with respect to the horizontal,
i.e., the Y -axis was also computed (In the Y Z-plane, Y -axis is the horizontal).
The following six scalar values θxy, θyz, θxz, `xy, `yz and `xz were computed
for each of the 25 joints tracked by the Kinect camera. In Table 6.1, these values are
illustrated for the first few joints. By concatenating these values for all the joints, a
150-dimensional vector was obtained. Each vector had a unique signature for each of
the exercises and hence this 150-dimensional vector was used as the feature vector.
Figure 6.4: Confusion Matrix achieved using the projections-based feature vector
84
Table 6.1: An illustration of the first 18 values of the 150-dimension feature vectorfor the Jumping Jacks exercise.
Joint θxy θyz θzx `xy `yz `zx
0 73.18 64.49 8.21 382.42 850.00 775.07
1 66.29 65.31 11.42 131.93 289.16 268.02
2 11.99 72.92 55.35 187.72 132.79 223.22
The feature vector described above were used to train a SVM and recognize
a new instance of an exercise. The accuracy of the SVM can be characterized by a
confusion matrix as shown in Fig.6.4. Each row of the matrix indicates the input
and each column indicates the output. Hence, the accuracy of prediction is along the
diagonals. It can be noticed that the accuracy of the SVM using projections-based
feature vector in recognizing the jumping jack exercise among 3 other exercises is
94.4%.
To illustrate exercising error detection using the Kinect camera, horizontal
shoulder adduction exercise is used as an example. The standing horizontal shoulder
adduction exercise requires the participant to maintain their posture as well as control
of the resistance band during the concentric and eccentric phases of the exercise.
This exercise primarily targets the chest and anterior deltoid muscles and requires
movement of the shoulder girdle with minimal movement of any other body joints.
Furthermore, the motion of the wrists should be smooth as the wrists approach the
end of the concentric phase. If the participant cannot handle tension in the resistance
band at the end of the concentric phase, the wrists tend to vibrate, which is classified
as an error.
85
Figure 6.5: Comparison of the Standing shoulder adduction exercise performed by anexpert participant(on the left) and a beginner participant(on the right). Observe lotof variations in the wrist position of the beginner participant. This shows that thebeginner participant is not in control of the exercise and is prone to injury.
Figure 6.5, plots the 3-D motion of 9 out of 25 joints as tracked by the
Kinect sensor. The figure on the left depicts the advanced exerciser while on the
right corresponds to the beginner exerciser. It can be seen that the motion of the
wrist joints of the advanced exerciser is uniform along the y axis while the motion
of other joints are minimal during full execution of the exercise. This indicates that
the participant is in full control of the motion and is using only the targeted muscles
while performing the exercise. Also observe in the figure, minimal variation of the
head, hip joints of the advanced exerciser compared to the beginner. This indicates
that the advanced exerciser, is in complete control of the exercise and maintains a
good body posture.
86
6.2 Multiple Kinect Cameras
The Kinect camera and the skeletal tracking algorithm provide 3D coordinates for the
location of twenty five joints. These coordinates are with respect to the location of the
camera as the origin. As already noted, when an exercise involves a complex sequence
of motions, not all the joints can be tracked by a single camera because of occlusions.
Multiple Kinect cameras are threfore used to accurately track the locations of all the
joints at all times. However, if a joint is occluded on one Kinect, the data from the
other Kinect is required to transform into one frame of reference. This problem is
solved using the technique Single Value Decomposition discussed in [23] and is out of
scope of the thesis.
To illustrate how the above transformation technique worked, the multiple
Kinect camera setup as shown in Figure 6.6 was laid out. A participant was asked to
perform Lunges exercise, while a front facing (Kinect Camera 1) and a side (Kinect
camera 2) facing cameras recorded the exercise activity. This exercise was selected
because during the exercise activity there would be brief moments during which the
back leg ankle joint is occluded in the Kinect camera 1. This behavior is shown in
Figure 6.7. It can be observed from the figure that when the participant back knee
bends down and nearly touches the ground, the knee joint occludes the ankle joint,
hence the data from the Kinect camera 1 goes from tracking(1) to un-tracked(0)
periodically. However, observe that the Kinect camera 2 has no problem tracking
this joint as evident in the bottom plot. The above discussed SVD technique is then
applied to Kinect camera 2 ankle joint data to transform into the Kinect camera 1
frame of reference as shown in Figure 6.8. Observe how the transformed data (green
data points) from Kinect camera 2 makes up for the untracked data frames from
Kinect camera 1(blue data points).
87
∆∆
Kinect Camera 1
Figure 6.6: The 3D Joint coordinates from each camera are provided by consideringthe location of the camera as the origin. When data for the same exercise are collectedusing multiple cameras, it is necessary to translate and/or rotate the coordinates fromone camera to the frame of reference of the other camera.
88
Figure 6.7: Demonstrating the need of multiple Kinect camera. While a participantperforming Lunges exercise, the Kinect camera 1 has un-tracked frames of ankle jointduring brief moments when the participant bent forward. During this period, Kinectcamera 2 has no problem tracking the joint as its positioned with an angle to theparticipant. Therefore Kinect camera 2 frames can be used in place of Kinect camera1, however they need to be transformed to Kinect camera 1 frame of reference.
89
Figure 6.8: Superimposed frames from Kinect camera 1 frames(blue data points)and transformed Kinect camera 2(green data points) data frames for the ankle jointduring a Lunges exercise. Observe the green data points cluster highlighted in thecircle appear in the absence of blue data points. The combined data can now be usedto analyze if the participant made an error during the exercise activity.
90
Tab
le6.
2:P
oten
tial
Err
ors
inJum
pin
gJac
ks
HIC
TE
xer
cise
s
Poss
ible
Pro
ble
mA
reas
of
HIC
TE
xerc
ises
NP
osi
tion
KeyJoin
ts
Not
landin
gon
the
bal
lof
the
foot
duri
ng
jum
p1
Fro
nta
lK
nee
[13,17] ,
Ankle
[14,18]
and
Foot
[15,19]
Not
keep
ing
hea
dst
raig
ht
orey
esfo
rwar
dw
hen
jum
pin
g1
Fro
nta
lH
ead[3] ,
Nec
k[2]
and
Spin
esh
ould
er[20]
Knee
sar
enot
flex
edat
the
tim
eof
landin
g1
Fro
nta
lK
nee
[13,17]
Arm
san
dle
gsar
enot
coor
din
ated
(not
synch
roniz
ed)
1F
ronta
lK
nee
[13,17]
and
Elb
ow[5,9]
Dis
tance
bet
wee
nth
ele
gsto
om
uch
orto
olitt
le1
Fro
nta
lF
oot
[15,19]
Jer
km
ovem
ent
ofth
eju
mpin
gja
cks
1F
ronta
lK
nee
[13,17] ,
Ankle
[14,18]
and
Elb
ow[5,9]
91
Tab
le6.
4:P
oten
tial
Err
ors
inW
all
Sit
HIC
TE
xer
cise
s
Poss
ible
Pro
ble
mA
reas
of
HIC
TE
xerc
ises
NP
osi
tion
KeyJoin
ts
Quad
rice
ps(
thig
h)
and
low
erle
gar
enot
90deg
rees
2Sag
itta
l[L
,R]
Hip
[12,16] ,
Knee
[13,17]
and
Ankle
[14,18]
Bac
kis
not
flat
agai
nst
the
wal
l1
Fro
nta
lSpin
ebas
e[0] ,
Spin
em
id[1]
and
Spin
esh
ould
er[20]
Bac
kis
arch
ed1
Fro
nta
lSpin
ebas
e[0] ,
Spin
em
id[1]
and
Spin
esh
ould
er[20]
Hea
d,
shou
lder
san
dhip
touch
the
wal
lflat
1F
ronta
lH
ead[3] ,
Shou
lder
[4,8]
and
Hip
[12,16]
Hee
lsnot
kept
ongr
ound
1F
ronta
lK
nee
[13,17] ,
Ankle
[14,18]
and
Foot
[15,19]
92
Tab
le6.
6:P
oten
tial
Err
ors
inP
ush
Up
HIC
TE
xer
cise
s
Poss
ible
Pro
ble
mA
reas
of
HIC
TE
xerc
ises
NP
osi
tion
KeyJoin
ts
Sta
rtin
gp
osit
ion
the
body
isnot
stra
ight
from
hee
lto
hea
d2
Sag
itta
l[L
,R]
Hea
d[3] ,
Nec
k[2] ,
Spin
ebas
e[0] ,
Hip
[12,16] ,
Ankle
[14,18] ,
and
Knee
[13,17]
Butt
ishig
hor
low
atst
arti
ng
pos
itio
n2
Sag
itta
l[L
,R]
Ankle
[14,18] ,
Hip
[12,16]
and
Spin
ebas
e[0]
Han
dp
osit
ion
isto
ow
ide
ornar
row
(must
be
shou
lder
wid
thap
art)
1F
ronta
lShou
lder
[4,8]
and
Han
d[7,11]
Dow
nan
dup
body
mot
ion
isnot
smoot
h(m
ust
godow
nan
dup
ason
eunit
)1
Sag
itta
lShou
lder
[4,8]
Elb
ows
stic
kou
tw
hen
the
body
goes
dow
n(e
lbow
sm
ust
be
clos
eto
body
duri
ng
dow
nan
dup
mov
emen
tof
the
body)
3F
ronta
lan
dSag
itta
l[L
,R]
Elb
ow[5,9]
and
Shou
lder
[4,8]
Hea
dis
hyp
erex
tended
duri
ng
upw
ard
mov
emen
tor
chin
isflex
eddow
nw
ard
1Sag
itta
lH
ead[3]
and
Nec
k[2]
93
Tab
le6.
7:P
oten
tial
Err
ors
inA
bdom
inal
Cru
nch
HIC
TE
xer
cise
s
Poss
ible
Pro
ble
mA
reas
of
HIC
TE
xerc
ises
NP
osi
tion
KeyJoin
ts
Bac
kis
not
flat
onth
egr
ound
duri
ng
crunch
1T
ransv
erse
Spin
ebas
e[0] ,
Spin
em
id[1]
and
Spin
esh
ould
er[20]
Knee
flex
ion
isto
om
uch
orto
olitt
leduri
ng
the
crunch
pos
itio
n1
Tra
nsv
erse
Knee
[13,17]
Hea
dis
flex
edto
om
uch
tow
ards
the
ches
t(m
ust
be
stra
ight)
duri
ng
the
crunch
(upw
ard
mot
ion)
1Sag
itta
lH
ead[3] ,
Nec
k[2]
and
Spin
eShou
lder
[20]
Fee
tar
enot
stat
ionar
yon
the
floor
when
crunch
acti
onis
don
e(m
ust
be
onth
egr
ound)
2Sag
itta
l[L
,R]
Knee
[13,17] ,
Ankle
[14,18]
and
Foot
[15,19]
Shou
lder
isro
unded
duri
ng
the
crunch
acti
on(m
ust
be
stra
ight)
2Sag
itta
l[L
,R]
Shou
lder
[4,8]
Han
dar
enot
stra
ight
duri
ng
the
crunch
acti
on2
Sag
itta
l[L
,R]
Shou
lder
[4,8] ,E
lbow
[5,9]
and
Han
d[7,11]
94
Tab
le6.
8:P
oten
tial
Err
ors
inSte
p-u
pon
toC
hai
rH
ICT
Exer
cise
s
Poss
ible
Pro
ble
mA
reas
of
HIC
TE
xerc
ises
NP
osi
tion
KeyJoin
ts
The
step
pin
gle
gon
the
chai
ris
not
90deg
rees
atth
ehip
,knee
and
ankle
2Sag
itta
l[L
,R]
Hip
[12,16] ,
Knee
[13,17]
and
Ankle
[14,18]
Duri
ng
clim
bth
eupp
erb
ody
isnot
stra
ight
1F
ronta
lSpin
ebas
e[0] ,
Spin
em
id[1]
and
Spin
esh
ould
er[20]
Body
isnot
stra
ight
afte
rth
eco
mple
tion
ofth
ecl
imb
1F
ronta
lH
ead[3] ,
Nec
k[2] ,
Spin
em
id[1] ,
Spin
ebas
e[0] ,
Hip
[12,16] ,
Knee
[13,17]
and
Ankle
[14,18]
The
step
pin
gle
g(dow
n)
isnot
flex
edat
conta
ct1
Fro
nta
lK
nee
[13,17] ,
Ankle
[14,18]
and
Foot
[15,19]
Aft
erth
est
eppin
gle
g(d
own)
conta
cts
the
grou
nd
the
chai
rle
gm
ust
be
90deg
rees
athip
,knee
and
ankle
1F
ronta
lH
ip[12,16] ,
Knee
[13,17] ,
and
Ankle
[14,18]
The
whol
eac
tion
isnot
exec
ute
din
”one
smoot
hm
ovem
ent”
from
star
tto
finis
h2
Sag
itta
l[L
,R]
all[1−25]
Chai
rhei
ght
isto
hig
hw
hic
hcr
eate
sex
acer
bat
ion
2Sag
itta
l[L
,R]
Hip
[12,16] ,
Knee
[13,17]
and
Ankle
[14,18]
95
Tab
le6.
9:P
oten
tial
Err
ors
inSquat
HIC
TE
xer
cise
s
Poss
ible
Pro
ble
mA
reas
of
HIC
TE
xerc
ises
NP
osi
tion
KeyJoin
ts
Sta
rtin
gfe
etp
osit
ion
isnot
shou
lder
wid
thap
art(
too
nar
row
orto
ow
ide)
1F
ronta
lF
oot
[15,19]
and
Shou
lder
[4,8]
At
Squat
pos
itio
nth
ean
gle
atth
eknee
and
ankle
isnot
90deg
rees
2Sag
itta
l[L
,R]
Hip
[12,16] ,
Knee
[13,17] ,
Ankle
[14,18]
and
Foot
[15,19]
Tru
nk
pos
itio
nis
not
stra
ight
and
ben
tat
hip
sfo
rwar
d1
Fro
nta
lSpin
ebas
e[0] ,
Spin
em
id[1]
and
Spin
esh
ould
er[20]
Knee
sar
enot
dir
ectl
yov
erto
es1
Fro
nta
lK
nee
[13,17]
and
Foot
[15,19]
Han
ds
are
not
stra
ight
and
par
alle
lto
the
grou
nd
2Sag
itta
l[L
,R]
Shou
lder
[4,8] ,
Elb
ow[5,9]
and
Wri
st[6,10]
Knee
sb
owin
war
dor
outw
ard
1F
ronta
lK
nee
[13,17]
96
Tab
le6.
10:
Pot
enti
alE
rror
sin
Tri
ceps
dip
onC
hai
rH
ICT
Exer
cise
s
Poss
ible
Pro
ble
mA
reas
of
HIC
TE
xerc
ises
NP
osi
tion
KeyJoin
ts
Sta
ring
pos
itio
nof
the
body
onth
ech
air
isnot
stra
ight(
hee
lto
hea
d)
3F
ronta
lan
dSag
itta
l[L
,R]
Hea
d[3] ,
Nec
k[2] ,
Spin
em
id[1] ,
Spin
ebas
e[0] ,
Hip
[12,16] ,
Knee
[13,17]
and
Ankle
[14,18]
Han
ds
tow
ide
apar
t1
Fro
nta
lH
and
and
Shou
lder
[4,8]
Han
ds
supp
orti
ng
the
chai
ris
not
stra
ight
atel
bow
s2
Sag
itta
l[L
,R]
Shou
lder
[4,8] ,
Elb
ow[5,9]
and
Han
d[7,11]
At
dip
the
arm
do
not
com
ple
teth
efu
llra
nge
ofm
otio
nin
flex
edp
osit
ion
3F
ronta
lan
dSag
itta
l[L
,R]
Shou
lder
[4,8] ,
Elb
ow[5,9]
and
Han
d[7,11]
Duri
ng
the
dip
the
body
isnot
goin
gst
raig
ht
dow
nw
ith
legs
stra
ight
3F
ronta
lan
dSag
itta
l[L
,R]
Knee
[13,17] ,
Hip
[12,16]
and
Spin
ebas
e[0]
Elb
ows
are
not
at90
deg
rees
atfu
lldip
3F
ronta
lan
dSag
itta
l[L
,R]
Shou
lder
[4,8] ,
Elb
ow[5,9]
and
Han
d[7,11]
97
Tab
le6.
11:
Pot
enti
alE
rror
sin
Pla
nk
HIC
TE
xer
cise
s
Poss
ible
Pro
ble
mA
reas
of
HIC
TE
xerc
ises
NP
osi
tion
KeyJoin
ts
The
conta
ctar
eaw
ith
grou
nd
isnot
flat
wit
hth
efo
rear
ms
3F
ronta
lan
dSag
itta
l[L
,R]
Han
d[7,11] ,
Elb
ow[5,9]
and
Shou
lder
[4,8]
For
earm
sar
enot
shou
lder
wid
thap
art(
too
nar
row
orto
ow
ide)
3F
ronta
lan
dSag
itta
l[L
,R]
Elb
ow[5,9]
and
Shou
lder
[4,8]
The
whol
eb
ody
isnot
stra
ight
from
hea
dto
hee
l(b
utt
inline
wit
hth
ere
stof
the
body)
3F
ronta
lan
dSag
itta
l[L
,R]
Hea
d[3] ,
Nec
k[2] ,
Spin
em
id[1] ,
Spin
ebas
e[0] ,
and
Knee
[13,17]
Angl
esat
ankle
,sh
ould
eran
del
bow
isnot
90deg
rees
3F
ronta
lan
dSag
itta
l[L
,R]
Han
d[7,11] ,
Elb
ow[5,9] ,
Shou
lder
[4,8] ,
Hip
Knee
[13,17] ,
Ankle
[14,18]
and
Foot
[15,19]
98
Tab
le6.
12:
Pot
enti
alE
rror
sin
Hig
hK
nee
s/R
unnin
gin
Pla
ceH
ICT
Exer
cise
s
Poss
ible
Pro
ble
mA
reas
of
HIC
TE
xerc
ises
NP
osi
tion
KeyJoin
ts
Upp
erto
rso
isnot
stra
ight(
upri
ght)
duri
ng
the
leg
dri
ves
1F
ronta
lSpin
ebas
e[0] ,
Spin
em
id[1]
and
Spin
esh
ould
er[20]
99
Tab
le6.
13:
Pot
enti
alE
rror
sin
Lunge
HIC
TE
xer
cise
s
Poss
ible
Pro
ble
mA
reas
of
HIC
TE
xerc
ises
NP
osi
tion
KeyJoin
ts
At
lunge
the
upp
erb
ody
isnot
stra
ight
1F
ronta
lSpin
ebas
e[0] ,
Spin
em
id[1]
and
Spin
esh
ould
er[20]
The
step
pin
gle
gis
not
90deg
rees
atan
kle
,knee
and
hip
2Sag
itta
l[L
,R]
Hip
[12,16] ,
Ankle
[14,18] ,
Knee
[13,17]
and
Foot
[15,19]
Knee
isnot
dir
ectl
yov
erth
eto
es1
Fro
nta
lK
nee
[13,17]
and
Foot
[15,19]
Rea
rle
gis
not
90deg
rees
atth
eK
nee
2Sag
itta
l[L
,R]
Hip
[12,16] ,
Knee
[13,17]
and
Ankle
[14,18]
Dip
duri
ng
the
lunge
isnot
stra
ight
dow
n1
Fro
nta
lSpin
ebas
e[0]
Not
step
pin
gat
the
consi
sten
tsp
otduri
ng
the
lunge
1F
ronta
lF
oot
[15,19]
100
Tab
le6.
14:
Pot
enti
alE
rror
sin
Push
-Up
HIC
TE
xer
cise
s
Poss
ible
Pro
ble
mA
reas
of
HIC
TE
xerc
ises
NP
osi
tion
KeyJoin
ts
Duri
ng
rota
tion
the
arm
s(b
oth)
do
not
mak
e90
deg
atth
eto
rso
2Sag
itta
l[L
,R]
Spin
em
id[1] ,
Spin
esh
ould
er[20] ,
Elb
ow[5,9]
Arm
saf
ter
rota
tion
are
not
inst
raig
ht
line
2Sag
itta
l[L
,R]
Han
d[7,11] ,
Elb
ow[5,9]
and
Shou
lder
[4,8]
Supp
orti
ng
arm
isnot
stra
ight
alon
gw
ith
the
shif
ting
arm
2Sag
itta
l[L
,R]
Han
d[7,11] ,
Elb
ow[5,9] ,
Shou
lder
[4,8] ,
Spin
esh
ould
er[20]
101
Tab
le6.
15:
Pot
enti
alE
rror
sin
Sid
eP
lank
HIC
TE
xer
cise
s
Poss
ible
Pro
ble
mA
reas
of
HIC
TE
xerc
ises
NP
osi
tion
KeyJoin
ts
The
conta
ctin
gfo
rear
mis
not
at90
deg
atth
eel
bow
and
shou
lder
1Sag
itta
lShou
lder
[4,8] ,
Elb
ow[5,9]
and
Han
d[7,11]
The
tota
lb
ody(h
ead
tosi
de
ofth
efo
ot)
isnot
ina
stra
ight
line
duri
ng
the
pla
nk
1Sag
itta
lH
ead[3] ,
Nec
k[2] ,
Spin
esh
ould
er[20] ,
Hip
[12,16] ,
Knee
[13,17]
and
Ankle
[14,18]
The
hip
sar
enot
inline
wit
hth
ere
stof
the
body(t
oo
low
orto
ohig
h)
1Sag
itta
lSpin
ebas
e[0] ,
Hip
[12,16]
and
Knee
[13,17]
Hea
dan
dnec
kar
enot
inline(
stra
ight)
wit
hth
ere
stof
the
body(t
oo
low
orto
ohig
h)
1Sag
itta
lH
ead[3] ,
Nec
k[2] ,
Spin
esh
ould
er[20]
and
Spin
em
id[1]
Res
ting
han
dis
not
stra
ight
atth
esi
de
ofth
eb
ody
duri
ng
the
pla
nk
1Sag
itta
lSpin
ebas
e[0] ,
Spin
em
id[1] ,
Spin
esh
ould
er[20] ,
Shou
lder
[4,8] ,
Elb
ow[5,9]
and
Han
d[7,11]
102
6.3 Estimating Ground Reaction Forces during a Vertical Jump
Figure 6.9 illustrates the kinematics of a single jump. This figure plots the spine base
joint position from the data collected using the kinect camera. Point A is the initial
start position from rest. During the time interval between B to C, the participant
squats down in preparation for jump. The interval from C to D is the take-off time
during which the participant exerts force on to the force plate. The point D is the
exact time instance at which the maximum force is applied on the force plate and the
participant takes-off into the air. The interval from D to E is the time for which the
participant is in the air. Time E is the exact instance at which the participant touches
down on to the Force plate. During E to F the participant is in the landing phase
and starts to exert force again on to the force plate. The interval from F to G is the
time it takes for the participant to recoil back to the rest position. To calculate the
ground reaction forces during the landing and takeoff phases, the impulse-momentum
method as described in section 6.3.1is used. In order to compare the accuracy of the
results from the data collected using Kinect 2.0, a traditional force place was used
that directly reported the ground reaction forces.
6.3.1 Impulse-Momentum Approach
Since the forces between the participant feet and ground are equal in magnitude and
opposite in direction, and since the times for which these forces act are equal in
magnitude, it follows that the impulses experienced by the participant and ground2
are also equal in magnitude and opposite in direction. As an equation during the
take-off and landing phases, this can be stated as
2The participant was required to stand on a force plate during the jump
103
Figure 6.9: Plot of spine base over a single jump.
∫ D
B
FGRF dt = m · vto +
∫ D
B
m · g dt Take-off phase (6.1)∫ F
E
FGRF dt = m · vtl +
∫ F
E
m · g dt Landing phase (6.2)
where, FGRF is the ground reaction forces captured using the force plate, vto is the
instantaneous velocity during the Take-off phase calculated from the data collected
from Kinect or Vicon, vtl is the instantaneous velocity during the Landing phase cal-
culated from the data collected from Kinect or Vicon, m is the mass of the participant,
and g is the gravitational acceleration (9.8 m/sec2).
Figure 6.10 verifies that our data is in accordance with what is expected from
equation 6.1. It can be observed from the figure that the impulse calculated at the
104
Figure 6.10: Impulse calculated using force plate, vicon and kinect data. The graphsshows that the impulse calculated using force plate data is approximately equal toimpulse calculated using kinect and vicon data. This shows that the change in mo-mentum is conserved during jumping phase.
force plate is approximately equal to the impulse calculated using the data collected
from Kinect and Vicon.
105
CHAPTER VII
DISCUSSION
In this thesis, the problem of Personalized Wellness Management was formulated as
a decision making under uncertainty problem, where an individual makes a series
of decisions in each time step, regarding the amount of nutrition calorie intake and
exercise calorie expenditure to indulge in each day. The outcome of such decisions
depends on various factors such as individual body metabolism, the ability to adhere
to the action decided, the motivation level of the individual, various metabolic changes
to the human body, etc. Under an appropriate set of assumptions, this problem was
mathematically formulated under Completely Observable Markov decision processes
and Partially-Observable Markov decision processes. Chapter 3 presented the system
model under each of these frameworks and the use of various commercially available
solvers to solve them.
To formulate the problem as a Markov decision processes, it was necessary for
the MDP State to be completely observable, and hence the weight of the participant
was chosen as it can be measured with out ambiguity. This required the transition
model between States depend on the individual body metrics and had to be derived
from carefully conducted and validated experimental studies. Since such data did
not exist, mathematical models of human weight dynamics discussed in Section 4.1.4
was used. These models however inherently possessed computation errors and did
not account for various other human factors such as ethnicity, disease condition etc.
Therefore an uncertainty model as described in Section 3.1.3 was included in the
model. Such a formulation allowed us to integrate domain expert knowledge and
106
user preferences into the framework through the use of reward functions. To demon-
strate this, three different reward functions were used namely 1) quick weight loss, in
which participant desired to reach target weight in the least amount of time 2) safe
weight loss, in which participant desired to reach target weight in a safely manner
recommended by experts and 3) personalized weight loss, in which the participant
stated his preferences in selecting exercise activity level. The outcomes of each strat-
egy was executed on an hypothetical participant and the results were described in
Section 3.3.1. Since the transition matrix and reward functions were stationary and
did not change with respect to time, the policy computed was also stationary. This
meant that if for some unknown reason the participant had trouble changing his state,
the action mapped by the policy did not change. This method also did not account
for the motivation level of the participant to adhere to the recommended action.
To account for the limitations of the Completely Observable Markov Decision
process, Partially Observable Markov decision process was explored. Since motivation
level is not directly measured, the adherence level of the participant to the recom-
mended action is used as the observation to maintain a probability distribution of
motivation level of the participant. However, this method required extensive data to
model the PWM problem through state transition matrices, observation transition
matrices. Such data was difficult to obtain and also required continuous monitoring
of the participant for long periods of time. Hence these matrices were hand con-
structed and solved using a widely accepted open source solver called pomdp-solve.
In POMDP modeling, the motivation distribution is used to map the intensity level
of the action. This intensity level is used as a filter to eliminate actions in the action
set of MDP. This filtered action set is used by the MDP to select an exercise and
nutrition action. Both the MDP and POMDP frameworks are computed for an op-
timal policy and therefore require large amounts of experimental data to formulate
107
the system dynamics. However, in the human wellness domain, we assume that the
participant is a rational agent and therefore good enough solutions suffice. The Goal-
Seeking paradigm that is presented in this thesis is one such paradigm that does not
optimize for actions, but instead selects the best action among the available choices.
The Goal-Seeking artifacts identified in section 2.7 formulated the system
model for the Personal Wellness Management domain. The three compartment model
of human weight dynamics was adapted as the reflection mapper artifact. A reward
function that captured safe weight loss dynamics was used to assign rewards to conse-
quences of each action served as the cost mapper artifact. Each action is run through
the reflection mapper to estimate the amount of weight change for the next time step
and was assigned a cost using the cost mapper. The design of the Evaluation Mapper
had multiple facets to it. In this investigation, we assumed that the decision-maker
can make the observation of whether the participant is adhered or not, by examin-
ing user reported data and physical examination. The design of more robust tools
that help the participant recognize exercising errors in real-time are currently under
investigation. The software design chapter 5 described the database, user interface
designs that were implemented for robust interaction of the participant, goal-seeking
software and the back-end database. To allow the participant to use goal-seeking
software services using either a smart phone or a PC, RESTful API’s were designed
that allowed for a flexible and scalable deployment. Several more aspects of this
system including human behavior models and tools for quantifying motivation along
with a robust tool to track nutrition behavior need to be developed. These tools must
be validated in field trials and revised before they can be commercialized.
108
CHAPTER VIII
CONCLUSIONS
This dissertation described the goal-seeking paradigm and presented a formulation
of a decision-maker for personal wellness management. The decision-maker estab-
lishes performance limits and measures the actual performance to adjust the wellness
prescription. Ultimately, the decision-maker seeks to transform individuals to be
intrinsically motivated to improve their personal wellness by recommending actions
that correlate with the adherence level of the participant. The comprehensive man-
ner in which the decision-maker incorporates knowledge from disparate domains, the
manner in which it integrates humans in the decision-making loop, and the focus on
good-enough solutions (instead of optimal solutions) make this a viable framework to
empower personal wellness management. In the future, this framework can be made
more robust by incorporating more complicated human weight dynamic models and
utility theory based action selection.
109
BIBLIOGRAPHY
[1] S.P. Keehan, G.A. Cuckler, A.M. Sisko, A.J. Madison, S.D. Smith, D.A. Stone,J.A. Poisal, C.J. Wolfe, and J.M. Lizonitz. National health expenditure pro-jections, 2014-24: Spending growth faster than recent trends. Health Affairs,2015.
[2] C.L. Ogden, M.D. Carroll, and B.K. Kit et. al. Prevalence of childhood andadult obesity in the united states, 2011-2012. The Journal of American MedicalAssociation, 311(8):806–814, 2014.
[3] L. Chapman. Meta-evaluation of worsite health promotion economic returnstudies: 2012 update. American Journal of Health Promotion, 26(4):TAHP1–TAHP12, 2012.
[4] P. Coorevits, M. Sundgren, G. O. Klein, A. Bahr, B. Claerhout, C. Daniel,M. Dugas, D. Dupont, A. Schmidt, P. Singleton, G. De Moor, and D. Kalra.Electronic health records: New opportunities for clinical research. Journal ofInternal Medicine, 274(6):547–560, 2013.
[5] D. Georges, S. Mats, and K. Dipak e.t. al. Using electronic health recordsfor clinical research: The case of the ehr4cr project. Journal of BiomedicalInformatics, 53:162–173, 2015.
[6] H. Tao, L. Liang, F. Xuexian, A. Peng, M. Junxia, and W. Fudi. Promisesand challenges of big data computing in health sciences. Big Data Research,2(1):2–11, 2015.
[7] F. Keith, D. Darcy, and C. Nitesh. Scaling and contextualizing personalizedhealthcare: A case study of disease prediction algorithm integration. Journalof Biomedical Informatics, 57:377–385, 2015.
[8] N. Kenney, A. Ghoting, S.R. Steinhubl, W.F. Stewartand B. Malin, and J. Sun.PARAMO: A PARAllel predictive Modeling platform for healthcare analyticresearch using electronic health records. Journal of Biomedical Informatics,48:160–170, 2014.
[9] S. Jimeng and C.K. Reddy. Big Data Analytics for Healthcare. In InternationalConference on Knowledge Discovery and Data, page 1525, 2013.
110
[10] H.A.Simon. Models of Man: Social and Rational- Mathematical Essays onRational Human Behavior in a Social Setting. John Willey & Sons, 1957.
[11] P.B. Shull, W. Jirattigalachote, M. A. Hunt, M. R. Cutkosky, and S.L. Delp.Quantified self and human movement: A review on the clinical impact of wear-able sensing and feedback for gait analysis and intervention. Gait and Posture,40(1):11–19, May 2014.
[12] C.C. Chow and K.D. Hall. The dynamics of human body weight change. PLoSComputational Biology, 4(3):1–13, 2008.
[13] M Chippa and S. Sastry. Battery systems management using a goal-seekingframework. In 9th Annual IEEE international Systems Conference (SysCon),pages 335–339, 2015.
[14] M. Chippa, S.M. Walen, S. Sastry, and F. Douglas. Goal-seeking formulationfor empowering personalized wellness management. In Medical Cyber PhysicalSystems, 2013.
[15] M.D. Mesarovic and Y. Takahara. Mathematical Theory of General Systems.Academic Press, 1974.
[16] M.D. Mesarovic, D.L. McGuiness, and D.A. West. Cybernetics of global change:Human dimension and managing of complexity, most policy paper. UnitedNations Educational Scientific and Cultural Organization, 1996.
[17] S. Sastry. Smartspace for automation. Assembly Automation, 24(2):201–209,2004.
[18] Y. Takahara and M.D. Mesarovic. Organization Structure - Cybernetic SystemsFoundation. Kluwer Academic, 2003.
[19] S.Sastry, O. Gimdogmus, T.T. Hartley, and R. J. Veillette. Coordinated dis-charge of a collection of batteries. Journal of Power Sources, 166(1):284–296,2007.
[20] H. Pidaparthy, S. Narayan, and S. Sastry. Automated exercise feedback systemusing a non-invasive sensor. In Workshop on Applications of Computer Vision,Graphics and Image Processing, dec 2014.
[21] S.P.R. Gaddam, M.K. Chippa, S. Sastry, A. Ange, V. Berki, and B.L. Davis.Estimating forces during exercise activity using non-invasive kinect camera. InInternational Conference on Computational Science and Computational Intel-ligence, pages 825–828, Nov 2015.
[22] P. Fernando. Hybrid recommender system for exercises. Master’s thesis, TheUniversity of Akron, Computer Engineering, 2016.
111
[23] S. Vankamamidi. Fusing joint information from multiple kinect cameras todetect errors in exercises. Master’s thesis, The University of Akron, ComputerEngineering, 2016.
[24] U.S Food and Drug Administration. Paving the way for personalized medicinefda’s role in a new era of medical product development, 2013.
[25] U. S. Office of the Actuary. National health expenditures projection. Technicalreport, Centers for Medicare and Medicaid Services, 2010.
[26] M. Barkhordari and M. Niamanesh. ScaDiPaSi: An Effective Scalable andDistributable MapReduce-Based Method to Find Patient Similarity on HugeHealthcare Networks. Big Data Research, 2(1):19–27, 2015.
[27] United States Department of Agriculture. www.choosemyplate.gov.
[28] S.R/ Colberg, R. J. Sigal, and B. Fernhall et.al. Exercise and type 2 diabetes.Diabetes Care, 33(12):e147–e167, Dec 2010.
[29] NIH Publication No. 98-4083. National institutes of health. clinical guidelines onthe identification, evaluation, and treatment of overweight and obesity in adults:The evidence report. National Heart, Lung, and Blood Institute;, September1998.
[30] R.H. Eckel, S.E. Kahn, and E. Ferrannini et.al. Obesity and type 2 diabetes:What can be unified and what needs to be individualized? The Journal ofClinical Endocrinology & Metabolism, 96(6):1654–1663, Jun 2011.
[31] V. Kotsis, S. Stabouli, and S. Papakatsika et.al. Mechanisms of obesity-inducedhypertension. Hypertension Research, 33(5):386–93, May 2010.
[32] C.J. Lavie, R. V. Milani, and H.O. Ventura. Obesity and cardiovascular disease:Risk factor, paradox, and impact of weight loss. Journal of the American Collegeof Cardiology, 53(21):1925–1932, May 2009.
[33] K. D Hall, G. Sacks, D. Chandramohan, C. C Chow, Y C. Wang, S.L. Gort-maker, and B. A Swinburn. Quantification of the effect of energy imbalance onbodyweight. Lancet, 2011.
[34] B.A. Swinburn, G. Sacks, and S.K. Lo et al. Estimating the changes in en-ergy flux that characterize the rise in obesity prevalence. American Journal ofClinical Nutrition, 89:1723–1728, 2009.
[35] J.O. Hill, H.R. Wyatt, G.W. Reed, and J.C. Peters. Obesity and the environ-ment: where do we go from here? Science, 299:853–55, 2003.
112
[36] B. Swinburn, G. Sacks, and E.Ravussin. Increased food energy supply is morethan sufficient to explain the us epidemic of obesity. American Journal ofClinical Nutrition, 90:1453–1456, 2009.
[37] K.D. Hall, J. Guo, M. Dore, and C.C. Chow. The progressive increase of foodwaste in america and its environmental impact. Plos ONE, 4(11):e7940, 2009.
[38] D. T. Levy, P. L. Mabry, Y.C.Wang, S. Gortmaker, T. T.K. Huang, T. Marsh,M. Moodie, and B. Swinburn. Simulation models of obesity: a review of theliterature and implications for research and policy. Obesity Reviews, 12:378–394,2010.
[39] A.R. Behnke, B.G. Feen, and W.C. Welham. The specific gravity of healthymen. Journal of the American Medical Association, 118(7):495–498, 1942.
[40] G.B. Forbes. Weight loss during fasting: Implications for the obese. TheAmerican Journal of Clinical Nutrition, 23(9):1212–1219, 1970.
[41] V. W. Antonetti. The equations governing weight change in human beings. TheAmerican Journal of Clinical Nutrition, 26(1):64–71, 1973.
[42] N.L. Keim, C.A. Blanton, and M.J. Kretsch. America’s obesity epidemic: Mea-suring physical activity to promote an active lifestyle. Journal of the AmericanDietetic Association, 104(9):1398–1409, 2004.
[43] P. Trumbo, S. Schlicker, AA. Yates, and M. Poos. Dietary reference intakesfor energy, carbohydrate, fiber, fat, fatty acids, cholesterol, protein and aminoacids. Journal of the American Dietetic Association, 102(11):1621–1630, 2002.
[44] B.E. Ainsworth, W.L. Haskell, S.D. Herrmann, N. Meckes, D.R. Bassett Jr,C. Tudor-Locke, J.L.Greer, J. Vezina, M.C. Whitt-Glover, and A.S. Leon.Compendium of physical activities: A second update of codes and met val-ues. Medicine & Science in Sports & Exercise, 43(8):1575–1581, 2011.
[45] K.R. Westerterp, J.H.Donkers, E.W. Fredrix, and P. Boekhoudt. Energy intake,physical activity and body weight: a simulation model. The British Journal ofNutrition, 73(3):337–347, 1995.
[46] N.J. Fuller, S.A. Jebb, M.A. Laskey, W.A. Coward, and M. Elia. Four-componentmodel for the assessment of body composition in humans: comparison with al-ternative methods, and evaluation of the density and hydration of fat-free mass.Clinical Science, 82(6):687–693, 1992.
[47] J.S. Garrow. Energy balance in man- an overview. American Journal of ClinicalNutrition, 45:1114–1119, 1987.
113
[48] S.A. Jebb, A.M. Prentice, G.R. Goldberg, P.R. Murgatroyd, A.E. Black, andW.A. Coward. Changes in macronutrient balance during over- and underfeedingassessed by 12-d continuous whole-body calorimetry. The American Journal ofClinical Nutrition, 64(3):259–266, 1996.
[49] G.B. Forbes. Body fat content influences the body composition response tonutrition and exercise. Annals of the New York Academy of Sciences, 904:359–365, 2000.
[50] J.P. Flatt. Carbohydrate-fat interactions and obesity examined by a two-compartment computer model. Obesity Research, 12(12):2013–2022, 2004.
[51] H. Rahmandad. Human growth and body weight dynamics: An integrativesystems model. PLoS ONE, 9(12):e114609., 2014.
[52] D.M. Thomas, C.K. Martin, S. Heymsfield, L.M. Redmanand A. Dale, andJ.A. Levine. A Simple Model Predicting Individual Weight Change in Humans.Journal of Biological Dynamics, 5(6):579–599, 2011.
[53] A. E.Dugdale and P. R. Payne. Pattern of lean and fat deposition in adults.Nature, 266:249–351, 1977.
[54] K.D. Hall. Body fat and fat-free mass inter-relationships: Forbes’s theory re-visited. The British Journal of Nutrition, 97(6):1059–1063, 2007.
[55] G.B. Forbes. Lean body mass-body fat interrelationships in humans. NutritionReviews, 45(8):225–231, 1987.
[56] K.D. Hall. Computational model of in vivo human energy metabolism duringsemistarvation and refeeding. American Journal of Physiology, Endocrinologyand Metabolism, 291(1):E23–E37, 2006.
[57] A. Keys, J. Brozek, A. Henschel, O. Mickelsen, and H.L. Taylor. The biology ofhuman starvation volume 1,2. Technical report, University of Minnesota, 1950.
[58] KD Hall. Mechanisms of metabolic fuel selection: modeling human metabolismand body-weight change. IEEE Engineering in Medicine and Biology Magazine,29(1):36–41, 2010.
[59] K.D. Hall. Predicting metabolic adaptation, body weight change, and en-ergy intake in humans. American Journal of Physiology, Endocrinology andMetabolism, 298(3):E449–466, 2010.
114
[60] W. H. M. Saris, S. N. Blair, M. A. Van Baak, S. B. Eaton, P. S. W. Davies, L. DiPietro, M. Fogelholm, A. Rissanen, D. Schoeller, B. Swinburn, A. Tremblay,K. R. Westerterp, and H. Wyatt. How much physical activity is enough toprevent unhealthy weight gain? outcome of the IASO 1st stock conference andconsensus statement. Obesity Reviews, 4(2):101–114, 2003.
[61] J. A.Harris and F. G. Benedict. A biometric study of human basal metabolism.In Proceedings of the National Academy of Sciences of the United States ofAmerica, 1919.
[62] N. Barrientos, J.E.D.E. Rivera, and L.M. Collins. A dynamical systems modelfor understanding behavioral interventions for weight loss. International Con-ference on Social Computing Behavioral Modeling, and Prediction, March 2010.
[63] I. Ajzen and T.J. Madden. Prediction of goal-directed behavior: Attitudes,intentions, and perceived behavioral control. Journal of Experimental SocialPsychology, 22(5):453–474, 1986.
[64] R.M. Ryan and E.L. Deci. Self-determination theory and the facilitation ofintrinsic motivation, social development and well-being. American Psychologist,55(1):68–78, 2000.
[65] J.O. Prochaska. Decision making in the transtheoretical model of behaviorchange. Medical Decision Making, 28(6):845–849, Nov 2008.
[66] T.A. Shimp and A. Kavas. The theory of reasoned action applied to couponusage. Journal of Consumer Research, 11(3):795–809, Feb 1984.
[67] N.K. Janz and M.H. Becker. The health belief model: A decade later. HealthEducation and Behavior, 22(1):1–47, March 1984.
[68] J.E. Navarro and D.E. Rivera. A dynamical systems model for weight changebehavioral interventions. Technical report, Control Systems Engineering Lab-oratory, Arizona State University, 2010.
[69] N. Barrientos, J.E.D.E Rivera, and L.M. Collins. A dynamical model for de-scribing behavioural interventions for weight loss and body composition change.Mathematical and Computer Modelling of Dynamical Systems,, 17(2):183–203,Apr 2011.
[70] N. Nandola and D.E.Rivera. Model-on-demand predictive control for nonlinearhybrid systems, with application to adaptive behavioral interventions. In 49thIEEE Conference on Decision and Control (CDC), Dec 2010.
115
[71] A. Zafra-Cabeza, D.E. Rivera, L.M. Collins, M.A. Ridao, and E.F. Camacho.A risk-based model predictive control approach to adaptive interventions inbehavioral health. In Proceedings of the 45th IEEE Conference on Decisionand Control, volume 19, pages 891–901, 2011.
[72] N. Nandola and D.E. Rivera. A novel model predictive control formulationfor hybrid systems with application to adaptive behavioral interventions. InProceedings of the 2010 American Control Conference, pages 6286–6292, 2010.
[73] A. Mahamadi and S.Sastry. Bond graph models for human behavior. In IEEEInternational Conference for Basic Science and Engineering, 2016.
[74] D. S. Alexiadis, P. Kelly, P. Daras, N. E. O’Connor, T. Boubekeur, and M. B.Moussa. Evaluating a dancer’s performance using kinect-based skeleton track-ing. In Proceedings of the 19th ACM International Conference on Multimedia,pages 659–662. ACM, 2011.
[75] E. Davaasambuu, C. Chiang, J. Y. Chiang, Y. Chen, and S. Bilgee. A microsoftkinect based virtual rehabilitation system. In The 5th International ConferenceFITAT, pages 44–50, 2012.
[76] T. Y. Lin, C. H. Hsieh, and J. D. Lee. A kinect-based system for physicalrehabilitation: Utilizing tai chi exercises to improve movement disorders inpatients with balance ability. In Modelling Symposium, pages 149–153, 2013.
[77] S. Obdrzalek, G. Kurillo, F. Ofli, R. Bajcsy, E. Seto, H. Jimison, and M. Pavel.Accuracy and robustness of kinect pose estimation in the context of coachingof elderly. In IEEE Annual International Conference of the Engineering inMedicine and Biology Society, pages 1188–1193. IEEE, 2012.
[78] D. Gonzalez-Ortega, F. J. Dıaz-Pernas, M. Martınez-Zarzuela, and M. Anton-Rodrıguez. A kinect-based system for cognitive rehabilitation exercises moni-toring. Computer Methods and Programs in Biomedicine, 113(2):620–631, 2014.
[79] D. Anton, A. Goni, A. Illarramendi, J. J. Torres-Unda, and J. Seco. Kires:A kinect-based telerehabilitation system. In Proceedings of 15th InternationalConference on e-Health Networking, Applications Services, pages 444–448. IEEE,2013.
[80] A. Ghose, K. Chakravarty, A. K. Agrawal, and N. Ahmed. Unobtrusive indoorsurveillance of patients at home using multiple kinect sensors. In Proceedingsof the 11th ACM Conference on Embedded Networked Sensor Systems, pages40–42. ACM, 2013.
116
[81] M. Robinson and M. B. Parkinson. Estimating anthropometry with microsoftkinect. In Proceedings of the 2nd International Digital Human Modeling Sym-posium, 2013.
[82] J. Han, L. Shao, D. Xu, and J. Shotton. Enhanced computer vision with mi-crosoft kinect sensor: A review. IEEE Transactions on Cybernetics, 43(5):1318–1334, 2013.
[83] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kip-man, and A. Blake. Real-time human pose recognition in parts from singledepth images. In IEEE Conference on Computer Vision and Pattern Recogni-tion, pages 1297–1304, 2011.
[84] Z. Zhang. Microsoft kinect sensor and its effect. IEEE MultiMedia, 19(2):4–10,2012.
[85] B. BonnechAre, B. Jansen, P. Salvia, H. Bouzahouene, L. Omelina, F. Moiseev,V. Sholukha, J. Cornelis, M. Rooze, and S. V. S. Jan. Validity and reliability ofthe kinect within functional assessment activities: Comparison with standardstereophotogrammetry. Gait and Posture, 39(1):593–598, 2014.
[86] A. Fern’ndez-Baena, A. Susin, and X. Lligadas. Biomechanical validation ofupper-body and lower-body joint movements of kinect motion capture datafor rehabilitation treatments. In 4th International Conference on IntelligentNetworking and Collaborative Systems (INCoS), pages 656–661. IEEE, 2012.
[87] C. Chen, Y. Zhuang, F. Nie, Y. Yang, F. Wu, and J. Xiao. Learning a 3d humanpose distance metric from geometric pose descriptor. IEEE Transactions onVisualization and Computer Graphics, 17(11):1676–1689, 2011.
[88] M. Hussein, M. Torki, M. Gowayyed, and M. El-Saban. Human action recogni-tion using a temporal hierarchy of covariance descriptors on 3d joint locations.In Proceedings of the International Joint Conference on Artificial Intelligence,pages 2466–2472. AAAI Press, 2013.
[89] R. Chaudhry, F. Ofli, G. Kurillo, R. Bajcsy, and R. Vidal. Bio-inspired dynamic3d discriminative skeletal features for human action recognition. In IEEE Con-ference on Computer Vision and Pattern Recognition Workshops, pages 471–478. IEEE, 2013.
[90] E. Ohn-Bar and M. M. Trivedi. Joint angles similarities and HOG2 for ac-tion recognition. In 2013 IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW), pages 465–470. IEEE, 2013.
[91] I. Jolliffe. Principal Component Analysis. Wiley Online Library, 2002.
117
[92] J. Wang, Z. Liu, Y. Wu, and J. Yuan. Mining actionlet ensemble for actionrecognition with depth cameras. In IEEE Conference on Computer Vision andPattern Recognition (CVPR), pages 1290–1297. IEEE, 2012.
[93] M. Muller. Information Retrieval for Music and Motion. Springer-Verlag NewYork, 2007.
[94] N. Kumar and R. V. Babu. Human gait recognition using depth camera: Acovariance based approach. In Proceedings of the Eighth Indian Conference onComputer Vision, Graphics and Image Processing, pages 21–26. ACM, 2012.
[95] R. Bellman. A markovian decision process. Journal of Mathematics and Me-chanics, 6, 1957.
[97] M. K. Chippa and S. Sastry. Markov decision process formulation of managinghuman weight loss. In IEEE Systems Confererence, 2016.
[98] l.P. Kaelbing, M.L. Littman, and A.R. Cassandra. Planning and acting in par-tially observable stochastic domains. Artificial Intelligence, 101:99–134, 1998.
[99] M.A. Alsheikh, T.H. Dinh, and N. Dusit et.al. Markov decision processes withapplications in wireless sensor networks: A survey. IEEE CommunicationsSurveys & Tutorials, 17(3):37, 2014.
[100] R. Gillani and A. Nasir. Incorporating Artificial Intelligence in Shopping As-sistance Robot using Markov Decision Process. International Conference onIntelligent Systems Engineering, pages 94–99, 2016.
[101] S.Feyzabadi and S.Carpin. Risk-aware path planning using hierarchical con-strained markov decision processes. In IEEE International Conference on Au-tomation Science and Engineering (CASE), pages 297–303, 2014.
[102] A.N. Burnetas and M.N. Katehakis. Optimal adaptive policies for markovdecision processes. Mathematics of Operations Research, 22(1), 1995.
[103] M. Sandri, P. Berchialla, I. Baldi, and D. Gregori et. al. Dynamic BayesianNetworks to predict sequences of organ failures in patients admitted to ICU.Journal of Biomedical Informatics, 48:106–113, 2014.
[104] K.J. Van Arendonk, K.H. Chow, and N.T. James et. al. Choosing the order ofdeceased donor and living donor kidney transplantation in pediatric recipients:a Markov decision process model. Transplantation, 99(2):360–6, 2015.
118
[105] T. Ben-Zvi, T. Chernonog, and T. Avinadav. A two-state partially observableMarkov decision process with three actions. European Journal of OperationalResearch, 254(3):957–967, 2016.
[106] T. M. Hansen, K.P. Edwin, S. Suryanarayanan, and A. A. Maciejewski. APartially Observable Markov Decision Process Approach to Residential HomeEnergy Management. IEEE Transactions on Smart Grid, 3053(July):1–8, 2016.
[107] T. Ayer, O. Alagoz, and N. K. Stout. A POMDP Approach to Personalize Mam-mography Screening Decisions. Operations Research, 60(5):1019–1034, 2012.
[108] S. Temizer and M.J. Kochenderfer. Unmanned aircraft collision avoidance usingpartially observable markov decision processes, 2009.
[109] P. Tseng. H-horizon, stationary markov decision problems in time proportionalto log(h). Operations Research, 9(5):287–297, 1990.
[110] A.M. Farahmand, R. Munos, and C. Szepesvari. Error propagation for approx-imate policy and value iteration. Advances in Neural Information ProcessingSystems, 2010.
[111] M. L. Littman, T. L. Dean, and L. P. Kaelbling. On the complexity of solvingmarkov decision problems. In In Proceedings of the Eleventh Annual Conferenceon Uncertainty in Artificial Intelligence, Montreal, Quebec, Canada,, 1995.
[112] J. MacGlashan. The brown-umbc reinforcement learning and planning (burlap).
[113] J.H. Ahn and J.C. Hornberger. Involving patients in the cadaveric kidneytransplant allocation process: A decision-theoretic perspective. ManagementScience, 42(5):629–641, 1996.
[114] M.F. Drummond, B. OBrien, G.L. Stoddart, and G.W. Torrance. Methods forthe Economic Evaluation of Health Care Programmes. Oxford University Press,1997.
[115] G.W. Torrance. Social preferences for health states: an empirical evaluation ofthree measurement techniques. Socio-Economic Planning Sciences, 10(3):129–136, 1976.
[116] G.W. Torrance, D.H. Feeny, W.J. Furlong, R.D. Barr, Y. Zhang, and Q. Wang.Multiattribute utility function for a comprehensive health status classificationsystem: Health utilities index mark 2. Medical Care, 34(7):702–722, 1996.
[117] P.B. Shull., W. Jirattigalachote, and M.A. Hunt et. al. Quantified self andhuman movement: A review on the clinical impact of wearable sensing andfeedback for gait analysis and intervention. Gait and Posture, 40(1):11–19,May 2014.
119
[118] V.E.D. Pinheiro and H.A. Simon. An operational model of motor skill diagnosis.Journal of Teaching in Physical Education, 11:288–302, 1992.
[119] J. K. Aggarwal and M. S. Ryoo. Human activity analysis: A review. ACMComputing Surveys (CSUR), 43(3):16–58, 2011.
[120] J. K. Aggarwal and L. Xia. Human activity recognition from 3d data: A review.Pattern Recognition Letters, 48:70–80, 2014.
[121] C.H. Chen, J. Favre, G. Kurillo, T.P. Andriachhi, R. Bajcsky, and R. Chel-lappa. Camera networks for healthcare, teleimmersion, and surveillance. IEEEComputer, 47:26–36, 2014.
[122] L. Wenmin, D. Wanchun, Z. Zuojian, and L. Chang. A cloud-based frameworkfor Home-diagnosis service over big medical data. Journal of Systems andSoftware, 102:192–206, 2015.
[123] J. H. M. Bergmann, V. Chandaria, and A. McGregor. Wearable and implantablesensors: The patient’s perspective. Sensors, December 2012.
[124] B. Klika and C. Jordan. High-intensity circuit training using body weight:Maximum results with minimal investment. ACSM Health and Fitness, 17(3):8–13, 2013.