Adaptive Deep Learning Model Selection on Embedded Systems€¦ · Adaptive Deep Learning Model Selection on Embedded Systems ABSTRACT The recent ground-breaking advances in deep

Adaptive Deep Learning Model Selection onEmbedded Systems

ABSTRACTThe recent ground-breaking advances in deep learningnetworks (DNNs) make them attractive for embeddedsystems. However, it can take a long time for DNNs to makean inference on resource-limited embedded devices.Offloading the computation into the cloud is often infeasibledue to privacy concerns, high latency, or the lack ofconnectivity. As such, there is a critical need to find a wayto effectively execute the DNN models locally on the devices.This paper presents an adaptive scheme to determine

which DNN model to use for a given input, by consideringthe desired accuracy and inference time. Our approachemploys machine learning to develop a predictive model toquickly select a pre-trained DNN to use for a given input andthe optimization constraint. We achieve this by first trainingoff-line a predictive model, and then use the learnt model toselect a DNN model to use for new, unseen inputs. We applyour approach to the image classification task and evaluate iton a Jetson TX2 embedded deep learning platform using theImageNet ILSVRC 2012 validation dataset. We consider arange of influential DNN models. Experimental results showthat our approach achieves a 7.52% improvement ininference accuracy, and a 1.8x reduction in inference timeover the most-capable, single DNN model.

CCS CONCEPTS• Computer systems organization → Embeddedsoftware; • Computing methodologies → Parallelcomputing methodologies;

ACM Reference Format:. 2018. Adaptive Deep Learning Model Selection on EmbeddedSystems. In Proceedings of (LCTES ’18). ACM, New York, NY, USA,12 pages. https://doi.org/10.475/123_4

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bearthis notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACMmust be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Requestpermissions from [email protected] ’18, June 2018, Pennsylvania, USA© 2018 Association for Computing Machinery.ACM ISBN 123-4567-24-567/08/06. . . $15.00https://doi.org/10.475/123_4

1 INTRODUCTIONRecent advances in deep learning have brought a steepchange in the abilities of machines in solving complexproblems like object recognition [8, 17], facialrecognition [34, 44], speech processing [10], and machinetranslation [2]. Although many of these tasks are importanton mobile and embedded devices, especially for sensing andmission critical applications such as health care and videosurveillance, existing deep learning solutions often require alarge amount of computational resources to run. Runningthese models on embedded devices can lead to long runtimeand the consumption of abundant amounts of resources,including CPU time, memory, and power, even for simpletasks [5]. Without a solution, the hoped-for advances onembedded sensing will not arrive.A common approach for accelerating DNN models on

embedded devices is to compress the model to reduce itsresource and computational requirements [11, 14, 15, 19],but this comes at the cost of a loss in precision. Otherapproaches involve offloading some, or all, computation to acloud server [25, 46]. This, however, is not always possibledue to constraints on privacy, when sending sensitive dataover the network is prohibitive, and latency, where a fast,reliable network connection is not always guaranteed.

This paper seeks to offer an alternative to enable efficientdeep inference1 on embedded devices. Our goal is to designan adaptive scheme to determine, at runtime, which of theavailable DNN models is the best fit for the input and theprecision requirement. This is motivated by the observationthat the optimum model2 for inference depends on the inputdata and the precision requirement. For example, if theinput image is taken under good lighting conditions and hasa simple background, a simple but fast model would besufficient for identifying the objects in the image –otherwise, a more sophisticated but slower model will haveto be employed; in a similar vein, if we want to detectcertain objects with a high confidence, an advanced modelshould be used – otherwise, a simple model would be goodenough. Given that DNN models are becoming increasinglydiverse – together with the evolving application workloadand user requirements, the right strategy for model

1Inference in this work means applying a pre-trained model on an input toobtain the corresponding output. This is different from statistical inference.2In this work, the optimum model is the one that gives the correct outputwith the fastest inference time.

LCTES ’18, June 2018, Pennsylvania, USA

M o b i l e n e t R e s N e t _ v 1 _ 5 0 I n c e p t i o n _ v 2 R e s N e t _ v 2 _ 1 5 20 . 00 . 51 . 01 . 52 . 0

+ *+*

+ *+ B e s t t o p - 5 s c o r e m o d e l

Infere

nce T

ime (

s) I m a g e 1 I m a g e 2 I m a g e 3B e s t t o p - 1 s c o r e m o d e l*

(a) Image 1 (b) Image 2 (c) Image 3 (d) Inference time

Figure 1: The inference time (d) of four CNN-based image recognition models when processing images (a) - (c).The target object is highlighted on each image. This example (combined with Table 1) shows that the best modelto use (i.e. the fastest model that gives the accurate output) depends on the success criterion and the input.

selection is likely to change over time. This ever-evolvingnature makes automatic heuristic design highly attractivebecause the heuristic can be easily updated to adapt to thechanging application context.This paper presents a novel runtime approach for DNN

model selection on embedded devices, aiming to minimizethe inference time while meeting the user requirement. Weachieve this by employing machine learning toautomatically construct predictors to select at runtime theoptimum model to use. Our predictor is first trained off-line.Then, using a set of automatically tuned features of the DNNmodel input, the predictor determines the optimum DNNmodel for a new, unseen input, by taking into considerationthe precision constraint and the characteristics of the input.We show that our approach can automatically derivehigh-quality heuristics for different precision requirements.The learned strategy can effectively leverage the predictioncapability and runtime overhead of candidate DNN models,leading to an overall better accuracy when compared withthe most capable DNN model, but with significantly lessruntime overhead. Using our approach, one can also firstapply model compression techniques to generate DNNmodels of different capabilities and inference time, and thenchoose a model to use at runtime. This is a new way foroptimizing deep inference on embedded devices.We apply our approach to the image classification

domain, an area where deep learning has made impressivebreakthroughs by using high-performance systems andwhere a rich set of pre-trained models are available. Weevaluate our approach on the NVIDIA Jetson TX2embedded deep learning platform and consider a widerange of influential DNN models. Our experiments areperformed using the 50K images from the ImageNet ILSVRC2012 validation dataset. To show the automatic portabilityof our approach across precision requirements, we haveevaluated it on two different evaluation criteria used by theImageNet contest. Our approach is able to correctly choosethe optimum model to use for 95.6% of the test cases, andnever picks a model that would give an incorrect inference

Table 1: List of models that give the correct predictionper image under the top-5 and the top-1 scores.

top-5 score top-1 score

Image 1 MobileNet_v1_025,ResNet_v1_50, Inception_v2,ResNet_v2_152

MobileNet_v1_025,ResNet_v1_50, Inception_v2,ResNet_v2_152

Image 2 Inception_v2, ResNet_v1_50,ResNet_v2_152

Inception_v2,ResNet_v2_152

Image 3 ResNet_v1_50,ResNet_v2_152

ResNet_v2_152

output. Overall, it improves the inference accuracy by 7.52%over the most-capable, single model but with 1.8x lessinference time.

This paper makes the following contributions:• We present a novel machine learning based approachto automatically learn how to select DNN models basedon the input and precision requirement (Section 3);

• Our work is the first to leverage multiple DNN modelsto improve the prediction accuracy and reduceinference time on embedded systems (Section 5). Ourautomatic approach allows developers to easilyre-target the approach for new DNN models and userrequirements;

• Our system has little training overhead as it does notrequire any modification to pre-trained DNN models.

2 MOTIVATION AND OVERVIEW2.1 MotivationAs a motivating example, consider performing objectrecognition on a NVIDIA Jetson TX2 platform.Setup. In this experiment, we compare the performance ofthree influential Convolutional Neural Network (CNN)architectures: Inception [23], ResNet [18], andMobileNet [19]3. Specifically, we used the following

3 Each model architecture follows its own naming convention.MobileNet_vi_j , where i is the version number, and j is a widthmultiplier out of 100, with 100 being the full uncompressed model.ResNet_vi_j , where i is the version number, and j is the number of layersin the model. Inception_vi , where i is the version number.

Adaptive Deep Learning Model Selection on Embedded Systems LCTES ’18, June 2018, Pennsylvania, USA

models: MobileNet_v1_025, the MobileNet architecturewith a width multiplier of 0.25; ResNet_v1_50, the firstversion of ResNet with 50 layers; Inception_v2, thesecond version of Inception; and ResNet_v2_152, thesecond version of ResNet with 152 layers. All these modelsare built upon TensorFlow [1] and have been pre-trained byindependent researchers using the ImageNet ILSVRC 2012training dataset [39]. We use the GPU for inference.Evaluation Criteria. Each model takes an image as inputand returns a list of label confidence values as output. Eachvalue indicates the confidence that a particular object is inthe image. The resulting list of object values are sorted indescending order regarding their prediction confidence, sothat the label with the highest confidence appears at thetop of the list. In this example, the accuracy of a model isevaluated using the top-1 and the top-5 scores defined bythe ImageNet Challenge. Specifically, for the top-1 score, wecheck if the top output label matches the ground truth labelof the primary object; and for the top-5 score, we check ifthe ground truth label of the primary object is in the top 5of the output labels for each given model.Results. Figure 1d shows the inference time per modelusing three images from the ImageNet ILSVRC validationdataset. Recognizing the main object (a cottontail rabbit)from the image shown in Figure 1a is a straightforward task.We can see from Figure 1 that all models give the correctanswer under the top-5 and top-1 score criterion. For thisimage, MobileNet_v1_025 is the best model to use underthe top-5 score, because it has the fastest inference time –6.13x faster than ResNet_v2_152. Clearly, for this image,MobileNet_v1_025 is good enough, and there is no need touse a more advanced (and more expensive model) forinference. If we consider a slightly more complex objectrecognition task shown in Figure 1b, we can see thatMobileNet_v1_025 is unable to give a correct answerregardless of our success criterion. In this caseInception_v2 should be used, although this is 3.24x slowerthan MobileNet_v1_025. Finally, consider the final imageshown in Figure 1c, intuitively it can be seen that this wouldbe a more difficult image recognition task, this main objectis a similar color to the background. In this case the modelwe should use changes depending on our success criterion.ResNet_v1_50 is the best model to use under the top-5score, completing inference 2.06x faster thanResNet_v2_152. However, if we instead use top-1 forscoring we must use ResNet_v2_152 to obtain the correctlabel, despite that it’s the most expensive model. Inferencetime for this image is 2.98x and 6.14x slower thanMobileNet_v1_025 for top-5 and top-1 scoringrespectively.

Feature Extraction Inference

1 2 3 4

Offline Profiling RunsMemory footprint

Training programs

Model Fitting

Feature Extraction

fMemory function

Feature values

5

Model SelectionImage Labels

Figure 2: Overview of our approach

YModel 1

Input features

Distance calculation Model 1?

NModel 2?

Model 2

NModel n?

Model n

NKNN-1 KNN-2 KNN-n

all models will fail

...

Y Y

Figure 3: Our premodel, made up of a series of KNNmodels. Each model predicts whether to use an imageclassifier or not, our selection process for includingimage classifiers is described in Section 3.2.

Lessons Learned. This example shows that the best modeldepends on the input and the evaluation criterion. Hence,determining which model to use is non-trivial. What weneed is a technique that can automatically choose the mostefficient model to use for any given input. In the next section,we describe our adaptive approach that solves this task.

2.2 Overview of Our ApproachFigure 2 depicts the overall work flow of our approach. Whileour approach is generally applicable, to have a concrete,measurable target, we apply it to image classification. At thecore of our approach is a predictive model (termed premodel)that takes a new, unseen image to predict which of a set of pre-trained image classification models to use for the given input.This decision may vary depending on the scoring methodused at the time, e.g., either top-1 or top-5, and we showthat our approach can adapt to different metrics.The prediction of our premodel is based on a set of

quantifiable properties – or features such as the number ofedges and brightness – of the input image. Once a model ischosen, the input image is passed to the selected model,which then attempts to classify the image. Finally, theclassification data of the selected model is returned asoutputs. Use of our premodel will work in exactly the sameway as any single model, the difference being we are able tochoose the best model to use dynamically.

3 OUR APPROACHOur premodel is made up of multiple k-Nearest Neighbour(KNN) classification models arranged in sequence, shown inFigure 34. As input our model takes an image, from whichit will extract features and make a prediction, outputting alabel referring to which image classification model to use.

4In Section 5.2, we evaluate a number of different machine learningtechniques, including Decision Trees, Support Vector Machines, and CNNs.

LCTES ’18, June 2018, Pennsylvania, USA

webcontent

Parsing Style Resolution Layout Paint Display

DOM Tree

Style Rules

Render Tree

Training Images

Inference Profiling

Feature extraction

optimum model

feature values

Learning A

lgorithm

Predictive Model

Kernel on CPU

Kernel on Accelerator

CPU config.

accelerator config.

host CPU config.

Figure 4: The training process. We use the sameprocedure to train each individual model within thepremodel for each evaluation criterion.

3.1 Model DescriptionThere are two main requirements to consider whendeveloping an inferencing model selection strategy on aembedded device: (i) fast execution time, and (ii) a high levelof accuracy. Having a premodel which takes much longerthan any single model would outweigh the benefit of usingit. We also require high accuracy to choose the optimuminferencing model, therefore reducing the oveall cost.

Following the above goals we chose to implement a seriesof simple KNN models, where each model predicts whetherto use a single image classifier or not. We chose KNN as ithas a quick prediction time (less than 1ms) and achieves ahigh accuracy for our problem. Finally, we chose a set offeatures to represent each image, the selection process ofthese features is described in more detail in Section 3.4.

Figure 3 gives an overview of our premodel architecture.For each DNN model we wish to include in our premodel, weuse a separate KNN model. As our KNN models are going tocontain much of the same data we begin our premodel bycalculating our K closest neighbours. Taking note of whichrecord of training data each of the neighbours corresponds to,we are able to avoid recalculating the distancemeasurements;we simply change the labels of these data-points. KNN-1 isthe first KNNmodel in our premodel, through which all inputto the premodel will pass. KNN-1 is used to predict whetherthe input image should use Model-1 to classify it or not,depending on the scoring criterion the premodel has beentrained for. If KNN-1 predicts that Model-1 should be used,then the premodel returns this label, otherwise the featuresare passed on to the next KNN, i.e. KNN-2. This process carrieson until the image reaches KNN-n, the final KNNmodel in ourpremodel. In the event that KNN-n predicts that we shouldnot use Model-n to classify the image, the next step willbe one of two depending on the user’s declared preference:(i) using a pre-specified model, so the user can have someoutput to work with; or (ii) do not perform inference andsimple informing the user of the failure.

3.2 Inference Model SelectionIn Algorithm 1 we describe our selection process forchoosing which inference models to include in ourpremodel. Essentially, this algorithm involves choosing thefirst model to include, which is always the one which is

Algorithm 1 Inference Model Selection ProcessModel_1_class =most_optimum_class(data)curr_class .add(Model_1_class)curr_acc = дet_acc(curr_class)acc_diff = 100while acc_diff > θ do

failed_cases = get_fail_cases(curr_class)next_class = most_acc_class(failed_cases)curr_class .add(next_class)new_acc = дet_acc(curr_class)acc_diff = new_acc - curr_acccurr_acc = new_acc

end while

optimal for the most of our training data, then iterativelyadding the most accurate model on the remainder of thetraining data until our accuracy improvement is lower thana threshold θ . Below we will walk through the algorithm toshow how we chose the model to include in our premodel.We have chosen to set our threshold value, θ to 0.5,

which is empirically decided during our pilot experiments.Figure 5 shows the percentage of our training data whichconsiders each of our CNNs to be optimal. There is a clearwinner here, MobileNet_v1_100 is optimal for 70.75% ofour training data, therefore it is chosen to be Model-1 forour premodel. If we were to follow this convention andthen choose the next most optimal CNN, we would chooseInception_v1. However, we do not do this as it wouldresult in our premodel being formulated of many cheap, yetinaccurate models. Instead we choose to look at the trainingdata on which our initial model (Model-1) fails; theremaining 29.25% of our data.From here on when adding new CNNs to our premodel

we exclusively consider the accuracy of each on thecurrently failing training data. Figure 6b shows the accuracyof our remaining CNNs on the 29.25% cases whereMobileNet_v1_100 fails. We can see that Inception_v4clearly wins here, correctly classifying 43.91% of theremaining data; creating a 12.84% increase in premodelaccuracy, and leaving 16.41% of our data failing. We thenrepeat this process, shown in Figure 6c, where we addResNet_v1_152 to our premodel seeing an increase in totalaccuracy of 2.55%. Finally we repeat this step one more time,to achieve a premodel accuracy increase of <0.5, therefore<θ , and terminate here.

The result of this is a premodel where: Model-1 isMobileNet_v1_100, Model-2 is Inception_v4, and, finally,Model-3 is ResNet_v1_152.

3.3 Training the premodelTraining our premodel follows the standard procedure, andis a multi-step process. We describe the entire training

Adaptive Deep Learning Model Selection on Embedded Systems LCTES ’18, June 2018, Pennsylvania, USA

Table 2: All features considered in this work.Feature Description Feature Description

n_keypoints # of keypoints avg_brightness Average brightnessbrightness_rms Root mean square of brightness avg_perceived_brightness Average of perceived brightnessperceived_brightness_rms Root mean square of perceived brightness contrast The level of contrastedge_length{1-7} A 7-bin histogram of edge lengths edge_angle{1-7} A 7-bin histogram of edge anglesarea_by_perim Area / perimeter of the main object aspect_ratio The aspect ratio of the main objecthue{1-7} A 7-bin histogram of the different hues

M . n e t _ v 1 _ 1 0 0

I n c e p t i o n _ v 1

R e s n e t _ v 1 _ 5 0

I n c e p t i o n _ v 2

R e s n e t _ v 2 _ 5 0

I n c e p t i o n _ v 3

R e s n e t _ v 1 _ 1 0 1

I n c e p t i o n _ v 4

R e s n e t _ v 2 _ 1 0 1

R e s n e t _ v 2 _ 1 5 2

R e s n e t _ v 1 _ 1 5 202 04 06 08 0

% of

being

optim

al

Figure 5: How often a CNN model is considered to beoptimal under the top-1 score on the training dataset.

process in detail below, and provide a summary in Figure 4.Generally, we need to figure out which candidate inferecingmodel is optimum for each of our training example (i.e.,images), we then train our model to predict the same forany new, unseen inputs.Generate Training Data. Our training dataset consists ofthe feature values of a set of images and the correspondingoptimum model for each image under an evaluationcriterion. To evaluate the performance of the candidate DNNmodels, they must be applied to unseen images. We chooseto use ILVRSC 2012 validation set, which contains 50kimages, to generate training data for our premodel. Thisdataset provides a wide selection of images containing arange of topics and complexities. We then exhaustivelyexecuted each image on each candidate model, measuringthe inference time and prediction results. Inference time ismeasured on an unloaded machine to reduce noise, and is aone-off cost – it only needs to be completed once. Becausethe relative runtime of models is stable, training datageneration can be performed on a high-performance serverto speedup the training data generation process. It is to notethat adding a new image classifier, simply requiresexecuting all images on the new image classifier whiletaking the same measurements described above.Taking the execution time, top-1, and top-5 results we

are able to generate a best image classifier for each image;that is, the model which achieves the accuracy goal (top-1or top-5) in the least amount of time. Finally, we extract thefeature values (described in Section 3.4) from each image,and pair the feature values to the best image classifier foreach image, resulting in our complete training dataset.Building theModel. The training data is used to determinewhich classification models should be used and the optimal

Table 3: Correlation values (absolute) of removedfeatures to the kept ones.

Kept Feature Removed Feature Correl.

perceived_brightness_rms 0.98avg_brightness 0.91avg_perceived_brightnessbrightness_rms 0.88edge_length4 0.78edge_length5 0.85edge_length6 0.82edge_length1edge_length7 0.77

hue1 hue{2-6} 0.99

hyper-parameters of the model. Since we chose to use KNNmodels to construct our premodel, the generated trainingdata is used to train our model using a standard supervizedlearning method. In KNN classification the training data isused to give a label to each point in the model, then duringprediction the model will use a distance measure (in our casewe use Euclidian distance) to find the K nearest points (inour case K=5). The label with the highest number of pointsto the prediction point is the output label.Training Cost. Total training time of our premodel isdominated by generating the training data. Generating thetraining data took less than a day using a NVIDIA P40 GPUon a multi-core server. This can vary depending on thenumber of image classifiers to be included. In our case, wehad an usually long training time as we considered 12 DNNmodels. We would expect in deployment that the user has amuch smaller search space for image classifiers. The time inmodel selection and parameter tuning is negligible (lessthan 2 hours) in comparison. See also Section 5.5.

3.4 FeaturesOne of the key aspects in building a successful predictor isdeveloping the right features in order to characterize theinput. In this work, we considered a total of 30 candidatefeatures, shown in Table 2. The features were chosen basedon previous image classification work [16] e.g., edge basedfeatures, as well as intuition based on our motivation(Section 2.1), e.g., contrast.

3.4.1 Feature selection. The time spent in making aprediction is negligible in comparison to the overhead offeature extraction, therefore by reducing our feature countwe can decrease the total execution time of our premodel.Moreover, by reducing the number of features we are also

LCTES ’18, June 2018, Pennsylvania, USA

M . n e t _ v 1 _ 1 0 0I n c e p t . _ v 1

I n c e p t . _ v 2I n c e p t . _ v 4

R e s n e t _ v 1 _ 5 0

R e s n e t _ v 1 _ 1 0 1

R e s n e t _ v 1 _ 1 5 2

R e s n e t _ v 2 _ 5 0

R e s n e t _ v 2 _ 1 0 1

R e s n e t _ v 2 _ 1 5 20 . 00 . 40 . 81 . 21 . 62 . 02 . 4

Infere

nce T

ime (

s) I n f e r e n c e t i m e

02 04 06 08 01 0 0 T o p - 1 a c c u r a c y

Top-1

Accu

racy (

%)

I n c e p t i o n _ v 1I n c e p t i o n _ v 2

I n c e p t i o n _ v 4

R e s n e t _ v 1 _ 5 0

R e s n e t _ v 1 _ 1 0 1

R e s n e t _ v 1 _ 1 5 2

R e s n e t _ v 2 _ 5 0

R e s n e t _ v 2 _ 1 0 1

R e s n e t _ v 2 _ 1 5 201 02 03 04 05 0

Top-1

Accu

racy (

%)

R e s n e t _ v 1 _ 5 0

R e s n e t _ v 1 _ 1 0 1

R e s n e t _ v 1 _ 1 5 2

R e s n e t _ v 2 _ 5 0

R e s n e t _ v 2 _ 1 0 1

R e s n e t _ v 2 _ 1 5 205

1 01 52 02 5

Top-1

Accu

racy (

%)

(a) top-1 accuracy and inference time (b) top-1 accuracy on the cases where MobileNet fails (c) top-1 accuracy on the cases where Mobilnet & Inception fails

Figure 6: (a) Shows the top-1 accuracy and average inference time of all CNNs considered in this work across ourentire training dataset. (b) Shows the top-1 accuracy of all CNNs on the images on which MobileNet_v1_100 fails.(c) Shows the top-1 accuracy of all CNNs on the images on which MobileNet_v1_100 and Inception_v4 fails.

Table 4: The chosen features.

n_keypoints avg_perceived_brightness hue1contrast area_by_perim edge_length1aspect_ratio

improving the generalization ability of our premodel, i.e.reducing the likelihood of over-fitting on our training data.Initially, we use correlation-based feature selection. If

pairwise correlation is high for any pair of features, we dropone of them and keep the other in order to retain most ofthe information. We performed this by constructing amatrix of correlation coefficients using Pearsonproduct-moment correlation. The coefficient value fallsbetween −1 and +1. The closer the absolute value is to 1,the stronger the correlation between the two features beingtested. We set a threshold of 0.75 and removed any featuresthat had an absolute Pearson correlation coefficient higherthan the threshold. Table 3 summarizes the features weremoved at this stage, leaving 17 features.Next we evaluated the importance of each of our

remaining features. To evaluate feature importance we firsttrained and evaluated our premodel using K-Fold crossvalidation (see also Section 5.5) and all of our currentfeatures, and recording premodel accuracy. We thenremove each feature and re-evaluate the model on theremaining features, taking note of the change in accuracy. Ifthere is a large drop in accuracy then the feature must bevery important, otherwise, the features does not hold muchimportance for our purposes. Using this information weperformed a greedy search, removing the least importantfeatures one by one. By performing this search wediscovered that we can reduce our feature count down to 7features (see Table 4) while having very little impact on ourmodel accuracy. Removing any of the remaining 7 featuresresulted in a significant drop in model accuracy.

3.4.2 Feature scaling. The final step before passing ourfeatures to a machine learning model is scaling each of thefeatures to a common range (between 0 and 1) in order toprevent the range of any single feature being a factor in itsimportance. Scaling features does not affect the distribution

a s p e c t _ r a t i on _ k e y p o i n t s

a v g _ p e r c . _ b r i g h t . c o n t r a s te d g e _ l e n g t h 10

51 01 52 0

lost a

ccurac

y (%)

Figure 7: The top five features which can lead to a highloss in accuracy if they are not used in our premodel.

or variance of their values. To scale the features of a newimage during deployment we record the minimum andmaximum values of each feature in the training dataset, anduse these to scale the corresponding features.

3.4.3 Feature analysis. Figure 7 shows the top 5dominant features based on their impact on our premodelaccuracy. We calculate feature importance by first training apremodel using all 7 of our chosen features, and note theaccuracy of our model. In turn, we then remove each of ourfeatures, retraining and evaluating our premodel on theother 6, noting the drop in accuracy. We then normalize thevalues to produce a percentage of importance for each ofour features. It can be seen that each of our features hold avery similar level of importance, ranging between 18% and11% for our most and least important feature respectively.The similarity of our feature importance is an indicationthat each of our features is able to represent distinctinformation about each image. all of which is important forthe prediction task at hand.

3.5 Runtime DeploymentDeployment of our proposed method is designed to besimple and easy to use, similar to current imageclassification techniques. We have encapsulated all of theinner workings, such as needing to read the output of thepremodel and then choosing the correct image classifier. Auser would interact with our proposed method in the sameway as any other image classifier: simply calling aprediction function and getting the result in return aspredicted labels and their confidence levels.

Adaptive Deep Learning Model Selection on Embedded Systems LCTES ’18, June 2018, Pennsylvania, USA

4 EXPERIMENTAL SETUP4.1 Platform and ModelsHardware.We evaluate our approach on the NVIDIA JetsonTX2 embedded deep learning platform. The system has a64 bit dual-core Denver2 and a 64 bit quad-core ARM Cortex-A57 running at 2.0 Ghz, and a 256-core NVIDIA Pascal GPUrunning at 1.3 Ghz. The board has 8 GB of LPDDR4 RAMand 96 GB of storage (32 GB eMMC plus 64 GB SD card).System Software. Our evaluation platform runs UbuntuUbuntu 16.04.3 LTS with Linux kernel v4.4.15. We useTensorflow v.1.0.1, cuDNN (v6.0) and CUDA (v8.0.64). Ourpremodel is implemented using the Python scikit-learnmachine learning package. Our feature extractor is builtupon OpenCV and SimpleCV.Deep Learning Models. We consider 14 pre-trained CNNmodels for image recognition from the TensorFlow-Slimlibrary [40]. The models are built upon TensorFlow andtrained on the ImageNet ILSVRC 2012 training set.

4.2 Evaluation MethodologyModel Evaluation. We use 10-fold cross-validation toevaluate our premodel on the ImageNet ILSVRC 2012validation set. Specifically, we partition the 50K validationimages into 10 equal sets, each containing 5K images. Weretain one set for testing our premodel, and the remaining 9sets are used as training data. We repeat this process 10times (folds), with each of the 10 sets used exactly once asthe testing data. This standard methodology evaluates thegeneralization ability of a machine-learning model.

We evaluate our approach using the following metrics:• Inference time (lower is better). Wall clock timebetween a model taking in an input and producing anoutput, including the overhead of our premodel.

• Energy consumption (lower is better). The energyused by a model for inference. For our approach, thisalso includes the energy consumption of thepremodel. We deduct the static power used by thehardware when the system is idle.

• Accuracy (higher is better). The ratio of correctlylabeled images to the total number of testing images.

• Precision (higher is better). The ratio of a correctlypredicted images to the total number of images thatare predicted to have a specific object. This metricanswers e.g., “Of all the images that are labeled to havea cat, how many actually have a cat?".

• Recall (higher is better). The ratio of correctlypredicted images to the total number of test imagesthat belong to an object class. This metric answerse.g., “Of all the test images that have a cat, how manyare actually labeled to have a cat?".

• F1 score (higher is better). The weighted average ofPrecision and Recall, calculated as 2× Recall×Precision

Recall+Precision .It is useful when the test datasets have an unevendistribution of object classes.

Performance Report. We report the geometric mean ofthe aforementioned evaluation metrics across thecross-validation folds. To collect inference time and energyconsumption, we run each model on each input repeatedlyuntil the 95% confidence bound per model per input issmaller than 5%. In the experiments, we exclude the loadingtime of the CNN models as the model only need to be loadedonce in practice. However, we include the overhead of ourpremodel in all our experimental data. To measure energyconsumption, we developed a lightweight runtime to takereadings from the on-board energy sensors at a frequencyof 1,000 samples per second. We then matched the energyreadings against the time stamps of model execution tocalculate the energy consumption.

5 EXPERIMENTAL RESULTS5.1 Overall PerformanceInference Time. Figure 8a compares the inference timeamong individual DNN models and our approach. MobileNetis the fastest model for inferencing, being 2.8x and 2x fasterthan Inception and ResNet, respectively, but is leastaccurate (see Figure 8c). Our premodel alone is 3x fasterthan MobileNet. Most the overhead of our premodel comesfrom feature extraction. The average inference time of ourapproach is under a second, which is slightly longer thanthe 0.7 second average time of MobileNet. Our approach is1.8x faster than Inception, the most accurate inferencemodel in our model set. Given that our approach cansignificantly improve the prediction accuracy of Mobilenet,we believe the modest cost of our premodel is acceptable.Energy Consumption. Figure 8b gives the energyconsumption. On the Jetson TX2 platform, the energyconsumption is proportional to the model inference time.As we speed up the overall inference, we reduce the energyconsumption by more than 2x compared to Inception andResnet. The energy footprint of our premodel is small,being 4x and 24x lower than MobileNext and ResNetrespectively. As such, it is suitable for power-constraineddevices, and can be used to improve the overall accuracywhen using multiple inferencing models. Furthermore, incases where the premodel predicts that none of the DNNmodels can successfully infers an input, it can skipinference to avoid wasting power. It is to note that since ourpremodel runs on the CPU, its energy footprint ratio issmaller than that for runtime.

LCTES ’18, June 2018, Pennsylvania, USA

M o b i l e n e t I n c e p t i o n R e s n e t O u r s0

1

2

Infere

nce T

ime (

s) i n f e r . m o d e l P r e m o d e l

M o b i l e n e t I n c e p t i o n R e s n e t O u r s012345 I n f e r . m o d e l P r e m o d e l

Joule

s

M o b i l e n e t I n c e p t i o n R e s n e t O u r s O r a c l e4 06 08 0

1 0 0

Accu

racy (

%)

T o p - 1 T o p - 2

P r e c i s i o n R e c a l l F 1 - s c o r e0 . 00 . 20 . 40 . 60 . 81 . 0

M o b i l e n e t I n c e p t i o n R e s n e t O u r s

(a) Inference Time (b) Energy Consumption (c) Accuracy (d) Precision, Recall & F1 score

Figure 8: Overall performance of our approach against individual models for inference time (a), energyconsumption (b), accuracy (c), precision, recall and F1 score (d). Our approach gives the best overall performance.

C N N K N N D e c i s i o n T r e e s S V M0 . 00 . 20 . 40 . 60 . 8

Runti

me (s

) R u n t i m e

02 04 06 08 01 0 0

Top-1

Accu

racy (

%) T o p - 1 A c c u r a c y

Figure 9: Comparison of alternative predictivemodeling techniques for building the premodel.

Accuracy. Figures 8c compares the top-1 and top-5accuracy achieved by each approach. We also show the bestpossible accuracy given by a theoretically perfect predictorfor model selection, for which we call Oracle. Note that theOracle does not give a 100% accuracy because there arecases where all the DNN models fail. By effectivelyleveraging multiple models, our approach outperforms allindividual inference models. It improves the accuracy ofMobileNet by 16.6% and 6% respectively for the top-1 andthe top-5 scores. It also improves the top-1 accuracy ofResNet and Inception by 10.7% and 7.6% respectively.While we observe little improvement for the top-5 scoreover Inception – just 0.34% – our approach is 2x fasterthan it. Our approach delivers over 96% of the Oracleperformance (87.4% vs 91.2% for top-1 and 95.4% vs 98.3%).Moreover, our approach never picks a model that fails whileothers can success. This result shows that our approach canimprove the inference accuracy of individual models.Precision, Recall, F1 Score. Finally, Figure 8d shows ourapproach outperforms individual DNN models in otherevaluation metrics. Specifically, our approach gives thehighest overall precision, which in turns leads to the best F1score. High precision can reduce false positive, which isimportant for certain domains like video surveillancebecause it can reduce the human involvement for inspectingfalse positive predictions.

5.2 Alternative Techniques for PremodelFigure 9 shows the top-1 accuracy and runtime for usingdifferent techniques to construct the premodel. Here, thelearning task is to predict which of the inference models,MobileNet, Inception, and ResNet, to use. In addition toKNN, we also consider CNNs, Decision Trees (DT) and Support

Vector Machines (SVM). We use the MobileNet structure,which is designed for embedded inference, to build theCNN-based premodel. We train all the models using thesame training examples. We also use the same feature setfor the KNN, DT, and SVM. For the CNN, we use ahyperparamter tuner [26] to optimize the trainingparameters, and we train the model for over 500 epochs.While we hypothesized a CNN model to be effectively in

predicting from an image to the output, the results aredisappointing given its high runtime overhead. We suspectthe low accuracy of the CNN is because the size of ourcross-validation training set (that contains 45K images) isnot sufficient for learning an effective CNN. Our chosen KNNmodel has a overhead that is comparable to the DT and theSVM, but has a higher accuracy. It is possible that the besttechnique can change as the application domain andtraining data size changes, but our generic approach forfeature selection and model selection remains applicable.

Figure 10 shows the runtime and top-1 accuracy by usingthe KNN, DT and SVM to construct a hierarchical premodelof three levels. A configuration is denoted as X .Y .Z , whereX , Y and Z indicates the modeling technique for the first,second and third level of the premodel, respectively. Theresult shows that our chosen premodel organization, (i.e.,KNN.KNN.KNN), has the highest top-1 accuracy (87.4%) andthe fastest running time (0.20 second). One of the benefitsof using a KNN model in all levels is that the neighboringmeasurement only needs to be performed once as the resultscan be shared among models in different levels. This meansthe runtime overhead is nearly constant if we use the KNNacross all hierarchical levels.

5.3 Impact of Inference Model SizesIn Section 3.2 we describe the method we use to chose whichDNN models to include. Using this method, and temporarilyignoring the model selection threshold θ in Algorithm 1, weconstructed Figure 11, where we compare the top-1 accuracyand execution time using up to 5 KNNmodels. As we increasethe number of inference models, there is an increase in theend to end inference time as expensive models are more

Adaptive Deep Learning Model Selection on Embedded Systems LCTES ’18, June 2018, Pennsylvania, USA

knn.k

nn.kn

nkn

n.knn

.dtkn

n.knn

.svm

knn.d

t.knn

knn.d

t.dt

knn.d

t.svm

knn.s

vm.kn

nkn

n.svm

.dtkn

n.svm

.svm

dt.kn

n.knn

dt.kn

n.dt

dt.kn

n.svm

dt.dt.

knn

dt.dt.

dtdt.

dt.svm

dt.svm

.knn

dt.svm

.dtdt.

svm.sv

msvm

.knn.k

nnsvm

.knn.d

tsvm

.knn.s

vmsvm

.dt.kn

nsvm

.dt.dt

svm.dt

.svm

svm.sv

m.kn

nsvm

.svm.

dtsvm

.svm.

svm

0 . 0

0 . 2

0 . 4

0 . 6

Runti

me (s

)

R u n t i m e

02 04 06 08 01 0 0

Top-1

Accu

racy (

%) T o p - 1 A c c u r a c y

Figure 10: Using different modeling techniques toform a 3-level premodel.

1 2 3 4 50 . 00 . 40 . 81 . 21 . 62 . 02 . 4

Infere

nce T

ime (

s)

# I n f e r e n c e M o d e l s

I n f e r e n c e t i m e

02 04 06 08 01 0 0 T o p - 1 a c c u r a c y

Top-1

Accu

racy (

%)Figure 11: Overhead and achieved performance whenusing different numbers of DNNmodels for inferencing.The min-max bars show the range of inference timeacross testing images.

n _ k e y p o i n t sa s p e c t _ r a t i o

c o n t r a s t h u e 1a r e a _ b y _ p e r i m

a v g _ p e r c e i v e d _ b r i g h t n e s se d g e _ l e n g t h 1 h u e 7

e d g e _ a n g l e 5e d g e _ l e n g t h 3

e d g e _ a n g l e 3e d g e _ a n g l e 6

e d g e _ a n g l e 4e d g e _ a n g l e 7

e d g e _ a n g l e 1e d g e _ l e n g t h 2

e d g e _ a n g l e 20

6

1 2

1 8

lost a

ccurac

y (%)

Figure 12: Accuracy loss if a feature is not used.

5 6 7 8 90 . 1 60 . 1 80 . 2 00 . 2 20 . 2 4

Top-1

accu

racy (

%)

Runti

me (s

)

# F e a t u r e s

R u n t i m e

02 04 06 08 01 0 0

T o p - 1 a c c u r a c y

Figure 13: Impact of feature sizes.

likely to be chosen. At the same time, however, the top-1accuracy reaches a plateau of (≈87.5%) by using three KNNmodels. We conclude that choosing three KNN models wouldbe the optimal solution for our case, as we are no longergaining accuracy to justify the increased cost. This is in linewith our choice of a value of 0.5 for θ .

5.4 Feature ImportanceIn Section 3.4 we describe our feature selection process,which resulted in using 7 features to represent each imageto our premodel. In Figure 12 we show the importance ofall of our considered features which were not removed byour correllation check, shown in Table 3. Upon observationit is clear that the 7 features we have chosen to keep are themost important; there is a sudden drop in feature

importance at feature 8 (hue7 ). Furthermore, in Figure 13we show the impact on premodel execution time and top-1accuracy when we change the number of features we use.By decreasing the number of features there is a dramaticdecrease top-1 accuracy, with very little change inextraction time. To reduce overhead, we would need toreduce our feature count to 5, however this comes at thecost of a 13.9% decrease in top-1 accuracy. By increasingthe feature count it can be seen that there is minor changesin overhead, but, surprisingly, there is actually also a smalldecrease in top-1 accuracy of 0.4%. From this we canconclude that using 7 features is ideal.

5.5 Training and Deployment OverheadTraining the premodel is a one-off cost, and is dominated bythe generation of training data which takes in total less than aday (see Section 3.3). This overhead can be speeded up usingmultiple machines. However, compared to the training timeof a typical DNN model, our training overhead is negligible.The runtime overhead of our premodel is minimal, as

depicted in Figures 8a. Out of a total average execution timeof less than a second to classify an image, our premodelaccounts for only 20%. In comparison to the most(ResNet_v2_152) and least (MobileNet) expensive modelswe consider in this work, this translates to 9.52% and 27%,respectively. Furthermore, our energy footprint is muchsmaller, making up 11% of the total cost. Comparing this tothe most and least expensive models, again, gives anoverhead of 7% and 25%, respectively.

6 DISCUSSIONNaturally there is room for further work and possibleimprovements. We discuss a few points here.Alternative Domains. This work focuses on CNNs becauseit is a commonly used deep learning architecture. To extendour work to other domains and recurrent neural networks(RNN), we would need a new set of features to characterizethe input, e.g., text embeddings for machine translation [48].However, our automatic approach on feature selection andpremodel construction remains applicable.Feature Extraction. The majority of our overhead iscaused by feature extraction for our premodel. Ourprototype feature extractor is written in Python; byre-writing this tool in a more efficient language can reducethe overhead. There are also hotshots in our code whichwould benefit from parallelism.Processor Choice. By default, inference is carried out ona GPU, but this may not always be the best choice. Previouswork has already shown machine learning techniques to besuccessful at selecting the optimal computing device [45].This can be integrated into our existing learning framework.

LCTES ’18, June 2018, Pennsylvania, USA

Model Size. Our approach uses multiple pre-trained DNNmodels for inference. In comparison to the default methodof simply using a single model, our approach would requiremore storage space. A solution for this would involve usingmodel compression techniques to generate multiplecompressed models from a single accurate model. Eachcompressed model would be smaller and is specialized atcertain tasks. The result of this is numerous models sharemany weights in common, which allows us to allowing usto amortize the cost of using multiple models.

7 RELATEDWORKDeep neural networks (DNN) have shown astoundingsuccesses in various complex tasks that previously seemeddifficult [7, 27, 30]. Despite the fact that many embeddeddevices require precise sensing capabilities, adoption of DNNmodels on such systems has notably slow progress. Themain cause of this slow progress is that DNN-based inferenceis typically a computation intensive task, which inherentlyruns slowly on embedded devices due to limited resources.Numerous methods have been proposed to reduce the

computational demands of a deep model by tradingprediction accuracy for runtime, via compressing apre-trained network [6, 15, 21, 24, 36, 42, 47], training smallnetworks directly [11, 37], or a combination of both [19].Using these approaches, a user now needs to decide when touse a specific model, in order to meet the predictionaccuracy requirement with minimal latency. This is becausedifferent models have different characteristics in terms ofprediction accuracy and running time. It is a non-trivial taskto make such a crucial decision as the application context(e.g. the model input) is often unpredictable and constantlyevolving. Our work alleviates the user burden byautomatically selecting the most appropriate model to usebased on the application constraint and input context.Neurosurgeon [25] identifies when it is beneficial to

offload a DNN layer to be computed on the cloud. UnlikeNeurosurgeon, we aim to minimize the on-device inferencetime without compromising prediction accuracy. Our workis useful in scenarios when sending data to the cloud isprohibitive due to e.g. poor network connectivity or privacyconcerns. The Pervasive CNN framework [41] generatesmultiple computation kernels for each layer of a CNN, whichare then dynamically selected according to the inputs anduser constraints. A similar approach [38] trains a modeltwice, once on shared data and again on personal data, in anattempt to prevent personal data being sent outside thepersonal domain. In contrast to the latter two works, ourapproach allows having a diverse set of networks, bychoosing the most effective network to use at runtime. They,however, are complementary to our approach, by providingthe capability to fine-tune a single network structure.

Recently, numerous software-based approaches havebeen proposed to accelerate CNN models on embededdevices. They aim to accelerate inference time by exploitingparameter tuning [29], computational kerneloptimization [3, 14], task parallelism [32] andpartition [28, 35], and trading precision for time [20] etc.Since a single model is unlikely to meet all the constraintsof accuracy, inference time and energy consumption acrossinputs [5, 13], it is attractive to have a strategy todynamically select the appropriate model to use. Our workprovides such a capability and is thus complementary toexisting approaches on DNN model acceleration.Off-loading computation to the cloud can accelerate DNN

model inference [46], but this is not always applicable dueto privacy, latency or connectivity issues. The workpresented by Ossia et al. partially addresses the issue ofprivacy-preserving when offloading DNN inference to thecloud [33]. Our adaptive model selection approach allowsone to select which model to use based on the input, and isalso useful when cloud offloading is prohibitively because ofthe latency requirement or the lack of connectivity.

Predictive modeling has been employed in prior works toperform various optimization tasks, including applicationscheduling [12], approximate computing [22], codeoptimization [43] and hardware-software co-design [4]. Nowork so far has applied this technique to dynamically selectdeep learning models to run on embedded devices. Ourapproach is also closely related to ensemble learning wheremultiple models are used to solve an optimization problem.This technique is shown to be useful on scheduling paralleltasks [9] and optimize application memory usage [31]. Thiswork is the first attempt in applying this technique tooptimize deep inference on embedded devices.

8 CONCLUSIONThis paper has presented an adaptive scheme todynamically select a deep learning model to use on anembedded device. Our approach provides a significantimprovement over individual deep learning models in termsof accuracy, inference time, and energy consumption.Central to our approach is a machine learning based methodfor deep learning model selection based on the model inputand the precision requirement. The prediction is based on aset of features of the input, which are tuned and selected byour automatic approach. We apply our approach to theimage recognition task and evaluate it on the Jetson TX2embedded deep learning platform using the ImageNetILSVRC 2012 validation dataset. Experimental results showthat our approach achieves an overall top-1 accuracy ofabove 87.44%, which translates into an improvement of7.52% and 1.8x reduction in inference time when comparedto the most-accurate single deep learning model.

Adaptive Deep Learning Model Selection on Embedded Systems LCTES ’18, June 2018, Pennsylvania, USA

REFERENCES[1] JJ Allaire, Dirk Eddelbuettel, Nick Golding, and Yuan Tang. 2016.

TensorFlow for R. https://tensorflow.rstudio.com/[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural

machine translation by jointly learning to align and translate. arXivpreprint arXiv:1409.0473 (2014).

[3] Sourav Bhattacharya and Nicholas D Lane. 2016. Sparsification andseparation of deep learning layers for constrained resource inferenceon wearables. In Conference on Embedded Networked Sensor Systems.ACM, 176–189.

[4] Bruno Bodin et al. 2016. Integrating Algorithmic Parametersinto Benchmarking and Design Space Exploration in 3D SceneUnderstanding. In PACT.

[5] Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. 2016. AnAnalysis of Deep Neural Network Models for Practical Applications.CoRR abs/1605.07678 (2016).

[6] Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger,and Yixin Chen. 2015. Compressing Neural Networks with the HashingTrick. In ICML.

[7] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, FethiBougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phraserepresentations using RNN encoder-decoder for statistical machinetranslation. In EMNLP.

[8] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, NingZhang, Eric Tzeng, and Trevor Darrell. 2014. DeCAF: A DeepConvolutional Activation Feature for Generic Visual Recognition. InICML (Proceedings of Machine Learning Research), Eric P. Xing andTony Jebara (Eds.), Vol. 32. PMLR, 647–655.

[9] Murali Krishna Emani and Michael O’Boyle. 2015. CelebratingDiversity: A Mixture of Experts Approach for Runtime Mapping inDynamic Environments. In ACM SIGPLAN Conference on ProgrammingLanguage Design and Implementation (PLDI ’15). 499–508.

[10] Dario Amodei et al. 2016. Deep Speech 2: End-to-End SpeechRecognition in English and Mandarin. In ICML (Proceedings of MachineLearning Research), Maria Florina Balcan and Kilian Q. Weinberger(Eds.), Vol. 48. PMLR, New York, New York, USA, 173–182.

[11] Petko Georgiev, Sourav Bhattacharya, Nicholas D. Lane, and CeciliaMascolo. 2017. Low-resource Multi-task Audio Sensing for Mobile andEmbedded Devices via Shared Deep Neural Network Representations.Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3 (2017), 50:1–50:19.

[12] Dominik Grewe et al. 2013. Portable mapping of data parallel programsto OpenCL for heterogeneous systems. In CGO.

[13] Tian Guo. 2017. Towards Efficient Deep Inference for MobileApplications. CoRR abs/1707.04610 (2017).

[14] Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark AHorowitz, and William J Dally. 2016. EIE: efficient inference engineon compressed deep neural network. In 43rd International Symposiumon Computer Architecture. IEEE Press, 243–254.

[15] Song Han, Jeff Pool, John Tran, andWilliam Dally. 2015. Learning bothweights and connections for efficient neural network. In Advances inneural information processing systems. 1135–1143.

[16] M Hassaballah, Aly Amin Abdelmgeid, and HammamAAlshazly. 2016.Image features detection, description and matching. In Image FeatureDetectors and Descriptors. 11–45.

[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deepresidual learning for image recognition. In Conference on computervision and pattern recognition (CVPR). 770–778.

[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.Identity mappings in deep residual networks. In European Conferenceon Computer Vision. Springer, 630–645.

[19] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko,Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.2017. Mobilenets: Efficient convolutional neural networks for mobilevision applications. arXiv preprint arXiv:1704.04861 (2017).

[20] Loc N. Huynh, Youngki Lee, and Rajesh Krishna Balan. 2017. DeepMon:Mobile GPU-based Deep Learning Framework for Continuous VisionApplications. In MobiSys. 82–95.

[21] Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han,William J. Dally, and Kurt Keutzer. 2016. SqueezeNet: AlexNet-levelaccuracy with 50x fewer parameters and <1MB model size. CoRRabs/1602.07360 (2016).

[22] Mohsen Imani, Yeseong Kim, Abbas Rahimi, and Tajana Rosing. 2016.ACAM: Approximate Computing Based on Adaptive AssociativeMemory with Online Learning. In ISLPED.

[23] Sergey Ioffe and Christian Szegedy. 2015. Batch normalization:Accelerating deep network training by reducing internal covariateshift. In ICML.

[24] Jonghoon Jin, Aysegul Dundar, and Eugenio Culurciello. 2015.Flattened Convolutional Neural Networks for FeedforwardAcceleration. (2015).

[25] Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, TrevorMudge, Jason Mars, and Lingjia Tang. 2017. Neurosurgeon:Collaborative Intelligence Between the Cloud and Mobile Edge. InASPLOS.

[26] Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, andFrank Hutter. 2016. Fast bayesian optimization of machine learninghyperparameters on large datasets. arXiv preprint arXiv:1605.07079(2016).

[27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012.ImageNet classification with deep convolutional neural networks. InNIPS.

[28] Nicholas D Lane, Sourav Bhattacharya, Petko Georgiev, ClaudioForlivesi, Lei Jiao, Lorena Qendro, and Fahim Kawsar. 2016. DeepX: Asoftware accelerator for low-power deep learning inference on mobiledevices. In Conference on Information Processing in Sensor Networks(IPSN). IEEE, 1–12.

[29] Seyyed Salar Latifi Oskouei, Hossein Golestani, Matin Hashemi,and Soheil Ghiasi. 2016. Cnndroid: GPU-accelerated execution oftrained deep convolutional neural networks on android. InMultimediaConference. ACM, 1201–1205.

[30] Honglak Lee, Peter Pham, Yan Largman, and Andrew Y. Ng.2009. Unsupervised Feature Learning for Audio Classification UsingConvolutional Deep Belief Networks. In NIPS.

[31] Vicent Sanz Marco, Ben Taylor, Barry Porter, and Zheng Wang. 2017.Improving Spark Application Throughput via Memory Aware TaskCo-location: AMixture of Experts Approach. InMiddleware Conference.95–108.

[32] Mohammad Motamedi, Daniel Fong, and Soheil Ghiasi. 2017. MachineIntelligence on Resource-Constrained IoT Devices: The Case of ThreadGranularity Optimization for CNN Inference. ACM Trans. Embed.Comput. Syst. 16 (2017), 151:1–151:19.

[33] Seyed Ali Ossia, Ali Shahin Shamsabadi, Ali Taheri, Hamid R Rabiee,Nic Lane, and Hamed Haddadi. 2017. A Hybrid Deep LearningArchitecture for Privacy-Preserving Mobile Analytics. arXiv preprintarXiv:1703.02952 (2017).

[34] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, et al. 2015. DeepFace Recognition. In BMVC, Vol. 1. 6.

[35] Sundari K. Rallapalli, H. Qiu, Archith John Bency, S. Karthikeyan, andR. B. Govindan. 2016. Are Very Deep Neural Networks Feasible on MobileDevices? Technical Report 16-965. University of Southern California.

[36] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and AliFarhadi. 2016. XNOR-Net: ImageNet Classification Using Binary

LCTES ’18, June 2018, Pennsylvania, USA

Convolutional Neural Networks. CoRR abs/1603.05279 (2016).[37] Sujith Ravi. 2015. ProjectionNet: Learning Efficient On-Device Deep

Networks Using Neural Projections. arXiv:1708.00630 (2015).[38] Sandra Servia Rodríguez, LiangWang, Jianxin R. Zhao, RichardMortier,

and Hamed Haddadi. 2017. Personal Model Training under PrivacyConstraints. CoRR abs/1703.00380 (2017).

[39] Olga Russakovsky et al. 2015. ImageNet Large Scale Visual RecognitionChallenge. In IJCV.

[40] Nathan Silberman and Sergio Guadarrama. 2013.TensorFlow-slim image classification library.https://github.com/tensorflow/models/tree/master/research/slim.(2013).

[41] Mingcong Song, Yang Hu, Huixiang Chen, and Tao Li. 2017. TowardsPervasive and User Satisfactory CNN across GPU Microarchitectures.In HPCA.

[42] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, andMartin A. Riedmiller. 2014. Striving for Simplicity: The AllConvolutional Net. CoRR abs/1412.6806 (2014).

[43] Kevin Stock, Louis-Noël Pouchet, and P. Sadayappan. 2012. Usingmachine learning to improve automatic vectorization. ACMTransactions on Architecture and Code Optimization (2012).

[44] Yi Sun, Yuheng Chen, Xiaogang Wang, and Xiaoou Tang. 2014. Deeplearning face representation by joint identification-verification. InAdvances in neural information processing systems. 1988–1996.

[45] Ben Taylor, Vicent Sanz Marco, and Zheng Wang. 2017. AdaptiveOptimization for OpenCL Programs on Embedded HeterogeneousSystems. In 18th ACM SIGPLAN/SIGBED Conference on Languages,Compilers, and Tools for Embedded Systems (LCTES 2017). 11–20.

[46] Surat Teerapittayanon, Bradley McDanel, and HT Kung. 2017.Distributed deep neural networks over the cloud, the edge and enddevices. In ICDCS. 328–339.

[47] Min Wang, Baoyuan Liu, and Hassan Foroosh. 2016. FactorizedConvolutional Neural Networks. CoRR abs/1608.04337 (2016).

[48] Will Y Zou, Richard Socher, Daniel Cer, and Christopher D Manning.2013. Bilingual word embeddings for phrase-basedmachine translation.In Conference on Empirical Methods in Natural Language Processing.1393–1398.