Efficient ML in 2 KB RAM for the Internet of Things Ashish Kumar (MSR) Saurabh Goyal (IIT Delhi) Manik Varma (MSR) Bonsai – A Compact Tree Model Prediction Accuracy vs Model Size Sparse Projection (Z) Yes No Yes No Bonsai – Key Ideas Prediction Costs on the Arduino Uno Comparison to Uncompressed Methods Prediction Accuracy vs Model Size 0.2 1 2 4 6 8 10 12 14 48 49 50 51 52 Model Size (KB) nDCG@1 FastRank Bonsai 0.2 0.3 0.4 0.5 52.68 52.7 52.72 Model Size (KB) nDCG@1 Bing L3/L4 Ranker Results Disadvantages of Tree Ensembles Bonsai Optimization • To build an efficient tree classifier • Which can be trained on the cloud • But which can make predictions on tiny IoT devices Arduino Uno Arduino Uno Objective 8 bit ATmega328P Processor at 16 MHz with 2 KB RAM & 32 KB read only Flash • Tree Ensembles might not fit in Kilobytes and might not be accurate • We design Bonsai to be a single, shallow, sparse tree with powerful nodes for accurate prediction • We reduce model size by learning Bonsai in a low- dimensional space into which all data is projected • We jointly learn tree and projection parameters so as to maximize accuracy within the given budget The Bonsai Objective Function Accuracy (%) 30 65 100 RTWhale-2 Chars4K-62 Chars4K-2 WARD-2 Bonsai GBDT kNN RBF-SVM Neural Nets Compressed : Uncompressed : Model Size (KB) RTWhale-2 Chars4K-62 Chars4K-2 WARD-2 1 GB 1MB 1KB Accuracy (%) Pred Time (ms) Pred Energy (mJ) Eye-2 RTWhale -2 0 1.67 3.33 5 4 11 18 75 82.5 90 Eye-2 2187 1312 0 3 2 6 10 50 60 521 313 Bonsai Linear LDKL NeuralNet Cloud-GBDT Bonsai’s Decision Boundaries Code for Bonsai code can be downloaded from http://manikvarma.org/ Model Size (KB) 0 5 10 15 Accuracy (%) 80 82 84 86 88 90 92 Eye-2 Bonsai BonsaiOpt GBDT Tree Pruning LDKL LDKL-L1 NeuralNet Pruning SNC Decision Jungle BudgetRF Model Size (KB) 0 50 100 150 Accuracy (%) 10 20 30 40 50 60 Chars4K-62 Bonsai BonsaiOpt GBDT Tree Pruning LDKL LDKL-L1 NeuralNet Pruning SNC Decision Jungle BudgetRF Model Size (KB) 0 5 10 15 Accuracy (%) 55 60 65 70 75 Chars4K-2 Model Size (KB) 0 5 10 15 Accuracy (%) 66 68 70 72 74 76 CIFAR10-2 Model Size (KB) 0 5 10 15 Accuracy (%) 80 82 84 86 88 90 92 Eye-2 Model Size (KB) 0 5 10 15 Accuracy (%) 86 88 90 92 94 96 MNIST-2 Model Size (KB) 0 5 10 15 Accuracy (%) 50 52 54 56 58 60 62 RTWhale-2 Model Size (KB) 0 5 10 15 Accuracy (%) 90 91 92 93 94 95 96 97 USPS-2 Model Size (KB) 0 5 10 15 Accuracy (%) 90 92 94 96 98 WARD-2