Performance-Aligned Learning Algorithms using Distributionally Robustness Principle Rizal Fathony Post-Doctoral Fellow @ Carnegie Melon University Joint work with: Anqi Liu, Kaiser Asif, Mohammad Bashiri, Wei Xing, Sima Behpour, Xinhua Zhang, Brian Ziebart, Zico Kolter.
22
Embed
Performance-Aligned Learning Algorithms using Distributionally ... · Rizal Fathony, Kaiser Asif, Anqi Liu, Mohammad Bashiri, Wei Xing, Sima Behpour, Xinhua Zhang, Brian D. Ziebart.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Performance-Aligned Learning Algorithms using Distributionally Robustness Principle
Rizal FathonyPost-Doctoral Fellow @ Carnegie Melon University
Joint work with: Anqi Liu, Kaiser Asif, Mohammad Bashiri, Wei Xing, Sima Behpour, Xinhua Zhang, Brian Ziebart, Zico Kolter.
Data
DataDistribution𝑃(𝒙, 𝑦)
𝒙1 𝑦1
𝒙2 𝑦2
𝒙𝑛 𝑦𝑛
…
Training
Supervised Learning | Classification
𝒙𝑛+1 ො𝑦𝑛+1
Testing
𝒙𝑛+2
…
Loss / Performance Metrics:loss ො𝑦, 𝑦 / metric( ො𝑦, 𝑦)
ො𝑦𝑛+2
Examples (depend on the task)
• Zero one loss / accuracy metric• Absolute loss (for ordinal regression)
• F1-score• Precision@k• Hamming loss (sum of 0-1 loss)
Example: Digit Recognition
…
1
2
3
…
accuracy ෝ𝒚, 𝒚 =1
𝑛
𝑖
𝐼( ො𝑦𝑖 = 𝑦𝑖)
Performance Metric: Accuracy
loss ෝ𝒚, 𝒚 =1
𝑛
𝑖
𝐼( ො𝑦𝑖 ≠ 𝑦𝑖)
Loss Metric: Zero-One Loss
Binary/Multiclass Classification
absloss ෝ𝒚, 𝒚 =1
𝑛
𝑖
| ො𝑦𝑖 − 𝑦𝑖|
Loss Metric: Absolute Loss
…
1
2
5
…
Predicted vs Actual Label:
Distance Loss
Example: Movie Rating Prediction
Ordinal Regression/Classification
F1𝑠𝑐𝑜𝑟𝑒 ෝ𝒚, 𝒚 =2 TP
AP + PP
Performance Metric: F1 Score
------
--
------
--
-
-----
--
------
--
------
--
------
--
--
-
---
-
--
+
+
+
+
+--
--
--- -
--
------
----
----
------
--
--
-
Confusion Matrix
Classification with Imbalance Datasets
Learning Tasks & Loss/Performance Metric
Machine Learning Tasks Popular Loss/Performance Metrics
Imbalance Datasets - F1-Score- Area under ROC Curve (AUC)- Precision vs Recall
Medical classification tasks - Specificity- Sensitivity- Bookmaker Informedness
Information retrieval tasks - Precision@k- Mean Average Precision (MAP)- Discounted cumulative gain (DCG)
Weighted classification tasks - Cost-sensitive loss metric
Align with the loss/performance metricby incorporating the metric into its learning objective
Perform well in practice
Adversarial Prediction Framework
A distributionally robust learning frameworkwith uncertainty set defined over the conditional distributions
Easy to integrate with deep learning pipeline
References
• Adversarial Cost-Sensitive ClassificationKaiser Asif, Wei Xing, Sima Behpour, and Brian D. Ziebart.Conference on Uncertainty in Artificial Intelligence (UAI), 2015.
• Adversarial Multiclass Classification: A Risk Minimization PerspectiveRizal Fathony, Anqi Liu, Kaiser Asif, Brian D. Ziebart. Advances in Neural Information Processing Systems 29 (NeurIPS), 2016.
• Adversarial Surrogate Losses for Ordinal RegressionRizal Fathony, Mohammad Bashiri, Brian D. Ziebart. Advances in Neural Information Processing Systems 30 (NeurIPS), 2017.
• Consistent Robust Adversarial Prediction for General Multiclass ClassificationRizal Fathony, Kaiser Asif, Anqi Liu, Mohammad Bashiri, Wei Xing, Sima Behpour, Xinhua Zhang, Brian D. Ziebart. ArXiv preprint, 2018.
• AP-Perf: Incorporating Generic Performance Metrics in Differentiable LearningRizal Fathony and Zico KolterIn submission, 2019.