Elicitability of Instance and Object Ranking

References

• Acerbi C, Szekely B (2014) Back-testing expected shortfall. Risk 27(11):76–81.Google Scholar
• Adomavicius G, Zhang J (2016) Classification, ranking, and top-k stability of recommendation algorithms. INFORMS J. Comput. 28(1):129–147.Link, Google Scholar
• Agarwal S, Sengupta S (2009) Ranking genes by relevance to a disease. Proc. Eighth Annual Internat. Conf. Comput. Systems Bioinformatics.Google Scholar
• Agarwal S, Graepel T, Herbrich R, Har-Peled S, Roth D (2005) Generalization bounds for the area under the ROC curve. J. Machine Learn. Res. 6(Apr):393–425.Google Scholar
• Alagoz O, Chhatwal J, Burnside ES (2013) Optimal policies for reducing unnecessary follow-up mammography exams in breast cancer diagnosis. Decision Anal. 10(3):200–224.Link, Google Scholar
• Amisano G, Giacomini R (2007) Comparing density forecasts via weighted likelihood ratio tests. J. Bus. Econom. Statist. 25(2):177–190.Crossref, Google Scholar
• Armstrong RD, Cook WD, Seiford LM (1982) Priority ranking and consensus formation: The case of ties. Management Sci. 28(6):638–645.Link, Google Scholar
• Bao Y, Ke B, Li B, Yu YJ, Zhang J (2020) Detecting accounting fraud in publicly traded us firms using a machine learning approach. J. Accounting Res. 58(1):199–235.Crossref, Google Scholar
• Berrocal VJ, Raftery AE, Gneiting T, Steed RC (2010) Probabilistic weather forecasting for winter road maintenance. J. Amer. Statist. Assoc. 105(490):522–537.Crossref, Google Scholar
• Bickel JE (2007) Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Anal. 4(2):49–65.Link, Google Scholar
• Bickel JE (2010) Scoring rules and decision analysis education. Decision Anal. 7(4):346–357.Link, Google Scholar
• Brier GW (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Rev. 78(1):1–3.Crossref, Google Scholar
• Brier GW, Allen RA (1951) Verification of weather forecasts. Malone TF, ed. Compendium of Meteorology (American Meteorological Society, Boston), 841–848.Crossref, Google Scholar
• Carvalho A (2016) An overview of applications of proper scoring rules. Decision Anal. 13(4):223–242.Link, Google Scholar
• Cassady CR, Maillart LM, Salman S (2005) Ranking sports teams: A customizable quadratic assignment approach. Interfaces 35(6):497–510.Link, Google Scholar
• Cheng W, Hüllermeier E, Waegeman W, Welker V (2012) Label ranking with partial abstention based on thresholded probabilistic models. Adv. Neural Inform. Processing Systems 25:2501–2509.Google Scholar
• Cheng W, Rademaker M, De Baets B, Hüllermeier E (2010) Predicting partial orders: Ranking with abstention. Balcázar JL, Bonchi F, Gionis A, Sebag M, eds. Joint Eur. Conf. Machine Learn. Knowledge Discovery Databases (Springer, Berlin), 215–230.Google Scholar
• Chu LY, Nazerzadeh H, Zhang H (2020) Position ranking and auctions for online marketplaces. Management Sci. 66(8):3617–3634.Link, Google Scholar
• Clark TE, Ravazzolo F (2015) Macroeconomic forecasting performance under alternative specifications of time-varying volatility. J. Appl. Econometrics 30(4):551–575.Crossref, Google Scholar
• Clemen RT, Fischer GW, Winkler RL (2000) Assessing dependence: Some experimental results. Management Sci. 46(8):1100–1115.Link, Google Scholar
• Clémençon S, Vayatis N (2007) Ranking the best instances. J. Machine Learn. Res. 8(Dec):2671–2699.Google Scholar
• Clémençon S, Depecker M, Vayatis N (2013) An empirical comparison of learning algorithms for nonparametric scoring: The TreeRank algorithm and other methods. PAA Pattern Anal. Appl. 16(4):475–496.Crossref, Google Scholar
• Clémençon S, Lugosi G, Vayatis N (2008) Ranking and empirical minimization of U-statistics. Ann. Statist. 36(2):844–874.Crossref, Google Scholar
• Cook WD, Kress M (1985) Ordinal ranking with intensity of preference. Management Sci. 31(1):26–32.Link, Google Scholar
• Dionne G, Giuliano F, Picard P (2009) Optimal auditing with scoring: Theory and application to insurance fraud. Management Sci. 55(1):58–70.Link, Google Scholar
• Dwork C, Kumar R, Naor M, Sivakumar D (2001) Rank aggregation methods for the web. Proc. 10th Internat. Conf. World Wide Web (Association for Computing Machinery, New York), 613–622.Google Scholar
• Fahandar MA, Hüllermeier E (2018) Learning to rank based on analogical reasoning. 32nd AAAI Conf. Artificial Intelligence.Google Scholar
• Fissler T, Ziegel JF (2016) Higher order elicitability and Osband’s principle. Ann. Statist. 44(4):1680–1707.Crossref, Google Scholar
• Fissler T, Ziegel JF, Gneiting T (2016) Expected shortfall is jointly elicitable with value at risk-implications for backtesting. Risk, 58–61.Google Scholar
• Fissler T, Frongillo R, Hlavinová J, Rudloff B (2021) Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals. Electronic J. Statist. 15(1):1034–1084.Crossref, Google Scholar
• Fürnkranz J, Hüllermeier E (2011) Preference Learning, vol. 19 (Springer, Berlin), https://doi.org/10.1007/978-3-642-14125-6.Crossref, Google Scholar
• Fürnkranz J, Hüllermeier E, Vanderlooy S (2009) Binary decomposition methods for multipartite ranking. Buntine W, Grobelnik M, Mladenić D, Shawe-Taylor J, eds. Joint Eur. Conf. Machine Learn. Knowledge Discovery Databases (Springer, Berlin), 359–374.Google Scholar
• Fürnkranz J, Hüllermeier E, Mencía EL, Brinker K (2008) Multilabel classification via calibrated label ranking. Machine Learn. 73(2):133–153.Crossref, Google Scholar
• Gneiting T (2011) Making and evaluating point forecasts. J. Amer. Statist. Assoc. 106(494):746–762.Crossref, Google Scholar
• Gneiting T (2017) When is the mode functional the Bayes classifier? Statist. 6(1):204–206.Crossref, Google Scholar
• Gneiting T, Raftery AE (2005) Weather forecasting with ensemble methods. Sci. 310(5746):248–249.Crossref, Google Scholar
• Gneiting T, Raftery AE (2007) Strictly proper scoring rules, prediction, and estimation. J. Amer. Statist. Assoc. 102(477):359–378.Crossref, Google Scholar
• Gneiting T, Ranjan R (2011) Comparing density forecasts using threshold-and quantile-weighted scoring rules. J. Bus. Econom. Statist. 29(3):411–422.Crossref, Google Scholar
• Gneiting T, Balabdaoui F, Raftery AE (2007) Probabilistic forecasts, calibration and sharpness. J. Royal Statist. Soc. Series B. Statist. Methodology 69(2):243–268.Crossref, Google Scholar
• Good I (1952) Rational decisions. J. Royal Statist. Soc. Series B. Statist. Methodology 14(1):107–114.Google Scholar
• Grushka-Cockayne Y, Lichtendahl KC Jr, Jose VRR, Winkler RL (2017) Quantile evaluation, sensitivity to bracketing, and sharing business payoffs. Oper. Res. 65(3):712–728.Link, Google Scholar
• Heinrich C (2013) The mode functional is not elicitable. Biometrika 101(1):245–251.Crossref, Google Scholar
• Herbrich R, Graepel T, Obermayer K (1999a) Regression Models for Ordinal Data: A Machine Learning Approach (Citeseer).Google Scholar
• Herbrich R, Graepel T, Obermayer K (1999b) Support vector learning for ordinal regression. Ninth Internat. Conf. Artificial Neural Networks (IET), 97–102.Google Scholar
• Hochbaum DS, Levin A (2006) Methodologies and algorithms for group-rankings decision. Management Sci. 52(9):1394–1408.Link, Google Scholar
• Hüllermeier E, Fürnkranz J (2010) On predictive accuracy and risk minimization in pairwise label ranking. J. Comput. System Sci. 76(1):49–62.Crossref, Google Scholar
• Joachims T (2005) Accurately interpreting clickthrough data as implicit feedback. Proc. 28th Annual Internat. ACM SIGIR Conf. Res. Development Inform. Retrieval (ACM, New York), 154–161.Google Scholar
• Jose VRR, Winkler RL (2009) Evaluating quantile assessments. Oper. Res. 57(5):1287–1297.Link, Google Scholar
• Kamishima T, Kazawa H, Akaho S (2010) A survey and empirical comparison of object ranking methods. Fürnkranz J, Hüllermeier E, eds. Preference Learning (Springer, Berlin), 181–201.Crossref, Google Scholar
• Kilgour DM, Gerchak Y (2004) Elicitation of probabilities using competitive scoring rules. Decision Anal. 1(2):108–113.Link, Google Scholar
• Kratz M (2017) Discussion of elicitability and backtesting: Perspectives for banking regulation. Ann. Appl. Statist. 11(4):1894–1900.Crossref, Google Scholar
• Lambert NS, Pennock DM, Shoham Y (2008) Eliciting properties of probability distributions. Proc. 9th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 129–138.Google Scholar
• Lichtendahl KC Jr, Winkler RL (2007) Probability elicitation, scoring rules, and competition among forecasters. Management Sci. 53(11):1745–1755.Link, Google Scholar
• Luce RD (1959) Individual Choice Behavior (John Wiley).Google Scholar
• Luce RD (1977) The choice axiom after twenty years. J. Math. Psych. 15(3):215–233.Crossref, Google Scholar
• McFadden D (1973) Conditional logit analysis of qualitative choice behavior. Zarembka P, ed. Frontiers in Econometrics (Academic Press, New York), 105–142.Google Scholar
• McFadden D (1986) The choice theory approach to market research. Marketing Sci. 5(4):275–297.Link, Google Scholar
• McSherry F, Najork M (2008) Computing information retrieval performance measures efficiently in the presence of tied scores. Macdonald C, Ounis I, Plachouras V, Ruthven I, White RW, eds. Eur. Conf. Inform. Retrieval (Springer, Berlin), 414–421.Google Scholar
• Merkle EC, Steyvers M (2013) Choosing a strictly proper scoring rule. Decision Anal. 10(4):292–304.Link, Google Scholar
• Miller N, Resnick P, Zeckhauser R (2005) Eliciting informative feedback: The peer-prediction method. Management Sci. 51(9):1359–1373.Link, Google Scholar
• Negahban S, Oh S, Shah D (2017) Rank centrality: Ranking from pairwise comparisons. Oper. Res. 65(1):266–287.Link, Google Scholar
• Nolde N, Ziegel JF (2017) Elicitability and backtesting: Perspectives for banking regulation. Ann. Appl. Statist. 11(4):1833–1874.Crossref, Google Scholar
• Osband K (1985) Providing incentives for better cost forecasting. Unpublished PhD thesis, University of California, Berkeley.Google Scholar
• Page L, Brin S, Motwani R, Winograd T (1999) The PageRank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, CA.Google Scholar
• Pfannschmidt K, Gupta P, Hüllermeier E (2018) Deep architectures for learning context-dependent ranking functions. Preprint, submitted March 15, https://arxiv.org/abs/1803.05796.Google Scholar
• Pickett KS (2006) Audit Planning: A Risk-Based Approach (John Wiley & Sons, New York).Google Scholar
• Plackett RL (1975) The analysis of permutations. J. Royal Statist. Soc. Series C. Appl. Statist. 24(2):193–202.Crossref, Google Scholar
• R Core Team (2021) R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna).Google Scholar
• Rose H, Rogers A, Gerding EH (2012) A scoring rule-based mechanism for aggregate demand prediction in the smart grid. Proc. 11th Internat. Conf. Autonomous Agents Multiagent Systems, vol. 2 (International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC), 661–668.Google Scholar
• Roy A, Mackin P, Wallenius J, Corner J, Keith M, Schymik G, Arora H (2008) An interactive search method based on user preferences. Decision Anal. 5(4):203–229.Link, Google Scholar
• Sapir M (2011) Bipartite ranking algorithm for classification and survival analysis. Preprint, submitted December 8, https://arxiv.org/abs/1112.1966.Google Scholar
• Schervish MJ, Seidenfeld T, Kadane JB (2009) Proper scoring rules, dominated forecasts, and coherence. Decision Anal. 6(4):202–221.Link, Google Scholar
• Shen H, Hong LJ, Zhang X (2021) Ranking and selection with covariates for personalized decision making. INFORMS J. Comput. 33(4):1500–1519.Abstract, Google Scholar
• Szörényi B, Busa-Fekete R, Paul A, Hüllermeier E (2015) Online rank elicitation for Plackett-Luce: A dueling bandits approach. Adv. Neural Inform. Processing Systems 28:604–612.Google Scholar
• Torgo L, Ribeiro R (2007) Utility-based regression. Kok JN, Koronacki J, Lopez de Mantaras R, Matwin S, Mladenič D, Skowron A, eds. Eur. Conf. Principles Data Mining Knowledge Discovery (Springer, Berlin), 597–604.Google Scholar
• Van Vlasselaer V, Eliassi-Rad T, Akoglu L, Snoeck M, Baesens B (2017) Gotcha! Network-based fraud detection for social security fraud. Management Sci. 63(9):3090–3110.Link, Google Scholar
• Waegeman W, De Baets B, Boullart L (2008) ROC analysis in ordinal regression learning. Pattern Recognition Lett. 29(1):1–9.Crossref, Google Scholar
• Werner T (2019) Gradient-free gradient boosting. PhD thesis, Carl von Ossietzky Universität Oldenburg, Germany. https://oops.uni-oldenburg.de/id/eprint/4290Google Scholar
• Werner T (2021a) A review on instance ranking problems in statistical learning. Machine Learn., ePub ahead of print November 18, https://doi.org/10.1007/s10994-021-06122-3.Google Scholar
• Werner T (2021b) Global quantitative robustness of instance ranking problems. Preprint, submitted March 12, https://arxiv.org/abs/2103.07198.Google Scholar
• Winkler RL, Murphy AH (1968) “Good” probability assessors. J. Appl. Meteorology Climatology 7(5):751–758.Crossref, Google Scholar
• Winkler RL, Grushka-Cockayne Y, Lichtendahl KC Jr, Jose VRR (2019) Probability forecasts and their combination: A research perspective. Decision Anal. 16(4):239–260.Link, Google Scholar
• Yoganarasimhan H (2020) Search personalization using machine learning. Management Sci. 66(3):1045–1070.Link, Google Scholar
• Yuan Y, Zhou QM, Li B, Cai H, Chow EJ, Armstrong GT (2018) A threshold-free summary index of prediction accuracy for censored time to event data. Statist. Medicine 37(10):1671–1681.Crossref, Google Scholar
• Zhou QM, Lu Z, Brooke RJ, Hudson MM, Yuan Y (2020) Is the new model better? One metric says yes, but the other says no. Which metric do I use? Preprint, submitted December 15, https://arxiv.org/abs/2010.09822.Google Scholar