Economic Behavior of Information Acquisition: Impact on Peer Grading in Massive Open Online Courses

References

• Abbas AE (2009) A Kullback-Leibler view of linear and log-linear pools. Decision Anal. 6(1):25–37.Link, Google Scholar
• Arrow KJ (1973) Information and economic behavior. Technical report 14, National Technical Information Service, U.S. Department of Commerce, Washington, DC.Google Scholar
• Arrow KJ (1984) Collected Papers of Kenneth J. Arrow, Volume 4: The Economics of Information (Harvard University Press, Cambridge, MA).Google Scholar
• Arrow KJ (1996) The economics of information: An exposition. Empirica 23:119–128.Crossref, Google Scholar
• Arrow KJ, Radner R (1979) Allocation of resources in large teams. Econometrica 47(2):361–385.Crossref, Google Scholar
• Balfour SP (2013) Assessing writing in MOOCs: Automated essay scoring and calibrated peer review. Res. Practice Assessment 8(1):40–48.Google Scholar
• Bordley RF (1982) A multiplicative formula for aggregating probability assessments. Management Sci. 28(10):1137–1148.Link, Google Scholar
• Brinton CG, Chiang M, Jain S, Lam H, Liu Z, Wong FMF (2014) Learning about social learning in MOOCs: From statistical analysis to generative model. IEEE Trans. Learning Tech. 7(4):346–359.Crossref, Google Scholar
• Calsamiglia C, Loviglio A (2019) Grading on a curve: When having good peers is not good. Econom. Edu. Rev. 73:101916.Crossref, Google Scholar
• Calvo-Armengol A, Patacchini E, Zenou Y (2009) Peer effects and social networks in education. Rev. Econom. Stud. 76(4):1239–1267.Crossref, Google Scholar
• Cho K, Schunn CD, Wilson RW (2006) Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives. J. Edu. Psych. 98(4):891–901.Crossref, Google Scholar
• Clemen RT (1986) Calibration and the aggregation of probabilities. Management Sci. 32(3):312–314.Link, Google Scholar
• Clemen RT, Winkler RL (1990) Unanimity and compromise among probability forecasters. Management Sci. 36(7):767–779.Link, Google Scholar
• Coursera Help Center (2018) Getting and viewing grades for peer-reviewed assignments. Accessed October 9, 2021, https://learner.coursera.help/hc/en-us/articles/208279946-See-feedback-and-grades-for-peer-reviewed-assignments.Google Scholar
• Dasgupta A, Ghosh A (2013) Crowdsourced judgement elicitation with endogenous proficiency. Proc. 22nd Internat. Conf. World Wide Web (ACM), 319–330.Google Scholar
• DeGroot MH (1974) Reaching a consensus 1974. J. Amer. Statist. Assoc. 69(345):118–121.Crossref, Google Scholar
• DeGroot MH, Mortera J (1991) Optimal linear opinion pools. Management Sci. 37(5):546–558.Link, Google Scholar
• Falk A, Ichino A (2006) Clean evidence on peer effects. J. Labor Econom. 24(1):39–57.Crossref, Google Scholar
• Falchikov N, Goldfinch J (2000) Student peer assessment in higher education: a meta-analysis comparing peer and teacher marks. Rev. Edu. Res. 24(1):287–322.Crossref, Google Scholar
• Gallus J (2017) Fostering public good contributions with symbolic awards: A large-scale natural field experiment at Wikipedia. Management Sci. 63(12):3999–4015.Link, Google Scholar
• Granger CW (1989) Invited review: Combining forecasts—twenty years later. J. Forecasting 8(3):167–173.Crossref, Google Scholar
• Grant A (2016) Why we should stop grading students on a curve. NY Times, September 10, http://nyti.ms/2cCFFfO.Google Scholar
• Howard RA (1966) Information value theory. IEEE Trans. Systems Sci. Cybernetics 2(1):22–26.Crossref, Google Scholar
• Kamble V, Marn D, Shah N, Parekh A, Ramchandran K (2018) A truth serum for large-scale evaluations. Preprint, submitted February 2018, updated August 2021, https://arxiv.org/abs/1507.07045v4.Google Scholar
• Keynes JM (1936) The General Theory of Employment, Interest and Money (Harcourt Brace and Co., New York).Google Scholar
• Khalil H, Ebner M (2014) MOOCs completion rates and possible methods to improve retention - a literature review. Proc. World Conf. Educational Multimedia Hypermedia Telecommunications (AACE), 1236–1244.Google Scholar
• Li HL, Xiong Y, Zang XJ, Kornbaher M, Lyu YS, Chung KS, Suen HK (2016) Peer assessment in a digital age: A meta-analysis comparing peer and teacher ratings. Assessment Evaluation Higher Edu. 41(2):245–264.Crossref, Google Scholar
• Luaces O, Diez J, Alonso-Betanzos A, Troncoso A, Bahamonde A (2015) A factorization approach to evaluate open-response assignments in MOOCs using preference learning on peer assessments. Knowledge Based Systems 85:322–328.Crossref, Google Scholar
• Luo H, Robinson AC, Park JY (2014) Peer grading in a MOOC: Reliability, validity, and perceived effects. Online Learning 18(2):1–14.Crossref, Google Scholar
• Michaels JW (1976) A Simple view of the grading issue. Teaching Sociol. 3(2):198–203.Crossref, Google Scholar
• Miller N, Resnick P, Zeckhauser R (2005) Eliciting informative feedback: The peer-prediction method. Management Sci. 51(9):1359–1373.Link, Google Scholar
• Morris S, Shin HS (2002) Social value of public information. Amer. Econom. Rev. 92(5):1521–1534.Crossref, Google Scholar
• Piech C, Huang J, Chen Z, Do C, Ng A, Koller D (2013) Tuned models of peer assessment in MOOCs. Preprint, submitted July 2013, https://arxiv.org/abs/1307.2579.Google Scholar
• Prelec D (2004) A Bayesian truth serum for subjective data. Science 306(5695):462–466.Crossref, Google Scholar
• Price E, Goldberg F, Robinson S, McKean M (2016) Validity of peer grading using Calibrated Peer Review in a guided-inquiry, conceptual physics course. Phys. Rev. Phys. Edu. Res. 12(2):020145.Crossref, Google Scholar
• Radanovic G, Faltings B (2015) Incentives for subjective evaluations with private beliefs. Proc. 29th AAAI Conf. Artificial Intelligence (AAAI, Palo Alto, CA), 1014–1020.Google Scholar
• Raman K, Joachims T (2014) Methods for ordinal peer grading. Proc. 20th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (ACM, New York), 1037–1046.Google Scholar
• Sacerdote B (2001) Peer effects with random assignment: results for Dartmouth roommates. Quart. J. Econom. 116(2):681–704.Crossref, Google Scholar
• Satopää VA (2017) Combining information from multiple forecasters: Inefficiency of central tendency. Preprint, submitted 2017, https://arxiv.org/abs/1706.06006.Google Scholar
• Satopää VA, Baron J, Foster DP, Mellers BA, Tetlock PE, Ungar LH (2014) Combining multiple probability predictions using a simple logit model. Internat. J. Forecasting 30(2):344–356.Crossref, Google Scholar
• Savva N, Tezcan T, Yildiz O (2019) Can yardstick competition reduce waiting times? Management Sci. 65(7):3196–3215.Link, Google Scholar
• Shah NB, Bradley JK, Parekh A, Wainwright M, Ramchandran K (2013) A case for ordinal peer-evaluation in MOOCs. Proc. NIPS Workshop Data Driven Education.Google Scholar
• Shleifer A (1985) A theory of yardstick competition. RAND J. Econom. 16(3):319–327.Crossref, Google Scholar
• Shnayder V, Agarwal A, Frongillo F, Parkes D (2016) Informed truthfulness in multi-task peer prediction. Proc. ACM Conf. Econom. Comput. 179–196.Google Scholar
• Walsh T (2014) The PeerRank method for peer assessment. Preprint, submitted May 2014, https://arxiv.org/abs/1405.7192.Google Scholar
• Young JR (2020) Will COVID-19 lead to another MOOC moment? Ed Surge, Higher Education, March 25, https://www.edsurge.com/news/2020-03-25-will-covid-19-lead-to-another-mooc-momentGoogle Scholar
• Zhang D, Allon G, Van Mieghem JA (2017) Does social interaction improve learning outcomes? Evidence from field experiments on massive open online courses. Manufacturing Service Oper. Management 19(3):347–367.Link, Google Scholar
• Zimmerman DJ (2003) Peer effects in academic outcomes: Evidence from a natural experiment. Rev. Econom. Statist. 85(1):9–23.Crossref, Google Scholar