Kaggle Competitions in the Classroom: Retrospectives and Recommendations

After increasing our course offerings in data science and machine learning at the United States Military Academy at West Point, we have been investigating innovative ways to motivate our students to engage with this challenging material and explore data outside the classroom. In addition, with the move to virtual classes and students spread across numerous time zones, we needed to find additional ways to build community and leverage our networks to add a flair to our teaching beyond the traditional in-class assessments. We found one such opportunity in Kaggle’s ability to host private competitions for educators.

Kaggle is an online data science platform best known for its public machine learning competitions. Since its inception in 2008, Kaggle has hosted thousands of competitions to advance research on a myriad of topics ranging from satellite imagery feature detection [1] to COVID-19 research analysis [2]. Besides hosting these global competitions, Kaggle has grown into an expansive data science educational platform that offers full courses, an open-source dataset repository, and a feature known as Kaggle InClass competitions (KICs) [3]. These KICs are contests specifically tailored for use in an academic setting that are completely customizable by the hosting professor or teaching assistant. In turn, the Kaggle platform provides a leaderboard, centralized location to access the competition data, and support for various popular code libraries.

Because all competition administration is controlled by the hosts, instructors can tailor the competition to their students or courses with a high degree of specificity. We recently added KICs as assessments to two of our courses: Applied Neural Networks (CS485) and Generalized Linear Models (MA478). One of the competitions was employed as a class project and the other as part of our term-end examination. This article describes the benefits we observed from implementing KICs and recommendations to overcome the challenges we encountered throughout the process.

Positive Student Response

Overall, students responded very positively to the inclusion of KICs. Instructors in both courses observed a heightened level of enthusiasm for course material and observed students initiating discussions and conducting self-guided research outside of class. This effect was amplified when we extended our competitions to the wider faculty, which motivated students to surpass individuals other than their classmates on the leaderboard. Additionally, the competition inspired students to look beyond the course scopes and apply more advanced techniques to progress on the leaderboard. Furthermore, most students expressed that they far preferred KICs over standard homework assignments.

After applying KICs across our two courses, and observing the qualitative effects in each, we can offer various recommendations to other educators looking to bolster their courses with some friendly competition. Firstly, despite the extensive functionality offered by Kaggle overall, we deem the effort required to set up a KIC to be nontrivial from a time-requirement perspective. While the level of complexity will vary based on the nature of the competition, educators are required to populate many different webpages with instructions, descriptions and rules. Furthermore, competition creators must select a scoring metric from a wide array of options, then provide solution and example submission sets. Given these requirements, instructors may face some technical challenges getting their competition off the ground. Our best recommendation here is for educators to start early when crafting their competitions.

Next, educators should thoroughly test their intended task and dataset to determine the range of scores students are likely to achieve. For the CS485 KIC, students were tasked with training an artificial neural network to produce text in the style of Dante Alighieri’s “Inferno.” Given the small size of the training corpus, students struggled to exceed 50% accuracy with their models. While this accuracy was enough to generate coherent words and phrases, an outside observer unfamiliar with machine learning and natural language processing may have judged this result as underwhelming.

On the other hand, in our MA478 competition, students could achieve upward of 97% accuracy by running a basic logistic regression model without any diagnostics. While attractive to the outside observer, this high level of accuracy for an initial model might discourage students from continuing to refine their models, especially with undergraduate students unfamiliar with machine learning. Thus, we recommend educators take great care in selecting data and crafting a competition that will inspire students, communicate the effectiveness of data science and machine learning techniques, and appear pedagogically sound to your colleagues.

Inspiring, Motivating Students

Regarding inspiring and motivating students, a key component of any assigned course homework or project is how it will be assessed. Accordingly, we want to stress that the leaderboard functionality of KICs should not be the only assessment tool used when implementing them. While the KIC offers a great platform to host some aspects of the assessment and serves as a motivator to encourage students to engage with the material, it does not offer instructors a complete perspective on the students’ performance on the assigned task and understanding of the underlying theory. Without an additional external assessment, such as a written report explaining their methodology, the KIC simply becomes a black box that ranks student submissions on a public leaderboard.

In both of our courses, the KIC was implemented as a graded assignment that required submission of a written report and the student’s code that they used to generate their submissions. These reports allowed instructors to use the students’ ability to articulate their model design methodologies as the greatest determinant in their final grades. We presume that conducting the assessment this way is especially important in courses where students are partaking in a KIC for the first time. In these scenarios, executing a competition without an additional graded submission might result in missing the opportunity for a thorough assessment of the students’ learning beyond final leaderboard positions.

We also want to address the risk of students finding KICs too interesting, resulting in a tendency to spend excessive amounts of time on the assignment. Students with less experience in data science and machine learning may not realize they are investing significant amounts of time to achieve results that produce only small performance improvement versus the amount of time and computation invested. To prevent students from spending hours retraining their models when their time would be better spent elsewhere, we recommend limiting the length of the competition to under two weeks. We implemented a two-week time limit in both courses and found that students had plenty of time to make multiple submissions without placing too much of a burden on their schedule and competing academic requirements.

Key Factor to Success

Lastly, our experiences with KICs led us to conclude that a key factor in their success is the course into which they are implemented. In our sample, the two courses, Applied Neural Networks and Generalized Linear Models, focused on two key areas of data science. The Applied Neural Networks course sought to showcase the practical application of artificial neural networks across a wide array of problems, leading us to integrate the KIC into one of the later blocks of instruction. The Generalized Linear Model course explored predictive models and techniques that enabled analysis of binary, binomial, multinomial, count and categorical data. In both cases, the classes were senior-level courses for students with programming experience in either Python or R. Students without this level of knowledge would likely be intimidated or overwhelmed by the requirements of a KIC, especially when it is a heavily weighted, individually graded event. We believe that KICs are best employed in advanced coursework after students have learned the basics of programming.

Ultimately, we found that Kaggle InClass Competitions are an interesting additional assessment tool that allows educators to provide motivation and inject friendly competition into their data science course in a way that leverages our connected educational environment. Given a reasonable amount of time for instructors to prepare, and effective consideration of expected results, these competitions have the potential to inspire students to excel, thus enhancing their learning and appreciation for course material. We plan to continue bolstering our curricula with modern tools such as these, and heartily recommend you give them a look within your endeavors.

References

1. V. Iglovikov, S. Mushinskiy and V. Osin, 2017, “Satellite Imagery Feature Detection using Deep Convolutional Neural Network: A Kaggle Competition,” http://arxiv.org/abs/1706.06169.
2. “COVID-19 Open Research Dataset Challenge (CORD-19),” https://kaggle.com/allen-institute-for-ai/CORD-19-research-challenge.
3. “Kaggle Competitions,” https://www.kaggle.com/c/about/inclass.