Which Data Sets Are Preferred by University Students in Learning Analytics Dashboards? A Situated Learning Theory Perspective

1. Introduction

As interests in exploiting the wealth of educational data generated by e-learning and web-based technologies for improving teaching and learning in higher education institutions (HEIs) increase across the globe (Waheed et al. 2018, Matcha et al. 2019), creating learner-centered designs and personalized dashboards has also gained considerable attention in the field of learning analytics (Lee et al. 2020, Valle et al. 2021). A learning analytics dashboard (LAD) “aggregates different indicators about learner(s), learning process(es) and/or learning context(s) into one or multiple visualisations” (Schwendimann et al. 2016, p. 37) so that teachers and learners can make sense of data at a glance (Few 2013), reflect on their practices, become better in their respective roles, and improve teaching and learning (Verbert et al. 2014, 2020). Generally, institutions collect data on students and their learning contexts unobtrusively from a range of data sources and provide engagement indicators through LADs. The benefits of LADs include enhanced information, improvement of learning outcomes, refinement of contents, development of data-driven courses, identification and prediction of students at risk, providing real-time feedback, improved conversations with personal tutors, closer monitoring of team assessment behaviors, optimization of teaching and learning processes, and personalization of learning (Banihashem et al. 2018, El Alfy et al. 2019). However, scholars (de Quincey et al. 2019, Selwyn 2019) lament that most of the LADs are teacher-facing and the students’ voice is absent in decision making about learning analytics (LA), thus ignoring the priorities and needs of students (West et al. 2020). This neglect results in low student use of LADs, which, in turn, diminishes the promise of LA as an enabler of deep learning (Kitto et al. 2017). Consequently, toward engaging students in the decision-making process about LA and to design LADs that better reflect students’ reality, we asked students to identify data sources that they consider to be relatively more meaningful than those currently used in HEIs (e.g., attendance records or library usage data). Our motivation is based on the evidence that students’ active engagement in LAD data set identification not only motivates them, but also affects their learning behavior positively (de Quincey et al. 2019) and improves the transfer of learning (Molenaar et al. 2020). Two research questions guided our empirical efforts:

1. Which of the author-identified data sets are perceived as important for student-facing LADs?

2. What additional student-generated data sets are perceived as important in LADs?

We make two contributions. First, in response to the criticism that most LADs lack theoretical foundations for their style and substance (Rogers et al. 2016, Jivet et al. 2018, Matcha et al. 2019), we use a situated theory of learning (Lave and Wenger 1990, Wenger 1999) as a lens and extend our understanding of situated learning–inspired data sets for LAD designs. This theory assumes that student groups share common interests in academic achievement or career success; they learn how to achieve those goals more effectively as they interact regularly in a specific socio-temporal context. In these groups, learning happens openly, informally, spontaneously, and incidentally as students help each other and share information with each other. Learning becomes a part of an everyday activity that is fluid, situated, and social and that is not confined to lecture halls and institutional systems. By adopting this theoretical perspective, we enable students to evaluate a wide range of data sets commonly used in LAD studies (e.g., time spent on watching video lectures). Thus, we shed light on a new set of data sets that are perceived as important for inclusion in student-facing LADs. We make an evidence-based case for situated-learning LADs that are theory-inspired and story-integrated.

Second, with the help of a card-sorting technique, we reveal students’ complex learning behaviors. Demonstrating the specifics of using a card-sorting technique—a gap identified in the literature (Sarmiento et al. 2020)—we call for LADs that better reflect learners’ social contexts and their everyday needs and priorities. In doing so, we provide a caution that offering genuine control and oversight to students in designing their own LADs might be more complex and challenging to implement in universities. We alert researchers to the fact that institutional overreliance on systems-based digital footprints as proxies for academic success may mean that we never fully capture the learning behaviors of students, who increasingly engage in self-directed, socially situated learning.

As a first step in this paper, we present the theoretical context of the study. We then describe our study design, analysis, and empirical findings. We conclude with a discussion on the implications and further directions of LAD design research.

2. Literature Review

The term “learning analytics” refers to “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” (Siemens and Long 2011, p. 34). This study is about the reporting stage of the LA process. We focus on students’ perspectives on LA dashboards so that students’ needs are placed at the heart of LAD designs. A review (Viberg et al. 2018) confirms that there is a shift toward a deeper understanding of students’ learning experiences in LA. As part of this shift, HEIs explore ways of providing students with access to the learning data that they generate when engaged in campus-based learning so that fresh opportunities, which encourage students’ agency, sense making, and reflection, are created for them. To this end, a field of research on student-facing LADs has gained prominence among LA researchers (Avella et al. 2016). In these dashboards, information tracing and digital footprints are aggregated and visually presented to students to enhance learning. Over the years, scholars propose frameworks and guidelines for designing LADs (e.g., Aljohani et al. 2019, Matcha et al. 2019), and many share their implementation experience (Kim et al. 2016, Chen et al. 2018). Nonetheless, reviewers continue to call for designs that are more meaningful to students so that, by having real-time access to relevant behavioral data, it could be possible to increase student awareness, reflection, and achievement (Bodily and Verbert 2017) as many LADs fail to motivate students who have different academic achievement levels (Kim et al. 2016), and they are not used (Bodily et al. 2018). Therefore, there are calls for more extensive, thorough, and rigorous research as a foundation for effective student-facing LADs (Roberts et al. 2016, 2017; Teasley 2017; Lim et al. 2019).

In response to these calls, researchers explored ways of designing dashboards that instruct students on what is useful to them and when they need it, also highlighting the actions students should take (Echeverria et al. 2018, Park and Jo 2019). What these student-facing LADs might look like and how they can combine the functions of various systems, also allowing extreme customization and displaying meaningful recommendations for each one, are questions that are yet to be fully answered. However, some reviews offer a clue to such designs. Bodily and Verbert (2017), in a review of 94 LAD studies, conclude that the following data sources are displayed in most LADs: “resource use” (the number of times students accessed course materials – 76%), “assessment data” (students’ academic performance – 36%), “social interaction data in online environments” (students’ engagement with the course blog and discussion board – 35%), “time spent data” (how long students accessed course materials – 31%), “other sensor data” (information gathered from sensors, such as face recognition, or biometric sensors – 7%), and “manually reported data” (students track and report their own performance data – 5%). Although these data sources may be useful to students to some extent, the reviewers confirm the need for integrating new data sources and conclude that future “research should focus on the impact of adding additional data sources to these systems” (Bodily and Verbert 2017, section 4.2). Schumacher and Ifenthaler (2018) establish the features that students really expect from LA: namely, features that support their planning and organization of learning processes (e.g., an integration of a personal schedule), provide self-assessments (e.g., real-time feedback), deliver adaptive recommendations (e.g., a customized prompt for task completion), and produce personalized analyses of their learning activities (e.g., comparison with fellow students). Another study indicates that features that help students perform better by recommending specific actions are favored by them (Rets et al. 2021). Sarmiento et al. (2020) advise that addressing issues of power imbalance and creating an equitable space for creative thinking are critical when engaging students as LAD designers. Broughan and Prinsloo (2020) go further to indicate that students should be seen as “equals” in this process. Our study builds on this small but growing number of empirical works. Next, we highlight the problematic understanding of learning that underpins most LAD studies and argue that a situated theory of learning might help us identify a range of newer data sources.

2.1. Why Have a Situated Theory of Learning in a Learning Analytics Study?

When reviewing the current data sets used in dashboard designs, most LAD studies tend to assume that learning is an individual process that is an outcome of classroom-based, teacher-centric activity of depositing contents on to students and that can be best understood by a close observation of the digital footprints that a student leaves in university-based platforms. However, as researchers and teachers, we recognize that a different kind of learning takes place among students, and in line with the situated theory of learning (Wenger 1999), learning is as much a part of the learner’s human nature as breathing and speaking. Learning, as a social phenomenon, happens as they engage in the collective process of learning in a shared domain of human endeavor, referred to as communities of practice. Students define their identity in their peer groups, build relationships on virtual platforms, adopt routines as members of a degree program, find meaning in student communities, learn practices in groups, and help each other solve problems creatively and more casually. Through continued participation in authentic social interactions and through regular engagement with peers, they learn, often unintentionally, wherever they are and whenever they can. They build relationships among each other through collaborative groupwork opportunities and learn from one another. Over time, as a community of practice, they develop a way of acting, behaving, or thinking that is shared by all members of the group. In sum, as social learners, they learn what is meaningful to their careers and lives (the “what” aspect of learning), often supplementing their course material by adopting a collaborative approach (the “how” aspect of learning) and in spaces that are more open, virtual, networked, and connected (the “where” aspect of learning).

This theoretical perspective challenges the institutional practice of using readily available data (such as lecture attendance) as proxies for student learning and holds promise to advance LA research and practice. To our knowledge, there is limited evidence for studies that explicitly adopt the situated theory of learning as a lens to explore dashboard design features. Wong et al. (2019), in their systematic review of learning theories’ application in 164 LAD papers published over a seven-year period, identify only one study (Carter and Hundhausen 2016) that adopts the situated, social theory of learning, which is used to explore a social programming environment. However, the authors use the theory only to examine student-to-student interactions within a closed, module-specific online environment. Although that study usefully concludes that student–student interactions can differentially affect grades, they fail to explore how contemporary students’ needs and their wider social learning influence LAD designs (not their intended research purpose). This is not to suggest that the social learning is ignored in LA literature.

Many authors (e.g., Shum and Ferguson 2012, de Laat and Prinsen 2014) promote social learning analytics: an area that focuses on understanding student connectivity and the development of social relationships and how this data can promote learning through social interaction. However, many LAD researchers using social learning analytics (e.g., Joksimović et al. 2015, Kaliisa et al. 2019), despite their social-constructivist orientation, define social presence narrowly and use what students do within institutional online environments—an “intrinsically social form of analytics” (Ferguson 2012)—as proxies for academic success (Hernández-García et al. 2015, Chigne et al. 2016, Popescu and Leon 2018). Many of these authors fail to view learning spaces as dynamic, democratic, distributed, and open entities produced by formal and informal social practices (Bayne et al. 2014). Although these studies are effective in predicting the academic success of some students in specific modules, their proxies ignore the learning that happens in the macrolevels and mesolevels, in open social contexts outside a module, through students’ active participation in peer groups, communities, and professional networks as they pursue their common goals and develop self-identity in informal environments of coaching and mentoring—“socialised analytics” in Ferguson’s (2012) words.

To summarize, previous research on student-facing LADs ignores the needs and priorities of students, often uses all-too-familiar proxies for tracking and measuring academic performance, fails to provide new data sets that might help to produce more useful LAD designs, and mostly lacks an articulation of a relationship between the learning theory that is applied and findings that are revealed. Therefore, recognizing the social learning practices of students, we explore what student-focused LADs look like from the perspective of students as social learners.

3. Institutional Context

Our university is a research-led institution in Northern Ireland with a successful record of utilizing predictive analytics for effective teaching, student retention, and decision making (Jaffrey 2019). It also places high value on student engagement in teaching and learning and invests resources in amplifying the student voice across its structures. With a view to enabling students to better monitor their own behaviors and performance, the university wanted to create student-friendly LADs. Therefore, we sought to understand what data sets students might want to see in student-focused LADs so that they are perceived as useful and relevant. At this time, students had not encountered any LADs, presenting us with an opportunity to learn about students’ perceptions. As part of a larger study that examines students’ perspectives on ethics, emotions, and design preferences in LA (Joseph-Richard and Uhomoibhi 2021, Joseph-Richard et al. 2021), we explored the following research questions from the point of view of students as social learners:

• Which of the author-identified data sets are perceived as important for student-facing LADs?

• What additional student-generated data sets are perceived as important in LADs?

4. Methodology

4.1. Research Plan, Ethics Approval, and Pilot Study

In this multimethod qualitative study, we used three data-collection strategies: 2 focus groups, 16 semistructured interviews, and 3 paired interviews to capture students’ perceptions on unique aspects of LADs in three sequential stages over a period of four months. We illustrate the entire research process of this study along with the many decisions we made during this four-month period in a flowchart (see Figure 1) and describe it in this section.

Situated learning theory was used to operationalize our data-collection processes. Our approach enabled us to examine (1) the meaning of particular contexts within which students learn, (2) how the contexts influence their learning, (3) unanticipated phenomena and influences that emerge spontaneously in open-ended conversations in ways that structured surveys cannot replicate, (4) the process by which learning takes place in particular situations, and (5) complex causal relationships (in this case, the varying and interacting causes of students’ situational learning) (Maxwell 2012). Our team consisted of the two authors who have expertise in the subject material and have experience using various qualitative methods, a learning technologist—a systems expert, and two research assistants who were trained in data-collection and analysis techniques. A health and well-being officer was available during the data-collection procedures to support students if required although her services were not needed during this study. On obtaining the institutional ethics committee’s approval and in line with British Educational Research Association (2018) ethical guidelines, we conducted a pilot study that included a focus group discussion with four undergraduates. After an introduction, we asked them what kinds of data they wanted to see in LADs. Having only limited knowledge of LA/LADs, they were not able to voluntarily suggest data sets or how they could be designed. This is not surprising as others report such experiences with students (Bennett 2018). With this experience, in line with the social constructivist stance, we wanted them to engage in a small interactive exercise based on the card-sorting technique to elicit their perceptions of LADs; this is known to be effective as a knowledge-elicitation tool (Wood and Wood 2008). Because good design requires partnership and user participation (Holtzblatt and Beyer 2012), the card-sorting method used in this study is, therefore, deemed suitable.

4.2. Background to the Card-Sorting Method

Card-sorting techniques (Rugg and McGeorge 2005) are effective ways of eliciting variation and commonality in research participants’ categorizations of features they perceive in a set of entities, such as cards, pictures, or objects. Although card-sorting techniques were relatively underutilized prior to 2018 (Viberg et al. 2018), the subject has recently attracted scholarly attention (Alvarez et al. 2020, Vezzoli et al. 2020); nevertheless, its “implementation [still] requires attention and specification” (Sarmiento et al. 2020, p. 1). Hence, we provide a rather more elaborate description of this method to serve as a tutorial to novice LA researchers. Card sorting involves asking participants to sort labeled cards into a hierarchy, groups, or categories, using a single criterion or a set of criteria. Card-sorting techniques are simple-to-use, user-centered techniques focusing on the terminology of participants (rather than that of researchers and experts) and known to elicit the semitacit knowledge that traditional interviews and questionnaires cannot access (Fincher and Tenenberg 2005). Users of these techniques assume that people make sense of the world by categorizing it, describing their own categorization of the world with reasonable validity and reliability “although people are not always able to do this” (Rugg and McGeorge 2005, p. 95) as revealed in our pilot study and, hence, the need for checking our assumptions in LAD studies. As participants categorize entities externally, researchers assume that their constructions reflect their internal, mental representation of how they experience, perceive, and imagine their world along with their kinesthetic capabilities (Rugg and McGeorge 2005). Thus, card sorts serve as an effective explorative tool in eliciting participants’ subjective, often semitacit understanding about certain aspects of phenomena in the world and their relationships to one another (Rugg and McGeorge 2005). It is recommended that participants have about 20–30 cards for managing the procedure. We decided to use “repeated single-criterion sorts,” a method recommended by Rugg and McGeorge (2005) for flexibility, ease of implementation, and utility in eliciting user beliefs and perceptions on various aspects of their worlds (Upchurch et al. 2001, Nawaz 2012). In our study, students sorted a set of cards repeatedly, each bearing a name of a data set (e.g., attendance data), using a single criterion (e.g., usefulness) each time. The next section details our procedures of applying the card-sorting technique.

4.3. Card-Sorting Procedure

Based on our review of the literature on student-facing LADs (e.g., Bennett 2018, Schumacher and Ifenthaler 2018, Sedrakyan et al. 2018), we identified a comprehensive list of 22 different data sets used in current LAD literature. In line with the published definitions, we labeled these “author-identified data sets” (such as an attendance record and library usage data) and created 22 A5-sized cards. We left several cards blank so that students could create their own labels. The names of the data sets were self-explanatory, but to help participants’ understanding of each data type, we also provided a short definition for each label. In Table 2, all data sets used in this study are grouped for ease of reference based on their familiarity to many LAD designers and practitioners along with their academic sources (Table 1).

Table 1. Author-Identified and Student-Generated Data Sets

Table 2. Characteristics of the Sample Engaged in the Three-Stage, Participatory Design

We numbered the cards on the reverse for recording the results at the end of the procedure. We prepared an instruction sheet to make it clear what we expected the participants to do and help the research team instruct participants. At the start, we asked students to look at the cards as a whole set before sorting them out, so they were aware of the kinds of data sets they could consider for sorting. We provided additional explanation for clarification. In line with our first research question, we asked them to assemble the chosen cards on a table based on perceived “usefulness” as a criterion and explained that they should use only those cards that they considered important for an LAD and abandon those that were less useful. To reduce the time involved in the statistical analysis of a sematic distribution and drawing from Jahrami et al. (2009), we suggested using a structure representing an importance continuum, which helped them rank the chosen cards (Figure 2).

In this structure, we assume the cards on the extremes at the top and bottom rows to represent the strongest views and those in the middle to contain the moderate views on the chosen criterion: utility. For example, if the “students’ grades” card is placed in the top row and the “building usage” card is in the bottom row, then their positions on an imaginary vertical axis represent students’ preference for viewing their grades as the most important data set and building usage as the least important data set on a student-facing LAD. Data sets that are placed in the middle row represent students’ neutral preference for those that are at the edges. Because a single criterion, utility, is prescribed as the basis for the sorts, we did not attribute any specific meaning to the cards that were placed on a horizontal axis from left to right. We encouraged participants to use this structure strictly when assembling their sorts, enabling us later not only to count the number of times a card is perceived as useful, but also to compare students’ preferred priority positions for the cards across all card sorts. Participants were observed and notes taken. Once an assembly was stabilized, we photographed the card sort (Figure 3).

4.4. A Three-Stage Implementation

As seen in Figure 1, we implemented this study in three stages.

4.4.1. Stage 1: Postgraduates.

We recruited postgraduate students enrolled in business and management degree programs during the academic year 2018–2019. Thirty-one postgraduate students from one cohort agreed to take part in two focus group interviews. Using a maximum variation sampling strategy, we purposefully selected (n = 20) volunteers who assured us they were comfortable sharing ideas about learning. During this stage, in the interest of accommodating postgraduate students’ in-campus availability, we conducted two focus groups, having 10 volunteers available at the same time in two different sessions during lunch breaks in the same place (following guidelines from Morgan 2019). They received an information sheet detailing the study and signed consent forms prior to the sessions. Upon clarifying students’ views on LA, we explained the types of data collected in universities and how LADs could use these data sets to support their learning. The focus group had two key questions: (1) If you could design a student-focused LAD, what data sets need to be present in them? (2) Based on your study habits and learning routines, what other additional data sets would you like to see in the LADs? Follow-up questions were used only to elicit more specific, in-depth information.

All participants in each of the focus groups worked together in one card sort, using one set of cards. We needed to set ground rules and give more detailed introductions about LA. As we had identified through the pilot study that students were unable to categorize their world easily, we defined all the data sets, particularly for international students (the majority), by using multiple examples and metaphors that cut across racial, regional, national, and socioeconomic boundaries. Pre-exercise discussions on their learning habits took around 45 minutes, bringing to light several interesting and unusual collaborative learning practices (explained later). During the exercise, the participants of the first focus group made use of the blank cards. Three different students proposed five new data sets, which everyone agreed to include after thoughtful discussion. These include “financial information” (money spent so far versus resources used by students and earning potential of a module if successfully completed), “part-time work info” (hours spent working during academic year), “DIY learning” (time spent in informal, self-initiated learning via cloud-based media tools), “networks created” (LinkedIn connections, industry links/contacts), and “mentoring hours” (hours spent in coaching/mentoring sessions at work). These newly created cards were added to the original set of cards for use by the first focus group and in all subsequent sessions.

Once the group finalized the card sort, we used a postsort discussion to learn more about the “why” behind their constructions. We pointed to two or three items at random (e.g., prior education data, historical module data, and building usage in Figure 2) and asked them why they considered them more or less useful than the other set. This procedure, known as dyadic and triadic elicitation, respectively (Rugg and McGeorge 2005), allowed us to further refine students’ perceptions on the card sorts. Then, to answer research question 2, we asked them to explain why they created the five additional data sets in their constructions. A research assistant handled logistics, took notes, observed seating arrangements, and monitored the audio-recording process. We facilitated both events, which took approximately 130 minutes each, and concluded the sessions with a summary. The entire focus group session happened during the card-sort exercise, and the postexercise interviews were audio-recorded.

4.4.2. Stage 2: Individual Interviews with Undergraduates.

In stage 2, we purposively sampled 20 of the undergraduates who responded to our invitation, using a maximum variation strategy. For practical reasons, after two failed attempts to organize focus groups, they opted to participate in individual interviews. Only 16 students attended the interviews during lunch breaks. We conducted the interviews, using the same questions and procedures as in stage 1; two research assistants were present, managing the whole process and observing postsort discussions and taking notes.

4.4.3. Stage 3: Paired Interviews with Undergraduates.

After the 16th one-to-one interview, some more undergraduates volunteered to take part in the study, not as individuals, but as pairs. Students convinced us that they normally do their reading and prepare for assessments in pairs. Because our focus, as qualitative researchers, was to fully understand the nature of social learning that occurs in students from their own perspectives and their behavior aligns well with the theoretical lens we used, our qualitative approach allowed us to shift our original design as acknowledged in Creswell and Creswell (2018). As part of the emergent design, we decided to conduct paired interviews following the procedures of Wilson et al. (2016). These interviews were conducted in the same place when the pairs were available, using the same questions we employed in stage 2. The difference was that, relatively, we spent less time discussing their social learning practices. The pair coconstructed one sort, using the same set of cards, including the five newly created cards. The research assistant recorded their conversations for later analysis, and as they did the sorting, we observed the nonverbal cues of the dyad and their interactions. When they encountered a difference of opinion in arranging the cards, they reconciled the conflicts with empathy and trust, allowing us to hear each of their perspectives. After the final sorts were photographed, the pairs remained available for a postsort interview. Data saturation occurred during this third paired interview. No one during stages 2 and 3 engaged in creating new data sets, but they rationalized that the cards at their disposal had contained all the topics that mattered to them. A professional firm transcribed all recordings verbatim.

4.5. Data Analysis

Our data comprised photographs of the card sorts; transcripts of card-sorting sessions; post–card sort conversations with individuals, pairs, and focus group participants; and handwritten notes from the lead researchers and the assistants. Our sample (see Table 2) contained male (43%) and female (57%) students: they were relatively young (range 18–36), had studied various business and management subjects (i.e., human resources management, marketing, innovation and enterpreneurship) within the business school as undergraduates (52%) and postgraduates (48%), and had direct industry work experience (range 2–10 years). In terms of their mode of study, among the postgraduate students who participated in stage 1, 80% of them studied part-time, and only 20% of them were in full-time study. However, among the undergraduates who engaged in stages 2 and 3, all of them were in full-time education. In terms of their origin, across the total sample, 52% of them were international students and the rest were home students (48%). More specifically, among the postgraduates who engaged in the stage 1 focus groups, 90% of them (n = 18) were international students (n = 2 (Europe), n = 9 (Asia), n = 3 (Middle East), n = 2 (Africa), n = 2 (Americas)). In contrast, among the undergraduates who participated in stages 2 and 3, only four were international students from different parts of the world (one Serbian female, one Nigerian male, one Japanese female, and one Pakistani male). These individuals represented different and diverse social, religious, cultural, and linguistic backgrounds, representing the rich diversity of student populations that is typical of the university. During the three stages of the data-collection process, the participants created 21 different constructions, in total, either in focus groups (n = 2), individually (n = 16), or in pairs (n = 3), and all of them had unique structures of their own.

Our analysis included merging, connecting, and embedding data (Creswell and Clark 2017) collected sequentially. Integrations occurred during the analytical process, and it culminated in the writing process. We then organized the results and the discussion sections around the substantive issues dealt with in the research (and not around the methods we used). In a preliminary stage, we as a team familiarized ourselves with the data by reading the transcripts in several team meetings. Then, in line with the guidelines provided in Rugg and McGeorge (2005), we content-analyzed the themes arranged in the participants’ constructions. Content analysis in qualitative studies involves establishing a set of categories and then counting the number of instances the data sets fall into each category (Silvermann 2020). Because all card sorts relate to a single phenomenon (i.e., personalized LAD), we considered all card sorts obtained from individuals, pairs, and groups together in this analysis. We scanned the distribution of data sets within each assembly and across all 21 card sorts. Analytical questions helped our counting process: “How many students ranked the same data set as their top priority?” “How much agreement is there between respondents about which data sets go in which rank when using the ‘utility’ criterion for dashboard designs?” We listened to the recorded audio conversation when reading the transcripts and scanning the card sorts as this process helped us consider the explanations provided for each data set. We used frequency counts to determine the number of times each of the data sets was prioritized and then codified the counts as patterns. This content analysis process permitted the comparison of the perspectives of students who do not share a common language or culture because of their lack of exposure to LADs.

We observed no difference in the patterns created by individuals, pairs, and groups. We identified the top and bottom five data sets ranked by the participants. We organized the quantitative results by displaying them along with the qualitative quotes that explain why students ranked specific data sets in certain ways in their constructions (Table 3). Then, we thematically analyzed (Clarke et al. 2015) the postsort discussion data and grouped them in light of the students’ justifications of how situated learning practices influenced their learning practices. We read the supplemental qualitative data captured in our notes and embedded them in the full sets of transcripts in order to understand how the exercise unfolded in real time. In this analytical integration, we noted how a comment was made (intensity and emotion), how often (frequency), and by how many students (extensiveness). To enhance the validity and reliability of our study, we used researcher triangulation, team vetting of themes, and reflexive journaling in the process. Specifically, both authors were mindful of the fact that our subject expertise and past experiences might shape the interpretations we make and make us actively look for evidence to support our assumptions about LADs. To be more reflexive in the analytical process, both authors wrote memos of personal reflections and how we developed codes and themes. The assistants kept records of all the raw data, field notes, memos, transcripts, team meeting decisions, and a reflexive journal of how and when we made analytical decisions as a team. We spent a prolonged period of four months with the students and developed an in-depth understanding of students’ learning practices. However, we limited our discussions about previous research experiences on this topic with the team members so that our past did not override the findings and interpretations of this study. During the peer debriefing sessions, the research assistants reviewed and asked questions about the definition of codes, cross-checked our content analyses, communicated regularly with the team the analytical interpretations, documented minutes of meetings, and thus enhanced reliability of the findings. In addition to using a rich, thick description to convey the findings, we also present discrepant information on students’ social learning practices that runs parallel to the main themes, thus further enhancing the validity of our findings. In the next section, the findings are presented, using illustrative quotes and pseudonyms.

Table 3. Students’ Choice of Datasets for LA Dashboards

5. Results

5.1. Which of the Author-Identified Data Sets Are Perceived as Important for LADs?

The card sorts allowed us to reveal the most preferred features in those dashboards. These data sets include frequency of grades, library history, online resource use data, historical students’ comparison data, and students’ financial information. Table 3 presents students’ preferred data sets and their reasons for the choices they made in the form of illustrative quotes. Their preferences were to see their grades (71%), the DIY approach to learning data (68%), online materials usage data (62%), the number of networks created (55%), and financial information (money spent so far versus resources utilized, earning potential of a course – 40%). We also found students’ perceptions on what not to include in dashboards. These include social media data (i.e., time spent on social network sites and an analysis of social media posts), building usage data, prior educational data, students’ physical and mental health data, and sports center access data.

Importantly, although all participants showed a desire to get involved in the procedure, there is less agreement among the participants on what to include or not to include in such personalized dashboards, showing a preference for having a greater autonomy and agency for self-selecting the data sets that are important to them. When discussing these features, the majority of participants showed stronger emotional connections with the data sets, particularly with those data sets generated by them, and let slip arguments that were autobiographical and value-laden to justify their decision on why something should be included or excluded in an LAD. Even more importantly, 60% of participants preferred a dynamic dashboard so that students could self-select which data sets they wish to see at a time of their choosing. Interestingly, 23% preferred an artificial intelligence–powered LAD system that could intelligently display data sets related to individual interests, needs, and priorities, similar to their Facebook page, which displays materials of interest to the account holder based on certain machine learning algorithms. It is worth noting that 42% of our participants expressed a need to include personal stories and narratives in LADs. They preferred vignettes of student stories (i.e., narratives describing past students’ career successes) linked to the display: “It would be good to see something on the dashboard to tell me ‘You are currently in place Z in your class; if you can achieve X% in this module, then the likelihood of getting an entry level job in six months is Y%. Read Dan’s story here.’ I will be motivated to work hard by reading Dan’s story” (Kumar). A student told us that if that kind of narrative could be made available in dashboards even before initial registration (enabled by machine learning capabilities that connect a fresh applicants’ data with similar alumni profiles), it would be helpful. Five students said that universities must “always allow” students to design their own LADs and that was “the right thing to do.” Overall, all 21 card sorts pointed to participants’ preference for individually focused, totally personalized dashboards, displaying only the data sets perceived as useful.

5.2. What Additional Student-Generated Data Sets Are Perceived as Important in LADs?

During the interviews, 63% of participants highlighted more frequently that they would like to see situated learning–inspired data sets. They emphasized that, for LADs to be valued highly, the designs should contain data on self-directed learning that happens socially, continuously, and in the vast landscape of practice and social life.

The instrumental use of TED talks, LinkedIn contacts, YouTube videos, and useful Tweets are highlighted by the participants: “We learn a lot from TED talks, and from our LinkedIn contacts. They offer us an alternative way of looking at the issues we discuss…Our mobile is our classroom. We learn a lot during our train journeys to the university. We discuss the stories we heard…we form opinions, develop our own ideas about a topic. Are you not including them here?” (Javier and Florian, pair 2). “YouTube is my learning space. I search, watch tutorials, and learn every day. Although I get the core knowledge from the lectures, my social media feeds teach me…I retweet useful postings to others and they share with me useful stuff. All of these helps me build on that core stuff” (Rhys).

We found that our students make use of learning opportunities that arise situationally, and at other times, they take a DIY approach to create such opportunities. They tend to take an active role in self-directed learning, and they do not expect teachers to deposit discrete packages of information and impart understanding: “We have a WhatsApp group. Our relationships are strengthened by our Snapchat connections. We learn all the time…as peers. In fact, I feel compelled to learn from them. They spark an interest in me” (Yu-Ting, a focus group participant). “My friend and I play video games together. We are better at problem-solving and multitasking. We learned about strategy, timing, and winning. Our strategizing has improved. We help each other to progress through the levels…Gaming keeps us smart and sane” (Risa).

As students engage in increasingly more complex tasks, they gain experience, learn the language of practicing managers, and begin to participate fully in communities of practitioners. Data indicate that their participation in authentic social interaction enables learning to occur; students demonstrate a productive engagement with both content and context, and they see learning as being integral to and an inseparable aspect of everyday practice, the workplace, family, and other social contexts. This is reflected in their quotes: “I was on a field trip recently, and I learned so much in that. We met business leaders, and they shared their challenges and struggles. We shared with them the theoretical frameworks and current evidence on HR-related topics. It is a collaborative learning experience” (Chen-Jui).

Students learn best in their social context, in which their learning is applied. For them, learning is not an isolated activity that leaves digital footprints only in campus computers. In fact, the lived‐in world of everyday activity (Lave and Wenger 1990) becomes their lecture hall where learning happens, and this is where identities and practice communities are produced and reproduced. Through these connections, they develop tools and techniques, online and off-line social networks, and mimic typical ways of handling their challenges in a context. Sadeen and Kumar, two postgraduate business students, were interested in the inclusion of workplace learning data: “I work as a part-time human resource assistant in a large company. In my coffee-room chats, in reading company blogs, and simply by speaking to more experienced HR practitioners…I develop my knowledge…I learn the lingo. I am also curious…I constantly look for better ways of doing things…I learn to impress my boss at work. The dashboards cannot ignore these ways of learning” (Sadeen). “I am doing a part-time industry placement, along with several other placement students. When we come together at work, we speak about how to tackle academic challenges we have in the university. We share our struggles and come up with ways of managing them. I want to see the indicators of the learning that happens in work placements” (Omer).

Taken together, our findings indicate that the participants learn, collaboratively, in socially situated ways, in field trips, work contexts, and industry placements and by learning from their mentors and practitioners in the real world. Social media tools and network events tend to facilitate their situated learning. We found a strong empirical base for the inclusion of such socially situated learning indicators in student-facing LADs.

5.3. Differences in LAD Preferences of Home vs. International Students and Undergraduate vs. Postgraduate Students

Our analysis revealed some variation in attitudes about the use of personal data among the international (52%) and home (48%) students. For example, some international students were more relaxed about the use of personal and medical records in predicting success: “My data is everywhere; I am not that fussy about them [being] used by my university” (Nurislam). However, the home students showed extreme resistance to the use of such data: “It is my data. Every time the university access it, university should ask my permission. ‘No consent’ means ‘Don’t touch my data’” (Darryl). Further, international students wanted to see displays of within-group comparison relatively strongly: “I want to see how my peers are performing and I want to see how [I am] doing when compared to their achievement” (Chen). This can be compared with the home students who emphasize individuality and personal values: “Success in learning is not necessarily about achieving a higher than average score. I don’t care to know how others are doing” (Mark). Further, data sets such as health and fitness and student leadership were considered important for undergraduates but were less so for the postgraduates. Postgraduate participants were concerned about the time commitments they had and the family and work pressures that reduce their ability to contribute fully to university life; they tend to perceive that current LADs do not respect these personal circumstances fully. Five undergraduates expressed the view that LADs tend to neglect the emotional aspects of educational experience: “I was disappointed to receive that feedback; I was sad and stressed. As a result, my approach to the next assessment was not that positive. I locked myself in my room, not speaking to anyone, not engaging with anything. I felt lonely…it took some time to regain my strength to get ready. This experience is completely neglected here” (Huan). Although postgraduates did not share affective experiences of learning with us, dashboards that incorporate students’ emotions might positively influence their behaviors as evidence on the effectiveness of emo-dashboards in changing at least teachers’ feedback reporting practices has begun to emerge (e.g., Ez-Zaouia et al. 2020). Although most of the undergraduates tend to believe the integrity of the data sets, a small minority (10%) doubted the accuracy and credibility of some of the data points by saying that their online behavior would never be fully and accurately captured by machines; they felt that what is captured could never say who learned what, in what circumstances, and why. For postgraduates, data accuracy and credibility is a matter of low importance. We summarize these differences in Table 4. To put it simply, we cannot make stronger statements about these differences in our study because of its limited sample size. Nonetheless, it indicates the potential influence of these attitudinal variations in shaping students’ expectations regarding the LADs and, by extension, making the design processes of student-facing LADs even more complex.

Table 4. Some of the Emerging Key Differences Between Undergraduates’ and Postgraduates’ Preference for LAD Personalizationa

5.4. Other Findings About Students’ Social Learning Practices

During different stages of data collection, students revealed that they had complex arrangements that result in less time being spent on the campus or on leaving limited “digital footprints” in institutional systems. They shared with us strange study behaviors that are essentially social; these included practices such as

• Print copy distribution: a student logs into a module online page to download all the materials at once and creates copies for everyone in the student’s inner circle.

• In-group sharing: students go to the library in small groups and borrow a wide range of books, exchanging them with each other as online digital books are shared among family members.

• Flatmates sharing: a student downloads module materials all at once on to the student’s tablet and then shares the tablet among all classmates who share the same house, thus leaving limited digital traces of their learning on institutional computers.

• Helpful note-takers: students receiving help from note-takers/scribes and from learning support assistants whose digital activity almost always happen in noninstitutional systems.

In addition to these, we encountered two cases of a translator and a scribe accessing online materials legitimately for two different students, and their digital footprints had not been visible in students’ records. Our research journal indicates that some participants (n = 4) at the end of the interviews, after the audio-recording was stopped, requested us to include in LADs, qualitative narrative stories of success, believing that, for example, viewing the stories of successful seniors who studied a degree program along with the details of their societal and economic background, might inspire them to recognize that someone else with that kind of background could also achieve the same success.

Taking together the findings presented in the last two sections, all these intricacies present a complex picture of students’ social learning practices and challenge the idea of conceptualizing learning as the one that always leaves observable traces in a formal virtual learning environment during an academic year.

6. Discussion

In our quest to understand students’ preferences for useful LADs, we learned at least three important lessons that affect future dashboard designs: (1) the need to include situated learning-inspired data sets in LADs, (2) the importance of including qualitative stories and the need to combine data sets for critical insights, and (3) the need to avoid an overreliance on a narrow set of data points leading to a neglect of affective dimensions of learning and the urgency to create new technological tools to capture real-life evidence of social learning.

First, when confirming the earlier findings on personalized dashboards (Leitner and Ebner 2017)—and complementing the findings of Rets et al. (2021) and Schumacher and Ifenthaler (2018)—we go further in two ways: identifying the most and least preferred features in those dashboards (see Section 5.1). Among the five most preferred data sets, four (grades, library data, online use, historical trends) have resonance with other scholarly works (Roberts et al. 2017). Our study highlights students’ preference to include their DIY approach to dynamic learning along with their part-time work data, the number of networks created in social contexts, and personal finance data in LADs. These data sets had not been identified as potential features of LADs in the existing literature, which used other stakeholders’ perspectives (e.g., teachers, learning technologists); we believe that using students as the lead designers of LADs is the reason for the uniqueness of the study’s findings. Displaying data on the money-making potential of courses or how much money is spent on fees in comparison with, for example, the extent to which they used the library sources, motivates 40% of our participants. This information when presented with the data about other students’ economic behaviors appears to be a powerful motivator of success. It is possible that, when these data sets are presented with students’ academic performance in comparison with that of their peers, as demonstrated by Aguilar et al. (2021), LADs could become more powerful catalysts of change. We also highlight students’ perceptions on what not to include in dashboards, such as building usage data or health data and sports center access. Besides the privacy concerns, students said that they did not see the usefulness of tracking this information. It is interesting to note that they overlooked the benefits of combining data sets for meaningful decisions; for example, combining sports center visits and health statistics with classroom engagement metrics and time spent on additional reading and research could show interesting behavioral patterns that may give clues to collective learning behaviors. In doing so, we demonstrate that a theory-inspired card-sorting technique has potential application for LAD research in that it can assist in uncovering key stakeholders’ perspectives on LADs and it holds potential for broader application as a knowledge-elicitation tool for consideration by researchers and designers.

Second, our study confirms the rich social learning landscapes that facilitate continuous education of our students in actual time (Wodzicki et al. 2012, Vaterlaus et al. 2016). We found that students preferred to learn from a landscape of practice, and that includes collaborative learning, mentoring sessions, industry networking, and the LinkedIn following of business leaders as well as learning from socially generated YouTube videos, Twitter feeds, Instagram live shows, WhatsApp groups, Facebook discussions, Snapchat communications, and TED talks, among others (see Section 5.2). They want LADs that include rich social learning data if useful and valuable. This study confirms that their approach to learning aligned well with the situated learning theory (Wenger 1999) and has influenced their design expectations. Although what was being learned (the content) was not always clear in the data, how the learning was carried out (i.e., the collaborative approach) and where the learning was taking place (i.e., in open, social spaces) was clear. We wondered how it might be possible to include learning data from socially generated videos, feeds, WhatsApp groups, and Facebook discussions into LADs. Our study triggers new questions about the feasibility of integrating students’ social media data on self-directed learning and also not using their social media data on recreational use in the LADs. Nonetheless, by clarifying how the students’ situated learning practices dictate LAD design choices, this study extends our understanding of what students as social learners want to see in the LADs. Our aim in highlighting these complexities is not to suggest how they can be effectively included in LADs, but rather to understand how recognizing students’ priorities could challenge our approach to designing LADs. We make it clear that students tend to develop their identities by actively participating in peer-level and/or practitioner communities (such as human resources managers or marketing executives, indicated by our sample in this study), which are equipped with shared procedures for talking and acting. As they participate in these communities, they begin to understand who they are and what potential they have. Because learning is a normal part of everyday practice—a central assumption of the situated learning theory used in this study and impossible to isolate—we invite LAD designers to reimagine the way they design and present learning data in LADs. Future research by interprofessional teams comprising of educational psychologists, system engineers, research-active teachers, LAD designers, managers, student representatives, and educational technologists with larger samples could help us understand how such social learning theory–inspired LADs could be designed. To that end, our study has taken a few steps into the hinterland of the students’ perspectives on useful LADs, and to that extent, it exposes some new data sets (e.g., data on DIY learning: self-reported data on time spent in self-directed learning via social media, network contacts, mentoring, etc.) that could be considered for student-facing LADs. For sustainable progress in this field, however, we may have to break down the walls between the practice-based and management-oriented community of LA and the academic-oriented community and have both communities begin to focus more on students and their learning and less on analytics (Guzmán-Valenzuela et al. 2021).

Finally, our study highlights the complexity of students’ lives, which may not easily be captured and measured by institutional LA systems. There is attitudinal variation among home and international students and among undergraduates and postgraduates (Section 5.3 and Table 4). Many of our participants also expressed that LA systems tend to overlook their off-campus lifestyles, digital activities by proxies, emotional experience of learning, and the ongoing education that they have from everyday experiences (Section 5.4). Based on these findings, we infer that the automated systems, using a limited set of easily measurable data points, could end up displaying information that is less useful to students and making inaccurate predictions of their academic success to teachers. Our study has a remarkable similarity to a global survey (Rosenhaus 2020) that revealed students increasingly adopt a DIY approach to learning. Therefore, as learning is increasingly becoming “personalised, socialised, [and] contextualized” (Moldoveanu and Narayandas 2019), if LADs need to be used for students’ behavioral change, we conclude that they have to display social learning indicators along with other data sets traditionally displayed. Consequently, it is critical that we invent technologies that enable development of such LADs that are informed by data points that capture social learning that happens in students’ communities and corridors. An overreliance on a narrow set of LA data points may lead to a neglect of affective and social dimensions of learning in LAD designs that will only be of limited value to students.

6.1. Limitations and Future Directions

Future research may explore how generalizable our findings are to other contexts. The preference for including part-time work hours and networks in LADs may be more prominent in our study given the strong emphasis our university places on developing employable graduates. Because LA is context-specific (McNaughton et al. 2017) and our small sample comes from one discipline, our local findings would benefit from a wider investigation in a multiuniversity context with a larger sample of students from a wider range of disciplines. Future studies could also explore whether LADs designed with these user-generated data sets would truly be used by students, supporting their success. Because our sample had no exposure to student-focused LADs, we could not examine this critical issue, and this is an area for future work. As universities across the globe pivoted to emergency remote teaching during the COVID-19 pandemic, it may be useful to see students’ expectations of LADs when studying online. Methodologically, we want to explore how we might quantify a card sort in terms of positive or negative assessment and strength of view of that assessment. We used content analysis techniques to quantify students’ views as falling within most and least preferred categories. Because these qualitative categories are based on a smaller sample size, no further analytical techniques could be applied to identify the strength of their preferences. In some cases, students found it difficult to strictly use the recommended hierarchical structure; on such occasions, more elaborate probing and longer interviews were needed to clarify their judgments, and these assemblies further complicated the analysis. Techniques such as Q-sort methodology combined with factor analysis could be used in future studies that rely on larger samples from different disciplines. Further, researchers could explore how LADs could link to qualitative narratives of off-campus learning-related practices of students—through hyperlinks, for instance—in ways that support their academic success. Newer interdisciplinary studies with more sophisticated methodological tools—inspired by cognitive neuroscience, for example—that are capable of recognizing students’ neurodiversity and their experiential variations hold promise for advancing the field (see some pioneering examples: Gillespie-Lynch et al. 2017, Hoefel and Gildner 2019). When these elements are displayed, how such visualizations may impact users of different ages, genders, and races and on their academic motivation and self-regulated learning is another area for future inquiries. In sum, our findings point to the need for generating new and more imaginative research on dynamically flexible LADs that display the situated nature of learning along with success stories of their peers and seniors.

7. Conclusions

This research explores how LADs for universities might be designed in a way that is oriented toward students’ needs and takes into account the social aspects of learning. By illustrating a complex picture of students’ learning journeys, we establish that their learning is much more social, situated within a more open, wider learning context than that which we normally describe in LAD-related studies. Students learn in groups as well as in formal education; hence, LAD researchers need to redirect their attention from narrowly observing campus-based learner traces to recognizing the generative process of students’ engagement with socially situated, everyday settings. We contend that what constitutes a useful, student-centered LAD is a design that better reflects the students’ realities by recognizing how modern students learn: using a DIY approach in situated social landscapes of connections, networks, and practices. Integration of these data sets in LADs, however difficult it might be, could help HEIs make more meaningful suggestions that help motivate students; thus, LA is positioned as a service and not as another interface made available to learners.

We acknowledge that, in some institutions, there may be an issue of independent and disconnected information systems that make valuable student data inaccessible or unusable to academics and other staff. To overcome the issue of data silos, institutions should, at a transactional level, conduct an inventory of existing systems, assess the data needs of different stakeholders, develop a data integration plan, ensure data security and privacy through implementation of access controls and other security measures, and provide training and support to those who access and use the integrated data (Rienties et al. 2015). However, larger constraints, such as policies around data privacy and security; limitations in the university’s information technology infrastructure; and a lack of resources or expertise in data mining, cleansing, and analysis; and other bureaucratic or cultural barriers, may make the process of creating LADs even more difficult. Therefore, at a more strategic level, institutions should focus on establishing a data governance framework, building a strong data infrastructure and analytics capability, promoting data literacy, and fostering a data-driven culture (Gašević et al. 2015, Sclater 2016, Rienties et al. 2017, Sclater et al. 2017, Herodotou et al. 2019).

We conclude that learning is dynamic, complex, and social; dashboards as generative artifacts are meaningful only in the contexts in which they are implemented, utilised, and acted upon. Because many LA systems opportunistically use data just because they are readily available, we believe that a more strategic approach that recognizes and manifests the social learning that happens in the landscapes of practice and civic life can lead to improved LA services. We hope that our findings might act as both a reminder for integrating the students’ voice as much as possible in all decisions related to LADs and a trigger for initiating strategic actions that improve institutional data infrastructure, security systems, talent, technology, and processes so that LADs become more reliable and relevant. We hope that our study offers some guidance in helping to shape the next generation of LADs in a way that makes them more useful to students.