Document View

The increase in the adoption of Internet-related technologies for online learning has been accompanied by a parallel, but separate, demand for greater accountability in higher education. Measures of student engagement offer valuable indicators of educational quality, yet have been limited to use in on-campus settings. The authors used key engagement dimensions that the National Survey of Student Engagement (NSSE) defined to measure student engagement in online courses from 3 universities. Online students were modestly engaged in selected NSSE dimensions and had a pattern of engagement that differed from on-campus students. [PUBLICATION ABSTRACT]

There is no change in higher education more sweeping than the transformation brought about by the advent of the Internet and Web. Maeroff (2003) maintained that developments in online learning are not "just a fad" but a "sea change" (p. 2). The amalgamation of knowledge and technology permits higher education to provide learning anytime, anyplace, and to anyone (Aggarwal & Bento, 2000; Maeroff; Pittinsky, 2003).

Concurrent with the phenomenal growth in online learning, stakeholders in education continue to demand greater accountability and evidence of effectiveness in teaching (Wilbur, 1998). Research in this area tends to focus on whether online learning is as effective as face-to-face learning in achieving learning outcomes. Studies on the effectiveness of online learning fall into three broad categories: (a) students' outcomes, focused on test scores and grades; (b) student attitudes about learning; and (c) overall student satisfaction with online learning. Findings largely support the view that the learning outcomes of students online are similar to those in face-to-face settings (Palloff & Pratt, 2001).

Bucy (2003) noted the volume of research that focuses on comparisons between traditional and online courses and concluded:

Rather than using research to help replicate what is done in the traditional classroom, researchers should focus on identifying what is done well in the online learning environment. Research should determine whether they (the students) are learning what we intended them to learn-NOT whether they are learning the same as in traditional methods. (p. 7)

Objective judgment of the student's knowledge and learning is still important for all stakeholders, including the learner, the instructor, and the educational institution (Valenti, Cucchiarelli, & Panti, 2001). However, the evaluation of online learning needs to go beyond these measures and consider the quality of the learning experience as a whole. Measures of student engagement offer such an evaluation.

Student engagement pertains to the time and physical energy that students expend on activities in their academic experience (Jacobi, Astin, Ayala, 1987; Kuh, 2003). Engagement pertains to the efforts of the student to study a subject, practice, obtain feedback, analyze, and solve problems (Kuh, 2003). The National Survey of Student Engagement (NSSE) measures dimensions of engagement on the basis of the widely cited Seven Principles of Good Practice in Undergraduate Education by Chickering and Gamson (Kuh, 2001). Though the NSSE was created for on-campus education, the Seven Principles of Good Practice in Undergraduate Education serves as its foundation, and the principles have been widely applied to online learning.

Chickering and Erhmann (1996) published guidelines on how online education can be done in ways consistent with these principles. Weiss, Knowlton, and Speck (2000) discussed the use of technology to facilitate the aims of the seven principles. Palloff and Pratt (2001) maintained that a sound, effective online course must abide by these same principles. Likewise, the findings of Thurmond, Wambach, and Connors (2002) indicated that the principles of good practice in education that Chickering and Erhmann described apply to the virtual classroom.

Much as application of the principles of good practice has been extended to the online setting, selected dimensions of the NSSE show promise for investigating student engagement in online learning. In the present study, we measured the level of student engagement in online learning in undergraduate education using specific dimensions of engagement considered to be effective educational practices: level of academic challenge, active and collaborative learning, student-faculty interaction, and enriching educational experiences (Kuh, 2001). That measurement was a first step toward the broader application of student engagement, from the traditional classroom to the online learning environment.

SAMPLE AND METHOD

In our survey research, students were required to complete an online selfadministered questionnaire relating to their experiences as students in online learning environments. The instrument was a modified version of the wellestablished NSSE survey, the widely accepted research initiative to assess student learning through measuring the level of engagement. First launched in 2000, NSSE was built on indexes of effective educational practice that came from the Seven Principles of Good Practice in Undergraduate Education (Kuh, 2001). The NSSE has enjoyed year-toyear growth in participation, from 276 colleges and universities in 2000 to more than 600 in 2007, with typical sample sizes of more than 100,000 students. NSSE represents a definite departure from the traditional approach to defining collegiate quality. Instead of test scores, student engagement focuses on the level of academic effort and quality of active and collaborative learning. Instead of indicating what resources are available to the student, student engagement indicates what the student does with the resources that are available. It is used to decipher the type and depth of interaction between the faculty and student. Facilitating, mentoring, and moderating are techniques promoting student engagement. Rather than measuring a concern about the reputation of the college, the measure of student engagement focuses on the quality of enrichment in the overall educational experience (Indiana University Center for Postsecondary Research [IUCPR], 2003).

NSSE was built on five benchmarks: level of academic challenge, active and collaborative learning, student interaction with faculty members, enriching educational experience, and a supportive campus environment. A supportive campus environment, as defined by NSSE, was deemed not applicable to the study. Modification of the NSSE instrument involved analyzing the applicability of each research question to the online learning environment. This was done on the basis of a review of literature regarding the deployment of Internet technology in the teaching-learning process and with careful attention to the guidelines by Chickering and Ehrmann (1996) in Implementing the Seven Principles of Good Practice in Undergraduate Education

The population of interest was undergraduate students from Oklahoma State University of Tulsa, Oklahoma; Capella University in Minneapolis, Minnesota; and Northeastern State University of Tahlequah, Oklahoma. All the students were enrolled in at least one totally online course. Our approach was a multicourse and multiuniversity strategy, which provided statistical benefits. Arbaugh and Hiltz (2005) affirmed the following:

Multi-course studies would provide several methodological benefits for ALN (Asynchronous Learning Networks) research. Two direct benefits would be increased external validity and statistical power. Because ALN research has been historically reliant on many studies based on individual courses, many of their findings may reflect idiosyncrasies of the instructor rather than provide accurate prescriptions for best practice in web-based course delivery. Multi-course studies increase the likelihood that these instructor-unique characteristics can be controlled for, thereby allowing for increased generalizability of findings. (p. 91)

Using multicourse, multi-institution samples helped us to increase the sample size, providing a more complete picture, and increased external validity and statistical power. It contributed to the generalization of findings (Arbaugh & Hiltz, 2005).

A total of 225 students participated. Of these, 24 responses were incomplete. Findings were drawn from the remaining 201 respondents.

RESULTS

Participants included 86 men (43%) and 115 women (57%). The typical profile of the respondent was White (84%) and less than 25 years old (85%). More than half (54%) of the participants were upperclassmen; 21% were freshmen; and 25% were sophomores. The overall engagement score was 5.65 on a 10-point scale. We used a variety of different scales (such as Likert-type and Semantic Differential) ranging from 4 to 7 points in the questionnaire. Scores were converted to a 10-point numeric scale to arrive at a global score. In all four benchmarks-level of academic challenge, student-faculty interaction, active and collaborative learning, and enriching educational experience-the engagement scores ranged from 5.13 to 5.87. The 2003 NSSE survey reported that on a 10-point scale derived from engagement scores, most colleges fall somewhere between 4 and 6. NSSE considered these indexes of modest levels of student engagement (IUCPR, 2003).

Although the intent of the survey was not to focus on comparisons with oncampus engagement, the 2006 NSSE scores were used as a benchmark to illustrate the depth of engagement online (see Table 1). Online students reported higher levels of engagement than both freshmen and senior on-campus students on each of the four benchmarks. Seniors from the top 10% of NSSE respondent institutions reported higher levels of engagement in the level of academic challenge, active and collaborative learning, and enriching educational experience. There was no difference in the level of student-faculty interaction. Compared with the freshmen from these highly engaging institutions, online freshmen respondents had greater student-faculty interaction and a more enriching educational experience. However, the level of academic challenge scored lower with online students. (IUCPR, 2006a; IUCPR, 2006b.)

A summary report of the engagement factors follows. This report provides an understanding of the type and depth of the engagement experience in the online environment as it pertains to each key benchmark area (see Tables 2 and 3). Findings conclude with a report on subgroup distinctions to highlight incidences where specific groups delineated by demographics (gender, age) and academic (grade performance, major) factors appeared to be comparatively more engaged.

Level of Academic Challenge

Level of academic challenge is whether students are putting forth enough academic effort, such as that spent studying, reading, writing, and preparing for class (Kuh, 2003). This benchmark measured engagement in academic rigor in areas of mental activities, type and quantity of homework, evaluations, and academic skill development.

Development of mental capacities to think at different levels has traditionally taken center stage in academic curricula (Barakzai & Fraser, 2005; Notar, Wilson, & Montgomery, 2005). All five levels of mental activities-namely memorization, analysis, synthesis, making judgment, and application-were prevalent in the online learning environment. Students were most engaged in analytical work. More than half of the respondents were at least often engaged in this way, whereas almost all respondents (96%) reported at least some engagement. This was followed by course work that required the application of theories and concepts to solve problems. Of respondents, 91% reported some, quite a bit, or very much emphasis. Memorization, synthesis, and making judgment enjoyed almost equal emphasis. Synthesizing and organizing ideas and information showed the lowest incidence (38%) of high levels of involvement.

The availability of technology serves to increase the opportunities to stimulate higher order levels of thinking. First, asynchronous networks allow the learner more time to think critically and reflectively, stimulating analysis, synthesis, judgment, and application. Second, multichannel communication through visuals, print, and virtual experiences promote thinking. Third, learning communities that are inevitable in the online classrooms are really communities of inquiry to advance mental thinking (Chickering & Ehrmann, 1996; Conrad & Donaldson, 2004; Lorenzo & Moore, 2002). Duderstadt, Atkins, and Houweling (2002) affirmed, "When implemented through active, inquirybased learning pedagogies, online learning can stimulate students to use higher order skills such as problem solving, collaboration, and stimulation" (p. 75).

In a typical week, the online student had to work on one to three problem sets that took more than 1 hr to complete and another one to three problem sets that required less than 1 hr of work. Each course typically required the student to read between one and three textbooks and between one and six articles. The majority of the students had to write between one and six reports. The latter requirement varied greatly. Whether the quantity of reading and writing assignments led to higher levels of learning depends on whether the materials assigned led the student to think and conceive persuasive arguments (IUCPR, 2003).

The online classroom presented challenging standards and expectations. Often, the majority of the students had to work harder than they thought they could to meet academic expectations. Tests and evaluations were even more demanding. The vast majority (more than 80%) of the students were challenged to do their best. This was an exemplary level of engagement. When high expectations are communicated, students rise to meet them (Reynolds, 1995). Since the Internet serves as the major platform for communication in the online classroom, evaluation criteria are usually published. This makes a more explicit communication of standards and expectations to spur the student's performance to greater heights (Chickering & Ehrmann, 1996).

With the exception of speaking skills, online learning made at least some contribution to the capabilities of the students to write clearly, think critically, and analyze quantitative problems. Almost half of the students reported that there was very little emphasis on speaking skills. Development of speaking skills was definitely not a strength of the online mode of education.

Students were highly engaged in the areas of critical and analytical thinking. Almost 90% reported at least some level of involvement. This was a good report because the ideal curriculum is one that encourages the learner to think critically and reflect on the content with the view to apply it to daily life (Chickering & Ehrmann, 1996). Critical thinking is a manifestation of an engaged environment (Conrad & Donaldson, 2004). Three quarters of the respondents reported the benefit of at least some development in writing skills and quantitative analysis. Online learning served to promote basic academic skill development, with the exception of speaking skills.

Student-Faculty Interaction

Student-faculty interaction relates to the nature and frequency of contact that students have with their faculty. Contact includes faculty feedback and discussion of grades and assignments, ideas, careers, and collaborative projects (Kuh, 2003). Faculty feedback was the most frequent type of interaction between the student and instructor. The level of contact was exemplary. With the exception of 4%, all students indicated they received feedback at least sometimes. As many as 66% indicated that the level of feedback was often or very often. Technology can be somewhat credited for this outcome because it provides various avenues for feedback (Chickering & Ehrmann, 1996). Prompt feedback is particularly vital for the online student who can otherwise suffer effects of isolation and detachment (Schwartz & White, 2000).

Grades and assignments were popular topics of discussion. Approximately 40% of the students interacted with the faculty on these issues often or very often and approximately 90% had these discussions at least sometimes. Online discussions on readings and class notes were at a modest level. Only a quarter of the students interacted often or very often on these matters. In all, 56% indicated that they sometimes had these discussions. How much student-faculty contact is deemed to be optimal depends on the issue that is at the center of the interaction. Both (a) communication about grades and assignments and (b) discussion of career matters require occasional interaction. However, feedback and discussion of ideas and class materials warrant higher levels of interaction (Kuh, 2003). The essence of engagement depends on students' participation in course-related materials (Wilbur, 1998). This circumstance led to the conclusion that the modest level of interaction on readings and class notes has room for growth.

Career-related issues were not frequently communicated between the faculty and student. Only 2% of students interacted with the faculty very often on career matters. Another 16% often interacted with the instructor on career issues. Approximately 40% of students had never discussed their careers with the faculty. This suggested that engagement in this area was less than optimal, because Kuh (2003) suggested that career plans merit discussion once or twice per semester.

Active and Collaborative Learning

Active and collaborative learning refers to efforts of the students to contribute to class discussions, work with other students, and engage in other class activities (Kuh, 2003). Collaboration is key to promoting learning in the online classroom (Conrad & Donaldson, 2004; Weiss et al., 2000). The online classroom has been commonly referred to as a learning community, implying the expectation that it is an environment that fosters collaborative efforts to promote learning (Barker, 2002; Benbunan-Fich, Hiltz, & Harasim, 2005; Dede, 2000). The present findings confirmed that online learners worked together on projects fairly frequently. Approximately 80% of students collaborated at least sometimes, and 40% worked together often to very often. In addition, the majority of the students (65%) acknowledged at least some benefits to growing their abilities to work effectively with others. More than a quarter of the respondents considered the contribution to be quite a bit to very much. Palloff and Pratt (2001) suggested that collaborative work empowers and engages the learner to the extent that it affects subsequent learning situations.

Group exchanges that solicit multiple perspectives and idea sharing are effective learning techniques (Benbunan-Fich et al., 2005; Conrad & Donaldson, 2004). In all, 62% of students had never made a class presentation online. Only 12% had often made online class presentations, whereas another 2% had very frequently made such presentations.

Learning with peers is the key to collaborative learning (Benbunan-Fich et al., 2005). Peer tutoring is a form of idea sharing in the collaborative learning environment in which students can freely conduct their own inquiry in self-directed groups (Kemery, 2000). Almost half of the students had never tutored their peers. Only 16% taught other students often or very often. This is in spite of the fact that the Internet and Web have removed time and place limitations to proliferate the channels for interaction and communication (Chickering & Erhmann, 1996).

Central to the success of online education are active learners who proactively take charge of their learning (Hiltz & Shea, 2005). Research confirmed that such students frequently visited the online library to seek out published resources to meet class assignments. Approximately 90% had at least visited sometimes. Almost half of the students made visits often or very often.

Enriching Educational Experience

An enriching educational experience involves the development of the person to learn to work effectively with people from different backgrounds and enables the use of technology to facilitate collaboration (Kuh, 2003). Avendano (2003) aptly described student engagement as having varying levels of quantitative and qualitative aspects of involvement. Learning online promotes applications to real-life situations and problem solving (Chickering & Ehrmann, 1996).

A conclusive benefit of online learning was the expertise that students developed in computer skills. It seemed intuitively clear that because students were highly engaged in using the computer to meet academic requirements, they developed an array of computer skills. Other than general gains in the use of computing and information technology, students acquired skills in quantitative analysis aided by computer technology, and approximately 90% of students gained at least some general expertise on use of computer and information technology. Approximately 60% of students considered the benefit to be quite a bit to very much. This was an exemplary level of engagement. Approximately 80% of respondents were at least sometimes engaged in using computer technology to facilitate data analysis, and approximately 40% were often or very often engaged in such quantitative work. The online course curriculum did not usually require the development of Web pages and creation of multimedia presentations. Approximately 40% of students were involved occasionally in this type of coursework, whereas 24% undertook such creative coursework at least frequently. In all, 35% had neither developed a Web page nor created a multimedia presentation.

Another definitive benefit of the online learning experience was the autonomy given to students so that they were able to learn effectively on their own. In all, 42% of students had benefited quite a bit from learning effectively by themselves. Another 34% believed online learning had very much helped them to learn effectively on their own. Almost all of the students (95%) acknowledged at least some assistance from this ability. Aggarwal and Bento (2000) proposed that online learning is a major channel for facilitating lifelong quality learning. Students in the online environment have to learn in a way that contributes to their daily life (Brown & Ellison, 1995).

Related to life skills, the engaged online learner is often a socially interactive student (Hiltz & Shea, 2005). Students interacted frequently on social and personal issues. Almost half the students communicated regularly on matters not related to the course. Approximately 20% never communicated socially with their peers. Online students might occasionally seek reading materials not related to their course through the online library resources. In all, 45% sometimes made these visits. In all, 32% never made any. This was a much lower percentage of engagement compared with similar visits made to meet requirements of the course.

Students acknowledged gains in knowledge and skill acquisition that facilitated their understanding of realworld and job-related problems. First, more than 70% of students had acquired at least some job- or work-related knowledge and skills. In all, 32% recognized quite a bit to very much gain. Second, almost 80% of students verified that online learning had rendered at least some help to their ability to solve complex real-world problems. Approximately 30% considered this benefit to be quite a bit to very much. Third, students were fairly engaged in online class discussions through e-mails, Listservs, and chat groups. Sharing of ideas and responding to the perspectives of others promoted thinking and understanding (T. Hatfield & S. Hatfield, 1995). Three quarters of the students participated in online class discussions at least sometimes. Approximately 40% were often or very often engaged in these discussions.

When technology is employed to foster a rich learning environment, meaningful experiences are created that contribute to a person's growth and development (El-Khawas et al., 2003). In the enriched environment, the learning becomes a part of the learner (Brown & Ellison, 1995).

Subgroup Distinctions

Students who accomplished an average of an A grade and students who were satisfied with their university experience reported higher levels of engagement in academic efforts and greater gains in educational outcomes. They reported considerably greater gains in academic skill development such as writing, speaking, and critical thinking. They were also more engaged in higher order levels of thinking in analysis, synthesis, making judgments, and application. These students also interacted more with the faculty and were consistently more engaged in online class discussions. They saw greater gains in their ability to work effectively with others and their capabilities to solve job-related and real-world problems. Students who were more accomplished (signified by their average grades of A or B) put in more effort to meet standards than did their counterparts. It seems an intuitive conclusion that students who were more accomplished and satisfied with their academic experience were more engaged.

Of all the majors, management and technology majors frequently reported higher levels of engagement and greater gains in various skills development. They acknowledged significantly more benefits in areas of writing, speaking, critical thinking, and quantitative skill development. Student-faculty discussions were more frequent for them. They were also more involved in class work that required analysis, synthesis, and making judgments. In particular, technology students were highly interactive online. They were more engaged in using technology for most types of class-related assignments. However, they viewed the contribution to their technology expertise from their online learning as somewhat less than other students.

Differences in engagement scores by gender and age group illustrated distinct patterns of engagement. However, these differences may not necessarily have been the outcome of gender and age diversities but the result of other variables such as the choice of majors and types of courses.

Men were disproportionately more engaged in memorization work, whereas women were somewhat more engaged in analysis, making judgment, and application work. Hiltz and Shea (2005) suggested that women are more comfortable in the online learning environment than are men and that this may be because of the female students' higher verbal skills and greater inclination toward collaborative learning approaches. Female students also reported more gains in writing skills, and male students indicated greater benefits from quantitative skill development. Male students interacted more with the faculty on readings, grades, and assignments, and they undertook Web page development and multimedia presentations more frequently. Female students reported higher levels of collaboration with peers on class projects. The students aged 25 years and older were geared more toward higher levels of thinking than the respondents who were younger than 25 years old; the former worked harder to meet expectations, were more engaged in participation in collaborative projects, and made more contributions to online class discussions. The older students tended to center their online discussions solely on course-related matters, whereas their younger counterparts (less than 25 years old) were more engaged in online social communication. Of all the age groups, students aged 25-34 years reported the most gains in many educational outcomes. They noted the most gains in academic skill development such as writing, critical thinking, and quantitative analysis. The older students interacted more frequently with the faculty than did any of the other age groups. There were substantially more benefits to their ability to work effectively with others. They were more involved in technology-aided coursework. Both these participants and senior students aged 45-54 years reported extensive benefits in learning to solve real-world problems. The senior group was the only one that was actively involved in peer tutoring. Students younger than 25 years old made more efforts to visit the electronic library outside of class requirements.

Impact of Research on Higher Education

These engagement results provided an understanding of the type and level of student engagement in online learning. Engagement happens along a continuum and is a matter of degrees of involvement (Avendano, 2003). Though the student has the responsibility to engage in academic activities, it is the faculty member's role to create purposeful course designs that promote interaction, participation, and communication in the online learning environment (Johnson, 2003; Weiss et al., 2000). Educators and instructors have to consciously and consistently sustain and grow the types and level of engagement in online education.

Institutions have an opportunity to increase learning through designing technological innovations to support the development of speaking skills. The Internet and related technologies make possible creative and cutting-edge pedagogy that permit innovation for the teaching-learning process (Burgstahler, 2000; Lowy & Ticoll, 1998). The prevalent concept of a learning community allows content to be delivered in many ways and eases the task of interaction (Palloff & Pratt, 2001).

Curriculum design should focus on increasing student-faculty interaction on readings and class-related materials. The prevalent level of engagement in these discussions was unexpectedly lower than other areas of interaction. The extent and quality of engagement is related to the efforts of the students to participate in and contribute to classrelated materials (Wilbur, 1998).

The online learning environment is an ideal setting to promote greater involvement in mental capacities. Slightly more emphasis can be given to incorporating assignments that require synthesis of and making judgments on course-related materials. Memorization of facts and information has to be minimized to the extent that there is a clear emphasis on higher order levels of thinking.

Students' participation in online class presentations should be promoted. This can include peer evaluation. Instructors frequently make online presentations to class members. The Internet and Web have been used effectively to enhance such presentations (Cole, 2000).

The online curriculum should actively engage students through challenging academic rigor, consistent and timely student-faculty interaction, a collaborative learning environment, and activities that enrich the development of the student. The Internet and related technologies have increased the opportunity for learning through the elimination of time and place constraints and the availability of flexible and innovative channels for interaction (Burgstahler, 2000; Chickering & Ehrmann, 1996).

The present findings serve to encourage the application of the concept of engagement to improve the quality of online instruction. The selected dimensions of the NSSE were found to be appropriate to measure engagement in an online setting, yielding useful results. Used as a complement to in-house evaluations, these findings can aid strategic planning and accreditation selfstudies for online education providers. Subsequent research built on this first study will provide comparative scores from which norms of engagement can be established and specific targets for improvements may be set.

FURTHER STUDIES

With the anticipated growth of online learning and its quick adoption in higher education circles, further studies are recommended. Subsequent surveys can track engagement trends across successive cohorts of students in higher education. Palomba and Banta (1999) suggested that because characteristics of the student body remain relatively constant over time, this approach reveals how programs and program changes are working. Replication of this study with a substantially larger sample will offer increased statistical power and greater insights.

Kuh (2001) suggested that engagement results in combination with other sources of information can direct higher education institutions to specific areas in educational practice for improvements related to learning. Engagement results combined with academic records provide campus-specific profiles of the undergraduate experience

Although the present study provides evidence of engagement in online learning, the next questions relate to what promotes engagement in the online environment and what relations exist between engagement data and other valid measures of student learning in Web-based learning. The present study also provides a framework from which the level of engagement in the graduate academic online experience can be studied.