Document View

The online participatory exam transforms the traditional exam into a constructivist, cooperative and engaging learning experience. Students learn from designing and answering exam questions, from evaluating their peers' performance, and from reading questions, answers and evaluations. This paper, aimed at faculty who teach online and at researchers interested in online learning, describes the procedures, advantages, and disadvantages of this new approach to the examination process. Five semesters of participatory exam research are analyzed. A majority of students preferred the participatory exam and believed that it increased their learning. [PUBLICATION ABSTRACT]

1. INTRODUCTION AND

Examinations are regarded as a standard way to measure mastery or achievement of education, and well examinations are a vital part of education (Lowel, 1926). It has perplexed educators to find and creative ways to test students' knowledge that will to learning rather than seeming like an interruption, especially for courses that use pedagogy on constructivist and collaborative or team-based Traditional education is based upon the objectivist model learning, which regards teaching as a process to knowledge from the expert to the learner (Leidner Jarvenpaa, 1995). Jonassen (1993) interprets this process as a mirror of reality rather than an interpreter of reality, so in a traditional examination, most students cram assiduously memorize the course materials in preparation examination questions designed by their instructors. and Jarvenpaa (1995) state that the traditional learning model often causes imperfect and knowledge transfer, so it is questionable whether the really comprehend, retain and master the subject with traditional examination preparation and processes.

Currently, asynchronous learning networks (ALN) are the most prevalent use of information technology in the higher education field. This is a term used to describe instructor-led online courses that include extensive studentstudent as well as instructor-student communication, and are taught mainly on an "anytime, anywhere" basis, though most also include other media (e.g., synchronous chats, recorded lectures, limited face to face meetings, or computer-assisted modules such as tutorials or simulations). The pedagogical emphasis in ALN courses is typically collaborative or teambased learning. Considering only the higher education sector in the U.S., overall online enrollment in for-credit university courses exceeded 2 million students in 2005 (Allen and Seaman, 2005). Compared to a traditional classroom, accurate and appropriate assessment learning quality in the "virtual classroom" (Hiltz, 1994) is more complicated even with cutting-edge information technology support. For example, IT technologies can effectively support multiple choice type online examinations derived from surface knowledge of a specific subject. However, such memorization is not the objective of ALN courses, and indepm research is needed to further understand the roles of instructors and students and the appropriate pedagogical assessment strategies for virtual learning spaces (Alavi and Leidner, 2001; Hiltz and Goldman, 2005; Benton 2007).

Our participatory examination research in ALN radically changes the roles of instructors and students as compared to traditional examinations, by shifting students from passive exam takers to active exam designers and judges. Our goal in designing the participatory examination is to motivate students' active engagement and deep learning in a virtual learning environment. The participatory examination transforms the traditional teacher-dominated exam process into a constructivist, cooperative and engaging learning experience. Students are authorized to participate in the entire exam life cycle including creating, grading as well as answering the exam questions, which all takes place in a virtual learning environment. Although each student individually answers his or her questions, all students in ALN can see the questions, answers and grading commentaries of others, thus fostering a collaborative learning environment.

There is a particular need to re-design the examination process in online courses, which emphasize self-paced learning, "deep" learning, and collaborative learning. Requiring students who are taking an asynchronous course online to travel to an examination site to take a timed examination in isolation from the rest of the class, for example, is contradictory to the pedagogical principles of effective online teaching (Alavi and Dufner, 2005). So is giving them an online quiz consisting of short answer questions, not for feedback and self-assessment but to serve as a summative measure of knowledge gained. The participatory exam is suitable for courses that are either totally online, or for blended courses that combine inclassroom meetings with the use of an asynchronous communication system for extended interaction among class members. Although the study presented here is based on a for-credit graduate level course, it should also be applicable to organizational training courses, with online courses for this purpose an important trend because of the globalization and advancement of computer communication technologies.

This paper should be of interest to faculty who teach online as well as to researchers who study the impact of the Internet and related technologies on higher education. It analyzes five semesters of participatory exam research conducted at a U.S. East Coast public research university. It aims to provide sufficient detail so that others may replicate our procedures in giving the exam and in measuring student reactions. After presenting our research questions in section 2, we review related learning, peer assessment, computermediated communication and exam research in section 3. We then detail our participatory process framework and exam procedures in section 4. Section 5 describes the research design and data collection methodology, while section 6 relates our data analysis and research results. In section 7, we discuss some issues raised and study limitations. We close with a vision of participatory exams as an integral part of learning across the curriculum in the years to come and future research directions.

2. RESEARCH QUESTIONS

Because the participatory examination is student-dominated and students have more power than in traditional exams, it is expected to be an enjoyable and active learning experience for them. The participatory examination in ALN is designed to be a unique learning process, and thus we are concerned with whether or not this exam process will improve students' perceived learning.

We will examine a set of indicators for each of the three following research questions (RQ) about participatory exams in ALN.

RQ1: Do students enjoy their learning experience in the participatory exam?

RQ2: Do students perceive more learning in the participatory exam compared to the traditional exam that students experienced in most of their other classes?

RQ3: Do students learn from all phases or only from some specific participatory exam processes (designing, reading, and answering and grading exam questions)?

To determine the validity of the above research questions, we designed a set of five-point Likert scale type questions, including "I enjoyed the flexibility in organizing my resources," "I was motivated to do my best work," and "I enjoyed the examination process" as the RQ1 indicators (see Table 4). To address RQ2 and RQ3 in terms of student "perceived learning" and learning phases in the participatory examinations, we explicitly examine whether students report learning from making up questions, reading other people's work, grading other students' answers, and whether they develop critical thinking and fact integration skills etc. (see Table 5).

3. LITERATURE REVIEW

Our participatory examinations utilizing computer-mediated communication (CMC) technologies are grounded in constructivist learning theory. In this section, we review related learning theories, and research on examinations and CMC.

3.1 Pedagogical Theories

Objectivist (Piaget, 1928) and constructivist (Vygotsky, 1978) are two major pedagogical approaches. The objectivist approach promotes teacher-centered learning. Students' performance is primarily assessed by their teachers. From a constructivist point of view, learning is student-centered. In this approach, actual learning takes place when students actively construct their knowledge through social interactions with their peers. The teacher's online role is that of a facilitator who guides students to absorb and integrate knowledge. Students have more freedom to actively engage in authentic learning activities, which are achieved both from individual engagement and social interactions with their peers. The role of students is not to passively accept the knowledge directly transferred from traditional lectures but to engage in-depth cognitive activities to build and create knowledge.

Participatory examinations aim to foster deep learning in ALN. According to Bloom's Taxonomy of Educational Objectives (Bloom et al., 1956), the cognitive levels of learning can be categorized as knowledge, comprehension, application, analysis, synthesis, and evaluation. The participatory examination process engages students in all levels of cognitive skills. For example, students need to grasp surface knowledge and have basic understanding of the subject to create exam questions for their peers (this involves knowledge, comprehension and application levels). Reversely, answering peers' questions requires in-depth analysis and synthesis skills. Furthermore, students need to grade the exam questions designed by themselves, which involves assessment skills. This results in deep learning in that the participatory exam learning process involves seeing logical relationships and patterns among pieces of information instead of simply memorizing surface knowledge (Entwistle, 2000; Hargreaves, 1997).

3.2 Authentic Assessment and Peer Evaluation

In traditional education, assessment is conducted entirely by the instructor. With participatory examination, the assessment is closely tied with student learning processes since students experience both assessing their peers' work and being assessed by their peers. Wright (2003) indicates that acceptance of assessment innovations is increasingly useful in today's education, in that changes in the assessment process can transfer classroom cultural practices to reach authentic assessment (Shepard, 2000).

Assessment skills are also important for students to work as a professional. The participatory examination provides an opportunity for students to practice authentic assessment. When students participate in assessment activities, they get an opportunity to build a metacognitive awareness of what constitutes excellent work (Frederikson and Collins, 1989). The practices of the assessment activities also facilitate students' intrinsic motivation to improve their work based upon peers' feedback (McConnell, 1999). A structured peer assessment approach helps students to understand the mechanism and implementations of working in teams (Goode and Teh, 2005).

3.3 Computer-Mediated Communication and Collaborative Learning

With the emergence of low-cost computer networks and the Internet, computer-mediated communication (CMC) technologies have been widely utilized in higher education and many other fields. CMC technologies support many communication elements for collaborative tasks. Asynchronous learning networks take advantage of CMC to achieve the promise of learning "anytime and anywhere" through asynchronous online discussions (Wu and Hiltz, 2004). CMC supports the collaborative (or group) learning through online social interactions, which is one of the most important implementations of the constructivist approach. The participatory examination process is conducted within an asynchronous CMC system and can utilize any asynchronous CMC platform that supports online classes.

3.4 Exam Research

Examinations are a standard assessment tool for both schools and students to achieve their goals. However, exam research has previously been carried out with only a few computer computing sciences related courses. For example, Woit and Mason (1998) found that students are better motivated to learn practical programming skills by taking a combination of both traditional and online exams in first year programming courses. Medley (1998) argues that online finals in computer programming courses can better represent students' learning and can provide clear and immediate feedback for students, while students reported considerable stress in online examinations (Woit and Mason, 2000). Simkin (2005) reported that student scores in collaborative exam teams in an entry-level computer programming class were significantly higher compared to individual scores. Recently, some participatory exam research has been carried out (Shen et al., 2005), which focuses on learning style, collaboration and community aspects of participatory examinations.

4. PARTICIPATORY EXAM PROCEDURE

This section describes in detail the innovative participatory exam procedures used in this study. This will enable researchers to understand this research, and practitioners who teach online to replicate the procedure if they so desire.

Five participatory exam field studies have been conducted in the graduate course CIS677, Information Systems Principles, a core course for Information Systems Masters and Ph.D. students at a US public research university, featuring intensive writing and reading assignments. Students from both blended (face-to-face plus online) and distance learning sections participated in weekly asynchronous (anytime, anywhere) online discussions throughout the course, as well as during the exam.

The major participatory exam communication platforms on which we conducted our field studies are Virtual Classroom(TM) software and WebBoard(TM) (www.webboard.com, see Figure 1). Any other ALN tools which support asynchronous and collaborative communications can also be utilized to conduct the participatory exams, for example, WebCT and Blackboard. During the participatory exam period, the instructor plays the role of exam procedure controller and students perform major learner roles including composing, answering and grading exam questions. The interaction among students and instructor occurs continuously. Figure 2 shows the two major processes in the participatory exam: the instructor's control process and the students' learning process.

4.1 Instructor Control Procedure

Set up participatory exam environment. First, the instructor creates exam discussion areas on the course WebBoard(TM) (Figure 1). Detailed examination instructions are posted.

Answer questions about the exam process. Throughout the whole exam period, students have the right to ask the instructor questions about the participatory exam procedure and issues surrounding it (e.g., is it fair to ask me to grade my peers?). The instructor is responsible for explaining and justifying all aspects of the exam.

Assign exam question IDs. When all students understand their roles in the participatory exam, the instructor will assign students exam question IDs. Postings are identified only by assigned IDs to ensure anonymity as shown in figure 1. Confidentiality is an important factor for people who are new to the peer feedback process (McGourty et al., 1998).

Edit the exam questions. Each student is required to design two exam questions that synthesize the course's multiple topics. In order to assure question quality, the instructor will review and if necessary edit the questions. The purpose is to control the quality of exam questions designed by the students and also to ensure a similar level of question difficulty for each question.

Assign who answers which questions. The instructor assigns two exam questions to each student. This process is random in order to minimize the possibilities that students who know each other might exchange their exam question IDs and answer each others' questions.

Assign level 1 and level 2 graders. There are two levels of student graders in the participatory exam. Usually the authors of the exam questions will be assigned as the first level graders; the doctoral students in this class are responsible for the second level grading. However, in summer 2002, because no doctoral students registered for CIS677, all masters students had a chance to work as secondlevel graders. Tevel 1 and level 2 grading is another way to minimize student cheating. If one student happens to get the exam questions designed by his/her friend, it is possible that the exam author might share exam answers plus the exam author (level one grader) will have a good chance to assign high scores for his/her friend. If this is the case, level 2 grader's grading mostly minimizes the possibilities of assigning high scores to poor answers.

Assign final grades. After the students finish both level 1 and level 2 grading, the instructor looks at the grades and comments and assigns final grades. As part of this process, the instructor also comments on and assigns a grade to the level 1 critiques and grading.

Handle student disputes. Students who disagree with their assigned final grades have an opportunity to dispute them. The instructor will review the student's justification and make a final decision. Disputes are an especially important feature. They help ensure the fairness of peerassessment, especially when instructors do not have time to carefully review each answer and evaluation. If a student believes the peer (or instructor) evaluations were incorrect, a dispute procedure ensures that the instructor focuses adequate attention to this specific solution.

4.2 Student Learning Procedure

Confirmation. Before the participatory exam starts, students confirm that they understand the whole exam procedure. (They can communicate with their instructor during the entire exam period.) They also confirm that they have received their exam question IDs.

Read other questions, answers, grade justifications and disputes. Throughout the whole exam process, students can read their peers' work. Because no questions are exactly the same, peers' answers help students broaden their understanding of course topics and motivate them to read more. Similarly, students learn through reading other questions, grade justifications and disputes - both specifically as they craft their own, and out of general interest and curiosity. The number-of-times-read count in figure 1 attests to how much of this peer reading occurred.

Make up exam questions. Students had four days to design two questions for their peers based on the course materials. For CIS677, questions require essay-length answers (up to 1500 words) that synthesize several topics from the course. Creating questions requires students to determine how to best assess the course material. Students post their questions anonymously by assigned question IDs.

Answer questions. Students had five days to post answers to their assigned questions. They could use any reference materials. Students had to submit their answers to the Turnitin service (www.turnitin.com) to reduce the possibility of plagiarism.

Level one grader grades answers. How should the students judge a peer's work? That is a new challenge for students. We provide very detailed grading instructions, which include multiple sub-scores on several criteria. Students are not only responsible for grading the two questions they designed themselves, but are required to provide two sentences of justification for each sub-score. Students have three days for level 1 grading.

Level two grader grades answers. In order to maximize grading fairness, doctoral students (Fall 1999, Spring 2000, Fall 2000 and Spring 2002 semesters) or the Master's students (Summer 2002) used the same grading criteria to provide a second opinion grade and justification.

Dispute. If the students disagree with the final grade that the instructor assigns, they can dispute by re-grading their own answer using the same explicit grading guidelines. This provides another opportunity for learning.

We designed the participatory exam to engage students in active and thoughtful participation throughout the exam lifecycle. Indeed, in this procedure they are involved in all aspects except administrating the online environment, developing grading criteria (which we discuss further in section 6), assigning final grades and resolving disputes (which they initiate). As fits with constructivism, the instructor plays the important role of mentoring the process, which includes ensuring the appropriateness of questions, answering queries and providing guidance. Participation exam scoring reinforces this; a portion of the exam grade can be assigned for the quality of the questions designed in addition to the quality of the critique and grading of the other students' answers. The grading procedure also allows the instructor to focus on particularly tricky answers where level 1 and 2 graders do not agree, and upon disputes where the student being graded does not agree. If well structured (and as we discuss in section 7, with better software to support the process flow), the participatory exam could free the instructor from some of the detailed work in developing, administering and grading exams, and instead allow a judicious refocusing of the instructor's time, which can make him or her a more effective mentor.

5. RESEARCH DESIGN AND DATA COLLECTION

From the Fall 1999 to Summer 2002 semesters, we conducted longitudinal field studies in CIS677 distance learning and blended (face-to-face supplemented by online) sections at a U.S. public research university. 240 students participated in the participatory exam studies. Our major data collection method is surveys; after taking the exams, the participants filled out post-exam questionnaires, which are the basis for our data analysis.

The first participatory exam, operated in the Virtual Classroom(TM) system, started in the Fall 1999 semester. Based on a successful experience, we continued conducting the participatory exam field studies on WebBoard(TM) for the following four semesters, with slight improvements of the procedure control. Based on student feedback each semester, we tried to improve the participatory exam control procedure, to provide a better constructivist learning environment (see table 1). Some interim results showed an overall positive student response to the participatory exams (Wu et al., 2004).

Among these participants, 61.1% are male and 38.9% are female. 40.8% have English as their native language, and the rest (59.2%) of the participants speak English as a second language. The majority (61.7%) had no prior online learning experience with the WebBoard(TM)/Virtual Classroom(TM) systems, 14.6% had taken one prior online course, 17.9% had two to four online courses, and only 5.8% had taken five or more online courses previously.

6. DATA ANALYSIS AND RESULTS

The major data analysis methods used in this research are descriptive and correlation analyses. In this section, we report the detailed data analysis and results from our fivesemester field studies.

Since Information Systems Principles is a required core course for Information Systems Masters and Ph. D. students, we were curious to know how students rate the course difficulty. The questionnaire data show there is a big gap between student expectations and the final feelings about course difficulty. When the students were asked to rate their expectations of the course difficulty, 39% thought it would be difficult or very difficult. However, when the students answered the question "How easy/difficult do you FIND this course is," the real course difficulty level was significantly higher, compared with student expectations. 79% found the course actually to be difficult or very difficult (see table T). Therefore, we postulate that perceived course difficulty is relevant to how the students rated the participatory exam.

Regarding the exam process quality (see Table 3), on the one hand, 73.4% of students agreed or strongly agreed that the grading criteria given by the professor were explicit enough. However, only 47.5% reported that the grading process was fair and 47.1% of students thought adding Ph. D. students as the second graders improved the grading fairness. This may stem from the students' sensitivity to being graded by peers and competition among students. The exam process quality index had a relatively low Cronbach Alpha value (0.61), possibly because we used slightly different questions in later semesters in our study. However, this Alpha value is still validated at the minimum level. 59.5% agreed that they did enjoy the exam process. This seemingly low enjoyment percentage probably was influenced by the WebBoard(TM) system crash in Spring 2000, which decreased the perceived exam enjoyment in that semester. Although the system crash caused student frustration in one semester, perceived exam enjoyment is still strongly demonstrated by perceptions of study motivation (66.4% agreeing) and flexibility (75.1%) in organizing their resources. The Cronbach Alpha value for the enjoyment index is 0.68 (see Table 4). These results give our research question #1 a positive answer: students did enjoy their learning experience in the participatory examination (a result most instructors would envy).

We are concerned whether the participatory examinations are a good learning experience for students (see Table 5). As students are in a unique constructivist learning procedure, in what ways do they learn from the participatory exams? What types of learning abilities do students attain from the participatory exams? The data analysis results show that 63.8% of students thought the exam successfully demonstrated what they learned from the course.

Almost 60% of students felt they mastered the course materials. Moreover, we found that students have learned from almost all exam phases, for example, 60.4% of students reported that they learned from making up exam questions, 60.8% of students gained knowledge from reading other people's answers, and 65.8% of students learned from grading other people's answers. In addition, students reported that their learning abilities and skills were strengthened in many ways. Specifically, 68.6% of students realized that their critical thinking skills were enhanced; 71% felt that their ability to integrate and develop generalizations was improved; and 69.5% learned how to value others' work. Meanwhile, 65.2% felt that they were stimulated to do additional reading. As a question set, we obtained a highly validated index for "perceived learning from the participatory exams", with a Cronbach's Alpha value of 0.88.

The above findings answered our research questions #2 and #3: students did perceive more learning in the participatory exams and they did learn from each of the exam learning phases.

Our series of participatory exams were conducted for five semesters with slight improvements (see Table 1). Subsequent to our first participatory exam field study on the Virtual Classroom system in Fall 1999, anonymous features were added, which helped decrease social pressure among student peers, thus providing a more democratic atmosphere Our series of participatory exams were conducted for five semesters with slight improvements (see Table 1). Subsequent to our first participatory exam field study on the Virtual Classroom system in Fall 1999, anonymous features were added, which helped decrease social pressure among student peers, thus providing a more democratic atmosphere for students to present their actual opinions. In addition, Ph. D. students were added to act as the second-level graders to further improve the grading fairness. A new system, Turnitin (www.turnitin.com), was used to detect possible exam plagiarism in Spring and Summer 2002. However, technical difficulties were experienced while our field studies were being conducting on WebBoard(TM) in Spring 2000, because too many students accessed it simultaneously and it crashed. Subsequently, the exam answering period was extended from one day to several days.

Some historical data (see Appendix) shows students perceived more satisfaction after we improved the participatory exam procedure each semester. For instance, 74% of students in Spring and Summer 2002 reported that the exam was successful in enabling them to demonstrate what they learned in class, compared to 59.1% in Fall 1999 and 65.3% in Spring 2000. Even during the system crash period, 51.4% of the students still reported enjoying the exam process. However, technical difficulties caused frustrations, which impacted the learning experience (Hiltz, 1994). 60% of students in Spring 2000 would have rather taken a traditional exam instead of our participatory exam. By comparison, many fewer students (25%) in Fall 2000 and only 14% in Spring and Summer 2002 would have preferred to take a traditional exam. In addition, comparing 44.8% of students in Fall 1999 who recommended the participatory exam with a 70% recommendation level in Spring and Summer 2002, the figure again demonstrates that improvement of the exam process did enhance satisfaction. Table 6 combines these results.

What is the relationship among these research variables? Table 7 shows all correlations among perceived learning, enjoyment, course difficulty, exam grades, exam grading process quality and recommendation for use. The correlation analysis results show that the exam grading procedure quality is significantly correlated with perceived learning (0.430**, p<=.01) and enjoyment (0.446**, p<=.01). Enjoyment is very highly correlated with perceived learning from the participatory exams (0.801**, p= <.01), and perceptions of learning are also correlated with recommendation for future use (0.634**, p<0.01). Recommendation for future use is correlated with final exam grades (.191**, p<.01), enjoyment (.689**, p<.01) and perception of fairness of the grading procedure (.424**, p<0.01). Interestingly, recommendation for future use is negatively correlated with course difficulty (-.148*), so probably the participatory exams provide desirable flexibility for students, which decreases the perceived course difficulty. This is perhaps because reducing the effort of memorizing course materials decreased the course workload, so that students recommended the participatory exam for future use.

7. DISCUSSION AND STUDY LIMITATIONS

Many interesting issues arise from giving students the responsibility to assess their own knowledge of the field and that of their peers. The main concern normally raised is fairness - assessment traditionally is the job of the professor and would students not be qualified to judge others? We counter with the argument that in their profession they will be called upon constantly to assess designs, products and people. We believe that graduate students (and in most cases undergraduates as well) are perfectly capable of judging one another. Plus the professor has an overview and final say over the process. (Unfortunately, for fear of corrupting study results, we could not tell students that theoretically they should learn more from the constructivist aspects of this process.) We still need to work harder to reassure students of the fairness of these aspects, and to give them the confidence that they have the right and privilege to be assessors.

Since our five-semester longitudinal field studies were conducted in a single information systems graduate course at a single university, the generalizability of the study results is an issue. However, the current positive results encourage us to explore the participatory examination assessment with more diverse courses and to collaborate with more schools. In addition, we did not set up an official control group to compare the learning effectiveness of participatory exam with other forms of exam, i.e., traditional exams and collaborative exams, and thus our current study results provide only a subjective comparison. A subsequent thesis building on this study uses an experimental design (Shen, 2005).

8. CONCLUSION, IMPLICATIONS AND FUTURE RESEARCH

Overall, our study results demonstrate that students enjoyed their online participatory examination assessment experiences, and they have learned from all phases of the exam process including designing, reading, answering and evaluating exam questions. Compared to the traditional exams, students reported that they preferred the participatory exams. Therefore, the participatory exam was proved to be an innovative and promising online assessment method, which can benefit online education and training.

Fairness issues arise in grading consistency among different student-graders. We conducted the participatory exam over five semesters, and each semester, the grading guidelines for the essay questions became more explicit (and longer). Still, it seems that some students grade easier or harder than others (just as different professors do). Grading is a skill, and hopefully students will become more confident, better and more consistent at it over time (Kerr and Park, 1995). In future research we shall also concentrate on improving this skill. Training students in the process of creating questions and grading solutions would ease their concerns, increase their overall learning, and make the entire process more enjoyable.

Involving students in designing the assessment criteria (Hersam et al., 2004) could also increase their comfort with grading and increase their buy-in with the process. Instructors could conduct this as a collaborative exercise in preparation for the exam.

In this paper, we primarily report the participatory examination studies conducted between 1999 and 2002. Based upon our positive preliminary results, a large further exam study focusing on collaborative aspects of participatory examinations was completed in 2005 (Shen, 2005). In future studies, we hope to broaden the types of exam questions (e.g., short essay, programming), the levels of students (high school through graduate) and the types of courses (e.g., engineering, humanities). We also plan to use this approach to engage students in the full lifecycle of other types of problems (e.g., quizzes, homework, labs, semester projects), which would give students additional experience across the semester. We shall look for collaborators to join us in this effort. We intend to increase the collaboration within the exam process by experimenting with teams of students designing questions, answering them, grading answers, and arbitrating disputes.

We also plan to structure our future studies to assess actual retained learning instead of only perceived learning. This may lead to longitudinal studies across courses and semesters to determine whether actual learning can be increased across a curriculum.

The workload is an important aspect for anyone considering adopting the participatory exam. With the correct structure (especially more collaborative) and online administrative tools, we believe that the process will be equal to or even less work for the instructor to manage than a traditional examination. In some semesters the workload did seem equal and so far we have been manually managing the process. Accordingly, future research will include designing and evaluating a general participatory exercise management tool to support many different kinds of exams and projects, which should reduce administrative overhead and streamline the process flow for both instructors and students. Preliminary examples include providing better scaffolds for assessment (e.g., templates and online training for the various grading criteria), linking students and instructors directly to pending items in the exam for their attention, and emailing students who are late in posting information.

While we have just conducted preliminary studies, we are greatly encouraged and excited by the results of the participatory exam. We envision this becoming a major part of pedagogy in the future, and look forward to a time when all students are confident and skilled in self- and peerassessment as part of their learning process.

9. ACKNOWLEDGEMENTS

We gratefully appreciate partial funding support for this research by the UPS Foundation, the New Jersey Center for Pervasive Information Technology, the New Jersey Commission on Science and Technology, and the National Science Foundation under grants DUE-0434581 and DUE0434998, and the Institute for Museum and Library Services under grant LG-02-04-0002-04. The opinions in this paper are solely those of the authors.