Assigning Wikipedia editing: Triangulation toward understanding university student engagement
First Monday

Assigning Wikipedia editing: Triangulation toward understanding university student engagement by Amy Roth, Rochelle Davis, and Brian Carver



Abstract
Professors across the United States participated in the first direct effort by the Wikimedia Foundation, the non–profit organization supporting Wikipedia, to engage the academic community and use Wikipedia in a class assignment. Three project participants, from different areas of study, conducted independent research into university student motivations for a Wikipedia assignment. We triangulate those data in this paper to describe how student motivations differ for a Wikipedia assignment from a traditional research paper assignment. Several themes emerged through the research and many of the dominant themes were linked. The global audience both motivated and intimidated students. Students appreciated the usefulness of contributing to Wikipedia and found satisfaction in making information accessible to the public worldwide. Students engaged with an online community and appreciated feedback and collaboration. Some recognized a degree of possessiveness that they felt toward the article. Both instructors and students observed that student research and writing skills improved. Qualitative data from both students and professors indicates that in learning basic writing skills, a Wikipedia writing assignment is comparable to a traditional research paper, however, students are more engaged in a Wikipedia assignment.

Contents

Introduction
Data sources and triangulation
Data synthesis
Conclusions

 


 

Introduction

How do university students engage with an assignment that contributes to Wikipedia? How do students evaluate their engagement with a Wikipedia assignment versus that found in a traditional research paper assignment? These questions piqued the curiosity of three researchers who shared a connection to Wikimedia’s Public Policy Initiative in 2010 and 2011. The collaborators gathered data independently and discovered each others’ research as the Initiative wrapped up. Discussion of the initial findings revealed striking overlaps and similarities in student engagement and motivation for a Wikipedia assignment. More complex findings emerged through triangulation of the data. This study can be included in the broader body of work assessing the value of using Wikipedia and other public platforms in university classrooms.

Background

Over the 2010–2011 academic year, the Wikimedia Foundation, the non–profit that supports Wikipedia, piloted the Public Policy Initiative. This Initiative was the first effort by the Wikimedia Foundation to reach out directly to academic circles. Wikimedia Foundation staff collaborated with faculty from 24 universities to improve Wikipedia content in articles related to United States public policy. Additionally, the project attempted to increase the use of Wikipedia as a teaching tool in university classrooms. Instructors from participating universities assigned their students to create or substantially improve a public policy related article in Wikipedia. Through the Public Policy Initiative, the Wikimedia Foundation developed a framework for supporting university instructors using Wikipedia in the classroom. Initiative staff coordinated volunteers through the ongoing Wikipedia Ambassador Program, which actively recruits and trains Campus Ambassadors to support professors on campus and Online Ambassadors to support students online. In addition to the Ambassadors, Wikimedia Foundation provided support materials and sample syllabi to the professors (“Public Policy Initiative,” 2011).

Through triangulation of all data we find that students are motivated differently by a Wikipedia assignment than a traditional research paper assignment. Some instructors report greater quality in student work. Both instructors and students report greater effort in a Wikipedia assignment. Students learn many technology–related skills through a Wikipedia assignment. Students learn to write in an unbiased and encyclopedic format. Students become excited about writing in Wikipedia when they have positive interactions with the Wikipedia community or when their work garners significant attention. Motivations and roles for both instructors and students are more complex in a Wikipedia assignment than a traditional research paper. Overall students and professors agree that in learning basic writing skills, a Wikipedia writing assignment is comparable to a traditional research paper, however, students are more engaged in a Wikipedia assignment.

Related work

Many in academia have regarded Wikipedia with suspicion (Giles, 2005; Read, 2006; Kittur, et al., 2008; Lucassen and Schraagen, 2010; Jenna Johnson, 2011; Wannemacher, 2011) since its inception in 2001 (“Wikipedia,” 2011). Others have pointed out that since the reliability of Wikipedia compares favorably with traditional encyclopedias (Giles, 2005), and fares better against the reliability of the information sources that people would likely be using if Wikipedia were not available (freely available Web sites returned as search results), then it is epistemically better for people to have access to Wikipedia (Fallis, 2008). Snyder (2010) surveyed how faculty and students use Wikipedia and found agreement among respondents that Wikipedia can be a useful tool for classroom use, and surprisingly, found that faculty were both utilizing Wikipedia more than students and that faculty have a more positive viewpoint toward Wikipedia usage in general education classes as well as major classes (the latter not being statistically significant).

Many educators see a need for evolution of American higher education systems and institutions to meet the increasingly complex needs of our rapidly changing environment (Shapiro, 2005; Jenkins, 2009; Zusman, 2011; Altbach, et al., 2011). Zusman (2011) describes this environment as follows: “the twenty–first century has brought with it profound challenges to the nature, values, and control of higher education in the United States. Societal expectations and public resources for higher education are undergoing fundamental shifts. Changes both within and outside the academy are altering its character — its students, faculty, governance, curriculum, functions, and very place in society.” Due to such changes, Jenkins (2009) asserts that students in the twenty–first century need a different set of skills for the future. In addition to the research and technical skills necessary to navigate the information–rich climate, students need to understand how media structure perceptions as well the economic and cultural contexts behind the production of mass media. Compiled together, these sets of skills go beyond media literacy and should be identified as social skills, as necessary means to interact with the larger environment.

But the changing environment is not just about the needs for students to develop new skills; rather the new environment covers the relationships between higher education and the production of knowledge, and pedagogical issues related to how we learn. Shapiro (2005) identifies an understood but not widely recognized social contract between universities and society; he states that universities have a responsibility to ask the questions society does not want to address and further to push society toward adoption of ideas that will improve the future. Altbach, et al. (2011) describe some of the new digital technologies available to professors and refer to the broader role of open educational resources toward improving the impact of education in the future. In an anonymous survey, Jones, et al. (2010) explore the role of various technologies in higher education. The four unnamed universities included in the study used multimedia and technology tools infrequently in classes. Based on the survey, they conclude that interactive software including wikis and blogs should be included in university learning tools because the benefits include an increase in student motivation for learning, the experience of a community of inquiry, and the independent learning required for new technology increases student employability.

Given the technological changes and new educational opportunities and experiences, higher education in the future will undoubtedly exhibit an increase in online teaching and tools, more fluency with computer–based technology by both professors and students, and a greater reliance on community–based learning and interaction (Altbach, et al., 2011). Dolan (2008) compares the results of teaching one section of an American government class face–to–face, with the results of an online section of the same course. She found that a “well–designed” online course was “at least as successful at providing university–level education,” [1]. Her study found that in addition to a higher average grade in the online course, online students also had higher satisfaction with the class. Johnson and Card (2007) describe the importance of “immediacy” behaviors or communication that indicates warmth and approachability in online courses. The immediacy behaviors resulted in higher quality in the writing assignments. The authors show that creating a warm online environment resulted in a feeling of responsibility among all students for their peers and students posted regularly to provide required reviews for fellow students. By focusing on immediacy behaviors, researchers demonstrate that online teaching and tools can be effective when the professor and students create a sense of community.

One advantage of incorporating an online community into a class assignment is feedback from community members. Van de Sande and Leinhardt (2008) describe “the Good Samaritan Effect” in a free, open, online homework help forum. The authors purport that the altruistic intentions of the tutors provides better help by encouraging multiple tutors to help, as opposed to traditional tutoring which is one on one, and that online participation typically results in a positive experience for the students. The authors compare the online tutoring environment to the behavior of the Wikipedia community. In another study related to online interaction, Cheng, et al. (2010) found that student participation in a voluntary online discussion forum was related to an improved course grade.

Many researchers have investigated the impacts of various technologies on student motivation. Martin (2008) provides an excellent definition of student motivation and engagement: “students’ energy and drive to engage, learn, work effectively and achieve to their potential at school and the behaviors that follow this energy and drive.” Cole (2009) reports on a failed experiment using wiki technology in the classroom. Her results demonstrate that it is not enough for instructors to just make technology available. She suggests that students need instruction on how to use technology and information about the benefits of its use. This study also finds that 70 percent of students report using blogs and wikis, but more than 90 percent have not actively contributed to blogs or wikis. In this case the use of technology was optional and students were not motivated by wiki technology. Ellison and Wu (2008) report similar findings for use of blogs in the classroom. They assert that sound instruction about blogging and proper criticism of other posts is necessary. They also found that while students were resistant to the new technology of blogging, it increased discussion among students. Neumann and Hood (2009) again found that success of wiki technology is dependent on proper implementation. Their results show how students participating in a class wiki became more engaged in learning, experienced more peer collaboration, and had higher attendance rates. Augar, et al. (2004) reveal that use of a wiki increased student engagement in an online course.

Research shows that when learning wiki technology is also tied to the public dissemination of information on a site such as Wikipedia, student engagement with assignments increases dramatically. Sormunen, et al. (2011) report that students showed higher motivation, but also greater stress for a Wikipedia assignment. Wannemacher and Schulenburg (2010) claim that “writing for Wikipedia helps students realize that scientific accuracy can be linked to dissemination of knowledge.” Forte and Bruckman (2006) conclude that students were motivated by the potential audience for their work in contributing to Wikipedia. They sum up use of Wikipedia in the classroom as “not without problems,” but it offers “a unique opportunity for student writers to enrich public discourse in a way that serves a real purpose and engages a real audience” [2]. Our separate research findings, compiled here in this paper, confirm many of these conclusions and show how professors design and evaluate Wikipedia assignments, how students engage with Wikipedia assignments in comparison with other written work, and how students generally evaluate their own work in classes with Wikipedia assignments. In addition, they show that the provision of additional supporting resources, such as teaching assistants, additional professor engagement and supervision of work, online and campus ambassadors, and specially designed assignments, is what makes the use of Wikipedia successful.

 

++++++++++

Data sources and triangulation

Data for this study came from multiple sources, geographic regions, and methods, including both quantitative and qualitative data. Two groups of data were collected from multiple university professors and students. First, the Wikimedia Foundation commissioned WestEd, an independent contractor and non–profit educational research organization, to perform interviews of six university instructors participating in the fall 2010 pilot of the Public Policy Initiative and nine participating faculty members in the spring 2011 term. WestEd also conducted a student survey. Interviews followed semi–structured qualitative interview procedures as described by (Richards and Morse, 2007; Myers and Newman, 2007; Creswell, 2009; Stake, 2010). In addition to recorded and transcribed records of the interviews, the interviewer provided the researchers with detailed notes (Appendix A contains the interview questions.). WestEd also administered a student survey through Zoomerang at the beginning of spring term, and once again at the end of the term (Appendix B contains the student survey questions.).

The second group of cross–university data was collected by Annie Lin, the Campus Team Coordinator for the Public Policy Initiative, who conducted focus groups of undergraduate and graduate students participating in the project at a sample of the participating universities: Boston University, Michigan State University, Simmons College, and Texas Southern University. The focus groups followed methods and principles described by Stewart, et al. (2007). Questions in the focus group interview started with general topics and then moved to specific questions about the assignment over the course of the interview. The moderator is experienced in qualitative interviews, and adopted a neutral persona to obtain student feedback. She directed her efforts toward encouraging conversation among participants (Appendix C contains the focus group interview guide.).

Two of the authors are university professors who were engaged in the project and also collected student responses to and evaluations of the Wikipedia assignments. Rochelle Davis, an Assistant Professor in the School of Foreign Studies at Georgetown University, required her classes of undergraduate and graduate students to contribute an entry to Wikipedia along with writing a research paper on the same subject and also to post blog entries about their experiences over the course of the school term. Georgetown’s Center for New Designs in Learning and Scholarship conducted student video interviews to gain an understanding and record of the undergraduate and graduate experiences with the Wikipedia assignments as designed by Professor Davis. Brian Carver, an Assistant Professor at the University of California, Berkeley School of Information, has required a Wikipedia–based assignment in his classes over three consecutive years. He conducted informal student surveys about the Wikipedia assignment at the end of each school term.

Often topics of qualitative studies are inherently difficult to quantify because these topics cannot be measured in discrete and tangible terms; the level of student engagement is one such topic. Triangulation is one means to improve reliability and validity of qualitative studies (Creswell and Plano Clark, 2011; Golafshani, 2003; Hussein, 2009; Olsen, 2004; Thurmond, 2004). Triangulation means incorporating multiple data types, sources, or methods to arrive at study conclusions (Creswell and Plano Clark, 2011; Golafshani, 2003; Guion, et al., 2011; Hussein, 2009; Olsen, 2004; Thurmond, 2004). Reliability is defined as the consistency of the result, and validity is the accuracy of the result, or how “true” it is (Golafshani, 2003). In general, five of types of triangulation are identified: data source, environmental, methodological, investigator, and theoretical (Guion, et al., 2011; Hussein, 2009; Thurmond, 2004). The unique aspects of this study allowed researchers to make use of all five triangulation types. The multiple data sources include both students and instructors involved in the program. The data come from several unique geographical locations across the United States. Focus groups, interviews, and surveys provide mixed methods to triangulate results. Multiple investigators with very different theoretical backgrounds (anthropology, law, and science) perform this research.

Researchers triangulated data from all sources to look for areas of convergence and divergence. Rather than testing specific hypotheses, collaborators approached the research with open questions about the success of a Wikipedia assignment and student engagement around that assignment. Starting from a general inductive approach (Thomas, 2006; Myers and Newman, 2007), collaborators used thematic analysis (Boyatzis, 1998; Braun and Clarke, 2006; “Demonstrating Rigor Using Thematic Analysis,” 2008) and triangulation (Mathison, 1988) to distill the qualitative and quantitative data into meaningful conclusions about university student motivations for a Wikipedia assignment.

 

++++++++++

Data synthesis

Whenever possible we provide quantitative results, however, since most of the data come from qualitative voluntary sources, it is not possible to give accurate response rates for the thematic analysis. Instead we provide several examples of each theme. Selected quotes from professors and students are included in the text and often in series at the end of section to give the reader a sense of the quality of the data and the enthusiasm expressed by the students. The qualitative methods captured a more complete picture of the student engagement spectrum than the quantitative survey. The quantitative survey indicates that students were most strongly motivated by a grade, often frustrated by learning wiki culture and technology, and in general had stronger reaction to the Wikipedia assignment than a traditional research paper, it does not capture the themes that emerged so strongly from all qualitative data sources. Together the quantitative survey and qualitative interviews and focus groups tell a complete story about student engagement. The survey confirms that the strongest motivator for a Wikipedia assignment was attaining a good grade, however when discussing the project, this fact rarely arose. That the grade was important was assumed but not of interest, students and instructors were more eager to talk about the unique aspects of the Wikipedia assignment. The qualitative data provide the basis for most of the thematic analysis described here. We share many of the quotes because we enjoyed them, and hope readers will as well.

In the spring term, 154 out of 614 students, a 25 percent response rate, completed WestEd’s survey about their experience doing a Wikipedia assignment for a university class. The survey results indicate that students’ primary motivation to participate and work on the Wikipedia assignment is achievement of a good grade. (Noting, at the same time, that such motivation measures the way the vast majority of undergraduates describe their motivation to do any assignment (Harackiewicz, et al., 1997.)) Although the survey indicates earning a good grade is the strongest motivation, this factor rarely comes up in the qualitative data. In discussion and blog posts, students focus almost entirely on other motivations which make the Wikipedia assignment unique. After achievement of a good grade, the survey indicates the following motivations, in order from strongest to weakest: contribution toward a useful public resource, interest in learning a new technology, opportunity to share work with a global audience, interest in the creation of knowledge, and collaboration with an online community. In the survey, students report that their skills improved in the following areas (starting with the greatest skill improvement): wiki editing, evaluating Wikipedia article quality, research, increasing adaptability for working with an online community, technology skills, collaborating, and writing.

The professor and student interviews and focus group data corroborate the survey results, but provide much greater detail about student motivation and engagement with a Wikipedia assignment. The motivational themes identified in the survey aligned with the following themes which consistently emerged through interviews and blogs:

  • Students express excitement about presenting their work to a global audience, but some indicate fear or additional pressure at presenting their work in a highly visible public forum.
  • Students appreciate that their work was useful and had a purpose beyond a grade; some showed it to friends and family members. Other students express interest in making information that is important to them accessible to the broader world, recognizing that knowledge can be a public good.
  • Students engage in an online community and appreciate feedback and collaboration; when interacting with the community, some recognize a degree of possessiveness toward the article.
  • Students often express an appreciation for learning a new technology, and frustration with limitations of wiki technology.
  • Instructors and students observe that student research and writing skills improved through the Wikipedia assignment.

Throughout this section, we describe several of the themes illustrated in Figure 1; we group the dominant themes when the qualitative data consistently link them together. Figure 1 presents a full qualitative analysis including the stakeholders, motivations, pressures and outcomes of a Wikipedia assignment. It illustrates the increased complexity of student motivation for a Wikipedia assignment. While the motivations related to the instructor as a stakeholder may be the strongest, this set of motivations and pressures is well understood and commonplace in the American academic system. The Wikipedia assignment introduces new stakeholders in the work product and with those stakeholders comes a novel and interesting set of motivations and pressures to produce a high quality work product.

 

Thematic analysis of student motivations for a Wikipedia assignment
 
Figure 1: Thematic analysis of student motivations for a Wikipedia assignment.

 

The global audience and fear

The Georgetown students’ blog posts, which were designed to elicit students’ evaluations and reflections on the assignment at its different stages, did not explicitly show that the greatest motivation for their work was for their grade (although undoubtedly this was the case). What the student narratives revealed instead was that the potentially huge global audience was their prime motivator and source of fear. An undergraduate student wrote, “I must admit, I had no idea what to expect when I found out that we would be writing Wikipedia articles for class. I was intimidated to say the least, as the prospect of having my work in a public forum of that size was, and is, a little daunting. But now that it is over, I realize that it was not that bad.” A graduate student commented that, “Creating a Wikipedia entry that will be fully visible to the internet–consuming public was certainly new for me. It is both exciting and frightening to think that something I wrote is now out in cyberspace and available for people to read! Although it will undoubted require revisions and clean up, I am proud of it as a preliminary article on what I believe is an important subject. Beyond learning the facts and figures relevant to my article, which at times had the side–effect of raising my blood pressure, I learned a great deal about the internet world.” In these two classes, the professor dealt with some of these fears by assuring the students that they are anonymous on Wikipedia (the students were counseled to chose non–identifiable usernames), and that their first grade was based on what they wrote, not on what happened to the material after they posted it. Their initial fears were often overcome when they posted their material and received feedback.

Evident in the students’ final evaluations of their work was that they were quite proud and excited to have been pushed to contribute their research and writing to one of the Web’s most heavily trafficked sites. A graduate student expressed that she found it, “extremely gratifying to put work into a project that will actually affect people and maybe interest someone or help strangers. My papers for class don’t usually go beyond me and my professor or one other reader, but this has a much wider scope.”

For the professor observing (and guiding) the students through the process, one of the interesting issues was to watch how they approached writing a Wikipedia article as they do their usual academic paper writing — which is a solitary research and writing experience. Then, when they posted their work publicly on Wikipedia, it went out into the world for anyone to see and comment on. This experience was unsettling for many, but over time they negotiated the comments (given by peers, the professor, online Wikipedia ambassadors, and Wikipedians), and re–worked their contributions. The professor witnessed the students’ growing awareness of how communities work. Developing this sense of a knowledge community and what it means was a goal for having students engage in Wikipedia article writing. One first year undergraduate student captured quite well this sense of individual accomplishment and communal belonging that he felt over the semester. “Seeing my article on the front page oddly excited me. It was cool having your own work on display in one of the most visited sites on the web. This was a real ‘nerd’ moment for me. I felt that by getting on the front page [Did You Know? Status], I had accomplished a great representation of the [subject] and Wikimedia Foundation Project community. Nevertheless the page was featured before I was completely finished and moving forward I used the community to polish my page. As the class all made entries, they became part of my community. With a peer edit from [another student] and a first grade/comments from the professor, I was able to complete my entry. Adjusting the final product with the suggestions of everyone who looked at my page I finished not only with my own confidence in the entry, but that of the community as well.”

Other students describe the motivation in writing for a global audience. “This is one of the most unique assignments that I’ve ever done, because it’s not just a research paper that you turn into to your professor, your professor grades it and then it’s pretty much done. You complete this assignment and your professor reads it and then it’s out there for the world to see. Because of that it’s kind of representing you in way, so you want to make sure that it remains how you intend it to.” Another student comments, “I really like the possible impact of the Wikipedia project. Unlike a course blog, which is unlikely to be read by anyone not in the class, it is considerably likely that someone surfing Wikipedia would stumble upon and be able to benefit from the project.”

Many students made similar comments about the audience for their work. “It was nice to go out there and put your work out there to be read by everybody, rather than just to get a grade. But I was a little nervous...” Another students comments, “I feel like I contributed something to the world in a way because all my other papers will be read only by a few people, maybe only one or two. So, this gives us a chance to interact with the broader world, because I know that over a thousand people have seen my page already.”

When asked if he would continue to contribute, a student who had one of the more memorable experiences writing a Wikipedia article responded as follows. “Yes, and I don’t know how I feel about it. I don’t know if I’m happy about it, that I’m obligated to do it. Because I did, I wrote the article, the whole thing, and it’s something that’s not in a stasis, it’s information that’s changing. For example, part of the article speculates on the future of the National Democratic Party and who will be the next president of Egypt. And this is all something that is very much related to foreign policy and something that’s going on now... Yeah, I will go back in a few months to make sure that when the next president is elected if it’s not what I say in the article that I change it to make sure that it’s still factually accurate.” The above student wrote the article on the National Democratic Party of Egypt in November of 2010. A big spike in hits occurred in this article around the Egyptian elections held in late November. When the Egyptian revolution occurred in early 2011, page views on that article went up to over 5,000 per day, and he was still actively editing the article to keep the information current.

Other students’ long–term relationship with Wikipedia contribution remains unclear. At the end of the assignment, students report that they expect to continue editing Wikipedia outside of class. When Berkeley students were asked how likely they thought it would be that they would edit Wikipedia again, on their own time, not as part of a class, roughly 85 percent, across five semesters, said it was somewhat or very likely that they would. However, unless students have adopted new usernames, Professor Carver, over a period now of more than two years, has seen virtually no subsequent contributions to Wikipedia from the student usernames from Spring 2009 and Fall 2009.

Usefulness of student work and the idea of knowledge as a public resource

In student motivation, the usefulness of the assignment end product often merged with having global audience for the work. Students linked the value of the information they created and made available, to the idea that at some point people would be interested in the topic and read the article.

Many students remark on the satisfaction that their work serves a useful purpose. One student comments, “Wikipedia project = a paper I’ve written that didn’t end up in a professor’s recycling bin. Awesome.” Another one states, “I really like the fact that the work done for this class won’t just get thrown away at the end like most homework.” Another student states that it “felt good doing something that wasn’t just an assignment, but that actually benefits outsiders.”

Many students commented on the value of their work and were more motivated by the opportunity to contribute information they care about to a public resource rather than just having a large audience for their work. They saw value in preserving the information, and felt pressure as stewards of that knowledge. One student demonstrates this theme in the following statement, “Most people don’t read about the subject I wrote about, and it doesn’t matter how many people read it. But what matters is that the material is really important to me.” Another student corroborates this idea, “The thing that made it entertaining for me was the fact that I was writing about something that was such a big part of my life.”

The following student links academic contributions to Wikipedia to the value of a public information resource, “I think it is a great idea. I especially like that it produces concrete, permanent enhancements to a real body of knowledge. As semesters advance, this class will greatly enhance this area of knowledge on Wikipedia.” Other students also put the assignment in the context of societal value, “It’s almost a fun experiment to see how the world reacts to something you create, and what parts get edited and what don’t,” and, “It’s awesome! Nice to contribute something to society.”

The quotes in the global audience section show that students recognize the broad sense of information accessibility: that by putting the information on Wikipedia they are making it available to the world. This undergraduate student articulates the importance of making information accessible through Wikipedia. “Access to information is important... I think [Wikipedia] is a place for all kinds of people in all kinds of economic situations to have that access. For that reason the articles need to be well done. There is way too much disinformation.” Other students differentiate between types of accessibility and comment on the unexpected difficulty of writing a neutral paper and developing text that is easily digestible. “One of the things I appreciated was that you found yourself being pushed out of academic–speak and really trying to write more clearly for lay people. And that goes back to the unseen bias... That was a nice thing to be able to do, make it more accessible.”

An undergraduate student who took the class specifically because of the Wikipedia component framed his evaluation of the Wikipedia project in terms of his accomplishments in his education: “This project was awesome. I feel like we have all made a tangible, solid contribution to human knowledge, as opposed to just our own knowledge. When you Google [the subject], my page is the first result. If that’s not a sign of a semester well–spent, I don’t know what is.” This sentiment about the larger accomplishments and the motivations to work hard because of the public forum were echoed by another undergraduate student who noted the contribution of the class: “In the past three months, our class has uploaded over 254,959 bytes of information onto Wikipedia, through 18 different articles. This includes pictures, charts, and hours of work that has gone into making this Wikipedia project a success. Knowing these numbers makes this feat seem remarkable, as it emphasizes that Wikipedia is a LIVING website. While encyclopedias of the ‘olden days’ (for lack of a better term) were stagnant and remained fairly constant between editions, Wikipedia has completely changed this way of ‘thinking.’ Today’s society lives minute–by–minute, and through websites like Wikipedia, the general populace can take the initiative to further the education of society. Knowing this has inspired me to create the best article possible.”

One expression of the student pride in their projects, which professors do not usually see from other assignments, is how students encouraged their family and friends to look at their Wikipedia articles. An undergraduate, who had been very skeptical at the outset, chronicled in her final blog post the following: ‘This project was unexpectedly cool. I must say I am quite proud to show the page when people ask about it ... . A friend of mine saw the article a couple weeks ago, not knowing I wrote it, and was really impressed when he discovered that I had contributed on Wikipedia ... I must admit ... I’m slightly impressed with myself, too.” Another student describes this same idea, also with a hint of humor, “My parents were very big fans of the article. Besides that, I’m not sure that anyone in the general public has actually read it. When they do, I suppose it’s nice to contribute to the general knowledge of, you know, the universe.”

Graduate students saw the value of their work in the creation of knowledge and commented on the fact that their work contributed to both detailed topics about which little is known and also by providing scope for broad topics. One student comments, “It’s surprising how certain subjects have so little written on Wikipedia and other subjects have so much written. When you add the links I was very surprised to find that some pages hadn’t been written yet, I would have thought were very important.” Another student echoes this idea, “I was surprised that there weren’t articles on a lot of the things that people were writing about. They seemed like noteworthy things that I would have thought there would be articles about. It really showed me how Wikipedia as a site is constrained to what interests people and what people want to write about.”

In a related conversation, graduate students explored the variation in the scope of their work, and expressed excitement about providing perspective on topics of global importance. One student describes contribution to a narrow topic, “It is pretty exciting... You are sharing information about topics that are not well known about. And you’re putting it out there, trying to transcend the academic world about topics that are part of our focus. We got to write on things that we chose, so things that are of interest to us. And being able to share that information that anyone can have access to it, is very exciting.” Another student agrees, “...there is not a lot of information about it. And I was very fortunate to be able to... enhance the amount of information that people will have access to about a place that not so well known about. Even though the issues there are very serious.” However, a different student immediately responded to the two comments above and said that in cases where classmates were not providing detailed information about little known topics, they were providing clarity to broad controversial issues. “Instead of taking small issues that no one’s really heard about... We’re taking big issues like ’democracy in the Middle East,‘ which we’ve heard so much about, and condensing in an understandable way, that will hopefully guide or clarify people’s points of interest that people could explore.”

Collaboration, community, feedback, and student possessiveness of work

During interviews, many students linked the ideas of collaboration, interacting with the Wikipedia community, getting feedback on their work, and a sense of possessiveness toward the articles.

For a few students, the online feedback came in the form of other users making significant deletions to their additions, which alarmed those to whom this happened. But even most of these students argued back (with the help of the online ambassadors), rewrote their work, improved their citations, and reconsidered their contributions to the entries. An undergraduate described his rather fraught experience in this way: “I’d say that this experience, overall, has been memorable. It had it’s ups... and it’s downs — like when I thought that the entire article had been deleted by someone and when someone accused me of plagiarizing a sentence. But on the whole, it taught me a new skill and exposed me to the intricacies of a form of information that we all use so regularly. I really mean it when I say that I valued the time I spent, even if the work could be incredibly tedious.” This student was horrified by the accusation of plagiarism and worked with the Wikipedia community and the online ambassadors to make the necessary clarifications to what was a very gray situation. From the professor’s standpoint, it allowed the student the opportunity to learn how to be a more careful researcher in a very tangible way.

In the WestEd survey, 89 percent of respondents reported that a Wikipedia assignment was equal to or better than a traditional research paper assignment in improving collaboration skills. Students reported learning collaboration skills through a Wikipedia assignment, a finding that Sormunen, et al. (2011) corroborate. Many students described collaboration in Wikipedia:

  • “Having someone change your writing without your knowledge online can be frustrating, but at the same time it can also be intriguing and a form of growth. When writing a Wikipedia article, I learned that all of the research and writing ability in the world doesn’t mean that you can make a perfect article, sometimes you just have to let the community do what it does best: make the compendium of human knowledge concise and in one place.”
  • “The traffic that came from the ‘Did you know?’ posting was really exciting but also brought with it a lot of people out to edit my work. I really appreciate all their help. However one person in particular sought out to change the tone of my article to lean towards the conservatives, which he claimed was not a bias just fact. It was interesting to be able to defend my work on such a large scale. The ambassadors really helped as well.”
  • “Having the opportunity to participate in this assignment has been truly remarkable. I have begun to really understand the merits to collegial scholarship under this form of collaboration and how Wikipedia is an organization that really does its best to solicit expertise from people around the globe.”
  • “Giving up control of your article and putting it onto the public domain is kind of a heart–racing experience, because you know that anyone can come in and just completely mess up your article... If you have sources supporting your information then you have the final word ... because you can say ‘this source supports what I say.’”

For most students the ideas of collaboration and the Wikipedia community were closely linked. “I checked over time and the community response to my page was marginally positive. Yes, there were critiques that it sounded like a brochure, I had too many headings, I needed to remove this line, etc., but those proved to be useful in the end. I addressed their responses and suggestions by making changes and trying my best to abide by the ‘laws’ of Wikipedia and information in the public domain. It was, in a way, like drafting a paper and simultaneously submitting it to millions of people for proofreading — although some may have misinterpreted the intent, it was still useful ... It was hard while researching to imagine how I could synthesize all of my various sources into a small article online, but in the end, it was the editing help of the other people that helped me do it.”

Students frequently commented on the idea of Wikipedia as a community. One graduate student describes it this way, “Wikipedia is like a direct democracy of knowledge, for better or for worse.” An undergraduate student offers another description of the community, “Writing a page for Wikipedia has taught me about the real cyber–community that exists and the potential for shared information.”

Some students perception of Wikipedia changed through the assignment. “I haven’t experienced this much, but talking to other people, the amount of community input, they’ve had in their article, and the amount of people that have called them out on every single aspect of what’s happening. You know, Wikipedia has this reputation for being this place where anything goes and anyone can post anything they want to and it doesn’t get checked. But I think the fact that all of our articles have been modified by 4 or 5 random people, some of us a lot more than that, kind of shows that Wikipedia is not this free for all place where anyone can write what they want. There is accountability, there is oversight. Anything that is there for any length has been approved by the community.” Another student describes how the experience with the community changed over time, “As time went on, I found myself to be part of the Wikipedia community. People that are on Wikipedia seem to genuinely want to help you upgrade your article. This made the Wiki task less intimidating and gave me incentive to want to do better. Their feedback is helpful and provided much insight to what the community needs. One of the people on the discussion page said that the photo I took of [a subject] was a great addition to the entry. It’s nice to know that people in the community recognize the effort you put into the entry. Overall, I found the Wikipedia project to be rewarding, even though it can be challenging at times.”

Woven into to the ideas of community and collaboration, another motivating factor for many students was the potential for feedback from other people who were also interested in the topic the student chose to write on. Many students reported hopes to receive content–related feedback and were disappointed that this did not happen. The students who did receive feedback about their topics often put in greater effort and developed a meaningful dialog with a broader community. The idea of getting feedback on the article was closely linked to the concept of connecting with an online community. Many students commented on this theme:

  • “I tried to get help from people in the content areas... and I haven’t heard from them.”
  • “For me it was disappointing not to have anybody in Wikipedia who was interested in policy and could tell me if it was good ... or tell me if it could be integrated into different groups.”
  • “The community was helpful for formatting and grammar, but I have yet to get feedback from subject matter experts.”
  • “I didn’t get any feedback on the subject matter ... but a lot of people were willing to help me with everything else. They were so nice to me.”
  • “I received a lot of encouragement before I put my article up ... But after I put my article up, I didn’t receive any comments. I even reached out to the community asking for someone to review it, and that has yet to be done.”
  • “It was kind of frustrating at first because when I first posted the bulk of my article, like almost immediately some guy came through and just erased a paragraph ... and I don’t know why ... so I put it back in. Literally it was like 45 seconds after I posted ...”

Even students who were disappointed with lack of interaction or experienced negative actions like deletion or edit reversion by Wikipedia were clearly invested and engaged in project and simply wished they would have had a more positive experience. One student explains, “Advice to the next class involved in this WP project, is really to just have a lot of patience and be diligent in your work on the article, despite what some of the Wikipedia Ambassadors may tell you. Because I went in to the Wikipedia Ambassadors after putting so much work into my article, I was all chipper, bright–eyed and bushy–tailed, and they just completely shot it down. And that’s a real shock to the system. You just really have to push forward.” Another student summarizes feedback this way, “It was fun to have that much more impact, instead of just one on one with a single professor. I did get feedback from a few mentors, and I just thought that there is something nice about opening up. And it seemed like there was a sense of that in the class, that we were involved in a group effort more than an individual effort ... Some of [my classmates] were electrified by the idea of sharing their research with a wider audience. Not just an assignment to get done, but a project with some value.”

Several graduate students were surprised by their feelings of possessiveness toward the articles they worked on. “I feel like I spent more time editing and monitoring the changes of my article made by others than I did researching and writing. I also underestimated how protective I would feel towards my Wikipedia article. One of the hardest parts of working in a forum like Wikipedia is the fact that anyone and everyone can change your article. I struggled with some of the changes that were made by others I perceived to be less qualified and knowledgeable on the topic. It was a challenge to engage diplomatically with such individuals that knew a lot about Wikipedia, but not about the subject.” Other students echo the idea of possessiveness, “I feel attached to the page, so I am happy to make changes if there are any needed even after this project comes to a close,” and “I was excited and yet very nervous to put it out on the web knowing that I have no control over other edits now.” An undergraduate student recognizes the possessive feelings, “It’s funny that you said ‘our articles’ ... because once we put them up they’re not really ours they’re like the whole Wikipedia community’s.” Berkeley students also expressed a possessiveness towards the work they did. When asked if they felt an “ownership stake” in the page they edited and whether they would monitor it from time to time to see if and how it changes, roughly 90 percent of Berkeley participants, across five semesters, have said they did feel such an ownership stake and would monitor the page.

Learning new technology skills

In the WestEd survey, 94 percent of respondents report that a Wikipedia assignment was equal to or better than a traditional research paper assignment in improving technology skills. In terms of learning wiki technology, very few of the Georgetown students had Wikipedia accounts before the class began, and those few were the only ones who had ever made any edits to Wikipedia before. Among the Berkeley students, across five semesters, virtually all reported having made fewer than 10 edits to Wikipedia prior to this assignment. Thus, students had a variety of reactions to the wiki technology: some found it easy to master, while others thought that learning and becoming proficient in the coding was a big hurdle to overcome and where they learned the most. A graduate student reflected that, “I think the best part of this assignment has been to learn how to write and edit on Wikipedia. I remember being very scared of it after the first workshop we had with the on campus Ambassadors. At that time I even considered writing the article in word format. I am very happy I discarded that idea and ventured on in the wiki–language, which, on the other hand has been very time–consuming.” For an undergraduate student using governmental statistical information, he not only had to figure out how to read and interpret it, but also how to present it in an easily digestible format. He reported that “Technically, I learned quite a bit through the project. The html codes required to make the charts and to cite sources on Wikipedia can be cumbersome, [but] I learned how to use these codes efficiently to spice up my article.” For most students, they figured out their own ways to meld together the skills they had in writing with the more technical elements of the wiki, although it added a large time burden (especially the citation interface). “Overall, the writing of this article has been interesting, but a bit frustrating,” a graduate student wrote, “mostly because I felt that learning the Wiki formatting was somewhat time–consuming (and I felt the quality of my writing suffered from the stop–and–go of figuring it out). Now that I’m about finished with the project I feel like I’m just getting to the top of the learning curve.”

Many students and instructors complained about the wiki markup formatting language. “The only real barrier was learning how to write with the wiki markup text. It’s not clear why you can’t just write like in Word.” This student comments on the difficulty of transferring work from a word processing program to Wikipedia. “I think one of [the disadvantages] was transferring, because I like to write in Word, and it was a really big bother. I had to lift all the references and then transfer all of them in the links, and that was very time consuming. I also listed out all the references in Word and it was a complete mess.” Wiki markup is difficult for a lot of new contributors, and developing a better user interface is a high priority on the Wikimedia Foundation’s goals (“VisualEditor,” 2011; Parscal, 2011).

Improvement in research skills and neutral writing

While it is not surprising that students’ ability to learn new technology and collaborate with others rated higher than traditional research paper assignments, the WestEd survey also indicates that students felt that Wikipedia assignments gave them more opportunities to improve their research and writing skills than traditional assignments.

In media literacy related questions, the WestEd survey reveals that students believed the Wikipedia assignment superior to a traditional assignment in helping develop the following skills: recognizing poor quality Wikipedia articles, evaluating the neutrality of a document, understanding what constitutes a credible source, and distinguishing between primary and secondary sources. In the WestEd survey:

  • 61 percent of respondents report that a Wikipedia assignment was equal to or better than a traditional research paper assignment in improving writing skills.
  • 81 percent of respondents report that a Wikipedia assignment was equal to or better than a traditional research paper assignment in improving research skills.
  • 86 percent of respondents report that a Wikipedia assignment was equal to or better than a traditional research paper assignment in improving editing skills.
  • 76 percent of respondents report that a Wikipedia assignment was equal to or better than a traditional research paper assignment in improving critical analysis skills.

These somewhat surprising results about student impressions of the skills that improved through a Wikipedia assignment may be related to the increased pressure to a good job reported by many students. For some, the potentially large audience created this pressure. “[A Wikipedia assignment] really makes you thorough. Knowing your facts, because you are reviewed by the whole world. That factor, your reputation as a writer or editor, is definitely in mind, and it helps keep Wikipedia very accurate, and lends more credibility to it. It’s the world’s biggest peer review.” A second student made similar remarks. “There was definitely a lot of pressure. I felt it when I was writing ... to make sure to cite everything, to back up any kind of even semi—controversial claim, to back it up with a source. In a way I think that was positive pressure. Even if it was a little daunting to think that so many people are going to potentially reference it or look at it or use it as a basis for further exploration, it encouraged, me at least, maybe the group, to really document very well.” For others the pressure came from Wikipedia’s guidelines. The biggest thing was working with Wikipedia’s restrictions. One is that everything has to be verifiable ... Another is no original research, you have to have documentation for everything.”

An undergraduate reflected on the process of learning of the Wikipedia assignment, choosing her topic, and then the research skills she developed to produce the written contribution: “When I first heard that we were going to be working on Wikipedia, I was very keptical about the value of such a project. We were taught that Wikipedia is about as scholarly and accurate as the Urban Dictionary, and is only good for looking up random things like the origins of the song ‘Cheeseburger in Paradise.’ So to hear that I would be writing an article for Wikipedia as a part of the class first came as a somewhat of a shock. However, I was even more surprised once I actually got to work on my article. Doing research for this was just like doing research for any other paper, and because it is a more obscure subject, I even had to get some books through WorldCat, which I had never done before. In fact, because Wikipedia is so strict about citations and doesn’t allow for any primary research and inference, I felt that I had to do even more research than usual.” In the Georgetown professor’s experience, the undergraduates learned more about the skills necessary to conduct research than the graduate students (all first year M.A. students), who were well–prepared. Three undergraduates expressed their experiences with doing research for their Wikipedia contributions as follows:

  • “There was not an abundance of sources available online, so learning to take what information I was given and then deriving more information from this (such as percentages and test scores) was a skill that I think will benefit me in the future.”
  • “At the end of it all, I can say that I’ve definitely learned something about the process of research and publishing in today’s modern age. The act of researching wasn’t just gathering information, but rather bringing together an impenetrable panzer of sorts that would withstand the constant picking of the Wikipedia community at large. It was definitely frustrating at times, but also rewarding when you could produce an article that stood the test of time.”
  • “The Wikipedia project, for the last 3 months, has been a great experience for me. I’ve learned so much in terms of how to research and write a balanced entry. The researching for my entry on the [subject] required me to dig deeper than what I already know. I went to the library and talked to the reference librarian, who was able to guide me in the direction I want to go. Many of the sources I’ve encountered are mostly about [one element of the subject] so I had to narrow the scope of my research. I found some of the anthropology journals to be very helpful. The library had many useful resources but I found the online journals and search engines such as Google Books to be more helpful in narrowing research.”

Many students expressed surprise at the number of of sources required for Wikipedia articles. The Wikipedia community is quick to apply “this article needs citations” banners, so the pressure to provide sufficient sources came from outside the classroom. “Getting enough research information was a big problem. The information was primarily based on going to the university’s websites; ProQuest, online journals. It helped me go through the actual legal brief.” Another student agreed, “Wikipedia demands more rigor, fact checking from students than papers.” One student talks about the value of using Wikipedia as a starting point for research. He learned how to use Wikipedia to find multiple types of sources for future assignments. “[I learned] the basics of an entire research method.”

Instructors also noticed the increase in research. “A lot of students had trouble with the research aspect. Students today are so used to relying on Google and internet sources ... I will have to reacquaint them with the library. Some of them have probably never used the library.”

Another instructor echoes this idea, “Particularly in terms of research it is valuable. I think it does increase the amount of research a student is forced to do.”

The neutral point–of–view style of writing was also a new experience for many of the undergraduates and graduate students. A graduate student described her challenge and what she felt she accomplished: “I found it rather difficult writing in the style that Wikipedia requires since I couldn’t use any opinion words. I am used to writing research papers with a thesis and an opinion. However, after writing my article, I felt more comfortable with the process of presenting dry facts without any judgment.” In the Georgetown classes, all of the students had to develop their Wikipedia contributions into an academic research paper afterward, and thus the Wikipedia contribution was framed as a way to do the background research and understand the subject deeply through a review of the literature. A graduate student articulated her experience as follows: “Procedurally speaking, I liked going through the process of presenting (or attempting to present) an objective summary of an issue without trying to build an argument first. I’m still working out exactly what I think about the [subject], whereas if I were writing my final paper without doing this first I would have found it much more difficult to think outside the preconceived notions I brought with me to the subject.”

For students who had strong opinions on subjects, the neutral point of view writing style was something they grappled with, particularly as the contributions they made went into the public sphere and people commented on them. One undergraduate worked through some of her own beliefs as she wrote the Wikipedia entry and the difficulty of the elusive neutrality. “The writing became more challenging as I went a long. When I started writing, I began with the facts of the text. What did [the subject, which is a text] say? I didn’t have a strong opinion either way because I didn’t know the arguments. It was easy to remain neutral, but the more I researched the stronger my opinion became. By the end, I had to keep checking myself by making sure my personal bias didn’t show through. Then I was concerned about overcompensating. It showed me just how difficult writing the facts can be when you really examine a issue. It became very clear that no matter how hard I tried to come off as a neutral party someone was going to project their own opinion on to my article.”

The neutral point–of–view theme arises from all data sources. The following statements come from participating students across the nation:

  • “Yes, it was a very different style of writing. In general you want to have a very good argumentative thesis. You’re always taught to convince your reader. When it comes to Wikipedia, that’s not the case. It’s more like a book review, this is what this person says, this is what happened. For me it was actually very difficult to keep my own personal bias out my writing for Wikipedia. I had to go back several times; it’s a very different experience.”
  • “The weird thing is that there can’t be any analysis. You have to totally remove yourself and anything other than what everything else has said about it from the article. Which is exactly the opposite of what you try to do with every other analytical paper that you write ... It’s a totally different style of writing.”
  • “Adjusting the writing style for Wikipedia is definitely a crucial aspect of this assignment. [Personal viewpoint] is something that WP heavily discourages, and universities worldwide encourage. So you’re really taking what you’re accustomed to and what’s ingrained in your general practice as a university student and discarding for the sake of making a workable Wikipedia article.”
  • “It was hard translating academic writing into a Wikipedia article.”

During interviews, the idea that that information must be presented accurately and without bias was often linked to the idea of creating knowledge or contributing to information for public record. One student commented, “[The global audience] really enforced the idea of being neutral in your viewpoints. I realized how un–neutral I was normally. It was fun to put that kind of collar on myself. You need to present all sides more even–handedly.” A second one echoes that idea, “[The global audience] makes you think about bias and not putting one in.”

The experience also led some students to read Wikipedia articles more critically in other contexts. This student comments on becoming a more critical Wikipedia reader. “I enjoyed the opportunity to learn the nuances of the Wiki planet, but now that I’ve contributed to it I am more critical of any info in articles I view. Not that I ever relied on it, but I’m more tempted than ever to start improving questionable entries.” One instructor described this benefit by saying that “Students become better consumers of information.”

 

++++++++++

Conclusions

The Wikipedia assignment presents both pros and cons for instructors and students. For instructors, the pros include: a higher level of engagement by students, a platform for students to interact with and learn from the world beyond the classroom, and by introducing students to an established online community there is greater potential for students to learn online etiquette, collaboration, and technology skills. For instructors the cons include: difficulty defining and grading the assignment and additional time investment to learn Wikipedia customs and technology, and to guide students students through community conflicts and technology training.

Students benefit from the knowledge that their research will be useful and many feel altruistic satisfaction from the Wikipedia assignment; students like showing their work to others, and put forth greater effort knowing that it is available to the public. Students benefit when they have positive interactions with the Wikipedia community. Drawbacks for students include: time and frustration in learning wiki technology and the potential for negative or no interaction with community members. Although, the quantitative survey data indicate that students are motivated primarily by grades and most do not intend to become regular editors of Wikipedia, the broad spectrum of qualitative data in this study give strong evidence that student engagement is much more nuanced than the quantitative result.

In general, students reported simply preferring the Wikipedia assignment to comparable alternatives. When Berkeley students were asked to compare this assignment to alternative writing assignments such as preparing reading summaries, posting to a class blog/mailing list/or forum discussion, roughly 90 percent, across five semesters, said that they would strongly prefer or somewhat prefer the Wikipedia project.

Although professors and students agree that their writing skills did not improve beyond what a traditional research paper would produce, there is substantial qualitative evidence that students put more effort into a Wikipedia assignment and learn writing skills and critical thinking skills by engaging with others online. The qualitative evidence indicates a higher degree of engagement in a Wikipedia assignment than a traditional assignment, especially among students who had positive interactions with the Wikipedia community. Students appreciate that with a Wikipedia assignment, their work has a purpose and value beyond the class assignment and grade. Students consistently state that they work harder on a Wikipedia assignment than a traditional assignment. Sormunen, et al. (2011) corroborate our findings that the global audience motivates students, students learn new technology skills, engagement increases when students write about topics of their choosing, and students improve in research and media literacy skills.

Some instructors report their impression that final student work product was superior to a traditional research paper assignment. The instructors also perceived that student engagement and motivations were greater on the Wikipedia assignment. Both instructors and students report a positive response to positive feedback on–wiki. Incentives such as the “Did you know?” feature, getting thoughtful feedback from a virtual stranger, and knowing that their article got a lot of page views all increased student engagement.

Our triangulation of data shows that students assigned to edit Wikipedia report greater engagement with such an assignment on many levels. They are both excited and trepidacious about the global audience for their work and motivated by the possibility of creating a public knowledge resource that will be useful to others and that will last beyond the school term. Those that are appropriately guided and learn the norms of the online community they are entering tend to have ultimately positive experiences collaborating with a global community outside of their student peer group. While reporting some frustration with the wiki technology, they also gain new media literacy and even improve their research and writing skills. Some also report a tendency to more critically assess the reliability of the information sources they encounter, both on Wikipedia and elsewhere. We believe that through engaging with Wikipedia, often questioned regarding its reliability, students can gain the critical assessment skills they need to assess Wikipedia or any information source and can improve the research and writing skills that traditional assignments have sought to reinforce. Even though the subject of this study, the level of university student engagement in a writing assignment, cannot be measured in discrete units, triangulation of the large amount of diverse qualitative data provides strong evidence that university students engage more in a Wikipedia assignment than a traditional research paper assignment. End of article

 

About the authors

Amy Roth is Research Analyst at the Wikimedia Foundation in San Francisco.
E–mail: ms [dot] amyelizabeth [at] gmail [dot] com

Rochelle Davis is Assistant Professor in the School of Foreign Service at Georgetown University.
E–mail: rad39 [at] georgetown [dot] edu

Brian W. Carver is Assistant Professor in the School of Information at the University of California, Berkeley.
E–mail: bcarver [at] ischool [dot] berkeley [dot] edu

 

Acknowledgements

The Stanton Foundation made the Public Policy Initiative possible through a private grant to the Wikimedia Foundation.

Georgetown’s Center for New Designs in Learning and Scholarship (CNDLS) conducted the student video interviews.

Cathy Ringstaff and Rasha El Sayed of WestEd created the survey questions and performed the instructor interviews.

The authors would like to thank all of the participants in the Public Policy Initiative, especially students who participated in the Georgetown video interviews or the focus groups, and CNDLS staff member Rob Pongsajapan for his assistance throughout. We would also like to thank the other Public Policy Initiative staff members: Annie Lin, Frank Schulenburg, LiAnna Davis, Mishelle Gonzales, Rodney Dunican, and Sage Ross.

 

Notes

1. Dolan, 2008, p. 390.

2. Forte and Bruckman, 2006, p. 187.

 

References

P.G. Altbach, P.J. Gumport, and R.O. Berdahl (editors), 2011. American higher education in the twenty–first century: Social, political, and economic challenges. Third edition. Baltimore: Johns Hopkins University Press.

N. Augar, R. Raitman, and W. Zhou, 2004. “Teaching and learning online with wikis,” Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference, pp. 95–104, and at http://www.ascilite.org.au/conferences/perth04/procs/augar.html, accessed 24 May 2013.

R.E. Boyatzis, 1998. Transforming qualitative information: Thematic analysis and code development. Thousand Oaks, Calif.: Sage.

V. Braun and V. Clarke, 2006. “Using thematic analysis in psychology,” Qualitative Research in Psychology, volume 3, number 2, pp. 77–101.http://dx.doi.org/10.1191/1478088706qp063oa

C.K. Cheng, D.E. Paré, L.–M. Collimore, and S. Joordens, 2011. “Assessing the effectiveness of a voluntary online discussion forum on improving students’ course performance,” Computers & Education, volume 56, number 1, pp. 253–261.http://dx.doi.org/10.1016/j.compedu.2010.07.024

M. Cole, 2009. “Using Wiki technology to support student engagement: Lessons from the trenches,” Computers & Education, volume 52, number 1, pp. 141–146.http://dx.doi.org/10.1016/j.compedu.2008.07.003

J.W. Creswell, 2009. Research design: Qualitative, quantitative, and mixed methods approaches. Third edition. Thousand Oaks, Calif.: Sage.

J.W. Creswell and V.L. Plano Clark, 2011. Designing and conducting mixed methods research. Second edition. Los Angeles: SAGE.

K. Dolan, 2008. “Comparing modes of instruction: The relative efficacy of on–line and in–person teaching for student learning,” PS: Political Science & Politics, volume 41, number 2, pp. 387–391.

N. Ellison and Y. Wu, 2008. “Blogging in the classroom: A preliminary exploration of student attitudes and impact on comprehension,” Journal of Educational Multimedia and Hypermedia, volume 17, number 1, pp. 99–122.

A. Forte and A. Bruckman, 2006. “From Wikipedia to the classroom: Exploring online publication and learning,” ICLS ’06: Proceedings of the 7th International Conference on Learning Sciences, pp. pp. 182–188.

D. Fallis, 2008. “Toward an epistemology of Wikipedia,” Journal of the American Society for Information Science and Technology, volume 59, number 10, pp. 1,662–1,674.

J. Giles, 2005. “Internet encyclopaedias go head to head,” Nature, volume 438, number 7070, pp. 900–901.http://dx.doi.org/10.1038/438900a

N. Golafshani, 2003. “Understanding reliability and validity in qualitative research,” The Qualitative Report, volume 8, number 4, pp. 597–607, at http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf, accessed 24 May 2013.

L.A. Guion, D.C. Diehl, and D. McDonald, 2011. “Triangulation: Establishing the validity of qualitative studies,” Department of Family, Youth and Community Sciences, Florida Cooperative Extension Service, Institute of Food and Agricultural Sciences, University of Florida, FCS6014, at http://edis.ifas.ufl.edu/pdffiles/FY/FY39400.pdf, accessed 24 May 2013.

J.M. Harackiewicz, K.E. Barron, S.M. Carter, A.T. Lehto, and A.J. Elliot, 1997. “Predictors and consequences of achievement goals in the college classroom: Maintaining interest and making the grade,” Journal of Personality and Social Psychology, volume 73, number 6, pp. 1,284–1,295.

A. Hussein, 2009. “The use of triangulation in social sciences research: Can qualitative and quantitative methods be combined?” Journal of Comparative Social Work, issue 1 for 2009, at http://jcsw.no/local/media/jcsw/docs/jcsw_issue_2009_1_8_article.pdf, accessed 24 May 2013.

E.J. Johnson and K. Card, 2007. “The effects of instructor and student immediacy behaviors in writing improvement and course satisfaction in a Web–based undergraduate course,” Mountain Rise, volume 4, number 2, at http://mountainrise.wcu.edu/index.php/MtnRise/article/view/81, accessed 24 May 2013.

J. Johnson, 2011. “Wikipedia goes to class,” Washington Post (29 May), at http://www.washingtonpost.com/local/education/wikipedia-goes-to-class/2011/05/03/AGNF4NEH_story.html, accessed 7 June 2011.

N. Jones, H. Blackey, K. Fitzgibbon, and E. Chew, 2010. “Get out of MySpace!” Computers & Education, volume 54, number 3, pp. 776–782.http://dx.doi.org/10.1016/j.compedu.2009.07.008

A. Kittur, B. Suh, and E.H. Chi, 008. “Can you ever trust a wiki? Impacting perceived trustworthiness in Wikipedia,” CSCW ’08: Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work, pp. 477–480.

T. Lucassen and J.M. Schraagen, 2010. “Trust in Wikipedia: How users trust information from an unknown source,” WICOW ’10: Proceedings of the 4th Workshop on Information Credibility, pp. 19–26.

A.J. Martin, 2008. “Enhancing student motivation and engagement: The effects of a multidimensional intervention,” Contemporary Educational Psychology, volume 33, number 2, pp. 239–269.http://dx.doi.org/10.1016/j.cedpsych.2006.11.003

S. Mathison, 1988. “Why triangulate?” Educational Researcher, volume 17, number 2, pp. 13–17.http://dx.doi.org/10.3102/0013189X017002013

M.D. Myers and M. Newman, 2007. “The qualitative interview in IS research: Examining the craft,” Information and Organization, volume 17, number 1, pp. 2–26.http://dx.doi.org/10.1016/j.infoandorg.2006.11.001

D.L. Neumann and M. Hood, 2009. “The effects of using a Wiki on student engagement and learning of report writing skills in a university statistics course,” Australasian Journal of Educational Technology, volume 25, number 3, pp. 382–398.

W. Olsen, 2004. “Triangulation in social research: Qualitative and quantitative methods can really be mixed?” Developments in sociology, volume 20, pp. 103–118, at http://www.ccsr.ac.uk/staff/Triangulation.pdf, accessed 24 May 2013.

T. Parscal, 2011. “VisualEditor/software design,” MediaWiki (14 September), at http://www.mediawiki.org/w/index.php?title=Visual_editor/software_design&oldid=434611, accessed 20 September 2011.

“Public Policy Initiative,” Wikimedia Outreach (29 July 2011), at http://outreach.wikimedia.org/w/index.php?title=Public_Policy_Initiative&oldid=22534, accessed 13 September 2011.

B. Read, 2006. “Can Wikipedia ever make the grade?” Chronicle of Higher Education, volume 53, number 10 (27 October), at http://chronicle.com/article/Can-Wikipedia-Ever-Make-the/26960, accessed 24 May 2013.

L. Richards and J.M. Morse, 2007. Readme first for a user’s guide to qualitative methods Second edition. housand Oaks, Calif.: Sage.

H.T. Shapiro, 2005. A larger sense of purpose: Higher education and society. Princeton, N.J.: Princeton University Press.

J. Snyder, 2010. “Wikipedia as an academic reference: Faculty and student viewpoints,” AMCIS 2010 Proceedings, at http://aisel.aisnet.org/amcis2010/17/, accessed 25 May 2013.

E. Sormunen, L. Lehtiö, and J. Heinström, 2011. “Writing for Wikipedia as a learning task in the school’s information literacy instruction,” paper presented at the ISSOME 2011 Conference (Abo/Turku, Finland), at https://www12.uta.fi/blogs/know-id/files/2011/08/ISSOME-abstract-Writing-for-Wikipedia-as-a-learning-task-in-the-school.pdf, accessed 24 May 2013.

R.E. Stake, 2010. Qualitative research: Studying how things work. New York: Guilford Press.

D.W. Stewart, P.N. Shamdasani, and D.W. Rook, 2007. Focus groups: Theory and practice. Second edition. Thousand Oaks, Calif.: SAGE.

V. Thurmond, 2004. “The point of triangulation,” Journal of Nursing Scholarship, volume 33, number 3, pp. 253–258.http://dx.doi.org/10.1111/j.1547-5069.2001.00253.x

C. van de Sande and G. Leinhardt, 2008. “The good Samaritan effect: A lens for understanding patterns of participation,” ICLS ’08: Proceedings of the 8th international Conference on International Conference for the Learning Sciences, volume 2, pp. 430–437, and at http://www.lrdc.pitt.edu/pubs/Abstracts/vandeSandeSamaritan.pdf, accessed 24 May 2013.

“VisualEditor,” MediaWiki (22 July 2011), at http://www.mediawiki.org/w/index.php?title=Visual_editor&oldid=420095, accessed 20 September 2011.

K. Wannemacher, 2011. “Experiences and perspectives of Wikipedia use in higher education,” International Journal of Management in Education, volume 5, number 1, pp. 79–92.http://dx.doi.org/10.1504/IJMIE.2011.037756

K. Wannemacher and F. Schulenburg, 2010. “Wikipedia in academic studies: Corrupting or improving the quality of teaching and learning?” In: M. Ebner and M. Schiefner (editors). Looking toward the future of technology–enhanced education: Ubiquitous learning and the digital native. Hershey, Pa.: Information Science Reference,, pp. 295–311.

“Wikipedia” (20 September 2011), at http://en.wikipedia.org/w/index.php?title=Wikipedia&oldid=451438800, accessed 20 September 2011.

A. Zusman, 2011. “Issues facing higher education in the twenty–first century,” In: P.G. Altbach, P.J. Gumport, and R.O. Berdahl (editors). American higher education in the twenty–first century: Social, political, and economic challenges. Third edition. Baltimore, Md.: Johns Hopkins University Press, pp. 109–149, at http://www.educationanddemocracy.org/Resources/Zusman.pdf, accessed 24 May 2013.

 

Appendix A

WestEd instructor interview protocol

  • Basic demographic information
    • Name
    • Position
    • Years of teaching
    • Background
    • Research interests
  • Motivation for their involvement in the project
    • Why did you decide to participate in this project?
  • Use and comfort with various forms of technology
    • Before becoming involved in this project, how would you describe your familiarity with various forms of technology such as wikis, blogs, social media (e.g., Facebook), blackboard platforms etc. ...?
    • Before becoming involved in this project, to what extent did you use these technologies for personal reasons?
    • Before becoming involved in this project, to what extent did you use these technologies in preparing to teach, or in actual classroom instruction?
    • Do you believe your personal or instructional use will change at all as a result of participating in the project?
  • Their personal familiarity with and use of Wikipedia before and during the project.
    • How frequently and for what reasons did you use Wikipedia before starting this project?
    • Did this change after you became involved with this project?
  • Perception of the value of Wikipedia
    • What do you believe are appropriate uses for Wikipedia? For what purposes should teachers and students NOT use Wikipedia?
    • How would you evaluate Wikipedia in terms of clarity, timeliness, and accuracy, appropriateness of material, neutrality, comprehensiveness? Do you think your views of Wikipedia have changed at all as a result of participating in the project?
  • Wikipedia assignment
    • What types of Wikipedia assignments did you give to your students?
    • How did these assignments compare to a more traditional writing assignment in terms of the level of time and effort it required for you, as the instructor, prepare the assignment?
    • How did these assignments compare to a more traditional writing assignment in terms of the level of effort it required for your students?
    • How did you organize students to work on the Wikipedia assignments? (e.g., groups, individuals)
    • How did you monitor student use of Wikipedia during the project?
    • Which Features of Wikipedia did you use, if any, to monitor students’ work on the assignment?
    • Process used to evaluate student work related to Wikimedia project
      • How did you evaluate student work related to Wikipedia project? What assessment tools did you use?
      • How did this assignment compare to a more traditional writing assignment in terms of the level of effort it required for you, as the instructor, to evaluate students’ work?
    • Extent to which they believe content of course topic on Wikipedia was improved by their participation and the participation of students
      • What is the value added, if any, of using Wikipedia in academic instruction?
      • Compared to a traditional writing assignment, to what extent did the Wikipedia assignment impact:
        • Student writing and editing skills
        • Student media literacy
        • Students’ ability to evaluate and validate sources
        • Students’ ability to collaborate
        • Students’ communication skills
        • Students’ ability to conduct research
    • To what extent did the Wikipedia assignment have an impact on students’ adaptability that is needed when working within an established online community?
    • As a result of your participation in this project, and the students’ assignment in this class, do you believe the content on Wikipedia has been improved?
  • Instructional barriers associated with using Wikipedia in their instruction and were elements missing that WMF should have improved to facilitate the use of Wikipedia as a teaching tool.
    • What instructional barriers associated with the use of Wikipedia hindered your classroom instruction? (e.g., Learning material, strategies to facilitate learning, learner interaction, assessment)
    • What would you recommend for the Wikimedia Foundation to add or enhance to facilitate the use of Wikipedia as a teaching tool?
  • Interest in continuing use of Wikipedia/classroom, personal, etc.
    • Do you expect to use Wikipedia in your classroom and for your personal use in the future? Why or why not?
    • How has your work on this project changed the way you view or use Wikipedia?
  • Usefulness of various support mechanisms provided by Wikimedia
    • How would you evaluate the support that Wikimedia has provided during the project? (Examples: Campus Ambassadors, Online Ambassadors, bookshelf materials, Wikipedia curriculum, Camps Team Coordinator, communications)
  • Perceptions about the model developed by the project for use in other universities and in other content areas.
    • How would you evaluate the model used for this project?
    • Do you recommend any changes to the model before using it in other universities and content areas?
    • Was this model suitable for domain of study, public policy?

 

Appendix B

WestEd student survey questions

1. First name
2. Last name
3. Email
4. Wikipedia ID
5. Age:

  • Under 18
  • 18–24
  • 25–34
  • 35–44
  • 45–54
  • 55–64

6. What year are you in school?

  • Freshman
  • Sophomore
  • Junior
  • Senior
  • Graduate student, master degree
  • Graduate student, doctoral degree

7. What is your field of study?

  • Business
  • Sciences
  • Education
  • Engineering
  • Humanities
  • Law
  • Public Policy
  • Other, please specify

8. How many classes have you taken in the field of study of your current class?

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10

9. Prior to taking this course, how frequently did you use the following:
options: Not at all, Once or twice a year, Once or twice a semester/quarter, Once or twice a month, Once or twice a week

  • Wikipedia
  • Social Media (e.g., Facebook, Twitter)
  • Blogs
  • Blackboard platforms
  • Skype
  • E–mail

10. Prior to taking this course, how frequently did you use Wikipedia to do the following:
options: Not at all, Once or twice a year, Once or twice a semester/quarter, Once or twice a month, Once or twice a week

  • Academic research (course–related)
  • Background information (personal use)
  • To find images
  • Write articles
  • Fix typos

11. Have you worked on a wiki article before?

  • I have never heard of a wiki
  • I have heard of a wiki but never read one
  • I have read a wiki
  • I have done coursework using a wiki
  • I have collaborated on a wiki
  • Other, please specify

12. Prior to taking this course, how frequently did you do the following:
options: Not at all, Once or twice a year, Once or twice a semester/quarter, Once or twice a month, Once or twice a week

  • Used collaborative software (e.g., wikis)
  • Used web based office suites (e.g., Google docs)
  • Participated in an online community
  • Participated in a group chat (e.g., IRC)
  • Posted to a blog

13. In learning how to use the tools or participate in the activities described in Question 12 (above), how often did you use the following:
options: Not at all, 2, 3, 4, Frequently

  • IRC chat room
  • Printed materials (e.g., brochures)
  • Online materials
  • Help videos
  • Course page on the WikiProject

14. Have you worked on a wikipedia assignment this semester?

  • Yes
  • No (skip to end)

15. To what extent did these skills improve as a result of Wikipedia assignment(s)? Please check all that apply.
options: Not at all, 2, 3, 4, To a great extent

  • Writing skills
  • Research (evaluating resources)
  • Wiki editing
  • Evaluating wikipedia article quality
  • Technology skills
  • Collaboration
  • Adaptability for working within an established online community

16. Have any of your other skills improved as a result of this Wikipedia assignment? If so, please describe.

17. How was the classroom organized to work on the Wikipedia assignment(s)? Check all that apply.

  • Individuals
  • Pairs
  • Small groups (3–5)
  • Big groups (6–10)

18. Which of the following tasks were assigned to you during your Wikipedia assignment(s)? Check all that apply.

  • Reviewing articles
  • Improving text of existing articles
  • Writing new articles
  • Creating visual aids
  • Performing research
  • Other, please specify

19. How many total hours did you spend working on your Wikipedia assignment(s)?

  • 1–3
  • 4–6
  • 7–9
  • 10–12
  • 13–15
  • 16–18
  • 19–21
  • Other, please specify

20. What is your impression of the quality of articles on Wikipedia with regard to the following?
options: Poor, 2, 3, 4, Excellent

  • Scope of topics
  • Timeliness
  • Reliability
  • Neutral point of view
  • Readability
  • Completeness
  • Appropriateness of images

21. To what extent did the following factors motivate you during your Wikipedia assignment(s):
options: Not at all, 2, 3, 4, To a great extent

  • Completion of coursework and achievement of a good grade
  • Collaboration with an existing online community
  • Interest in how knowledge is created and communicated in an online worldwide project like Wikipedia
  • Contribution of information for public use through a worldwide project like Wikipedia
  • Interest in learning a new technology skill
  • Ability to show family and friends final class product
  • Opportunity to share work with a global audience

22. What is your impression of the Wikipedia community? Check all that apply:

  • Petty
  • Difficult to work with
  • Motivated by self–importance
  • Unresponsive
  • Open to new ideas
  • Motivated by the impact of Wikipedia on the world
  • Helpful
  • Supportive
  • Competent
  • I have not formed an opinion

23. Would you be interested in becoming a Campus Ambassador?

  • Yes
  • No
  • Maybe

24. Please rate the usefulness of the Wikipedia assignment to your future career.
options: Not useful at all, 2, 3, 4, Very useful

25. We are interested in understanding how traditional assignments differ from Wikipedia assignments. A traditional assignment would be a paper of similar length to your Wikipedia assignment. Please use the choices below to compare the impact of traditional and Wikipedia assignments on the following:
options: Traditional assignment was significantly better, Traditional assignment and Wikipedia assignment provided approximately the same benefit, Wikipedia assignment was significantly better

  • Improving your writing skills
  • Improving your research skills
  • Improving your editing skills
  • Improving your collaboration skills
  • Improving your technology skills
  • Improving your critical analysis

26. To what extent do you agree with the following statements:
options: Not at all, 2, 3, 4, To a great extent

  • I prefer being accountable to my school professor rather than a global audience for an assignment.
  • I prefer working collaboratively with an online community rather than working in student groups.
  • Academic papers helped me understand the course topic more comprehensively than Wikipedia assignments.
  • Traditional assignments were more relevant to my degree than Wikipedia assignments.
  • Traditional assignments took more time than Wikipedia assignment.
  • Wikipedia assignments helped me to better understand how knowledge is created compared to traditional assignments.
  • Wikipedia assignments helped me to better distinguish between primary and secondary sources compared to traditional assignments.
  • Wikipedia assignments helped me to better recognize poor quality Wikipedia articles.
  • Wikipedia assignments gave me a better understanding of what constitutes credible information sources compared to traditional assignments.
  • Wikipedia assignments helped me to better evaluate the neutrality of a document compared to traditional assignments.

27. If you had the choice in your next assignment, which would you choose? Briefly explain why.

  • An assignment that does not involve the use of Wikipedia
  • An assignment that does involve the use of Wikipedia

28. Please rate the usefulness of the materials provided to support your work on the Wikipedia assignment(s).
options: Not useful at all, 2, 3, 4, Very useful

  • IRC chat room
  • Printed materials
  • Online materials
  • Help videos
  • Course page on the WikiProject

29. How often did you use the materials provided to support your work on the Wikipedia assignment(s).
options: Rarely, 2, 3, 4, Often, N/A

  • IRC chat room
  • Printed materials
  • Online materials
  • Help videos
  • Course page on the WikiProject

30. Please indicate any other materials that you found useful in completing your Wikipedia assignment(s).

31. Please rate the usefulness of the materials presented by the Campus Ambassador.
options: Not useful at all, 2, 3, 4, Very useful

32. How helpful was the Campus Ambassador?
options: Not helpful at all, 2, 3, 4, Very helpful

33. How approachable was the Campus Ambassador?
options: Not approachable, 2, 3, 4, Very approachable

34. How often did you consult the following people for assistance while working on the Wikipedia assignment?
options: Rarely, 2, 3, 4, Often, N/A

  • Campus Ambassador
  • Online Ambassador
  • Instructor
  • Fellow Students

35. Would you contribute to a Wikipedia article in the future?

  • Yes
  • No
  • Maybe
  • Additional comment

36. Which of the following barriers would hinder you from contributing in the future to a Wikipedia article?

  • Time
  • Irrelevant to coursework
  • Lack of payment
  • Technically challenging
  • Working environment does not meet my needs
  • Bad experience with this assignment
  • Other, please specify

 

Appendix C

Wikimedia student focus group interview guide

  1. I. Technology familiarity
    1. What forms of technology or social media do you regularly use, and of those, which have you used in a classroom environment?
    2. How much Wikipedia experience do you have?
      1. How often do you read Wikipedia? When do you generally use it in your research process?
      2. Have you edited an article before this class? Why or why not?
      3. Do you have a Wikipedia username?
  2. Description of Wikipedia Assignment
    1. How do you feel about this assignment?
      1. What excites you?
      2. What concerns you?
      3. What was your initial reaction when you heard you would be writing a Wikipedia article?
    2. How do you think this assignment is different from a traditional assignment?
      1. What are the pros and cons of this assignment vs. a traditional assignment?
    3. What skills relating to the Wikipedia assignment do you think may apply to your future career?
      1. Using a wiki?
      2. Collaboration?
      3. Online communication/etiquette?
  3. Motivations for a Wikipedia Assignment
    1. What motivates you most about this assignment? (try to not prompt these unless the students are really stuck)
      1. getting a good grade?
      2. contributing to Wikipedia mission “sum of all knowledge”?
      3. contributing to a resource you use?
      4. the satisfaction of writing a useful assignment?
    2. How are your motivations for doing this assignment different from your motivations for traditional assignments?
    3. What would make make your experience working in WP better?
      1. What do you hope to get out of it?
      2. What would make you want to contribute after the class?
      3. What would make you not want to contribute?
  4. Project Support Materials
    1. What have done so far in the class related to Wikipedia?
      1. What will you do later in the academic term with Wikipedia?
    2. What is your impression of the Campus Ambassadors?
      1. In your understanding, what is their role?
      2. Are they helpful? Are they knowledgeable?
      3. How were their presentations?
    3. What is your impression of the Online Ambassadors?
      1. In your understanding, what is their role?
      2. Are they helpful? Are they knowledgeable?
    4. Were the printed and online materials useful? (e.g., brochures, handouts, videos)
    5. What other materials would be beneficial to you?
  5. If there’s time
    1. Do you think this assignment fits in the course? Why/Why not?
      1. Does it make sense in the broader perspective of the class?
      2. Is it out of place, or does it help bring the application of skills together with the information in the course?
  6. Wrap up
    1. How has your perception of Wikipedia changed?
    2. Do you think Wikipedia belongs in higher education classrooms at all? Why/why not?
    3. Is there anything else you'd like to add that we haven’t already covered?

 


Editorial history

Received 16 January 2013; revised 1 May 2013; accepted 8 May 2013.


Creative Commons License
“Assigning Wikipedia editing: Triangulation toward understanding university student engagement” by Amy Roth, Rochelle Davis, Brian Carver is licensed under a Creative Commons Attribution–NonCommercial–ShareAlike 3.0 Unported License.

Assigning Wikipedia editing: Triangulation toward understanding university student engagement
by Amy Roth, Rochelle Davis, and Brian Carver.
First Monday, Volume 18, Number 6 - 3 June 2013
http://firstmonday.org/ojs/index.php/fm/article/view/4340/3687
doi:10.5210/fm.v18i6.4340





A Great Cities Initiative of the University of Illinois at Chicago University Library.

© First Monday, 1995-2014.