Badges, credits, points, karma…
June 9th, 2010 at 0:44Reputation-building seems to be all the rage these days, at least for online communities. This is not a bad thing; indeed, I think that many of us looking at the growing size and personalization of the Web have felt that the development of working reputation systems will be key step. In education, reputation is king, and highly reputed institutions command steep tuition and fees, and highly reputable professors are well compensated for sharing their knowledge. In education, desirability is measured by exclusivity – the lower the percentage of interested people who actually gain access, the greater the perceived reputation (and value) of that educational offering.
It stands to reason that reputation will also be crucial in open education, but here the rules have been changed, since reputation borne of exclusivity is no longer an option. The challenge, then, is to develop reputation systems that are honest, in the sense that the “currency” of reputation is authentic, and that external consideration (and perhaps validation) of reputation is transparent. In its most basic form, such a reputation system will have to demonstrate that a given educational resource (course of study, textbook, learning module, tutor, what have you) in fact improves the targeted skills and knowledge for any given learner. In other words, you need to be able to determine the student’s competency prior to the educational intervention, and then evaluate the gains made, and the difference should be positive. In my mind, this is one of the more important potential ramifications of open educational practice. Seat time and simple exposure to certain types of materials will not be (and really, never has been) sufficient.
So what are our options here? A first step might be to simply list as many web-based reputation systems as we can find. For example, Notemonk, a site that supports virtual study groups and exchanges around books, has a point system to reward people who “contribute to the community.” Similarly, the Huffington Post has a “HuffPost Badge” system to “recognize and empower HuffPost readers and users.” And we are all familiar with ratings systems, such as Digg rankings, eBay seller ratings, customer review (star ratings) for both products and sellers on Amazon, and myriad crowdsourced review sites like Yelp, Zagat, Angie’s List, and more. Some traditional reputation metrics simply migrated online, such as institutional branding (people tend to trust physics lessons from MIT more than from a random physics lessons site), popularity indices (page views, downloads, subscriber numbers, etc), and certification by agencies that position themselves as arbiters of quality (think of the Good Housekeeping Seal of Approval). This last category seems ripe for expansion and innovation, particularly if educators and associated institutions are willing to pay for the service of evaluating and filtering some subset of the vast pool of content on the Internet.
To ground this exercise a bit, we want to consider possible reputation systems with an eye towards specific implementation in P2PU. We are gathering pointers to existing online reputation systems on the Mozilla wiki. Any system might be of interest, regardless of the type of organization using it and the way in which it has been implemented. In the meantime, we are already fleshing some of these systems out so that P2PU can better support their use and experimentation.
Tags: accreditation, badges, Drumbeat, metrics
July 6th, 2010 at 11:45 am
It seems to me that you are talking about two different categories of objects – resources, and people. While there are many interesting ways of evaluating resources in a distributed fashion, I don’t think resources can have a “reputation” in the way a person can.
I am very skeptical to your statement:
“In its most basic form, such a reputation system will have to demonstrate that a given educational resource (course of study, textbook, learning module, tutor, what have you) in fact improves the targeted skills and knowledge for any given learner.”
Certainly I am very interested in attempts to measure the “educational value added” of a course, in order to evaluate how we organize courses, and provide a base-line to measure whether we are improving or not. However, many of the resources that I use for learning every day would probably fail this test. I could learn incredible amounts about audio manipulation from Audacity. An article about Eisenhower in Wikipedia could teach me a lot about American politics. I could even analyze porn in order to gain a better understanding of gender dynamics. But how would you ex-ante evaluate that?
The problem is that to evaluate quality, we have to evaluate purpose, but the wonderful thing about resources (especially open ones) is that they can be used in ways never imagined by the creators (or the evaluators).
Reputation of individuals is something quite different – I think this is very interesting and important. In this process, what worries me is when we choose something to be an indicator of something else (the number of thumbs-up as an indicator of your helpfulness in a community, number of articles published as an indicator of your academic standing), and especially when this indicator becomes high-stakes (when a scholarship or your job depends on it), that people will become gaming these indicators (which happens so often in other places, people writing what their professor wants to read, not what they are most interested in – professors thin-slicing their papers, etc)… And then we loose the entire valuable peer-review and feedback mechanism. Would I still give honest feedback to a peer-student, if I knew that my good or bad feedback might help decide whether he got that job or not?
July 14th, 2010 at 1:12 am
Excellent comments Stian. I should have been clearer regarding “resource reputations.” Those unquestionably require sufficient context to understand exactly how and why we might expect a given resource to be effective for a particular audience. The fact that a resource has a great reputation for a given purpose is very valuable, and it does not change the fact that the same resource might be a very poor fit in a different setting or for a different purpose. There is some happy middle ground where sufficient information is provided to give a sense of whether a resource might be appropriate for a particular need, but not so much detail that such determinations become hopelessly fuzzy and burdensome. No one has hit upon the magic formula yet. But having the resources be “open” to such experiments is a key step.
Your concerns about the reputations of individuals are really important, and something that I think the P2PU community will need to start thinking about seriously as we ramp up the social tools. Most academic institutions take on a certain level of responsibility for the credits and recognition they bestow to their students – their reputations rest on continual external validation of those claims. P2PU is too new for us to take on much of that responsibility, unless we are willing and able to exert a substantial level of oversight to prevent people from cheating or whatever. Tricky business, that. In any case, these issues should be, and are, key considerations as we start rolling out mechanisms for reputation-building in P2PU.