The New York Times recently published an article entitled, “Scholars Test Web Alternative to Peer Review.” The article describes how the mainstream journal, Shakespeare Quarterly, opened up the review process for a special edition to whomever was interested, and was even able to compare the results of that process to a more typical panel of appointed experts by way of understanding the risks and rewards for authors. The results were positive all around, and now other humanities journals are looking to experiment as well.

This is good news. The article does a good job of comparing a few of the key distinctions between traditional peer review and more open methods. To wit:

“The traditional method, in which independent experts evaluate a submission, often under a veil of anonymity, can take months, even years. Clubby exclusiveness, sloppy editing and fraud have all marred peer review on occasion. Anonymity can help prevent personal bias, but it can also make reviewers less accountable; exclusiveness can help ensure quality control but can also narrow the range of feedback and participants.”

At its heart, I think much of the question of peer review boils down to whether people fundamentally believe in the core precept of transparency and replicability for ideas and research. In science, anyway, one of the first things you learn is that you need to build a corpus of evidence before you can start to claim something as true. Obviously that applies for any given study, where you should perform the experiment in such a manner that you can show that it is statistically valid, but it also applies across studies, where people should be given sufficient information to determine for themselves whether the claims are valid, even to the point of replicating the work wholesale. We have not adhered to this core principle for a long time, most notably in our failure to publish all of the raw data associated with any given study.

In the end, we have to ask ourselves as scientists and researchers, what is it we fear from transparency?

Another misperception many academics have about open peer review is that there will be hordes of lay people with little understanding of the subject matter who will see fit to comment on or attack their work. While we can certainly expect that some non-experts may read and comment on an article, the fact is that most people don’t read things that are not interesting to them or that they cannot understand. These days, expertise in any given subject area is pretty broadly distributed…. there simply aren’t enough faculty jobs around to corral all of the experts into academia. And besides, most of us spend significant chunks of our time pursuing interests as amateurs rather than professionals. I love to cook, but I have only rarely harbored aspirations to be a professional chef. Examples abound of “citizen science” initiatives where surprisingly sophisticated research is performed by volunteer communities, usually via a distributed network. While some forms of research and scholarship are not likely to lend themselves so well to this approach, it nonetheless proves that interest in seemingly esoteric research questions and capacity to meaningfully engage abounds among the masses.

Fortunately, P2PU is being built from the ground up with transparency and open peer review as core operating principles. It is this fealty to these principles that is making it possible for us to explore tricky and long-standing questions around authentic assessment (especially for “soft skills”), alternative routes to accreditation, and new pedagogies for deep engagement with learning. If our goal is to actually facilitate learning, for everyone, then we need to know if learning is not happening. The relevant data should be available to anyone so that anyone can get in the business of improving the outcomes. It would not surprise me if the insights gained at P2PU ultimately inform educational practice generally, on and offline. Indeed, these insights, or at least insights gained through a similarly comprehensive commitment to open peer review and data-sharing, will probably be the only research outcomes worth paying any attention to.

We have been trying to decide on a few key skills to experiment with in terms of assessment for the upcoming round of courses for the School of Webcraft. After deliberating with a variety of people, we have settled on “Good at answering people’s questions,” and “Good community builder.” There are other so-called “soft skills” and “hacker’s habits” which are equally fascinating and listed here. These various skills of interest to the web development community are not mutually exclusive, so we’re hopeful that our work on these two initial skills will pave the way for addressing the other skills in short order.

When considering the “Good at answering people’s questions” skill, it was reasonably straightforward to develop some key elements of that skill which are amenable to measurement and nearly universally applicable. For example, it is hard to imagine that clarity, one of the component parts of this skill, is not necessary in any circumstance where answering people’s questions is desirable.

However, in considering the “Good community builder” skill, it quickly became apparent that the component parts that might identify someone as being an excellent community builder are likely to vary depending on the circumstances. For example, someone who is adept at building a virtual community of web developers is likely to exhibit a different suite of behaviors than a person who is adept at building a community of neighborhood activists. Even if we restrict ourselves to the open web-development space, the specific manifestation of “community building skill” will probably depend on the type of project, the stage of project development, and other factors.

At first, this might seem to be a real problem. How can you evaluate whether or not someone is a good community builder if the component parts of being a good community builder are not universal? However, in reality this shouldn’t be an issue. A “community builder,” just like a “business leader,” or “writer,” or “actor,” or any other number of skilled professions and roles, is not a homogeneous category. When a person is identified as an “excellent writer,” the natural follow-up question is, “What kind of writer?” We recognize that excellent writers come in many flavors, even as we recognize that there is some shared expertise among the different categories of writers.

Here are a few quick diagrams to illustrate this point further, one that visualizes the writing analogy, and two focused on community building, where one diagram distinguishes among different activities, and the other diagram distinguishes among different stages of project development, any one (or more) of which might lead someone to be recognized for their community building skills.

Clearly, these are highly stylized diagrams. In reality, the contexts will sometimes overlap a lot, sometimes not at all, and they will vary in their generality to the “community builder” space, meaning that the interior circles will be bigger or smaller depending on the situation.

Regardless, I think the way this makes sense for P2PU is to simply acknowledge that “community building” is a heterogeneous construct, and that we will accommodate that heterogeneity by design. We can do that in a number of ways, including:

• allow people to accumulate certain numbers of sub-skills or behaviors to certain thresholds, at which point they are awarded some level of community-building distinction. For example, if there are 30 different ways that people can exhibit community-building behavior, perhaps mastery of any 10 of those ways garners a “community builder” mark.
• categorize subsets of behaviors into logical groupings according to specific contexts, and then award the “community builder” mark to those people who exhibit every behavior in a category.
• simply ask members of the community who seemed most like a “community builder,” acknowledging that different people and communities may have different reasons for that determination.

There are others as well. The great thing about P2PU in this case is that it is naturally set up to provide correlative evidence of those skills and behaviors that are associated with community building. In other words, if someone is flagged as being a good community builder, we are in a position to study what types of activities that person does in that community which lead people to call him or her a community builder. The possibilities here are nearly endless! You can help frame up some of the ways that we think about community building by visiting the wiki and jumping in.