Ahrashs Blog

The New York Times recently published an article entitled, “Scholars Test Web Alternative to Peer Review.” The article describes how the mainstream journal, Shakespeare Quarterly, opened up the review process for a special edition to whomever was interested, and was even able to compare the results of that process to a more typical panel of appointed experts by way of understanding the risks and rewards for authors. The results were positive all around, and now other humanities journals are looking to experiment as well.

This is good news. The article does a good job of comparing a few of the key distinctions between traditional peer review and more open methods. To wit:

“The traditional method, in which independent experts evaluate a submission, often under a veil of anonymity, can take months, even years. Clubby exclusiveness, sloppy editing and fraud have all marred peer review on occasion. Anonymity can help prevent personal bias, but it can also make reviewers less accountable; exclusiveness can help ensure quality control but can also narrow the range of feedback and participants.”

At its heart, I think much of the question of peer review boils down to whether people fundamentally believe in the core precept of transparency and replicability for ideas and research. In science, anyway, one of the first things you learn is that you need to build a corpus of evidence before you can start to claim something as true. Obviously that applies for any given study, where you should perform the experiment in such a manner that you can show that it is statistically valid, but it also applies across studies, where people should be given sufficient information to determine for themselves whether the claims are valid, even to the point of replicating the work wholesale. We have not adhered to this core principle for a long time, most notably in our failure to publish all of the raw data associated with any given study.

In the end, we have to ask ourselves as scientists and researchers, what is it we fear from transparency?

Another misperception many academics have about open peer review is that there will be hordes of lay people with little understanding of the subject matter who will see fit to comment on or attack their work. While we can certainly expect that some non-experts may read and comment on an article, the fact is that most people don’t read things that are not interesting to them or that they cannot understand. These days, expertise in any given subject area is pretty broadly distributed…. there simply aren’t enough faculty jobs around to corral all of the experts into academia. And besides, most of us spend significant chunks of our time pursuing interests as amateurs rather than professionals. I love to cook, but I have only rarely harbored aspirations to be a professional chef. Examples abound of “citizen science” initiatives where surprisingly sophisticated research is performed by volunteer communities, usually via a distributed network. While some forms of research and scholarship are not likely to lend themselves so well to this approach, it nonetheless proves that interest in seemingly esoteric research questions and capacity to meaningfully engage abounds among the masses.

Fortunately, P2PU is being built from the ground up with transparency and open peer review as core operating principles. It is this fealty to these principles that is making it possible for us to explore tricky and long-standing questions around authentic assessment (especially for “soft skills”), alternative routes to accreditation, and new pedagogies for deep engagement with learning. If our goal is to actually facilitate learning, for everyone, then we need to know if learning is not happening. The relevant data should be available to anyone so that anyone can get in the business of improving the outcomes. It would not surprise me if the insights gained at P2PU ultimately inform educational practice generally, on and offline. Indeed, these insights, or at least insights gained through a similarly comprehensive commitment to open peer review and data-sharing, will probably be the only research outcomes worth paying any attention to.

We have been trying to decide on a few key skills to experiment with in terms of assessment for the upcoming round of courses for the School of Webcraft. After deliberating with a variety of people, we have settled on “Good at answering people’s questions,” and “Good community builder.” There are other so-called “soft skills” and “hacker’s habits” which are equally fascinating and listed here. These various skills of interest to the web development community are not mutually exclusive, so we’re hopeful that our work on these two initial skills will pave the way for addressing the other skills in short order.

When considering the “Good at answering people’s questions” skill, it was reasonably straightforward to develop some key elements of that skill which are amenable to measurement and nearly universally applicable. For example, it is hard to imagine that clarity, one of the component parts of this skill, is not necessary in any circumstance where answering people’s questions is desirable.

However, in considering the “Good community builder” skill, it quickly became apparent that the component parts that might identify someone as being an excellent community builder are likely to vary depending on the circumstances. For example, someone who is adept at building a virtual community of web developers is likely to exhibit a different suite of behaviors than a person who is adept at building a community of neighborhood activists. Even if we restrict ourselves to the open web-development space, the specific manifestation of “community building skill” will probably depend on the type of project, the stage of project development, and other factors.

At first, this might seem to be a real problem. How can you evaluate whether or not someone is a good community builder if the component parts of being a good community builder are not universal? However, in reality this shouldn’t be an issue. A “community builder,” just like a “business leader,” or “writer,” or “actor,” or any other number of skilled professions and roles, is not a homogeneous category. When a person is identified as an “excellent writer,” the natural follow-up question is, “What kind of writer?” We recognize that excellent writers come in many flavors, even as we recognize that there is some shared expertise among the different categories of writers.

Here are a few quick diagrams to illustrate this point further, one that visualizes the writing analogy, and two focused on community building, where one diagram distinguishes among different activities, and the other diagram distinguishes among different stages of project development, any one (or more) of which might lead someone to be recognized for their community building skills.

Clearly, these are highly stylized diagrams. In reality, the contexts will sometimes overlap a lot, sometimes not at all, and they will vary in their generality to the “community builder” space, meaning that the interior circles will be bigger or smaller depending on the situation.

Regardless, I think the way this makes sense for P2PU is to simply acknowledge that “community building” is a heterogeneous construct, and that we will accommodate that heterogeneity by design. We can do that in a number of ways, including:

• allow people to accumulate certain numbers of sub-skills or behaviors to certain thresholds, at which point they are awarded some level of community-building distinction. For example, if there are 30 different ways that people can exhibit community-building behavior, perhaps mastery of any 10 of those ways garners a “community builder” mark.
• categorize subsets of behaviors into logical groupings according to specific contexts, and then award the “community builder” mark to those people who exhibit every behavior in a category.
• simply ask members of the community who seemed most like a “community builder,” acknowledging that different people and communities may have different reasons for that determination.

There are others as well. The great thing about P2PU in this case is that it is naturally set up to provide correlative evidence of those skills and behaviors that are associated with community building. In other words, if someone is flagged as being a good community builder, we are in a position to study what types of activities that person does in that community which lead people to call him or her a community builder. The possibilities here are nearly endless! You can help frame up some of the ways that we think about community building by visiting the wiki and jumping in.

In any group of students, in any educational setting, we can assume that different students have different motivations towards learning. Some students are going to be deeply interested in the material, striving to learn no matter how challenging. Some students will have very specific needs, usually motivated by some project or responsibility which they are seeking to complete. Some students are going to be motivated primarily by the grades and degrees and other externally visible metrics that might support their larger ambitions. Still other students will just go through the motions of learning, perhaps to fulfill requirements or expectations set by others. And in some cases you may have students actively resisting all opportunities for learning or engagement. We are fortunate in P2PU that we are unlikely to encounter the last two categories of student given that this is an informal, opt-in system; however, as P2PU grows and meaningful routes to accreditation and competency are developed, this situation might change… something to watch out for.

Here’s one way of visualizing this diversity for P2PU:

In the diagram above, the blue ellipse is the “P2PU community,” which can be construed however you like. For the sake of argument, let’s pretend it contains all registered and potential participants, and all running courses, in a P2PU “semester”.

The small circles represent registered individuals in the system. Each individual has a particular portfolio, including a suite of interests in P2PU and a suite of prior experiences/expertise which might affect their choices of courses as well as the manner in which they engage with each course. Much of this diversity will be opaque – those individuals are rendered in the background blue color. However, some individuals will have opted into P2PU-supported pathways which identify their particular orientations. In this case, we can imagine that the colors represent the expectations each individual might have for “certified outcomes” of any given course, as follows:

Note that not every course will be able to easily accommodate the full diversity of aspirations. For example, if a course is designed to require high levels of peer engagement and collaboration, then it may not work for someone to just survey the course. Similarly, some courses may not be set up to enable accreditation or certification in any robust manner.

The dashed squares represent individual P2PU courses. Obviously, the specific placement (and the cap on 16 participants) is arbitrary. There are six total courses represented here. The main point of the diagram is to illustrate that any given course is likely to contain a heterogeneous body of participants, and much of the variation is likely to be invisible. In some cases, it may be necessary to ask potential participants to reveal more about themselves up front so that they can be properly oriented within the structure of the course. But this also presumes that course facilitators will have considered the ways in which the course can differentiate among these different participants and still operate as a cohesive peer group. Since that is likely to be difficult for many presumptive course facilitators, we should also build independent pathways to portfolio-building and accreditation which students can opt into even without any overt support from their course peers or facilitator. In this case, we will need to provide a lot of general information to potential P2PU participants detailing how they can get more out of courses and how much of what they want is their responsibility to see through. My recent post on the effort:outcomes relationship considers this challenge more closely.

There. That’s the charge. Now we just have to make it real.

The Chronicle of Higher Education ran an article on June 30 entitled, “Is Globally Democratized Learning Always a Good Thing?” by Ben Wildavsky of the Kauffman Foundation. The question is a good one. And while both Mr Wildavsky and the Kauffman Foundation have admirable records of engaging with and supporting educational innovation, I read his post with some dismay given the simplistic and stale straw-men that he used by way of addressing some of his concerns.

Many of my concerns were ably addressed by Geoff Cain in his comments, so I will just add a few more observations. I think Mr Wildavsky would be hard pressed to find anyone who simply dismisses expertise as irrelevant to the educational enterprise. Among the many colleagues I have who are actively seeking to democratize education at a global scale, expertise is highly valued and eagerly sought after. The problem with too much of our formal educational system is that the manner in which that expertise is tapped is inefficient at best and ineffective at worst. And we have decades of solid research pointing to the hierarchical nature of our classrooms, and the resulting pedagogies that we practice, as one of the most important culprits in perpetuating these sorry outcomes. For too long we have been operating on the assumption that education is something we do to people, rather than something we pursue together.

The new DIY, peer-learning, and/or tech-enabled educational opportunities do not eschew expertise; on the contrary, they provide the means for learners to assemble a body of expertise (as resources or people) which best meets their needs and preferences. There is no question that some people will need more guidance in this process than others. Indeed, some people will definitely have to be pushed through an educational program, especially when concerned with knowledge and skills that are deemed crucial to functioning in society (like basic reading and math). But for the majority of us, the democratization of education means that we can pursue and maintain our innate interests in learning, rather than have it denied to us outright or beaten out of us in stultifying educational settings that are out of touch, too expensive, and demeaning to all concerned, especially the experts.

In our ongoing work on alternative routes to accreditation, it has become clear that the solutions we provide are going to require a “contract” of sorts with P2PU course participants who desire these higher-value outcomes. I have tried to visually capture the relationship between effort and outcomes in the diagram below.

The “Low-value, high effort” section is presumably of no interest to anyone. We have sayings for this section, like “counting grains of sand on a beach,” and the like.

The “High-value, low effort” section is mostly a fiction, though I would argue that this is essentially what our current higher educational system promises to people. Actually, the promise is more akin to “You have already put in your time, money, and effort – reap the rewards of your performance to date by coming here, acquiring our good brand, and benefiting accordingly.” Note that this category requires that some entity vouch for the quality of the learning outcomes because there is little or nothing more to go on. I would say that P2PU should take pains to avoid this category; it’s a rat hole.

It is the “Value mostly reflects effort” section which is most interesting, and most relevant for P2PU. The key is to develop and honor an honest contract of sorts between P2PU and course participants whereby each party understands what is actually expected of them. For participants, this means recognizing that the “value” of a P2PU course will depend substantially on the person’s ambitions and effort. For P2PU, it means not promising that which cannot be offered; even if P2PU is not the accrediting agent, it is still necessary to facilitate passage towards accreditation if a claim is made that such a thing is possible with P2PU courses.

Thoughts?

This project, like so many projects, would benefit from employing some robust, data-driven, dynamic visualization tools to help us better map out competencies, illustrate relationships among concepts, and share insights. This is always a struggle. If you know of any good tools for taking networked information (like in a relational database) and rendering it easily into a concept map or network diagram, let us know.

As an example, I generated a quick diagram of the content and pedagogy curricular categories using the Text 2 Mind Map application, pasted below. It’s really simple and cool, but it can’t actually diagram networked relationships, only hierarchical ones. Oh well.

From P2PU

Reputation-building seems to be all the rage these days, at least for online communities. This is not a bad thing; indeed, I think that many of us looking at the growing size and personalization of the Web have felt that the development of working reputation systems will be key step. In education, reputation is king, and highly reputed institutions command steep tuition and fees, and highly reputable professors are well compensated for sharing their knowledge. In education, desirability is measured by exclusivity – the lower the percentage of interested people who actually gain access, the greater the perceived reputation (and value) of that educational offering.

It stands to reason that reputation will also be crucial in open education, but here the rules have been changed, since reputation borne of exclusivity is no longer an option. The challenge, then, is to develop reputation systems that are honest, in the sense that the “currency” of reputation is authentic, and that external consideration (and perhaps validation) of reputation is transparent. In its most basic form, such a reputation system will have to demonstrate that a given educational resource (course of study, textbook, learning module, tutor, what have you) in fact improves the targeted skills and knowledge for any given learner. In other words, you need to be able to determine the student’s competency prior to the educational intervention, and then evaluate the gains made, and the difference should be positive. In my mind, this is one of the more important potential ramifications of open educational practice. Seat time and simple exposure to certain types of materials will not be (and really, never has been) sufficient.

So what are our options here? A first step might be to simply list as many web-based reputation systems as we can find. For example, Notemonk, a site that supports virtual study groups and exchanges around books, has a point system to reward people who “contribute to the community.” Similarly, the Huffington Post has a “HuffPost Badge” system to “recognize and empower HuffPost readers and users.” And we are all familiar with ratings systems, such as Digg rankings, eBay seller ratings, customer review (star ratings) for both products and sellers on Amazon, and myriad crowdsourced review sites like Yelp, Zagat, Angie’s List, and more. Some traditional reputation metrics simply migrated online, such as institutional branding (people tend to trust physics lessons from MIT more than from a random physics lessons site), popularity indices (page views, downloads, subscriber numbers, etc), and certification by agencies that position themselves as arbiters of quality (think of the Good Housekeeping Seal of Approval). This last category seems ripe for expansion and innovation, particularly if educators and associated institutions are willing to pay for the service of evaluating and filtering some subset of the vast pool of content on the Internet.

To ground this exercise a bit, we want to consider possible reputation systems with an eye towards specific implementation in P2PU. We are gathering pointers to existing online reputation systems on the Mozilla wiki. Any system might be of interest, regardless of the type of organization using it and the way in which it has been implemented. In the meantime, we are already fleshing some of these systems out so that P2PU can better support their use and experimentation.

One of the growing challenges we face as researchers is managing the deluge of relevant information out there – never mind the irrelevant stuff that also obviously consumes huge quantities of time. So much of what I know about a topic comes from so many different places – conversations, blog postings, peer-reviewed articles, conference presentations… it’s totally overwhelming. And for those of us working outside of traditional academic settings, this problem is even worse since so many of the “authoritative” sites are (still) inaccessible, making less formal modes of communication even more important.

As someone trained in academic discourse, I feel it is important to ground my work in the existing knowledge base. But at the same time, especially given the wide range of my interests, I have little capacity to ensure that I am fully informed of the latest thinking in any one field, at any breadth or depth. This can engender a form of academic paralysis, where I think I might have something interesting to share, but I am afraid that someone else will have already said it and it might be perceived that I am unfairly claiming the idea as my own.

What to do?

Well, first of all, I added a note to the project blog (see to the right) that clarifies that any absence of referencing does not indicate any unique claim by me to the posted ideas. We all know that ideas build on each other, and existing referencing customs are designed to ensure that the generators of prior knowledge are given credit. I believe that credit will be given here as well, but perhaps as an after-the-fact acquisition from the broader community instead of as an a priori review by myself.

Second of all, I think we need to reconsider our idolization of the idea originators. It is my sense that people naturally find ways to distinguish themselves and their ideas from other people. The harder task is finding and building on the common ground. Besides, if someone generates a lot of new(ish) ideas, that person will acquire recognition regardless of what legal or community norms are in practice. In fact, I would submit that ideas are relatively cheap – it is their contextualization and implementation that is difficult and worthy of support. In the Internet age, we can imagine systems that reward “idea movers” more than “idea generators.” I suppose we have to be watchful for simply rewarding whomever shouts the loudest, but the archive that is the Internet will help us sort out who has done most to conceptualize or promote an idea, without getting too bogged down figuring all of that stuff out beforehand.

Finally, I am hoping that anyone and everyone interested in this project will contribute to a crowdsourced reference database, growing here. Ideally, references listed here should include links to the particular postings (or other media) where the reference seemed relevant. There might also be some cool way to accomplish similar aims using Zotero, semantic wikis, or the like. Feel free to make suggestions. In the meantime, instead of taking offense if you or your favorite reference is left out somewhere, just add it in!

I’ve uploaded a basic description of the P2PU/Mozilla Open Accreditation project and the tentative scope of work to the wiki. Check it out, share insights, get involved.

Many of you already know that P2PU and Mozilla have teamed up to develop pathways to competency in open web skills, or web craft. The idea is that many potential participants in P2PU are seeking to do more than simply satisfy random or hobby urges – in many cases, they are seeking bona fide courses of study for development of competency in areas of possible employment or other tangible uses. The initial focus is on open web skills, or web craft. Several of us are hashing through these ideas and seeking engagement and feedback from the broader P2PU and Mozilla communities.

In that vein, this blog will be a forum for me (Ahrash Bissell, project lead on open accreditation for P2PU/Mozilla) to share my musings, post questions and entreaties, and otherwise just keep people up to date. I expect I will be blogging about authentic assessment, open accreditation, effective pedagogical models, peer learning, digital media and learning… well, blogging about a whole mess stuff related to Peer to Peer University, open education, and assessment/accreditation.

Much of the work on this project will be taking place on the P2PU section of the Mozilla wiki.

Feel free to get in touch!

-Ahrash