Ahrashs Blog

OK, so this is a lengthy post, but we are now approximately one week away from the start date of the Fall 2010 P2PU course offerings, and there is much to say. We have an incredible slate of courses to choose from, including a solid corpus of courses occupying some portion of the School of Webcraft curriculum. Exciting stuff!

And we believe that we are largely still on track to launch an initial test of some of our assessment work. To sum up the work to date:

• we are compiling a variety of perspectives on the opportunities and risks in online, authentic assessment (as well as the related process of accreditation),
• we have some initial schemas in place for understanding how the activities within P2PU can benefit from and foster assessment designs,
• we have a list of key skills related to web development which transcend core knowledge and instead illustrate the “soft skills” or “hacker’s habits” which define great web developers,
• and we have some initial specs for how we might actually assess those skills in the P2PU ecosystem.

In parallel, the P2PU team has generated a fantastic set of resources (and an associated boot camp, of sorts) for course facilitators, including pointers for incorporating relevant assessments and considering how they want to authenticate learning outcomes. We are collaborating with the folks at Mozilla, MacArthur, Carnegie, and elsewhere to design new ways of personalizing learning and accreditation, and for adapting tested reputation and evaluation systems for P2PU. It seems that things are just getting started around here…

Given that we are now implementing many of our initial designs, it seems useful to step back and think a bit about the different categories of assessment that we are dealing with, and how those categories affect our priorities and plans. Here is an initial attempt:

Automated data capture

This category encompasses all of the data that we can capture as a byproduct of people using the P2PU site (and, perhaps, other sites as well). The data might include the number of logins or page views (as a proxy for general interest), the number of courses taken, the number of uploaded resources, the number of comments to the site, etc. We might also be able to leverage back-end stats (like Google metrics) as a way to gauge broader interest and impact for certain things, like course happenings and participant creations.
For the most part, these are weak assessment metrics in that they leave a lot of room for different interpretations, and they only rarely have any direct relevance to a skill or goal that we are interested in measuring. Nonetheless, especially at scale, they may provide powerful evidence of certain types of outcomes, and they are essentially unique to online learning platforms.

Integrated and targeted data capture

This category consists of those tools and operations that are embedded in the P2PU platform and are generally systemic, as opposed to course-specific. For example, we can enable commenting features for community posts, or rating systems for courses, and so on.

These types of activities require us to actually encode the relevant tools into the system, and they can be tailored to be more less assessment oriented. Ideally, the implementation should be somewhat generic and adaptable so that the same basic tool can be used for different purposes depending on the need. As an example, we can imagine that a course facilitator might ask participants in the course to contribute a series of blog posts to the P2PU site on a specific topic, and also ask those course participants to critique those posts. The critiquing tool can be the same tool that is used in a generic fashion to “like” or “dislike” comments about P2PU, but a different version of the tool would appear for registered users of that specific course. We are working on implementing something like this already for this next round of courses.

Bolt-on assessments

This category encompasses all of the “typical” types of assessment that we usually think of: tests, quizzes, essays, etc. These are assessments are primarily designed to assess someone for some specific knowledge or skill. As such, they only make sense in context, which means that they are likely to be specific to an individual course or assignment.

Right now, the easiest way to incorporate such assessments is to do so independently; for example, a course facilitator can easily ask participants to craft an essay as an assignment and then attach it to an email or post it to a discussion board. Deep integration into the P2PU platform is not necessary. However, this “bolt-on” design makes it difficult for the community to benefit from the work, either as curious onlookers or as educational researchers. In particular, it will be difficult to compare outcomes across courses, even if it is the same course offered in different semesters, so much of the wisdom garnered in a prior course (e.g., on better or worse assignments, good pedagogical designs, etc) is likely to be lost.

It may be possible for P2PU to integrate some level of functionality for these types of assessments which enables course facilitators to select from among the various options, adapt the assessments to suit, and then rest assured that the resulting data will at least be available for other uses. But some of you who are reading this will probably be thinking, “Wait, isn’t that basically what a LMS (learning management system) does?” Yes, indeed, that approach starts to look a lot like Blackboard, Moodle, Sakai, and a host of other learning platforms. Maybe there is something different about P2PU that makes this a worthwhile pursuit, or maybe we find a way to get P2PU and Moodle to talk to each other, or maybe we explore the other categories of assessment more fully to see whether this category needs as much attention as the rest of the education community seems to think.

Note that this category includes even very lightweight types of assessment, such as course reviews. If you plan to ask participants in your course whether they liked the course, then you are employing this type of assessment.

Third-party tools

This category differs from the prior category in that the assessments unambiguously leverage external applications and sites. For example, writing courses might utilize Calibrated Peer Review for peer-based assessment, or photojournalism courses might utilize Flickr as a repository and display platform for student work.

In these cases, one thing we would like to be able to do is to facilitate the process by which course instructors can determine which sites and tools on the Internet might best serve their goals. Perhaps we will eventually create a clever guide (or discover one that already exists), but for now simply asking the community for help probably works just as well. Another goal might be to get P2PU and these external applications to talk to each other more easily, via APIs or otherwise. If a course facilitator can direct participants to an external application with some confidence that the resulting output (assignments and associated data) will flow into the P2PU system, that seems ideal.

There may be other categories as well, or perhaps these categories need some additional parsing and clarification. Regardless, these categories can help us to make better decisions about priorities for implementation. To start, most of our effort has been focused on category 2, the integrated and targeted data capture. The reason for this is that course facilitators cannot easily capture these types of data on their own, and these assessments can provide value to both individual courses as well as the P2Pu project as a whole. We will also be working to direct the automated data we can capture to useful ends, such as the portfolio pages for registered P2PU users. We will be working on at least one or two implementations of category 3 assessments, but oriented more towards course feedback and community commentary, rather than traditional assessment designs. We are in the early phases of thinking about category 4 assessments and conversing with other projects and organizations about the possibilities there.