Building off the last post, badges are nothing more than .png files unless they are backed by some assessment and value.  I have been working on defining what assessment looks like in these peer learning, open education environments and it has really been a mind-blowing journey so far. When I first started trying to grasp the task at hand, I realized very quickly that ‘assessment’ mean a lot of different things - it can be the thing that you do to prove that you have learned something (like taking the exam), the design of that thing (question type/writing), the delivery of that thing (paper or online, ‘assessment engines’), the act of comparing the work/answers to some rubric (grading the exam), or the end product itself (the grade).  So needless to say, there are a lot of moving parts to think about when approaching the concept of assessment in general.  But then when thinking about it for these participatory, peer learning environments, there is much further to go.

These environments are intentionally atypical, and with that comes benefits and limitations (in general, but that’s another post, for this one +/- for assessment):

They are open and accessible to anyone with network access.  

What this means for assessment: There will be more people across many different levels and proficiencies that view and/or participate in these courses. The assessments should provide options for these levels and help learners build on their existing skills and develop new ones.  Further, because these courses are open, there is the likelihood that people will float in and out and assessments should allow them to do so, and ‘check’ their knowledge without forcing them to complete the course (if the topic or skills are redundant their existing capacities), but at the same time, assessments should provide milestones to motivate learners to stay engaged in the course as well. 

They are decentralized, meaning that there are not “core” courses or particular paths/sets of courses that people are forced to take.

What this means for assessment: The concept of prescribed degrees does not work here because learners will have unique learning paths across various courses, and even various websites or platforms. Further, the set of courses is not predefined and there may be overlaps, meaning different learners may learn the same skill in different places in different ways.  So the assessments need to be granular enough to capture the learning wherever it occurs, and flexible enough to allow learners to demonstrate the skill in contextual and relevant ways.  Assessments should also be relevant outside of the assessment context itself, and allow people to submit existing work or challenge them to create something meaningful to them to demonstrate competency.

They are peer-driven, and the person organizing the course is not necessarily an expert, but simply guide or facilitator.  Their main goals are to foster a community of learning and provide some scaffolding to guide that community through collaborative learning of a particular topic.  Therefore, there is not the authority figure or typical concept of an instructor.  

What this means for assessment: Short answer, grades won’t work. The simple reason grades ‘work’* in formal environments is that we are preconditioned to expect/accept the instructor-student relationship. The instructor is the expert that pushes information on us and give us top-down ratings of our work and learning**.  But that doesn’t work here.  There are no authority figures - peers are learning from each other and from the interactions and activities. So the assessments need to reflect those relationships and should capitalize on peer assessment as much as possible.  Also, output of the assessment should be more than a flat grade or mark, but should be focused around feedback and guidance.  Also, because these are not expert-driven environments, the assessments need to build in or account for trial-and-error types of approaches.  Learners should be able to learn from the assessment and refine work if they have not met the requirements, etc.

They depend on community development and engagement to be successful. 

What this means for assessment: Again, peer assessment should be incorporated as much as possible.  But we should think about skills and behaviors that support community and build those into the assessment scheme as well.  Perhaps there are lightweight ‘assessments’ based on interactions with peers or automatic assessments and feedback/awards based on behavior through the online learning environment.

I am sure there is more.  And you have noticed that I have intentionally kept badges out of the conversation here.  That’s because badges and assessments are different things.  The badge is the signal of a skill or competency and the assessment is the way to demonstrate/validate those skills.  In our model, each assessment will be tied to a badge, but also in some cases multiple assessments will be tied to a single badge, giving people flexibility in how they demonstrate the skill and earn the badge.

So in summary, for our pilot, the key assessment considerations are:

• Incorporate peer assessment as much as possible
• Provide levels of assessments/badges to meet various needs, as well as help motivate people to build skills or continue participating in courses
• Provide multiple assessment options or paths to the badge
• Assessments should be relevant outside of the learning context - and should allow for submission of existing work, new interesting and relevant work, and/or peer recommendations or nominations. 
• Learners should be able to seek out assessments on their own - nothing forced.  (although there may be cases for automatically assessed and issued badges to promote community behaviors)
• The badge should link back to the work submitted for the assessment, and any feedback or endorsements from the assessors.
• …

I will share the plan over the next couple of weeks and we forge forward.

-E

*I actually started this post with a diatribe against grades and traditional forms of assessment but so many others have expressed it so much better.  I particularly love Cathy Davidson’s (of HASTAC) thoughts on the limitations and obsolescence of grades:

http://dmlcentral.net/blog/cathy-davidson/thought-experiment-why-grade-why-test-what-if

http://www.hastac.org/blogs/cathy-davidson/those-who-dont-grade-learn

http://www.hastac.org/blogs/cathy-davidson/my-response-ny-times-quest-explain-grading

**I have definitely drunk the student-centered kool-aid. From the existing literature and research (not cited here but I can definitely provide), we know that students learn more when they can construct their own understanding of ideas and connect them to their own lives.  We know that people learn MORE and when they can collaborate and interact.  We know that students are more engaged when they have more control within the learning environment. We know that deeper understanding comes from trying out various strategies, getting things wrong, revising, etc.  It’s not enough to have some one push information on us, we need room and flexibility to mash up that information, get our hand dirty, connect it to something that we care about, hear the interpretations of our peers, etc.  I have written and spoken a lot about this to date and I am sure it will make it into the blog over time. But this is one of the reasons I love P2PU and other social learning efforts that recognize and embrace this shift to student-centered, participatory learning.  It’s the future man. 

I am currently working with Peer-2-Peer University (P2PU) and Mozilla Drumbeat to integrate assessment and badges into the open and peer learning environments on P2PU, specifically the School of Webcraft. We’ve been doing a lot of thinking about this and I am finally getting around to capturing my thoughts here.  I should get a badge.

What are badges? 

Come on, you’ve seen them before.  Boy Scouts. World of Warcraft. Foursquare.  I do something, demonstrate some skill, defeat some monster, show up in some location, meet some predefined criteria or assessment…and I get a badge.  If I know about the badge, I might be motivated to do the necessary behaviors or meet the requirements to get the badge, or if the badge is a surprise, I might be motivated to keep exploring or trying out various things to earn or unlock more badges. Once I have the badge, I can display it so that others can see it and thus demonstrate my skills or achievements.

There are many crossovers here with learning - motivation, feedback, exploration, achievement.  

Why do we need badges?

Well, we need something.  Is it badges?  Maybe, maybe not.  But there is no question that we need an alternative form of assessment and certification (although I hate that word…it conjures up images big, mean Microsoft gorillas). Here are a few reasons why we need a change:

• In the current system, the institutions (schools, universities, etc.) have the all the control. They decide what types of learning are “official” and what “counts”.  But most learning doesn’t happen within those confines and constraints and there are lots of examples of people learning outside of the system: open education courses and materials, afterschool programs, peer discussions, books, Wikipedia, the Web in general, LIFE…learning happens everywhere.  But it only counts if it happens through an institution.  Why? Why shouldn’t the learner have control?
• Current models of assessment (grades, rankings, etc.) currently don’t work well for many kinds of learning - in fact, many argue that they don’t work well for most learning.  In peer learning environments, grades and rankings do not encourage participation and information sharing, and in fact, can constrain the interaction and learning.  In informal learning environments, these models make it feel like school, squashing the inherent value and engagement.  In many open education environments, there is not often a dedicated instructor or authority figure to issue the top-down grade. And so on.
• There are so many important skills and competencies, some age-old and some new(ish) in today’s world, that are not currently captured or acknowledged. Things like the often referenced 21st Century Skills, or New Media Literacies, which cover everything from information organization and evaluation, to negotiation and trial-and-error prototyping. Or the “soft” skills like critical thinking and teamwork.  None of these skills are captured in my credit, grade or degree.  And yet, these skills are critical to most careers and are often some of the key things that employers are looking for. As a learner, it is difficult, or impossible, to know to seek out or hone these types of skills because they aren’t acknowledged or encouraged…and yet they will be glaringly apparent the first time I fub up in a critical situation that involves one or more of these competencies. When I am applying for a job - my resume and education history tells potential employers nothing about my full set of skills and if I have any of these other competencies. And when I am looking to hire someone, I have to come up with clever questions to try to get a complete picture of someone (above and beyond the resume and education history which everyone knows is a limited resource) in 30 minutes. 

Badges?!

What if there were badges for various skills that you could collect across learning experiences, carry them with you and then share out to various audiences as needed?  You may earn badges that represent more traditionally recognized behaviors or skills like completing a course or mastering a mathematical model, but you could also earn badges for softer skills like critical thinking, teamwork and information analysis.  You could earn badges from authorities, like Mozilla, from course organizers where appropriate, from peers or even from yourself.  The badges would be associated with assessments that once successfully completed, earns you the badge.  There might be multiple assessment paths to a single badge, giving you the flexibility to have a unique and personalized learning path.  But you could also look at the badges of other people to discover things to learn or try for…or what skills to develop or hone for particular disciplines or jobs.  You could even (possibly) carry the badges back to the institutions with you to get credit or help them cater that experience to your interests and needs. 

So that’s what we are currently exploring.  Of course, there are many unanswered questions, some of which I am sure are springing to mind as you read this.  Questions like: What skills should we assess? Are there skills that are better left unassessed?  What do we want to encourage?  How do we avoid encouraging the “wrong” behavior? Who gets to decide which skills to assess? How much influence should outside stakeholders, such as employers, have on badges?  Should they be able to design assessments and badges that are relevant to them?  How can we let them have a say without creating an imbalance in the system or constraining the learning? How granular should badges be? For example, our HTML5.0 badge is at the level of the entire language mastery, but would we want HTML tag level badges?  What granularity is the right level?  Do badges aggregate into larger or higher level badges? Should badges expire?  How do we deal with skills that need to be refreshed or renewed?  How can the badge system grow with learners? How does the introduction of badges affect learner motivations?  If learners were initially intrinsically motivated, how do we avoid “crowding out” those motivations with an extrinsic badge system? How will people game the system?  How much will they do so? How can we discourage gaming or recognize when it happens? Will these badges translate to formal learning environments? And if so, how?  What would be required to make schools or institutions value or accept badges?  Can we meet those requirements without changing the nature of the learning environments?

There are a lot of questions and a lot of unknowns, but we need a change…we need to give the learners the control.  So this is one way we are hoping to accomplish that.  We are building a badge/assessment pilot in the January session of the School of Webcraft, which is a subset of P2PU courses focused around web development and endorsed by Mozilla.  We are hoping to have a core set of badges and assessments, as well as the initial infrastructure to support the issuing, collection and displaying of badges over the next month (or less).  We plan to learn a lot and start to answer the questions above.  But we can’t possibly answer all of these questions alone.  We hope to encourage more interest in badges and these new approaches, get more people researching them and issuing them (within the same open infrastructure ideally) and figure this out together.

I’ll keep you updated as much as possible here.  So buckle up!  Next up, thoughts on assessment and the open badge infrastructure…

-E