I have been intimately involved with P2PU since the first courses started in September 2009, working on supporting course organizers, designing and developing some of the technology, and thinking about the models of learning interactions that we wanted to support. However, I have still not taught a single course myself. Time to change that!

I am currently doing my PhD at OISE, University of Toronto, as part of the Encore lab. Our work is located within the field of Computer-Supported Collaborative Learning, a very fascinating field which I have been spending the last year getting into, and whose major conference I will attend for the first time this summer in Hong Kong. However, when getting into this field, I faced one challenge – OISE does not offer any courses directly on CSCL. Instead, you have to kind of “pick it up” by talking to fellow students, and your supervisor, doing the core readings yourself, etc.

I figured this would be a great topic for a P2PU course. Much of the foundational literature in the course is open access (both because it is a relatively young field, and because many of the core researchers support OA – the core journal in the field is entirely OA, for example, meaning that we would be able to link to key readings without worrying about people being able to access them). In addition, the idea behind P2PU is “peer-learning”, and this would be a true example: I am not organizing this course because I am an expert, but because I want someone to learn with.

I was lucky enough to find Monica Resendes, a PhD student with the IKIT lab, also at OISE/University of Toronto. It’s been very interesting discussing our different ideas about how we should approach this, and I really appreciate not doing this by myself.

To get a feeling for the field, we began by surveying the key academics (as identified by being on the board of ijCSCL, the annual conference committee, etc) and looking at the people who had posted their syllabi online. We found about fifteen faculty who had done this (and often they had course syllabi going ten years back, many times with open wikis so we could see their students’ work as well). This gave us a sense of how core faculty see “the field”, and which topics we should cover. There was much more than could be covered in an eight-week course, so we decided to create a separate “analysis and assessment” course, which will hopefully run sometime in the fall.

We began writing up a course outline on Google Docs, with tentative readings, and a structure for the course. We are doing some interesting things with badges (see also our current badge platform, and the badge whitepaper), as well as with core and network participants, inspired by the Wiley wikis on one hand, and the Massive Open Online Courses (more) on the other.

I hope you will join us – either as a core participant (sign up is open for another week), or as a part of our network (go to the course website, and click Follow). We would love for the discussions in the course to take on a life outside of the course – maybe someone will write a blog entry, record a YouTube video, or edit a wiki page that gets retweeted, reblogged and discussed about far outside the confines of the course. The topics should certainly be interesting to a wide variety of folks.

Here is the introductory video (we’ll aim to do one of these a week, usually much shorter though):

Hope to see you, and looking forward to all the conversations!

Stian

This term, I took a Knowledge Media and Design Institute class on values and design. Part of the class was a group assignment that had us choose a public venue, and do unobtrusive participant observation (this could be a public square, a museum or another place where people congregate). Based on what we found through our ethnographic ministudy, and the readings and ideas from the class, we would come up with a design intervention to further a certain value. My group chose the value of “collaboration”, and used Peer2Peer University as the “venue”.

We were able to do the study in P2PU, because all interactions are archived publicly – the ethics approval for the class would not have allowed us to do research on a class that was protected by password. For me, it was fun having a group of people with a lot of ideas around design and knowledge media, but who did not know much about P2PU. Of course, this was a small project in one class, so it cannot be seen as a full-scale research project, more to get your feet wet with ethnography and design, but it was still a lot of fun, and hopefully could inspire further work.

The group visited several current and archived P2PU courses (here’s an example “notification” that I posted in one of the courses to let them know that we were having a peek). We did not do any rigorous analysis of the data gathered, but from the notes that the group members took, their feeling was that course organizers were trying very hard to promote collaboration and peer-learning, but that participants did not take enough cognitive responsibility, which led to conversations that were very course organizer-centric.

We discussed many possible reasons for this, and brainstormed ways of improving this state of affairs. We came up with a number of ideas, like designing some kind of an “orientation” for new participants, to let them understand better the P2PU principles. In the end though, we wanted to create a design intervention. Based on the readings from class, we thought that perhaps the linear design of the discussion forums, which lead to good ideas being “buried”, could be a factor hindering deeper engagement from the students.

To demonstrate our idea, we were asked to create a “future scenario” video. Creating a video always takes much longer than you think, but it was a fun assignment, and we were lucky to have a very photogenic “protagonist”.
We also wrote a small paper describing some of the literature we consulted, and our thinking around the video.

I am very interested in the theory around visual representations of discourse in online learning. I will be giving a talk about this tomorrow (Wednesday, April 5th, 2011) at the CCK11 course. It will also be an important component of the P2PU course on Computer Supported Collaborative Learning which I will co-organize on P2PU starting later this month.

Thanks to Arlo, Rebecca and Eleonore for a fun collaborative project.

Stian

Existing approaches to course-based OER

There are generally two approaches to course-based “big OER” (institutional OER projects, as opposed to resources released by individual professors or others). The first is the MIT OpenCourseWare approach (which has been replicated by universities across the US, and the world). Given that professors are already developing a set of materials to be used in their face-to-face teaching, let’s grab these and upload to the web. The result is a curriculum, maybe some PowerPoints, sometimes lecture recordings, some quiz sheets, etc. From the perspective of self-learners, this is rarely enough material. Only a fraction of all OCW courses provide lecture recordings, and even if they do, most of the resources listed in the syllabus will be unavailable (books that cost $100s of dollars, articles that are only available through university libraries).

The other approach is more common for distance universities which tend to develop much more of the material by themselves, using a more industrial approach to curriculum development (with teams of subject experts working with instructional designers, web and media specialists, etc). Because they have developed more of the material in-house, and for online presentation, they are able to share more coherent and accessible packages – OpenUniversity UK is a great example of this. MIT OCW Scholar is an attempt at making a few selected OCW courses into more complete packages, with additional resources necessary for students to learn. Most of this is still generated by MIT, but they also link to some outside resources.

Of course, there are also plenty of OER resources which do not take the shape of “college-level 12-week courses”, from projects like Connexions, an online authoring platform for educational modules, to Free High School Science Textbooks in South Africa, and even resources that we often don’t think of as OERs, such as Wikipedia, and Directory of Open Access Journals. However, for an independent self-learner, it can be very difficult to put together a sequence of learning by picking and choosing from these sites, especially in a subject that is not familiar.

The one thing common to most of the approaches listed above, is that they focus on producing and sharing their own materials. In the case of many institutions, it’s a point of pride that “MIT videos”, “Yale videos”, etc. are being watched by people around the world. They are branded products, sharing the “excellence of the institution” with the wider world. The predictable result is that we might have ten or twenty “Economics 101″ courses, all skeletal and incomplete, all containing the material from only one institution. It might be much more beneficial to the world if a Yale professor spent his time improving, and adding to a course created by an MIT professor, instead of just putting out his own material – but that might not bring as much attention and publicity.

This dilemma was one of the reasons why we started Peer2Peer University. Our model is basically based on three pillars. First, course organizers create course outlines that only link to resources that are freely (gratis) available on the web. These resources can be from OpenCourseWare collections, from open access journals, from Wikipedia articles, YouTube videos, newspaper articles, etc. These course outlines are published on P2PU.org, and made available under an open license – anyone can access them, and begin learning by themselves, whether or not a course is running right then, or not. The second pillar is to create a community of learners around this course, which goes through the resources together, discuss the ideas, and support each others’ learning. The final pillar is recognition of learning and accreditation, which we are still experimenting with in several ways.

I wrote my MA thesis about a large project for publishing open courses in China, which resulted in more than 12,000 courses being published by more than 700 universities. When I talked to groups of Chinese students and professors in the open education field, they often complained that the quality of these courses was not high enough, and that students would not be interested in visiting them. I encouraged them to think of these courses as resource collections, and curate curricula that were excellent – find a great video from this course, a great reading from that course, put it all together. I also suggested making this easier, when I was invited to give a talk to the Top Level Courses Resource Portal team, at the Higher Education Press.

Saylor Foundation Free Education Initiative

Today I came across the Saylor Foundation Free Education Initiative, and was extremely impressed. The Saylor Foundation was started by Michael J. Saylor, co-founder of a company called MicroStrategy (apparently an interesting guy), and had assets of around $14 million in 2009. The mission of the foundation is to provide access to a free college-level education for all (they say that high school and post-graduate courses might be coming in the future). Their strategy for achieving this is:

By developing, soliciting, and disseminating free online academic materials in a structured and intuitive format, we will be an alternative and a complement to mainstream education providers, especially for students who cannot take advantage of educational opportunities because they cannot afford them.

They have identified the ten majors with the highest enrolments in the US:

• Biology
• Business Administration
• Chemistry
• Computer Science
• Economics
• English
• History
• Mathematics
• Mechanical Engineering
• Political Science
• Psychology

And for each, they’ve endeavoured to create a full compliment of courses. For example, the Economics major lists fourteen courses, seven in the core program, and seven electives. All but three of these are complete.

Core Program

• ECON101: Principles of Microeconomics (100%)
• ECON102: Principles of Macroeconomics (100%)
• ECON103/MA101: Single-Variable Calculus I (100%)
• ECON104/MA121: Introduction to Statistics (0%)
• ECON201: Intermediate Microeconomics (0%)
• ECON202: Intermediate Macroeconomics (0%)
• ECON203: Econometrics (100%)

Elective Courses

• ECON301: History of Economic Ideas(100%)
• ECON302: Money and Banking / Financial Economics(100%)
• ECON303: Labor Economics(100%)
• ECON304: Economic Development(100%)
• ECON305: Public Finance(100%)
• ECON306: Industrial Organization(100%)
• ECON307: International Trade(100%)

What’s unique about these courses is that they are curations of material freely available on the web, put together in a very well thought-out structure. For example, the course History of Economic Ideas consists of five units. Each unit has a brief introduction, learning objectives, and a list of carefully selected resources. Here is the first unit:

Unit 1: Ancient Economic Thought

• 1.1 Political, Economic, and Social Thought in the Near East and Greece
• 1.1.1 Ancient Egypt to Mesopotamia
• Lecture: iTunesU: University of Wisconsin-Madison: Professor Charles Anderson’s “Political, Economic, and Social Thought: Egypt, Mesopotamia, and Ancient Israel”
• 1.1.2 Greece
• 1.1.2.1 Xenophon
• Reading: Project Gutenberg’s version of Xenophon’s The Economist
• 1.1.2.2 Aristotle
• Reading: Project Gutenberg’s version of Aristotle’s A Treatise on Government
• 1.2 China
• Reading: Wikipedia’s version of Fan Li’s “Twelve Golden Rules and Safeguards”
• 1.3 India
• Reading: Fordham University’s version of selections from Chanakya’s Arthashastra

As you can see, the foundation does not aim to produce all the material themselves, rather they link to resources from OpenCourseWare, Project Gutenberg, Wikipedia, and other sources (some of the math courses link to Khan Academy videos).

I have to include another module from this course, the last one, about visionary thinkers and economic utopias:

• 5.1 Fictional Utopias
• 5.1.1 Sir Thomas More
  • Reading: Project Gutenberg’s version of Sir Thomas More’s Utopia and University of Wisconsin: J.P.Sommerville’s “Thomas More, Christian Humanism, Catholicism, and Utopia”
• 5.1.2 Edward Bellamy
  • Reading: Project Gutenberg’s version of Edward Bellamy’s Looking Backwards, 2000-1887 and Bolling Green State University: Wayne Sneath’s “Edward Bellamy”
• 5.1.3 Modern Discourse
  • Lecture: iTunesU: Walker Art Center: Stephen Duncombe’s “Utopia is no Place: the Art and Politics of Impossible Futures”
• 5.2 Economic Experiments
• 5.2.1 The Shakers
  • Reading: The University of Virginia: Kathleen Hogan’s “The Shakers”
• 5.2.2 The Qarmatians
  • Reading: Bahrain Government’s “al-Qaramita”
• 5.2.3 The Anarcho-Syndicalists
  • Reading: Anarchosyndicalism.net’s “Anarcho-Syndicalist FAQ”
• 5.2.4 Marxists
  • Reading: Marxists.org’s Rosa Luxemburg’s The Russian Revolution, Chapter Six
• 5.3 Buddhist Economics
  • Reading: The Centre for Bhutan Studies’ “Gross National Happiness”

Really exciting stuff – I would have loved to take this course as part of my undergrad!

These course outlines were designed by hired professors – here is an ad in The Chronicle of Higher Education for “College-Level Course Designers for Free Education Initiative”. They are licensed under Creative Commons BY license, and almost all the material is available as very nice HTML pages (except, for some reason, for the reading comprehension questions, and model answers).

There’s also school.saylor.org, a Moodle install where you can take the final exam in each course as a Moodle quiz. I took the final exam in the course mentioned above, about 50 multiple choice questions. I got 72% – a C-, but I didn’t exactly study for it. Not sure what the intention with this site is – it isn’t advertised anywhere, but is perhaps the first step in a process of offering more social features, or a pathway to accreditation.

Here’s Saylor’s Alana Harrington presenting their project as part of the OER University discussions, and she mentions that one of their outstanding problems is the accreditation.

These course resources would work great for P2PU classes – bringing together people to go through the material together (I can imagine some great discussions between people who have been reading about Buddhist economics, anarcho-syndicalists and the Shakers!)

Stian

• @gsiemens In which building of the Banff Centre is #lak11 preconference tomorrow?
• @courosa I fell off the lak course too early, from what I gathered though it’s quite a grab bag. The conference will be interesting #LAK11
• Finally at #Lak11, great place. Already buzzing with conversations and excitement. This will be fun!
• Should be of interest to #lak11 participants RT @eknight A Working Badge Paper (for your feedback and collaboration): http://bit.ly/dG4Q9k
• @gsiemens recommends newcomers in #elearning field to “go data” – that’s where the careers will be. All the big companies hiring. #lak11
• #lak11 “doing data analysis “to” the learner, vs. “for” the learner”
• #lak11 One of my concerns is that a lot of the “open learner models”, etc, is based around guided discovery, tutoring. How about 1/2
• #lak11 more open-ended inquiry based, constructivist learning? Training vs education vs discovery vs creation?
• #lak11 “As soon as you start measuring something, you change people’s behaviour” – yes! One of my big concerns.
• First mention of latent semantic analysis at #lak11. Great – how does it tie in with semantic web and social web?
• @dgasevich Could you share a copy of the ’06 “Learning Object Context on the Semantic Web”? UToronto doesn’t have access to fulltext. #lak11
• RT @sbskmi: Learners won’t care about algorithms — until an algorithm thinks you will fail your course and blocks your course admission… #LAK11
• .@dgasevich shows software tools loco-analyst http://bit.ly/fF59m6 and intelleo http://bit.ly/e8dRXb – fascinating #lak11
• Great conversations around coffee break@ #lak11. Corporate training, #CSCW, business analytics, research collab, #OA, … All connected!
• For fun, I’ll try to take notes in #etherpad, feel free to jump in and add stuff: http://bit.ly/hxleEj #lak11
• Wish presenters at #lak11 could use short urls on their slides, make it easier to type in on our side, before you change slides.
• The #lak11 visualization shows how little time I spent in the course, and how quickly I fell off. ariadne.cs.kuleuven.be/monitorwidget-lak11
• “altruistic learning (MOOCs etc) vs competitive learning (for grades, curved, zero-sum game)” – see differences in collab. behaviour? #lak11
• #lak11 “Participation is nice, but it doesn’t really tell you what learning is happening – content analysis as well”
• #lak11 DataTel already hosts big datasets of learning interaction – wants more. Very interesting to #P2PU.
• #lak11 Is there a common format to describe learning interactions across different platforms, to make it available to lot’s of vis tools?
• #lak11 So many fun new tools to play with / explore. Hope these will be gathered somewhere (rather than having to mine all the PPTs)
• #lak11 Would be great to have a “speed-geeking” session where people can showcase their tools etc more intimately.
• #lak11 Oh oh, @psychemedia is up. I have a feeling my fingers are going to hurt from all the notes I need to take…
• Great intro – “how can non-programmers do this kind of stuff, extract and analyze data” #lak11
• #lak11 @psychemedia : “Data is a dish best served raw” – great slogan for the #od #opendata movement
• #lak11 “A way of having a conversation with data in a visual form” – something we are very good at (doing through vision vs through math)”
• A few people are starting to help me keep notes from #lak11 on #p2pu’s #etherpad. Very fun, and useful. http://bit.ly/hxleEj
• RT @malpaso: HIrst neglected to mention “transformation” (one structure to another) as a step. Not a trivial step for non-programmers. #lak11
• @aribadernatal Maybe. But would be useful with a list of data that these tools need, so we can know what to export. #lak11
• #lak11 Note to self: read up on and play with ManyEyes.
• How do you link your own data, if there is no unique identifier? If I want to writ about OU academics, how do I make it easy to link? #lak11
• #lak11 Try to use consistent usernames. Not perfect (might not be available, someone could set up acct w/your username on new soc service)
• #lak11 Again, Twitter following isn’t necessarily meaningful. I follow more than 800 ppl, but I don’t read all of their tweets every day!
• #lak11 Retweets might be a better metric, but also not perfect. Would be neat if there were “twitter trends” showing me whose tweets I read
• #lak11 But Twitter doesn’t know, because Nambu pulls down all the tweets and filter them for me.
• #lak11 It’s funny how incredibly much easier it is to map Twitter followers, then to map co-citations in academic papers!
• @gconole We need unique academic IDs, and unique paper IDs, and citations/papers need to use them. There are initiatives but … #lak11
• RT @sbskmi: @psychemedia is the mashup DJ spinning his discs here at #lak11 But is current SNA adequately tuned for LEARNING?
• RT @malpaso: Endless data. Endless patterns. Endless visualizations. What’s relevant for decision and action? #lak11
• @gsiemens Yes, there are others too. But haven’t seen a single journal including this info yet. #lak11
• @sbskmi Philosophizing abt diff. ways of live collab. notes – wonder how would be different in Compendium etc. http://bit.ly/hxleEj #lak11
• RT @niklas_karlsson: Data from the course #Lak11 can easily be misinterpreted when personal interpretations are made. http://bit.ly/fnAiLe #lak11
• #lak11 Wonderful lunch buffet, and great conversations about the amazing surveillance potential of data analytics. We know your appliances!
• #lak11 One of the frustrating things to me is the disconnect between schools of education and teach&learn in higher education.
• #lak11 No link between Office of Teaching Advancement and School of Education at University of Toronto.
• When where? :) RT @gsiemens Who’s up for dinner/beverages later tonight? #lak11 @mweller is organizing :)
• One of the core questions of #P2PU!! RT @rmitchell Who Does? ;) RT @aribadernatal: “Students don’t *do* optional.” #LAK11
• Brilliant. RT @dougclow @houshuang @rmitchell @aribadernatal Students don’t do optional, but many learners do! d:-) #lak11
• Unfortunately #p2pu etherpad seems to be unstable, moving notes to #piratepad: http://bit.ly/hH3EHN #lak11
• RT @technostats: @sbskmi – socio-cultural discourse analysis as a potential measure of knowledge acquisition. Very interesting concept. #LAK11
• Great talk by @sbskmi at #lak11. Very excited by new platforms for “reflective deliberation” Use at #p2pu? Potential4analysis&deep learning
• I heard rumours in the corridor about hot springs tonight – I’m interested :) #lak11
• @gsiemens http://bit.ly/f65vfu #lak11 Hot springs location
• #lak11 I find some of todays talks very interesting, others not at all. Wonder if others feel the same. Is learning analytics one thing
• #lak11 or several things? Are there clear overlaps/synergies? Or is this a collection of two different groups of people?
• #lak11 @gsiemens: Do we need different analytics for different disciplines? Is there no universal “learning science”?
• #lak11 gsiemens: Learners will game metrics. Me: only if the metrics are high stakes (ie. does it also apply at P2PU?)
• #lak11 gsiemens: Interesting disciplinary differences between socsci/humanities and CS in conference steering committe.
• Great talk RT @sbskmi Theory-based Learning Analytics: #LAK11 talk slides at http://bit.ly/i8y4Ag
• RT @psychemedia: @gsiemens Presentation graphics may well be different to visualisations used in more analytic phase…? #lak11
• RT @CosmoCat: Don´t miss @dougclow ´s “The Learning Analytics Cycle” http://t.co/6BCewhf – and notes from @houshuang http://bit.ly/fTTLW6 #lak11
• Starting collaborative note taking today here: http://bit.ly/g2b2wC, yesterday’s notes are here: http://bit.ly/hH3EHN #lak11
• #lak11 @gsiemens: “@psychemedia is the McGyver of data” – absolutely
• RT @lgully: Hirst: irony of course descriptions written for grads not students seeking info #LAK11. Need discoverable info.
• RT @lgully: RT @CrudBasher: #lak11 Totally agree -> Hirst – learning should be lifelong. Universities should keep a learning relationship with grads
• #lak11 “Predictors vs causes:predictors of learning outcomes may be useful for “diagnostic” purposes, but need not be causally related”
• RT @opencontent: @JonElmSherrill We almost always have ~more~ data online, it’s just significantly impoverished… Pick your poison? #lak11
• #lak11 Finally a presenter using short urls, thank you @aribadernatal
• #lak11 Ah, it was http://bit.ly/egV5Rt Mahout – thanks Andrew.
• #lak11 Reputation as a proxy for learning in informal learning contexts – Doug Clow. Very interesting to #p2pu!
• RT @Anna_De_Liddo: @dougclow I LOVE iSpot! We would need an iSpot for learning! what about laSpot: spot new learning analytics tools/theories! #lak11
• What does @dougclow have against power-laws anyway? :) #lak11
• Very dense talk by Dan Suthers on multi-level analysis. Just read his paper last night – lot’s I don’t understand, but fascinating #lak11
• #lak11 Interesting design of sessions here – very little time to ask questions or involve presenters in discussions around their prezos.
• For new arrivals, collaborative notes on #lak11 sessions being taken here: http://bit.ly/g2b2wC Feel free to join.
• RT @psychemedia: SNAPP bookmarklet code http://bit.ly/hSmBy3 #lak11
• .@cteplovs @dreff Wish you both could have been here! But Ravi Vatrapu is doing a great job. #lak11
• RT @cab938: Bears as learners, what a nice example for a conference in banff #lak11
• Varatrapu: “Many ICT managers say that their IT systems are UNESCO heritage – cannot be touched”. Now cloud-based, avoid problem #lak11
• CommonLibrary on Sourceforge (Phil Ice mentioned this): http://bit.ly/gc0IfA #lak11
• That wraps up some pretty intense note-taking. Check out http://bit.ly/g2b2wC, feel free to add info. Will post on blog tonight. #lak11
• RT @psychemedia: Really nice idea – give students a VM instance within which to do course related activities… and track them while they’re at it ;-0 #lak11
• @dougclow Not competition, complementarity :)Was just saying to Andrew that it would be neat to combine them to get sth more complete #lak11
• Internet seems a bit more stable – I’m trying collab notes on Etherpad: http://bit.ly/dZtfdM #lak11
• Great conversation over lunch with @sbskmi, Dan Suthers, Ravi Vatrapu and @anna_de_liddo, lot’s of new ideas. #lak11
• Hoping someone will announce some collective activities tonight (dinner, drinks, springs, night skiing, improv? :)) #lak11
• #lak11 Sorry about note taking, feeling a bit overwhelmed by great ideas and things to explore. Luckily this is all being captured.
• @dougclow Or maybe we should collaboratively create an interpretive snow sculpture that embodies the key lesson from the conference? #lak11
• #lak11 Interesting UoC has drop-out rate of 40%. Wonder if #p2pu can aim to beat that…
• @xaoch It’s a really nice hike, but I think it gets dark around 6:15, and there is no lighting there. #lak11
• @sbskmi This is key even for #p2pu, which uses multiple external platforms for courses. #LAK11
• @xaoch Easier if people could use same e-mails. Also need to get all the info in, not all services provide RSS etc. #lak11
• .@gsiemens Summing up #lak11 with “intimate encounter with tree” metaphor. Field is indeed moving fast, and lot’s of opportunities 4 collab
• We’re thinking of going to springs at 6 (meet in lobby), and then for dinner/drinks if people want. #lak11
• Just back in Toronto after great #lak11 conference. Now lot’s of follow-up and thinking about conference themes. Stay in touch all!
• RT @markmelia: Glass – a key component to ubiquitous learning in the future – maybe – http://youtu.be/6Cf7IL_eZ38 – thanks to tony bates for sharing #lak11
• Notes from second day of #lak11 posted: http://bit.ly/gkfwxV. Hope to write more reflective blog posts soonish. #oer

During the second day of the Learning Analytics Conference, I continued taking notes in Etherpad, just like I had done during the pre-conference, and day 1. After lunch, I felt quite burnt out however, after taking quite detailed notes for two and a half day already. In addition, I had some very interesting conversations during lunch, and my head kept spinning around those ideas, rather than focusing on the current speakers, so the notes below are by no means complete. Luckily Doug Clow took notes from the afternoon sessions.

All in all, it was a really great conference – beautiful venue, lot’s of opportunities to interact, and a ton of new ideas. I hope to write some longer more reflective pieces about themes that I saw in the conference, and how they relate to my own research, once I get my todo list under control.

Erik & Hannah Duval – keynote

attention – what do people pay attention to, when they learn?
someone is, or is not, interacting with what is going on. Can we capture what the person does, can we use what we capture, to get better at doing what we do?

Human readable attention stream
yammer – intranet twitter, gives you the “pulse” of the team.

human, explicit, nice, but doesn’t scale – overwhelming, like LAK11

using attention to filter & suggest, provide awareness & support social links
wakoopa – analytics. plugin. tracks everything you do. awareness of what you have been doing.

Software recommendations – other people with similar behaviour are using different applications, etc.

Find out when friends start using new software. If the people in this room could keep each other informed about the tools we use, in a very light-weight way.

Can we do something like that, for learning?

Physical exercise – notion of capturing data automatically, and using them to help you get better at what you want to do – very big community, RunKeeper for example. Run with your device.

“I want to run a marathon in September” – out comes a training program, and tracks it – hey you’re not on track. How would this look like for learning, especially language learning etc.

RescueTime – you can set yourself goals. How much time to spend on email etc. It will tell you if you go over.

Google tracks searches.

Contextualized attention metadata.
Responsive Open Learning Environment (ROLE), EU funded project.

Pull a number of components together, like widgets, and build your own learning environment. We try to keep track of everything going on in these environments.

We can build tools that visualize what’s going on (Visualizing PLE Usage, Erik Duval et al)

http://www.role-showcase.eu/role-tool/cam-zeitgeist

Awareness for learners & teachers

Hans Poldoja is a PhD student of Erik Duval – his EduFeedr. People post things on their blogs, software figures out if they are doing what they should be doing.

Through tools, starting to collect datasets about how people interact with learning.

Figure out what recurrent patterns are, and what they mean.

TELEurope.eu – teleurope.eu/pg/groups/9405/data…

Would love to wire my students, and measure what goes on in the brain (but small ethical problems).

The quantified self

Dangers?

Scary if the university or the organization owns the learning metadata – 1984…

AttentionTrust.org -

Total recall. Book about E-Memory concept

Jeff Jarvis – the benefits of leading your life in an open way, tracking a lot of stuff, making that available. – book upcoming. “Public parts”

Motivation and self-efficacy among students – are they doing something for their own, rather than doing it for your professor. Strategy with own students: very open learning, people outside of the class see what you do, can be very motivating to students.

My students will auto-report what they think I want to know, so they get a better grade – which doesn’t necessarily have anything to do with what they are doing. How to really track this?

Comment from audience: This seems very related to the “game layer”, social competition etc.

Katja Niemann – Usage Contexts for Object Similarity: Exploratory Investigations

The self-regulated learner needs support to decide which learning object fits his needs best in current context

Recommend suitable learning objects according to
- learning goal
- competence level
- preferred learning style

Problems with finding those objects
- expert metadata: expensive
- automatically generated metadata: good results for texts, but not for other media types
- social metadata: sparse, ambigous, faulty

Contextualized Attention Metadata (CAM)
Linguistic basic unit: word – sentence
CAM: action/object – session

Use methods from linguistics on these sessions from CAM.

Paradigmatic relations
two words that often appear in the same context might be semantically similar
ex “beer” and “wine”

SO
do objects with similar usage have similar context?

Each object holds a usage context profile (UCP) comprising all its usage contexts
C consists of pre- and post-contexts

UCP similarity – compare pre-context and post-context

http://portal.mace-project.eu MACE – testbed to connect lerning objects in field of architecture

Using learning analytics to assess student’s behaviour in open-ended programming tasks- Paulo Bilkstein

If we don’t come up with ways to give teachers incentives to assign projects to encourage 21st century problem solvers, they won’t do it.

Anna De Liddo – Discourse-Centric Learning Analytics

discourse as indicator of learning – key indicator of meaningful learning is the quality of contribution to discourse

sociocultural perspective on learning
discourse as a tool to think collectively

through which people can compare their thinking, explore ideas, shape agreement

chronologically vs logically rendered dialogue environments
(most online environments represent discourse as a timeline)

You have to read the entire thread to find the key items that have been discussed – not scaleable.

Online Deliberation: Emerging Tools Workshop (http://www.olnet.org/odet2010)
Essence: E-Science, Sensemaking & Climate Change

Demo of Cohere

Have to explicitly choose the kind of contribution they are making. Can annotate and include webpages.
Make connections – search database, and pick post you want to connect to. Have to associate a semantic to the connection – what kind of link is this?

Ways of filtering posts, visualize in different ways.

Online discussion – ask students to classify what contributions you are making, and how this connects – unrelated to where your post appears.

Analytics per learner – Cohere personal notebook, all the notes, annotated websites, connections made, people connected etc. Different tables: post types (how learner contributed to discourse). What kinds of rhetorical moves are they making when they connect through posts?

Discourse network structure = concept network + social network

Concept network – nodes are posts, edges are semantics of connections. Normal network analysis: identify hub topics or hub posts. Who authored these posts? (In our case studies, the hub posts were questions).

Social network: tells you if there are sub-groups of learners that are not talking to each other.

Outdegree = measure of users’ activity – you created a lot of activity pointing to others
Indegree = indirect measure of relevance of a user’s post – how many connections have been done to posts authored by you

We are interested in the rhetorical role that a user’s contribution is making to a document or conversation and the nature of the connection to other contributions using semantic relationships.

Future:

• embed learning analytics within Cohere’s UI
• investigating computational linguistics tool for automatically detecting rhetorical gestures within text documents (with Xerox, Agnes Sandor, http://olnet.org/node/512)
• ability to set software agents to monitor the discourse network – moving toward user-defined sematnic network analysis

Dan Suthers: We did this in the 1990′s, problem we ran into: reliability of learner self-categorization. Often, everyone would just choose the category on the top of the list.

The learner needs to see the value of using these tools.

This is building on Dan Suthers’ work, and Scardamalia and Bereiter’s work, etc. It’s a challenge for learners

• to know what the role of their contribution is
• to be motivated to put in the role

This is how you make your thinking visible – if this is being assessed, that might be an incentive.

Learning Analytics and Exploratory Dialogue – Simon Buckingham-Shum

Hours of material – how can LA help spot critical, knowledge-building discourse?

How many points in the webinar triggered learning/knowledge building.

Text chat is very challenging, because there are fragments.

Data source: OU online conference.

3 kinds of talk

• disputational talk (arguing, discussing)
• cumulative talk (positive, not too critical, building on people’s stuff, confirming and elaborating)
• exploratory talk (ideal type – we try to scaffold this – engage critically but constructively, making thinking visible, these may be challenged and counter-challenged, but challenge are justified and alternative hypothesis are offered)

Comes from Mercer (2004) Sociocultural discourse analysis (J Appl. Ling) studying children in classrooms. http://politicaltheology.com/index.php/JAL/article/viewArticle/1443

Indicators of exploratory talk?

• “good example” – could be “good example” or “never heard anything less like a good example” – but implies evaluation

Indicated 94? indicators. Some of the obvious ones are quite misleading.

Future research needed to

• check reliability of this form of analysis
• check validity
• differentiate exploratory talk abt content, tools, process, people
• investigate relationship between chat and audio/video
• automate process of analysis

Today was the first day of the conference, with a lot of very interesting sessions. Too much to process at once, but hopefully these notes will be useful. They were taken in Etherpad, and others, especially Andrew Barras, helped out. Doug Clow also took very extensive notes from the sessions (morning, afternoon). More below the fold.

Feel free to jump in and help us take notes anyone. Link to pre-conference notes: http://piratepad.net/lak11-collaborative-notes (and archived here: http://reganmian.net/blog/2011/02/27/notes-from-learning-analytics-conference-2011-pre-conference/).

This was mentioned by someone: http://www.clips.ua.ac.be/pages/pattern Python tool for textual analysis

You can download all of StackOverflow data for analysis: http://meta.stackoverflow.com/questions/2677/database-schema-documentation-for-the-public-data-dump-and-data-explorer

Timeflow

Tony Hirst: Pragmatic analytics: insight, representation and structure

“Scoring Points” – book about about how Tesco used loyalty program to collect data about shoppers, and offer better services to shoppers (http://www.amazon.com/Scoring-Points-Continues-Customer-Loyalty/dp/0749453389/ref=sr_1_1?ie=UTF8&qid=1298908493&sr=8-1)

Segmentation – different groups of shoppers. 

JISC – business intelligence (http://www.jiscinfonet.ac.uk/bi)

Marketing companies already know huge amounts about you – deliverable at post code, or address level. Difference is that now you can get access to data without paying huge sums (through social networking analysis etc).

After graduation, we should engage learners as life-long learners, and offer subscription services – we already know a lot about them. Not just put them in the “advancement/fund raising” bucket. 

Course choice analytics. 
Two years ago, Google was the dominant way for people to find OU courses – now Facebook is becoming increasingly important. 

How does this work with OER discovery? How are people finding your OER? What kinds of descriptions are you using. Are you describing the course with language that could only be understood by people who have already completed the course? :) What search terms are people using to find your course?

Descriptive reports
Prescriptive models (common sense model of how people behave)
Predictive voodoo (you don’t know what’s going on inside)

Library
Dave Pattern (@daveyp) – added “people who borrowed this book also borrowed this other book” into library catalogue. Clear stats: increase in books borrowed. 
Also looked at whether engagement with library improved people’s degree qualifications. Correlation between use of library and qualification.

“Negative feedback, closed loop control system”

If you make changes, and there is no measurable change in output, how do you know your change had any effect?

Using Google Analytics to analyze online course, block by week. Extract data from GA, build a model, find unique visitors to different resources, get a better feel for how people are moving through the course. Then you can experiment with for example moving course assessment, see whether weeks are overloaded or underloaded.

Time Series Data demonstrates
-Trend
-Seasonality
-Noise

The concept of “detrending” data… To be able to get to periodicity. 

Fourier analysis – any time series signal can be made from combinations of sinusoidal curves. Segment time series data into different periodicities. 

books: O’Reilly
Collective Intelligence
Visualizing Data
Data Analysis

People who use Google Analytics often just view the default reports. Do you try to relate the behaviour from GA to behaviour reported in other systems (virtual learning environments with their own reporting, etc – or is this data produced in “silos” and looked at by different people?) 

Looking at the behaviour of course websites AS websites – how effective are they AS websites? Are they visiting pages, are they clicking on links. Not necessarily to gauge student learning.

Must be careful using Google Analytics. The graphs can be misleading
Always be suspicious of means. Also look at bin sizes.

Anascombe’s Quartet http://en.wikipedia.org/wiki/Anscombe%27s_quartet

Simpson’s paradox. http://en.wikipedia.org/wiki/Simpson‘s_paradox

Segmentation is critical. 
Facebook Course profiles. Students can provide info about courses they are taking

Using tags on Twitter to visualize networks, create animations over time. 
Get a pretty good way of the structure of the students, and their social networks, and how we can communicate with them. 

Data can make people uncomfortable (and close-up videos of eyes can do this as well)

David Wiley – BYU – Learning analytics as Interpretive Practice

“Warning voice”. Download slides: http://slideshare.net/opencontent

Interpretation != science?

Confusion of science with positivism
Social scientists have “physics envy”, quant > qual

Educational measurement: what does he/she know?
Research that is mediated by observation – can’t crack open George’s head (and wouldn’t want to if I could)… People engage in behaviours, and we take those behaviours.
Online learning is even worse – can’t look at George to see if he is paying attention. Second layer of abstraction. 

All observable behaviour online is expressed in this very restricted vocabulary to key presses and mouse clicks. Two layers removed.

Westerman’s argument: Quantitative inquiry is interpretive. (http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VD4-4MD9G50-2&_user=10&_coverDate=12%2F31%2F2006&_rdoc=1&_fmt=high&_orig=gateway&_origin=gateway&_sort=d&_docanchor=&view=c&_searchStrId=1658861033&_rerunOrigin=google&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=02dcae8f717baa616689b174b85b3fe6&searchtype=a)

Construct operational definition of “they’re happy”, “they know calculus” – an operational definition is like a diving mask – you can’t see anything else. 

Calculating time on task online – we have now clue of whether people are even looking at the screen. We have a “common sense” idea about time on task, dangerous.

“Letting the data tail wag the theory dog” (Vic Bunderson)

Can we call it success, if we can predict, but we don’t understand why? 

If not positivism… then what? 
Hermeneutics – meaning and interpretation

Problems with metaphor – information processing model – only works when the brain is operating like a computer. Breaks down when creativity is involved

Reductionism – “nothing more to be said when neurophysiology has had it’s say”

Behaviour in context, social practice. How do we observe behaviour in online environments? 

Structural equation modeling
Multilevel data structures
Continous, longitudinal measures

Tasks nested within practice

Learning analytics is an ethical activity – what happens if people actually follow the recommendations we make? 

Stephen Fancsali – Variable Construction for PRedictive and Causal Modeling of Online Education Data (Dept of Philosophy, CMU also with Apollo Group)

We only have access to complex, raw, log data

Predictors vs causes – predictors of learning outcomes may be useful for “diagnostic” purposes, but need not be causally related to outcomes

Difference between diagnostic and causal

Predictive analytics – identify high-performing students
If we are interested in changing products to change (enhance) student performance, we nee dto know causes of student learning outcomes (causal knowledge)

Causal graphs. Lot’s of work has been done on this during last 20-30 years. Focused on data provided at appropriate level/unit of analysis. How do we deal with log data etc.

SO

rely on intuition/expert opinion to construct ad hoc variable (detailed research in conference paper)

OR

devise a data-driven search for variable construction

Data from grad level econ course – data from messaging and access to “resource” (chapter)

variables:
student public and group forum message count
intructor private forum message count
chapter “view” count

learning outcomes:
final exam score (independently graded by textbook exam)
course grade (grade points out of 4.0)

excluded demographic variables because interested in actionable interventions

search for variable construction

START assembled data per student -> operators -> prune -> causal graph search -> causal predictive modeling -> assess: prune or STOP -> operators (start over)

operators: sum, max, var, per day, etc. logarithm, discretize, interactions.

if two operators are highly correlated (for example mean and median), take the one that is the most predictive. 

determine sets of causal graphs that could explain data (use method that allows for unobserved common causes)

“average causal predictability”

aboveMedian(log(min(message world length))) – for example

http://www.phil.cmu.edu//projects/tetrad – causal discovery algorithms implemented in OSS Tetrad project

Ari Bader-Natal (Grockit)

What are people doing. What’s effective/engaging? How can we do analysis effectively? What do we do about data that doesn’t leave a trace in the system. How do we ask questions that are testing hypotheses. How do we do this in a way that’s easy enough that it leads to action?

How do we feed this back for different audiences?

What are people actually doing? 
Just poke around in the database

What is effective / engaging?
Duration and frequency of discussion differ between different levels of students
Which of the various interventions in Grockit lead to largest learning gains?

Resources
bit.ly/exp-long
bit.ly/exp-short
bit.ly/one-line-split

Make it very easy to run experiments in the code, A/B testing etc, including generating reports. 

Webbased analytics interface – showing all reports etc. Decision makers can subscribe to different reports. 

Self-reported SAT scores etc used to “calibrate” system.
Apache system “Mahout”  OSS http://mahout.apache.org/ – thanks! :)

Doug Clow (OU) iSpot – d.j.clow@open.ac.uk – @dougclow – http://dougclow.wordpress.com

“A place to share nature”. 

How do we connect people who want to watch tv on nature, with OER lessons on nature?

Interesting “badge system” at iSpot – you can see if they have taken a course, if they are active in a society, if they are experts etc. Similar to video games – Also similar to what P2PU is trying to build, see Erin Knight’s “badge paper”: http://erinknight.com/post/3545583326/a-working-badge-paper – building a distributed authenticated badge system, to let you take badges from for example iSpot “with you” to other platforms

Underpinning theory
“Fairy rings of participation” Makriyannis and De Liddo 2010 – http://olnet.org/node/353

Reputation and learning
informal learning context. assessment very important for learning, very heard to provide in informal learning context. 
“reputation as a proxy measure of learning. – not (just) social approval

If you make an identification, and an expert agrees with you, your reputation +1. If you agree with someone else, theirs goes up by your reputation/1000. (Experts get 1000).

Reputation analytics
How does this work – is reputation mostly increased by experts, or what?
Long-tail distribution of engagement means a functioning social network?

Not everything that looks like a power-law is a power-law. 

Learning analytics cycle – learners -> data -> metrics/analytics -> intervention -> learners (all over again)

Main feedback cycle is the reputation generated. 

Observations and reputation received (learning) are highly unequally distributed (fat tail)

Reputation given is even more highly unequal – experts have an amplified effect

Any correlation between: agreements given and received, or reputation given and received, are weak, highly nonlinear, and distinct.

Informal learning context – feedback direct to other learners, not mediated by specialists
Participation pattern is typical of social software

Effective informal learning assessment by reputation

Future
Adapt reputation system to other domains
More sophisticated fitting
Social network analysis
Identifying learning (reputation vs formal course)
More qualitative research

Role of experts
Incentives? How are they identified? They do volunteer – it’s an important thing to do. We talk to places where we can get these people, people in the project have lot’s of relationships. The expertise in natural history resides in these amateur groups. Feature them on the site. 

Initial purpose was not to create scientific data, but to create more people who can create scientific data. But actually, some of the observations have been useful – trying to enable exporting of information to amateur societies. 

Serious games – games with a purpose.

Xavier – Ecuador slides: http://www.slideshare.net/xaoch/learnometrics-keynote-lak2011

Learning respositories are not working because growth is linear, not exponential like youtube

connexions is exponential http://cnx.org/

why do OCW users contribute more than Merlot users? Answer: Engagement – there must be a value proposition

Reuse is the main feature of Learning Objects but very little is known about actual reuse rates

Registry of Open Access Repositories http://roar.eprints.org/

Dan Suthers – publications: http://lilt.ics.hawaii.edu/research/pubs.html

Unified Framework for Multi-Level Analysis of Distributed Learning

multiple theories about learning in social settings
- social as stimulus to social entity as learning agent
-networked individualism to maintaining  a joint conception of problem
- diffusion of innovation to knowledge building

All involve uptake (Suthers, ijCSCL 2006, learning epistemologies). 

Uptake is evidenced by how individual actions are observably contingent on the actions of others in their socio-technical context

How learning takes place through interplay between individual and collective agency
- situated accomplishment of individuals and small groups
- local accomplishments giving rise to larger phenomena in networks
Requires coordinated multi-level analysis

distributed across multiple media and sites (chat, whiteboard, etc)
“Distributed activity may be analytically cloaked”

Abstract transcript representation

Adjacency pairs – each event is related to the one before. 
Contingency graphs – empirical relationships between events that collectively evidence uptake (dependencies?) (Garfinkel: contingently achieved accomplishments)

Media dependency: to reply to a message, it must first be written
Read events – you must read, to be able to write
Temporal contingency, or events that contain the same actor – more powerful contingency (they did something right after reading a message)

Lexical or semantic overlap – reuse of noun phrases

Collections of contingencies as evidence of uptake. 

Associograms: directed afiliation network of actors and artifacts
Mediation model: how actors’ associations are mediated

A round trip, interaction patterns. (Something that you cannot see in threading structure, but is shown by contingency graphs). 

Relationships – associograms and pariwise associations (relationship model)

Multimedia associations 
Characterize pairwise relationships in terms of distribution across media
Compare roles of various media in supporting associations

Social ties – enables application of Social Network Analysis methods

Tool enables us to go from log data to ties – get representations to a level where you want to do your analysis. Keep tie to data, so you can go back to the evidence at any point. Multi-level analysis.

Use contingency graphs for
- microanalysis
- semi-automated analyses of graph manipulations to find pivotal moments

Tapped In (SRI International)
- network of educators, professional development and peer support
- 20k educators, 8k user-created spaces etc
- lot’s of media (threaded discussion, chats, wikis, resource sharing)

Imported all activity for a two-year period into framework. Generate contingencies. 

Workshop at CSCL 2011 in HK: http://www.isls.org/cscl2011/cfp-ws-suthers.htm

Bakharia – SNAPP – A bird’s eye view of temporal participation interaction

Diagnostic instrument allows teaching staff to evlauate student behavioural patterns against learning activity design objectives and intervene as required in a timely manner

SNA can be used to identify
    – learner isolation
    – creativity
    – community formation

How can we realize the potential of real-time SNA?
making the analysis transparent to the user… as per doug clow suggested earlier. it gives the data the potential of being the intervention

Two forums, same number of messages and participants. Threaded view – doesn’t tell us if they are structurally different, temporal activity.

From relationship point of view: in forum A, no interaction, all via tutor.

We are representing the data with wrong visual metaphor – we need different ways of representing the data (live). (To learners? OR just to tutors?) – can we embed these sociograms within the forums themselves?

Tool:
Integration w/ LMS (Moodle, BB, D2L)
Render a sociogram as alternate representation of the threaded discussion view

Difficulties with integrating with LMSes – APIs didn’t let you interrogate discussion forums, not allowed to directly access database, etc. Plugins limited to adding new features, not modifying existing features.

To install, drag button onto toolbar (so it’s a bookmarklet)
You visit a forum, click the bookmarklet. SNA diagram appears. 

Can also annotate, add a date when you are trying a new strategy for example, and it will keep that as a log. 

Learner isolations – dense interactions between central nodes, etc
Facilitator Centric patterns

My critique: who you reply to is not a great indicator of whom you are interacting with. I might read a whole thread, and reply to the last post, but include replies to all the previous postings in my post – something like Suther’s uptake model.

Future directions:
Content analysis
Behavioural modeling
Topic modeling

Ravi Varatrapu

NEXT-TELL project (http://www.next-tell.eu)

High-density classrooms, rich personalized learning environments, one-to-one laptop projects
Information overload – how can teachers take advantage of all this data?

Learning ecologies
    students have access to large network of information resources, tools, and social resources
    students know different things, and know differently
    
Challenge of teaching adaptively and personalised in the high-density classroom combined with a rich information

NEXT-TELL – innovation platform for formative classroom assessment 

Teaching analytics
- learning sciences (interactional pathways to learning outcomes)
- learning analytics (systemic metrics)
- visual analytics (tools & techniques)

Dynamic diagnostic pedaogical decision-making
- leanring activity designa nd formative assessment
- Classroom Information Systems (CIS)
    – data provenance and process provenance
    – meaningful
    – actionable
- methods, tools, and training for a new “professional vision”

Design Based Research Expert
Teaching Expert

Teaching analytics
- evidence-centered activity & assessment design
- learning activity & assessment tracing
- open learner models (OLM)
- learning analytics
- visual analytics

Open Learning Models (from intelligent tutoring systems)

inspectable, scrutable by learners.
“reflection of a bear in a pond”

teacher uses ECAD planner to deploy assessments, activites recorded, formative assessments recorded, into analytic engine, a visualization on task progress – this visualization is available ot learner.

Empirical design-based research, first led by researchers, then led by teachers

Computational social science laboratory (CSSL) – eye tracking, neurological, physiological data collection equipment.
60-80 classrooms

We need to invent visualizations / representations. Eye tracking: get at good designs for visualizations. Put eye tracker in classroom, see how are teachers using it? How does communication layer hold up in real time in the classroom?

“Good data comes from good instrumentation. Need multiple measures to correlate.”

Phil Ice – Multi-level Institutional Application of Analytics – American Public University System

Dashboard that lists all 86,000 learners, sorted by most likely to disenrolling during the next five days. Uses 86 different factors, let’s you drill down, and directly engage with action.

Semantic analysis – granularity model
Purpose – accreditation of institution, showing that you meet all course objectives etc. Mapping resources to objectives.

Federation, disaggregation, relational mapping, ontological ordering

Injection engine (on SourceForge). http://sourceforge.net/projects/commonlibrary/ Injest any kind of content, strip content out of anything (including PDF), and turn it into XML. Then disaggregate the content – if you have series of JPEGs associated with text, video – disaggregate them. Utilize metadata of JPEG, natural language processing of txt, order them against ontologies that you specify. Adobe has tools for audio-to-text analysis for video. Apparently very difficult to learn – give one person 3 months to play with it…

Gap analysis – shows which goals have not been fulfilled.

Compared to two independent human coders – 93% accuracy (three passes of refinement of LSA).

92.7% savings in time – $83k saved

Roundtripping – take in student work and have it subjected to LSA, to match students’ work to pre-formed ontologies – actual evidence of learning outcomes

Chris – lecture capture

Every 30 seconds sends heartbeat back to server – who is watching, what are they watching, where are they in the video etc. 

Automatically capturing slide transitions etc

Using logging data points, we can almost perfectly reconstitute their viewing behaviour. 

Note-taking panel, both individual notes, and global notes – by slide (not much takeoff)

opencast Matterhorn project – lecture capture solution, open and free, built for higher ed. Want to build in all this analytics tracking. http://www.opencastproject.org/project/matterhorn

– IDEAS AND INNOVATIONS –

Teplovs, Fujita and Varatrapu – Generating Predictive Models of Learner Community Dynamics (teplovs@cvs.dk)

Latent semantic analysis – too many dimensions for traditional analysis – visualization might be a solution.

Knowledge Space Visualizer – http://chris.ikit.org/ksv/

Research by Chris Teplovs – http://chris.ikit.org/ (including his PhD thesis, about this topic)

Explicit links and implicit links, cosine between vectors.
Chronology and authorship can also be included. 

Advantages
    – flexible tresholds

Use this to generate a learning model – “you are what you write”

Vector representation of each user. Define similarity between any two user models. Hypothesis under which we could expect to see productive interactions (for example Vygotsky’s Zone of Proximal Development)

Latent Semantic Analysis + Interaction-based User Models
“potential for productive interaction”
“actual interaction”
can start looking at interplay between potential and actual interactions

Can we use something like game theory to understand community dynamics?
    would need to understand the “payoffs” that accrue to interactions, the strategies (perhaps multiple) that participants employ, and the repertoires of strategies

Offer of summer visits to Copenhagen for PhDs or post-docs.

Ravi (again) Cultural Consideration sin Learning Analytics – varatrapu@cbs.dk, http://www.itu.dk/people/rkva

culture? what is it? 200 definitions of culture. 

social aspects of HCI (Reeves and Nass 1996, The Media Equation)

culture and CSCL?
how people do with technologies, outcomes don’t differ – different interaction pathways, but not different product?

problem solving with conceptual representation – how do people use tools/affordances. How do they interact socially, discourse presence, social/cognitive presence. What do they think of the participants after the collaborative session?

American-American, American-Chinese, Chinese-Chinese etc
No difference in learning outcomes. But very different how they go there.

Borrowing a lot from other disciplines – need for an integrative theory of culture and sociotechnical interactions. 

Interacting with technologies and interacting with others via technology

Structures of technological intersubjectivity

Affordances – arguing for a tight link between perception and action
- meaning making opportunities and action taking possibilities in an actor-environment system in a particular situation, relative to actor competencies and system capabilities

Appropriation of affordances. In some Asian classrooms, not appropriate to ask very difficult question to teacher (face saving). 

Intentional utilization of affordances is culture-sensitive, context-dependent (“God’s must be crazy”)

Intersubjectivity

Combining this with Dan Suthers work on uptake

Mike Sharkey – Academic Analytics Landscape at U Phoenix

435,000 students at U of Phoenix

30+ databases  430+ tables
1.5 TB increasing by 100Gb/month

All data is copied into a central repository from external DBs

Tablo – data visualization tool – expensive

Presentation from Spain (Abelardo?)

Using a virtual machine, with built in “spying”, which captures compiling, errors, URLs etc. Huge amount of data about how students are working. 

This week, I am participating in the first Learning Analytics conference in Banff, Alberta. I’m interested in this topic both for my PhD research, and it’s also something that P2PU has been very interested in pursuing. I started taking notes in Etherpad, and invited others to join me – most of the notes below were taken by me, but a number of others also contributed (thanks to all of you)!. These notes are very raw, but you might find some of it useful. I’ve also focused more on the things that are interesting to me, rather than trying to provide a comprehensive overview. I will keep taking notes for the next two days of the conference (as long as I keep having good access to a power bar, my aging MacBook has a battery life of about 1,5 hours by now).

If you want to follow the conference, there is a fair amount of tweeting at #lak11, and the conference is streamed at UStream.

Notes below the fold.

George Siemens

The intelligent curriculum – personalizing experience for individual students – adaptive curriculum

TEKRI – has very deep data about student interaction

Q: Who is going to succeed, who will fail? Not percentage-wise, individuals!

“Who talks to whom, about what, and to what effect” (sociology)

“Who’s connecting, what are they talking about, what’s the impact in the long run?”

Analyzing emails, who should be connected – business analysis – IBM

Marissa Meyer – the physics of data

function of scale

map reduce, hadoop

computing speed

digital data – data exhaust, trails

learning analytics – measurement, collection, analysis, reporting of data about learners and their contexts, or purposes of understanding and optimizing learning, and the environments in which it occurs

knowledge analytics – text / data mining, info retrieval machine learning, linked data) for processing data to provide representations in forms of which conclusions can be drawn in an automated and domain-aware way

model:
learners data + intelligent curriculum > profile, analysis > prediction > personalization, adaptation, intervention

Dragan

ontology – basic structure or armature around which knowledge base can be built

semantic web cannot exist without social web – cannot be easily produced.

DBpedia as ontology

BBC – semantic data – BBC music, leveraging data from MusicBrainz, DBpedia

learning ecosystems

authoring + reusability > packaging > learning and collaboration (community, peer-review, presenting, community, learners)

learning context ontology – content structure ontology, content type ontology, user model ontology, learning design ontology, domain ontology (DBpedia)

(IEEE International Conference on Advanced Learning Technologies – “Learning Object Context on the Semantic Web”)

sparlq – language for data mashups – can pull in data from many sources

loco-analyst http://jelenajovanovic.net/LOCO-Analyst/
harmonization of personal and organizational goals

http://intelleo.eu

ubiqitous learning analytics – from lot’s of platforms, formats – aggregates and integrates
cannot exist without advanced learning analytics

Ideas from lunch:

use latent semantic analysis to go through all published papers by all professors at a given university – find interesting linkages (topics, methods, common citations) of professors in different departments – sugget that they get to know each other. “you guys should talk”

FYI (from Joe): LSA might not be the best method to use here, I generally prefer “Concept Forest” (but need to make some experiments!).

Katrien

visualizing activities for self-reflection and awareness

-learning resource recommendation

dataTEL inititative

research on recommendation for learning

LAK11 pre-course data: http://ariadne.cs.kuleuven.be/monitorwidget-lak11

login: choose any users name

views average time spent of all users, and can highlight individual users (how do they calculate time spent? which resources were they able to suck in? ability to use with P2PU data?) is it OSS?

how does this modify student activity?

privacy issues for universities – this is the nice thing about LAK and other MOOCs

can anonymize – see the lines, but not who the students are

George: How real-time is the data? Currently pre-loaded, we can update every day, but not on the fly. Performance issues.

Time – estimates. Track all interactions.

“Altruistic learning situations vs competitive learning situations (grades are curved, etc)”

• Participation metrics

  Participation metrics might be more meaningful in altruistic learning situations – my example from MOOC: I signed up, but was only active during the first week. There is no penalty to dropping out – so people leave very easily. Presumably I would have learnt a lot more if I had been actively engaged with the course over the six weeks.

Compare to a competitive learning situations, where there might be a participation mark to make sure students generate a number of posts per week – in this case, pure participation statistics would be a lot less meaningful, and you’d need to look much more closely at LSA, discourse analysis to see if they are really doing deep thinking, or just “writing”… (This will be useful in open learning situations too – but we need to get the engagement and participation first. Plus it’s harder, because people have more leeway to define their own learning outcomes… Maybe link to individual learning plans etc?)

Contextualized Attention Metadata

deployed in ROLE-PLE, RWTH-Aachen engineering, Moodle

Track every kind of data such as user clicks

http://bit.ly/laksurv – survey about tool/ LAK11

“Happy to load your data into the tool.” Again: how does this work with distributed learning – this is the holy grail (funny how the MOOC people are so anti-LMS, but yet it’s an LMS that enables this to happen) – (gs: good point. Have you looked at social media monitoring tools? They are starting to address distributed interactions: see: http://www.diigo.com/user/gsiemens/monitoring

sh: @gs thanks! – this will be a core focus of the new P2PU platform – having a solid user-friendly core, which enables us to pull in statistics and info from lot’s of outside platforms, wikis, twitter, blogs, Wikipedia, github – for people to view activity, learners to create portfolios, assessment, and data analysis. (Hoping to make data-dumps of all learner activity available to all researchers on a regular basis / real-time)

We used to think that we could just use tags, and suck in #lak11 stuff from lot’s of platforms into a widget. Two problems: spam (all my blog posts get reposted on lot’s of splogs, with the tags intact, and popular tags are often spammes on Twitter etc), and connecting a learner identity in the LMS to an external contribution (learners might not use same login across different systems) – how to do this easily?
Would be interesting to come up with a standard data format for sharing “learning interaction logs”, so that visualizers etc could plug directly into data from P2PU / MOOCs, or even BlackBoard… Do any examples exist?

September RecSysTel 2010: http://adenu.ia.uned.es/workshops/recsystel2010/

dataTel: bit.ly/ieqm (too fast)

request for data sets, they already have datasets, Mendeley (2 million users), APOSDLE, ReMashed, .edunet, Mace, Melt

ROLE: Responsive Open Learner Environment

can suggest new tools of resources on the fly

tools that can support learner self-assessment

The Data Shop – A data analysis service for the learning science community http://pslcdatashop.web.cmu.edu #lak11

Tony Hirst

How can non-programmers do these kinds of things – lowering barriers

Data is a dish best served raw

*Workflow:

- Discovery

- Acquisition as data

APIs – offered by many social networks

(currently P2PU doesn’t offer any API – wonder what kinds of APIs would be useful to others, how to design them, etc)

screenscraping – extracting data from loaded webpages straight from html

scraperwiki – tool for this – http://scraperwiki.com/

import to HTML in Google Spreadsheet (grab table, or list?) identify table by the number of the table on the page (or number of list) – http://googlesystem.blogspot.com/2007/09/google-spreadsheets-lets-you-import.html

- Representation

- Cleansing

Data is notoriously unreliable – different dataformats, spelling mistakes, correcting errors (outliers might be valid though)

Hirst answered my question now about transformation. A possible tool is Google Refine.

Google Refine – load in a CSV, find strings that look similar and should be similar

http://code.google.com/p/google-refine/

Stanford Data Wrangler: http://vis.stanford.edu/wrangler/

Alfred Essa adds: transformation (one structure to another)

(Visual) analysis

– we can stop trends and variations visually much better than in raw tables

generate Google “heatmap”

Yahoo Pipes ( http://pipes.yahoo.com ) useful for generating KML format used by GoogleMaps and GoogleEarth

JSON = Web interchange language (mostly javascript based)

The really neat thing is that there is a tool to convert Yahoo Pipes to Python, which means that if Yahoo ever pulls the plug (not impossible, given how many services Yahoo shuts down, you won’t loose your tools): http://blog.ouseful.info/2010/09/30/yahoo-pipes-code-generator/

ManyEyes ( http://www-958.ibm.com/software/data/cognos/manyeyes/ )

Scraping “In Our Times”, finding members of the OU (not in a linked data form, neither list of academics in OU). lot’s of work to make things join up.

Gephi is an interactive visualization and exploration platform for all kinds of networks and complex systems http://gephi.org/ <- looks very cool

Use common usernames on different social platforms – not perfect, what if the name is taken on a service, or if someone takes your common username and use it to open an account at a new service. But maybe “good enough” for now. Would be great if there was a unique person identifier, which you could put into a field on all social services (and where you could do reverse lookup) – I guess this tends to be e-mail. Does Twitter have an API, give me the Twitter stream of shaklev@gmail.com?

if I wanted to follow “best practices” when writing about OU Academics, how would I “link” this data? Perhaps hyperlink their names to their faculty pages? What is the “unique identifier”? (Having personal URLs is great, but not sustainable – your own DNS, maybe you forget to renew it – your institution, maybe you change jobs)…

There are more common login services now that may help. For example, facebook logins.

I wonder how these help coordinate data – is there a way for me to find all the accounts that a Facebook user has logged into? Privacy issues? Also, Facebook isn’t the most generous at giving data out (they love “sucking it in” to Facebook though) Good point.

Create Google Custom Search Engine over just the followers in a certain area. “Roll your own” social search engine. Same with creating a “blog roll” in a certain topic.

I wonder if you can extract semantic relationships via twitter or delicous data?

Academic publishing
Funny how much easier it is to create networks of people who retweet each other, or follow each other, than to create a graph of people who co-author papers, or are cited by the same papers… We need unique author IDs, and unique paper IDs. And these need to be used every time a citation / author is mentioned in a paper. There are experiments in both categories, but nothing comprehensive yet.

Not related to anything
Interesting to think about the ideal design of such a collaborative note taking tool… Ways of keeping an emerging structure, and inserting notes at appropriate locations. (I’d almost like several small windows on the screen for different kinds of notes – one for speaker notes, one for books to look up, one for process notes etc). Ways of graphing visually (like @sbskmi does with Compendium, live meeting notes)… ways of going through at the end and collate everything – with ppts, notes, tweets etc…

LUNCH

Linda Baer

Systematic adoption of learning analytics

When looking at analytics, you have to look at past, present and future trends

Past: What happened?

Present: What is happening now?

Future: What will happen?

Analytical DELTA

Accessible high quality Data

Enterprise orientation

Analytical Leadership

Strategic Targets

Analyst

Tom Davenport books on analytics http://www.tomdavenport.com

Have a look at Doug Clow’s conference notes: http://dougclow.wordpress.com/2011/02/27/lak11-learning-analytics-and-knowledge-banff/ Oh very nice find! Thanks to Doug!

Stian – hope it’s ok to mention this here as well: http://www.learninganalytics.net/?p=122 cfp learning and knowledge analytics in ETS (delete if it isn’t)

Theory-based learning analytics (Simon Buckingham-Shum)

learning analytics and sensemaking – important and different from normal chat and social network analysis… google analytics etc designed by people who don’t know much about education

New learning theory – that wasn’t possible before we had learning analytics (ref. big science)

KMI – similar to AU’s TEKRI http://projects.kmi.open.ac.uk/hyperdiscourse/

(Note to all: Knowledge Cartography is brilliant, available here: http://www.springerlink.com/content/978-1-84800-148-0 for people who have access. Dan Suthers, who wrote a chapter, will be here too)

theory-based analytics: assumptions, evidence-based findings, statistical models, instructional models, as well as more academic “theories”theory-based analytics: assumptions, evidence-based findings, statistical models, instructional models, as well as more academic “theories”

The question is whether this has INTEGRITY as a meaningful indicator, and WHO/WHAT ACTS on this data

Another summary/notes of LAK11: http://alfredessa.com/?page_id=617

A theory might predict future patterns based on analytics.

RAISEonline: learning analytics in English schools

UK higher ed system: analytically “oppressed”

Previous OU study data are best predictors of future success

“operationalize” a definition of “at risk” – complex intersection of well-documented factors that threaten completion rates.

this is a theory of learning – model, empirically based findings, we can make predictions. (model of learning or of engagement?)

Sensemaking (Karl Weick): comprehending, patterning, interaction in pursuit of mutual understanding

Collective intelligence tools and analytics should be designed specifically to minimise breakdowns in sensemaking

Example

Experts might fail to recognize a novel phenomenon… SO tools should pay particular attention to edge-cases, or scaffold critical debates between contrasting issues

Complex systems only seem to make sense retrospectively

Use narrative theory to detect and analyse knowledge-sharing/interpretive stories

Coherent pathways through the data ocean are important

Much knowledge is tacit, shared through discourse, no formal cofidications. Trust is key to flexible sensemaking when the environment changes.

SO scaffold the formation and maintenance of quality learning environments.

Many small signals can build over time and become a significant force/change

SO highlight important events and connections → aggregation and emergent patterns

learning-to-learn analytics

set of generic dispositions and skills that characterize good learners – if we can teach them a language for this, they can learn to become better learners. deal with ambiguity, ask good questions, etc.

7 dimensions of learning power : http://www.vitalhub.net/index.php?id=8

Changing and Learning

Meaning Making

Curiosity

Creativity

Learning Relationships

Resilience

Strategic Awareness

Effective Lifelong Learning Inventory (Ruth Deakin Crick)

http://www.vitalhub.net/index.php?id=674

When making a blog post – tag it with the relevant indicator, dashboards for teachers.

(Idea: link this with work on badges at P2PU)

Discourse analytics

disputational talk

cumulative talk

exploratory talk

knowledge is made more publicly accountable and reasoning is more visible in the talk (ref to research papers).

use Elluminate to analyze exploratory talk. separate learning from social interaction. Taking Mercer’s framework for Exploratory talk – find “canned phrases” that signal the presence of these, run these on the transcript of the text chat. Preliminary evidence: yes, it works.

Xerox Incremental Parser: sophisticated text parser technology. Run on scientific paper – can pick out categories of knowledge level claims.

For example

background knowledge

…the previously proposed…

…is universally accepted…

contrasting idas

generalizing

novelty

significance

http://olnet.org/node/512

Integrate this with Cohere human annotation interface. Webcast about this (possibly at URL above)

What comes after threaded-discussion forum?

Analyst-defined visual connection language

Reflective deliberation platform (others being developed at MIT). Similar to Scardamalia & Bereiter (http://www.ikit.org/kbe.html)

Check this chapter out – very good overview of different ways of designing deliberative platforms for deeper learning: http://lilt.ics.hawaii.edu/papers/2008/suthers-2008-cartography.pdf

Videos from ODET 2010: Online Deliberation Emerging Tools – via @sbskmi

http://olnet.org/odet2010

George Siemens

dashboards must be learner-facing, learners should know everything about themselves that institutions know about them.

If people know what your metrics are, they will game them – like content farms.

I first became aware of the explosion in interest around foreign open courses in China when I was asked for an interview by a Chinese reporter writing about this phenomenon (interview in Chinese). Instead of the traditional 开放式课程 (kaifangshi kecheng) – quite a literal translation of “open courses / open courseware”, the new term being used is 公开课 (public / open courses).

A little linguistic aside: Here is what the ABC Chinese Dictionary says about kaifang: “1) come into bloom 2) lift a ban/etc. 3) open to traffic or public use; open to the world 4) be turned on, be in operation 5) hand over a government monopoly to private operations”. For gongkai: “V. make public; make known to the public. S.V. open; overt; public”.

In addition to the above, my initial associations with the two words above (as a non-native speaker) is the following: kaifang was used a lot for the liberalization and reform movement in the 1980′s and going forwards. It can also be used about people to mean that they are liberal, even loose, or “open minded”. It has traditionally been used for open in OER, open source, open courseware, etc. Seeing the dictionary definitions above, it is something that “has been opened” as a conscious act, by someone. Gongkai to me seems to reflect something that is “public”, almost by its nature, where access to it is almost a right.

Either way, there is now a very strong public interest in these courses, with this new catchy name, and a number of website portals have been created, which feature the video lectures from Yale, Stanford, MIT, etc.

While browsing, I came across CICIStudy, and found it to be a site with very clean design, and nice functionality. I was very curious as to who had started it, what their purpose was, etc., and sent an interview to the e-mail listed on their website asking for an interview – which they graciously granted.

I found their take on the developing interest in open courses in China to be very interesting, and I wish them the best of luck in the future!

(The original interview was conducted in Chinese, and the full text is available. I did the translation, and CICIStudy has not had a chance to review it for accuracy. Since the term gongkaike does not differentiate between OpenCourseWare and other open course projects, I have translated it as “open courses” below.)

1. Who are you? What is your background? What kind of an organization is CICIStudy? Is it a part of a larger company, or a company that was just started for this purpose? Or are you just doing it as a group?

Currently our group is not part of any company, and we have not created a company ourselves, everything is in the beginning phase. But this doesn’t mean that it is difficult for us to create an excellent service, the members of the group have backgrounds from large companies such as Microsoft, and are extremely familiar with web development, which guarantees that we are able to effectively develop CICIStudy.

2. How did you first hear about, or get interested in open courses courses? Why do you think they are useful and important to Chinese students?

At the beginning of 2010, I had only barely heard about open courses, and watched a few course recordings. I though it was great, but didn’t pay that much attention. I had no idea that the phenomenon of open courses had grown so large in other countries. In April, I happened to watch a news report about open courses, and suddenly realized the importance of providing a local platform for these resources. That evening, I began to seek out resources, and plan the development of CICIStudy.

Three days later, I began seriously planning and designing the site. After launching the site, I began to understand open courses more in depth. Whether these courses are transmitting knowledge, or spreading moral values, they touch people profoundly.

So far, these courses have not been able to have a large impact on Chinese students – they might watch a lecture after they are finished repeating homework, going to class, and all the other activities of a university student.

However, the courses are still very important – being able to provide access to wisdom for students who seek it is already of extraordinary value. And the real value created is not just limited to students – from the feedback we have received to the site, we know that 60 year olds are equally interested in the courses.

3. What is the goal of CICIStudy? What kind of functionality do you provide?

What CICIStudy wants to provide is a stream of information that contains condensed wisdom. The goal of the platform is for people to be able to very easily receive high quality education (unequal distribution of educational resources has always been a problem). At the same time, learners should be able to receive guidance and communicate with other learners, so that the learning becomes truly participatory, and not just limited to downloading and viewing.

This means that for CICIStudy, how to enable everyone to learn more effectively becomes the key question, and the development of all features must be centred around this goal. Also, we are not at all restricted to only offering open course resources, even if open courses are currently the best, and most attractive resources available, which is why we chose to present them as the main feature.

4. Lately, Chinese white-collar workers have taken an explosive interest in open courses, why? Open courses have existed since 2003, why is it only now that so many people in China became interested in them?

I think it all began with some subtitling groups, which about half a year ago began to subtitle videos from Yale and other universities, on their own initiative. Since this had removed the language barrier that prevented Chinese net users from accessing the videos, and given the wide renown of the universities involved, this led to much interest, and extremely high download/view numbers from video sharing sites. This in turn led to the first newspaper article, which began a continuous stream of news coverage, only serving to increase the general interest in these videos.

In 2003, these special conditions did not exist. The scale and reach of subtitling groups was still very small, and video sharing sites and community portals were also in the very beginning of their development trajectory. At that time, there wasn’t the level of reflection around educational issues as there is today, and finally, the media was not as interested at that time, as they are today. All of these factors, as well as other direct and indirect factors, led the open course phenomenon to explode this year – it wasn’t random.

5. There are other websites that do very similar things to you, for example NetEase Open Courses. What is special about your site, how can you compete with these other sites?

The NetEase Open Courses site is a part of the NetEase video site. Currently, they are focusing on how to overcome the linguistic obstacles, by hiring translation agencies to subtitle videos. CICIStudy is more concerned about becoming a platform that enables learning. Displaying the videos to the users is fulfilling one step of sharing open resources, but it’s not enough. It would be irresponsible to ignore the need for learner initiative and enthusiasm during the learning process.

We hope that the platform will enable us to explore and research effective ways of enabling all students interested in acquiring more knowledge to do so effectively with rich communication and collaboration. On one hand, virtualizing the traditional classroom with it’s socratic interactions provides an atmosphere of deep thought and reflection, on the other hand. On the other hand, utilizing the web’s potential for rich media and immediacy enables us to amplify the advantages of traditional distance education.

Currently, we are still very far from achieving our goal, but whether we will be successful or not, this kind of research is still very valuable.

6. What is your business model? Will you let people pay for accessing courses in the future?

Open Courses are based around Creative Commons licenses, and so far the materials I have seen all require sharing to be in a non-commercial situation. How to find an effective business model in such a situation is still a paradox. We would like to be able to consult with the rights owners to ease the situation a bit, so that we could keep the site running. Of course, if there were foundations or other organizations that could provide financial support, that would be of huge help, but currently we don’t have any leads in this field.

One thing that is certain, is that there will never be a situation where people need to pay to be able to access resources on our site, they will always be freely available to anyone who are interested. Even if we acquired materials that we owned the copyright to, we would not close off access – that would go against our initial principles.

7. Foreign open courses use open licenses, so using them is not “piracy”. Is that important to you? How do you make sure that your website respects the requirements of these licenses?

You are right, this is very important to us, we really hope that we are respecting the licenses properly. We show the CC logo and the fact that this resource is licensed under a CC-license on every video page, but there might still be some oversights.

From my personal point of view, the understanding of CC licenses in China is still very weak. We are still in the phase of stressing the importance of copyright protection, and the concept of the creative commons has not gotten wide recognition yet. People have not yet reflected much about the meaning of knowledge sharing, and abuses of open licenses.

8. What do you think about the Chinese National Top Level Courses Project? Are there excellent resources there as well? Will you consider including resources from Top Level Courses on your website in the future?

It’s very difficult for me to give an accurate assessment of the Top Level Courses, since I have not reviewed these resources systematically, although I am sure that the courses contain many excellent resources. I have never participated in the production of Top Level Courses at university-level, but I helped out as a student in the production of video courses at my high school. The teachers who were delivering the video lectures would spend a lot of time preparing, beginning almost a month before the lecture took place, and they made sure to use multimedia appropriately.

This left a deep impression on me, at that time I even created a Flash website as assist in the learning. The Top Level Courses Project probably also has a lot of really great websites that promote interaction. However, when I went to the main portal site, I only saw a point-based system for downloading resources, which limits the distribution of these resources.

In the future, if the licensing problem gets resolved, I hope that we can put these resources on our website and make them available to all of our users. There are many great lecturers in China, and their Top Level Courses can be great for learning. It’s a real shame that we are currently unable to share these courses.

9. What is your future development plan for your platform, and group? New features, contents, services?

The development of CICIStudy has already reached milestone number 3, and between the 10th and 15th of January, we will release new functionality to the site. This will include taking course notes, and rating these among users, collaborative translation, search functionality, better organization of the lecture videos, etc. We are currently exploring and researching better and richer ways of studying.

Thank you very much to CICIStudy for generously agreeing to this interview, and for providing Chinese learners with more opportunities.

Building off the last post, badges are nothing more than .png files unless they are backed by some assessment and value.  I have been working on defining what assessment looks like in these peer learning, open education environments and it has really been a mind-blowing journey so far. When I first started trying to grasp the task at hand, I realized very quickly that ‘assessment’ mean a lot of different things - it can be the thing that you do to prove that you have learned something (like taking the exam), the design of that thing (question type/writing), the delivery of that thing (paper or online, ‘assessment engines’), the act of comparing the work/answers to some rubric (grading the exam), or the end product itself (the grade).  So needless to say, there are a lot of moving parts to think about when approaching the concept of assessment in general.  But then when thinking about it for these participatory, peer learning environments, there is much further to go.

These environments are intentionally atypical, and with that comes benefits and limitations (in general, but that’s another post, for this one +/- for assessment):

They are open and accessible to anyone with network access.  

What this means for assessment: There will be more people across many different levels and proficiencies that view and/or participate in these courses. The assessments should provide options for these levels and help learners build on their existing skills and develop new ones.  Further, because these courses are open, there is the likelihood that people will float in and out and assessments should allow them to do so, and ‘check’ their knowledge without forcing them to complete the course (if the topic or skills are redundant their existing capacities), but at the same time, assessments should provide milestones to motivate learners to stay engaged in the course as well. 

They are decentralized, meaning that there are not “core” courses or particular paths/sets of courses that people are forced to take.

What this means for assessment: The concept of prescribed degrees does not work here because learners will have unique learning paths across various courses, and even various websites or platforms. Further, the set of courses is not predefined and there may be overlaps, meaning different learners may learn the same skill in different places in different ways.  So the assessments need to be granular enough to capture the learning wherever it occurs, and flexible enough to allow learners to demonstrate the skill in contextual and relevant ways.  Assessments should also be relevant outside of the assessment context itself, and allow people to submit existing work or challenge them to create something meaningful to them to demonstrate competency.

They are peer-driven, and the person organizing the course is not necessarily an expert, but simply guide or facilitator.  Their main goals are to foster a community of learning and provide some scaffolding to guide that community through collaborative learning of a particular topic.  Therefore, there is not the authority figure or typical concept of an instructor.  

What this means for assessment: Short answer, grades won’t work. The simple reason grades ‘work’* in formal environments is that we are preconditioned to expect/accept the instructor-student relationship. The instructor is the expert that pushes information on us and give us top-down ratings of our work and learning**.  But that doesn’t work here.  There are no authority figures - peers are learning from each other and from the interactions and activities. So the assessments need to reflect those relationships and should capitalize on peer assessment as much as possible.  Also, output of the assessment should be more than a flat grade or mark, but should be focused around feedback and guidance.  Also, because these are not expert-driven environments, the assessments need to build in or account for trial-and-error types of approaches.  Learners should be able to learn from the assessment and refine work if they have not met the requirements, etc.

They depend on community development and engagement to be successful. 

What this means for assessment: Again, peer assessment should be incorporated as much as possible.  But we should think about skills and behaviors that support community and build those into the assessment scheme as well.  Perhaps there are lightweight ‘assessments’ based on interactions with peers or automatic assessments and feedback/awards based on behavior through the online learning environment.

I am sure there is more.  And you have noticed that I have intentionally kept badges out of the conversation here.  That’s because badges and assessments are different things.  The badge is the signal of a skill or competency and the assessment is the way to demonstrate/validate those skills.  In our model, each assessment will be tied to a badge, but also in some cases multiple assessments will be tied to a single badge, giving people flexibility in how they demonstrate the skill and earn the badge.

So in summary, for our pilot, the key assessment considerations are:

• Incorporate peer assessment as much as possible
• Provide levels of assessments/badges to meet various needs, as well as help motivate people to build skills or continue participating in courses
• Provide multiple assessment options or paths to the badge
• Assessments should be relevant outside of the learning context - and should allow for submission of existing work, new interesting and relevant work, and/or peer recommendations or nominations. 
• Learners should be able to seek out assessments on their own - nothing forced.  (although there may be cases for automatically assessed and issued badges to promote community behaviors)
• The badge should link back to the work submitted for the assessment, and any feedback or endorsements from the assessors.
• …

I will share the plan over the next couple of weeks and we forge forward.

-E

*I actually started this post with a diatribe against grades and traditional forms of assessment but so many others have expressed it so much better.  I particularly love Cathy Davidson’s (of HASTAC) thoughts on the limitations and obsolescence of grades:

http://dmlcentral.net/blog/cathy-davidson/thought-experiment-why-grade-why-test-what-if

http://www.hastac.org/blogs/cathy-davidson/those-who-dont-grade-learn

http://www.hastac.org/blogs/cathy-davidson/my-response-ny-times-quest-explain-grading

**I have definitely drunk the student-centered kool-aid. From the existing literature and research (not cited here but I can definitely provide), we know that students learn more when they can construct their own understanding of ideas and connect them to their own lives.  We know that people learn MORE and when they can collaborate and interact.  We know that students are more engaged when they have more control within the learning environment. We know that deeper understanding comes from trying out various strategies, getting things wrong, revising, etc.  It’s not enough to have some one push information on us, we need room and flexibility to mash up that information, get our hand dirty, connect it to something that we care about, hear the interpretations of our peers, etc.  I have written and spoken a lot about this to date and I am sure it will make it into the blog over time. But this is one of the reasons I love P2PU and other social learning efforts that recognize and embrace this shift to student-centered, participatory learning.  It’s the future man. 

I am currently working with Peer-2-Peer University (P2PU) and Mozilla Drumbeat to integrate assessment and badges into the open and peer learning environments on P2PU, specifically the School of Webcraft. We’ve been doing a lot of thinking about this and I am finally getting around to capturing my thoughts here.  I should get a badge.

What are badges? 

Come on, you’ve seen them before.  Boy Scouts. World of Warcraft. Foursquare.  I do something, demonstrate some skill, defeat some monster, show up in some location, meet some predefined criteria or assessment…and I get a badge.  If I know about the badge, I might be motivated to do the necessary behaviors or meet the requirements to get the badge, or if the badge is a surprise, I might be motivated to keep exploring or trying out various things to earn or unlock more badges. Once I have the badge, I can display it so that others can see it and thus demonstrate my skills or achievements.

There are many crossovers here with learning - motivation, feedback, exploration, achievement.  

Why do we need badges?

Well, we need something.  Is it badges?  Maybe, maybe not.  But there is no question that we need an alternative form of assessment and certification (although I hate that word…it conjures up images big, mean Microsoft gorillas). Here are a few reasons why we need a change:

• In the current system, the institutions (schools, universities, etc.) have the all the control. They decide what types of learning are “official” and what “counts”.  But most learning doesn’t happen within those confines and constraints and there are lots of examples of people learning outside of the system: open education courses and materials, afterschool programs, peer discussions, books, Wikipedia, the Web in general, LIFE…learning happens everywhere.  But it only counts if it happens through an institution.  Why? Why shouldn’t the learner have control?
• Current models of assessment (grades, rankings, etc.) currently don’t work well for many kinds of learning - in fact, many argue that they don’t work well for most learning.  In peer learning environments, grades and rankings do not encourage participation and information sharing, and in fact, can constrain the interaction and learning.  In informal learning environments, these models make it feel like school, squashing the inherent value and engagement.  In many open education environments, there is not often a dedicated instructor or authority figure to issue the top-down grade. And so on.
• There are so many important skills and competencies, some age-old and some new(ish) in today’s world, that are not currently captured or acknowledged. Things like the often referenced 21st Century Skills, or New Media Literacies, which cover everything from information organization and evaluation, to negotiation and trial-and-error prototyping. Or the “soft” skills like critical thinking and teamwork.  None of these skills are captured in my credit, grade or degree.  And yet, these skills are critical to most careers and are often some of the key things that employers are looking for. As a learner, it is difficult, or impossible, to know to seek out or hone these types of skills because they aren’t acknowledged or encouraged…and yet they will be glaringly apparent the first time I fub up in a critical situation that involves one or more of these competencies. When I am applying for a job - my resume and education history tells potential employers nothing about my full set of skills and if I have any of these other competencies. And when I am looking to hire someone, I have to come up with clever questions to try to get a complete picture of someone (above and beyond the resume and education history which everyone knows is a limited resource) in 30 minutes. 

Badges?!

What if there were badges for various skills that you could collect across learning experiences, carry them with you and then share out to various audiences as needed?  You may earn badges that represent more traditionally recognized behaviors or skills like completing a course or mastering a mathematical model, but you could also earn badges for softer skills like critical thinking, teamwork and information analysis.  You could earn badges from authorities, like Mozilla, from course organizers where appropriate, from peers or even from yourself.  The badges would be associated with assessments that once successfully completed, earns you the badge.  There might be multiple assessment paths to a single badge, giving you the flexibility to have a unique and personalized learning path.  But you could also look at the badges of other people to discover things to learn or try for…or what skills to develop or hone for particular disciplines or jobs.  You could even (possibly) carry the badges back to the institutions with you to get credit or help them cater that experience to your interests and needs. 

So that’s what we are currently exploring.  Of course, there are many unanswered questions, some of which I am sure are springing to mind as you read this.  Questions like: What skills should we assess? Are there skills that are better left unassessed?  What do we want to encourage?  How do we avoid encouraging the “wrong” behavior? Who gets to decide which skills to assess? How much influence should outside stakeholders, such as employers, have on badges?  Should they be able to design assessments and badges that are relevant to them?  How can we let them have a say without creating an imbalance in the system or constraining the learning? How granular should badges be? For example, our HTML5.0 badge is at the level of the entire language mastery, but would we want HTML tag level badges?  What granularity is the right level?  Do badges aggregate into larger or higher level badges? Should badges expire?  How do we deal with skills that need to be refreshed or renewed?  How can the badge system grow with learners? How does the introduction of badges affect learner motivations?  If learners were initially intrinsically motivated, how do we avoid “crowding out” those motivations with an extrinsic badge system? How will people game the system?  How much will they do so? How can we discourage gaming or recognize when it happens? Will these badges translate to formal learning environments? And if so, how?  What would be required to make schools or institutions value or accept badges?  Can we meet those requirements without changing the nature of the learning environments?

There are a lot of questions and a lot of unknowns, but we need a change…we need to give the learners the control.  So this is one way we are hoping to accomplish that.  We are building a badge/assessment pilot in the January session of the School of Webcraft, which is a subset of P2PU courses focused around web development and endorsed by Mozilla.  We are hoping to have a core set of badges and assessments, as well as the initial infrastructure to support the issuing, collection and displaying of badges over the next month (or less).  We plan to learn a lot and start to answer the questions above.  But we can’t possibly answer all of these questions alone.  We hope to encourage more interest in badges and these new approaches, get more people researching them and issuing them (within the same open infrastructure ideally) and figure this out together.

I’ll keep you updated as much as possible here.  So buckle up!  Next up, thoughts on assessment and the open badge infrastructure…

-E