Tag Archives: Online Course

https://flic.kr/p/aH8CPV

Reading: How should we measure online learning activity?

I’m very interested in learning analytics in general, but probably more so in their application to how they can be used to benefit both the academic team (to help guide and improve the offered course) and the students so they can be used to help them guide and motivate them through the learning materials.

This paper ‘How should we measure online learning activity?‘, from the ALT journal Research in Learning Technology, is a good place to start looking at the metrics involved, and how we should measure/interpret them too. If you’ve time, read it too. Let me know what you think?

The proliferation of Web-based learning objects makes finding and evaluating resources a considerable hurdle for learners to overcome. While established learning analytics methods provide feedback that can aid learner evaluation of learning resources, the adequacy and reliability of these methods is questioned. Because engagement with online learning is different from other Web activity, it is important to establish pedagogically relevant measures that can aid the development of distinct, automated analysis systems. Content analysis is often used to examine online discussion in educational settings, but these instruments are rarely compared with each other which leads to uncertainty regarding their validity and reliability. In this study, participation in Massive Open Online Course (MOOC) comment forums was evaluated using four different analytical approaches: the Digital Artefacts for Learning Engagement (DiAL-e) framework, Bloom’s Taxonomy, Structure of Observed Learning Outcomes (SOLO) and Community of Inquiry (CoI). Results from this study indicate that different approaches to measuring cognitive activity are closely correlated and are distinct from typical interaction measures. This suggests that computational approaches to pedagogical analysis may provide useful insights into learning processes.

O’Riordan, T., Millard, D., & Schulz, J. (2016). How should we measure online learning activity?. Research In Learning Technology, 24. doi:http://dx.doi.org/10.3402/rlt.v24.30088

Image source: Neil Cummings (CC BY-SA 2.0)

Learning Technologist pt.15

What is a Learning Technologist? Pt. 15

In May 2015 I joined Warwick Business School, WBS, as an eLearning Consultant. In September the same year I was awarded the ‘highly commended’ Learning Technologist of the Year award from the Association for Learning Technology (ALT). The strange thing is, that was the last time I posted about being a Learning Technologist here. After 14 posts I stopped.

There’s no reason for it, I didn’t even realise I’d done it until a few tweets last night from Clare Thompson (@ClareThomsonQUB) and Sue Beckingham (@suebecks) reminded me about it. Yes, I’ve continued to write about work and wider reading of the industry we’re in, but this is Clare’s tweet that prompted me to write here again, about being a Learning Technologist:

In the last two years I thought of and collaborated on, edited and then self-published  The Really Useful #EdTechBook. I’ve developed, supported, mentored, facilitated, and bled/wept over the creation of two MOOCs for the University of Warwick (Big Data and Literature and Mental Health). I’ve facilitated a total of 15 runs/presentations of all five Warwick MOOCs. I’ve two other MOOCs in development at the moment, one of which took myself and colleagues to Italy recently to interview and film important individuals for case study and ‘thought’ pieces who were attending an event in Prato Centre, Monash University, Italy. Oh, and I’ve met & interviewed Sir Ian McKellen and Stephen Fry, all part of the day job!!

YouTube: Literature and Mental Health

Outside of work on MOOCs I’ve been included on the EdTech Magazine list for 2015 and 2016 lists for ‘Top 50 IT Blogs Influential Blogs in Higher Education’ and the 50 Most Influential HE Professionals Using Social Media list. I was interviewed for the published work on How has Apple transformed your classroom? Part I, the Teacher’s Practical Guide to the Flipped Classroom and wrote this article on ‘Facilitating the Unknown’ in a Special Issue: Open Facilitator Stories, based on the amazing online course BYOD4L, and been involved in multiple weekly tweet-chats from/on LTHEchat.

And these are just the thing I can remember off the top of my head. Perhaps I should be more organised and keep better notes? Oh, and I still Sketchnote.

The great thing is that I have the interest and passion to do all this, all the time. I love being connected and in a position to collaborate or share knowledge and experience. I love that I can swap roles and identities so quickly depending on what the day brings – technical support, pedagogic support, management or administration, etc. No day is the same. No email asks for the same thing. No meeting covers the same thing (well, not very often).

Like Clare I find that IRL meetings can be awkward, conferences can be draining, events can be difficult to get everything done I want and see everyone I want to see AND still have time for the event itself. I’ve been reading the Quiet Revolution, about introverts, and engaging in their regular tweet chat. Having the time to reflect on events and conversations is important, and sometimes events can be so hectic there simply isn’t the time:

Oh, one final thing .. there’s always room for Lego! Pictures of Lego, actual Lego kits, or just talking about Lego.

Image source: clement127 (CC BY-NC-ND 2.0)

what happened to all the MOOCs?

What happened to the MOOCs?

MOOC = Massive Open Online Course

You knew that already, yes? Here are my thoughts for a Friday afternoon.

Massive – Yes, these courses are usually large. But anything that isn’t constrained by the number of chairs in a room has this potential. A course that has 200 people on it from a provider that has rooms that can cope with no more than 150 people would call this ‘massive’. We know that FutureLearn has had the largest ever online course (I’m not calling them MOOCs anymore) with 440,000 registered for one course, but we’ve yet to see the stats about how many who signed up actually started it, completed more than one week, even completed the course? I wonder if the larger numbers are reflected in the percentages of these ‘completors’ and whether they’re better/worse than those with (much) lower numbers (e.g. 5-15k sign-ups) or figures from other providers?

It’s a little strange that the Guinness World Record website has no mention of FutureLearn or online course, does that still make it a valid record?

Open – Yes, they’re open, but it’s increasingly difficult to find the ‘open’ version (especially on Coursera), you’ve really got to hunt for the link in amongst all the ‘specialization’ and ‘pricing’ links. If you didn’t know what to look for you’d be forgiven for thinking these MOOCs are not free. Open is also about the lack of requirement/prerequisites to already be educated to a particular level. Open, in this way means we can all try something we’d otherwise have to complete an application for (and pass).

Online – Yes, they’re online. Well, they are available to everyone, so long as everyone has access to a computing device and an internet connection. I would like to say personal access to a device and access to a reliable internet connection, but I appreciate this isn’t always the case.

Course – Yes, they’re a course; a collection of articles, videos and activities, maybe with discussion points dotted here or there (for social learning), and probably a test or end ‘assignment’ to prove you’ve learned something to qualify you for a certificate.

My point here is that I am seeing less and less, on the courses or platforms I see, that resemble MOOCs I saw two years ago. MOOC providers have to make money, yes, so there needs to be a way for them to make it, and statements and certificates and the like is a good way to do this. I’m just not sure we’re creating MOOCs for the reason we started – are we trying to force the learning into a model that is, essentially, for-profit now? What about courses that are really just about learning something new, not for CPD or to further a career, those that don’t have something that can be tested? Do you force a test or assignment just so an arbitrary mark can be assigned, therefore completing the numbers & stats in order for a mark/grade or completion rate for a certificate to be awarded?

Are MOOCs (and what we used to refer to as MOOCs) about learning or, as it seems now with exams, tests, assignments, certificates, etc. about the testing and payment options?

I’ll hang my hat firmly on the peg and say that, in the original and purest ‘ideal’ of a MOOC, MOOCs should be about expanding your knowledge, in any subject, for your own reasons and in your own time. Whether there is a paid-for option at the end (provided it’s still free for all at the start and everything inside the course is the same) shouldn’t matter. But it feels like it does. It feels like the commercial aspect is taking over. I hope not.

Image source: ms.akr (CC BY 2.0)

MOOCopoly

MOOCopoly

While searching for something else, I found this fantastic image/resources from Alan Levine – MOOCopoly:

“MOOCOPOLY is the game of teaching massively, openly online and includes the range of players from DS106, CCK08 to the Stanford trio, MITx, and more. Take your chance and many you will have community or not. See more about the making of this monster cogdogblog.com/2014/05/03/moocopoly-the-game/

Fantastic. While it’s based in the DS106 and other specific courses, there is much here for everyone who uses or is involved in MOOCs to take away and adapt. I can see a conference poster coming from this … ;-)

MOOCopoly

Click for original Flickr image: Alan Levine (CC BY 2.0)

MOOC success

Making a MOOC ‘successful’

Designing a ‘successful’ MOOC is one thing. Making a MOOC ‘successful’ is something completely different.

Much has been written by far better and more eloquent people than me (here and here and here and here and here) on what makes a successful MOOC – all about interactions, journeys, optimum length, appropriate materials, platform, etc.. But what about making a MOOC successful? To me, there is a difference.

This isn’t about making / building / designing a MOOC, it’s about making / encouraging / promoting / informing people about the MOOC.

The argument about MOOC success, learner retention, completion numbers, registrations, etc., is one that will rage on and on, everyone has an opinion, everyone looking at something different, all very valid, and all very important questions. There isn’t a definitive answer – each MOOC is different, for a different audience, for a different demographic (maybe), and designed in a way that different learner ‘profiles’ can get something different out at the end. If indeed they reach the end, which of course they don’t have.

No, making a successful MOOC requires more than a lead academic(s) subject knowledge, learning technology, instructional/education design, assessments, an appropriate learning goal/journey, working platform, etc. You need all the other stuff as well.

The other stuff you need? Well, try:  Continue reading

MOOC quality

MOOCs – question on purpose, quality, student retention, feedback, etc.

Ahh, questions around the purpose, quality, value, etc. in and around MOOCs have started again, and justly so.

  • Disclaimer: Like many I have opinions, but not answers.

The recently raised questions, started by Fred Riley on the ALT mailing list, have produced a good set of resources for those of us who are starting to ask these questions, needing a more comprehensive or value-added answer.

Fred’s original query was:


Does anyone on this list know of any recent research and/or articles on the teaching quality of MOOCs? I’m thinking of things such as:

  • student retention, with MOOC drop-out rates being notoriously high (I plead guilty to that myself :( )
  • student surveys and qualitative feedback
  • how many students in a MOOC platform (eg FutureLearn) go on to take further courses in that platform

I’m sure that there are many other indicators of quality – those are just off the top of my head. I’m not in the MOOC game myself as yet, other than as a punter, but I’m looking to get into the development side of things.


In some instances, especially around the data of students/learners taking further courses (across MOOC platform providers as well as within) is difficult, but I hope we can get to a stage where this kind of data is available and open to interrogation (if only for the individual partner to  query their own courses).

Here are some of the resources shared, in response to Fred’s original query:

If you have any further links or resources that would help Fred and the ALT mailing list, please reply to the thread on the mailing list. If you don’t have access then please leave the link or your comment below for everyone to have the opportunity to read.

Yes, OK. Fred’s question also raises the question around the ‘quality’ of a MOOC, the validity in the data of learner retention or ‘steps completed’ as triggers for saying a MOOC is of a certain quality, or the student was ‘successful’ on the course, but these are for another post. Fred answered this quite clearly on the ALT mailing list that, for him “retention is IMO and indicator of quality as perceived by the student – the better retention, the more students are engaged with the course and its materials. If they don’t like a course, they’ll drop out.”

NB: I’ve helped run several runs of the Warwick/FutureLearn ‘Shakespeare and his World’ MOOC and use this as an example I use where the statistics provided for the 10 week course don’t necessarily match the actual experience. Case in point is the number of learners who complete the course, in that they take all the tests and mark at least one step as complete in each of the 10 weeks. We know from the learners themselves, from their comments, feedback, tweets, etc., that they take what they want from the course – one learner may only like Shakespeare’s comedy’s, another likes on his tragedy’s, so they will omit the plays/weeks they don’t like. They should still be viewed as a successful learner, and I’;m sure they think that of themselves, as in their own mind (and in ours!) they got what they wanted from the course, yet did not actually ‘complete’ it.

If there is one question for 2016 and MOOCs, it’s whether there is any way to really truly, honestly, understand the ‘value’ of a MOOC?

Image source: State Library Victoria (CC BY-NC 2.0)

ALTC 2015

The Interview Process #altc

From this year’s ALT conference I enjoyed (finally) meeting Wayne Barry, EdTechBook contributor, and chatting about his ALTC presentation.

Wayne’s presentation looked at a different way of interviewing candidates for Learning Technologist positions using standard questions and short presentations, but also the inclusion of a short role-play exercise. Each candidate is given advance notice that they will engage with an ‘academic’ who is interested in introducing elements of distance learning to their module. During the short exercise (many people took issue with the use of the term ‘role-play’) candidates will exhibit both knowledge of their discipline as well as the ability to listen, engage, problem solve, and debate with a member of the team taking the role of an academic.

So, how do you find out if someone will fit in to your office and team environment? Can you do this by just questions? Do competency based questions offer enough space for someone to fudge their way through the process, or rather offer the interviewers enough insight to see the tRuth behind the candidate?

This reminds me of this video, from Heineken: Job Interview. Slightly over the top, but you get the idea – by changing the process you find out many different things (hopefully good) about the candidates. Enjoy!

YouTube: Job interview at Heineken

Digging Deeper into Learners’ Experiences in MOOCs:

Reading: Digging Deeper into Learners’ Experiences in MOOCs

One aspect of working on MOOCs is that there is no clear way to measure it’s success. Do you use the stats and logs that indicate clicks and time-on-page, or look at the nature of the conversations and/or comments made?

That’s why this paper loaded to Academia.edu by George Veletsianos piqued my interest – is there something in here that can help me understand the metrics we need to use in order to measure the learning and/or success of a MOOC?

“Digging Deeper into Learners’ Experiences in MOOCs: Participation in social networks outside of MOOCs, Notetaking, and contexts surrounding content consumption.”

Unsurprisingly the authors highlights the lack of literature around MOOCs that look into the metrics of MOOCs that are not captured on the MOOC platform (EdX, Coursera, FutureLearn, etc.), notably the social engagements, note-taking, and content consumption. Something I’d not considered before is the “availability of large-scale data sets appears to have shaped the research questions that are being asked about MOOCs.”  Continue reading

What makes a good online course?

What makes a good online learning experience?

Is it possible to define the qualities of what makes a good online learning experience, or a good MOOC? Is there a check list we could have pinned to the wall which we could use as we design and build our courses?

Here’s a few items I think the list needs, feel free to add your own ideas in the comments field below:

Presentation: Is the student able to relate to the subject and the presenter / educator? This is not always easy as the platform (Blackboard, Moodle, FutureLearn, Udacity, etc.) often controls how the materials are ‘presented’. Even with these constraints you do have options on designing your materials and laying them out in ways which make them easy to navigate or interact with.  Continue reading

#FLbigdata

Big Data Videos #FLbigdata

I’ve already posted these videos before, but I thought I’d post them here again, in one place, as a good resource for the learners on the Big Data FutureLearn course that started today.

All of these have one thing in common … do you know where your data goes, or who is watching/listening/capturing your data?

Tom Scott – I know what you did five minutes ago / YouTube


 

Hot on your trail: Privacy, your data, and
who has access to it / YouTube

Tom Scott – Social Media Dystopia / YouTube

Jack Vale – Social Media Experiment / YouTube

Digital Dirt / YouTube

Everyone knows Sarah / YouTube

Do you have any more you’d like to showcase and bring to your fellow FutureLearners? Drop a comment with the link below.