Asking the right questions

I mentioned previously that I’m hosting a series of seminars at the Open University, with the intention of showcasing our own expertise (internally and externally) and also getting us an institution to engage with current ed tech developments.

Last Thursday my colleague Prof Bart Rienties looked at analytics. Not so much real time, but rather digging into large data sets across multiple courses to explore what it is that students actually do. This is always difficult to ascertain, but particularly so for a distance education university when you don’t get to see what they’re doing. We ask them of course through surveys, which are useful but these aren’t always reliable – people often forget what they actually did, over or under-estimate times spent on certain activities and sometimes give answers they think you want to hear. There are problems with analysing data sets too as you are sometimes inferring what the data means in terms of student behaviour (eg is spending a long time in the VLE a sign of engagement or that they’re struggling to understand?), but combined with other tools it provides some fascinating insights.

Bart framed his talk around ‘6 myths‘ we have regarding OU student behaviour:

    1. OU students love to work together
    2. Student satisfaction is positively related to success
    3. Student performance improves over time (eg from first to third level courses)
    4. The grades student achieve is mostly related to what they do
    5. Student engagement in the VLE is mostly determined by the student
    6. Most OU students follow the schedule when studying

The answer to all of these is “No” (or at least “Not that much”). You can see the talk below to see Bart expand on them, and it’s not my place to expand on his work. But I’ve been doing a lot of thinking about these since the talk. Initially they might be deemed a concern, but perhaps they are also highlighting positive behaviour. For example student behaviour is largely determined by what we set out in the course (no 5). That seems like an effective outcome of good learning design, particularly for distance education students. Similarly, while students don’t slavishly follow the course schedule (no 6), with many studying ahead or just behind the calendar, this can be framed as part of the flexible (and accessible) design. Some online courses from other providers are very strict – you have to do task X on Monday, contribute to discussion on Wednesday, complete the quiz on Friday, etc. For our students such a regimented approach would not work, many students know they will be unavailable at certain times for instance and so study ahead.

And so on for each of these. What this highlights for me is that the work Bart and his colleagues do at the OU is essential in getting us to ask the right questions. We are all guilty of making assumptions about student behaviour I think. The kind of analysis undertaken here is not seeking to automate any process or remove people from the system (often the fear of an analytical approach), but rather encourages us to dig deeper into what is happening in the very real lives of our students. And from this we can design better courses, support mechanisms, assessment, etc.

Here is Bart’s talk:


  • Val Bentinck

    Interesting set of facts thank you. When you refer to teaching quality this seems to be strongly related to course design and not the role of associate lecturers or the ‘scaffolding’ undertaken by them in learning events and the copious feedback on assignments, so was this factored in? You have very much focused on numbers rather than how students feel about their achievement? Also I am not too sure manipulating the audience with Vodka reward is not too different to the cookie manipulation? Look forward to your reply.

  • Val Bentinck

    Hi Martin, I had been provided with this link and another by my Ed Doc supervisor, so it was really interesting to see how tech and online work is viewed from central rather than AL perspective although in the past I have had one of your books pass through my ever-growing references I was directed to keep my focus more to practice issues, however my analysis is that there can be no separation in reality. Yes, I agree humour is needed! A refreshing change. Bart has kindly offered to signpost my work as I was being supervised by John Richardson so I should hopefully be able to chat with Bart at some point. Thanks for replying.

    • mweller

      Hi Val, sorry I replied in a hurry last time so was a bit terse, I didn’t mean to be. In agree it’s difficult to separate the two, indeed for one of Bart’s myths I thought it actually demonstrated that design and what students did with their AL was pretty closely matched. Not sure, but this paper on course design and retention may be relevant for your research http://oro.open.ac.uk/57277/

Leave a Reply

Your email address will not be published. Required fields are marked *