Last post for Evaluation for Best Practice paper

Well I've done it - I passed!!!

Thanks to Bronwyn for being so understanding while I struggled to catch up constantly these past few months, but I made it in the end.

On reflection I found that although I cannot say I enjoyed the study, I actually did like analysing the final evaluation of the data and producing the final report.

I got some really useful data from my study and hope to implement this both for my final paper - the PROJECT - and also furthering on from that in future online resource development in the Tourism and Travel area at UCOL.

The last paper of my qualification is not blog based, so I probably won't be posting much here at least for the rest of the year, but may do an odd posting if I get time.

Onwards and upwards...
:-)

Final Report - Assignment 4

I think I've done it! Has taken me so much longer than I had hoped but here is the final report for my evaluation. The link to Google docs is below
http://docs.google.com/Doc?docid=0ARpw4K3nyVioZGhwOGhtYjJfM3JuOHc2NmM&hl=en

I can't say I have really enjoyed this one, I don't like being on the back foot all the time, and seemed to struggle to catch up continually...

Will put my final thoughts on this evaluation process soon.

Feedback on Data Analysis

Hi all
The link below takes you to Bron's feedback for my data analysis. I have taken her comments on board and will adjust the final report accordingly. (I had already done some of what she said already as I've already started doing the final plan, but will change some other parts for Assignment 4.

http://docs.google.com/Doc?docid=0ARpw4K3nyVioZGhwOGhtYjJfMmdzenE4ZGNq&hl=en

Thanks also to Joy for giving me great feedback too.

No idea why but my lovely tables and graphs are not in the Google doc version below...

Have emailed Bronwyn my original document, it looks so much nicer when formatted in its original form.

Data analysis

Finally I've finished analysing my data. Here is the link to Google docs (the formatting has been thrown off a little though)

http://docs.google.com/Doc?docid=0ARpw4K3nyVioZGhwOGhtYjJfMmdzenE4ZGNq&hl=en

Six questionnaires were handed out and completed by the Certificate in Global Travel students by our programme administrator on June 22nd after students attended the last assessment for their programme (therefore providing a 100% response rate).

The purpose of the questionnaire was to assess the effectiveness of the current UCOL online resources used for the Tourism and Travel “Destination” unit standards as part of the Certificate in Global Travel Level 4, and to identify areas for improvement.

The questionnaire helped gain an insight into what the students found useful and appropriate for their on line learning. However the focus group generated much more beneficial discussion about the usefulness, appropriateness and ability of online resources to help in students learning and information retention. Students were receptive to using new technologies and had many practical and constructive comments to be taken into consideration when planning and implementing the new on line resources to be developed for introduction in 2010.

So once Bronwyn (or anyone else) has given me feedback on this, I can get onto the final report! Phew, this has been a hard uphill struggle for me!

Draft Evaluation Plan (version 2)

What a month it has been! After being all enthusiastic at getting back into the paper I then hit the ultimate brick wall - marking! Seriously I have been marking solidly for 4 weeks now (albeit it with a 5 day break in between to Queenstown for a trip away with the students to mark the end of their course!) - yes I know - poor me!


Anyway I now am trying frantically to get back into it (again).

This is my second draft on Google docs of the Evaluation Plan after Bronwyn's comments and includes the appendices (the questionnaire and the Focus Group questions). Here's the link:
http://docs.google.com/Doc?docid=dhp8hmb2_1fw6p22ch&hl=en

I have the results to analyse next... yes I know I'm so far behind!! I will get there eventually!

Draft Evaluation Plan

I've been working really hard this week to catch up, using every spare minute (helped by the fact a couple of my classes did not run this week) so have managed to get a Draft done of my Evaluation Plan.

I know I'm way behind most of you, but any feedback you have time to give would be most welcome!
I hope this link to Google docs work - I'm not sure what I'm doing, but just in case, a copy is also below... http://docs.google.com/Doc?docid=dhp8hmb2_0c5mfpvvg&hl=en

==============================================================
Draft Evaluation Plan

Introduction
This evaluation plan will evaluate current students’ opinion on the quality of some of the BlackBoard resources currently being used at UCOL in the tourism and travel section, as well as highlighting possible areas for improvement.

The final report of this evaluation will be forwarded to UCOL management however the main reason for this evaluation is as an assessment for the Evaluation of eLearning for Best Practice paper, which the author is completing as part of the Graduate Certificate in Applied eLearning from MIT.

The author of this report and evaluator of data is Elaine Dittert.

Background
The UCOL Certificate in Global Travel Level 4 incorporates the National Certificate in Travel Level 4. Two unit standards taught in this programme are Unit 18213 Destination United Kingdom & Eire, and Unit 3727 Destination Pacific Islands. Both these units are 10 credits each, so are considerably large in their content as well as assessment value. UCOL had previously been approached by several part time students working in industry, wishing to complete these papers on their own without attending formal classes, and so four years ago an "e-learning solution" was developed on the Learning Management System (LMS) currently used by UCOL - BlackBoard. This site has since been continually refined to the point now where not only is it used by "on-line students" but it is now also used as a basis for the face to face Destination classes conducted in the Level 4 programme. Positive comments have been received from these f2f students on how useful they find this resource in completing their assignment work.
The current Level 4 students have had to rely on this information much more than normal to complete their work this year due to the staff member being ill, so they are prime candidates for this evaluation study.

Further to this UCOL management has put in a request to the tourism and travel section to transfer any information currently on BlackBoard into Moodle which is the new preferred LMS at UCOL. This provides an ideal opportunity for tourism and travel staff to update and improve the resources available to the Level 4 students.

Purpose
The purpose of this evaluation is to assess the effectiveness of the current BlackBoard online resources used for the “Destination” unit standards and to identify areas for improvement.

Limitations
There is a tight time restriction, as the students to be targeted leave UCOL at the end of June, so evaluation will have to be conducted very quickly. Also the number in the Level 4 group has dwindled to only 8 students, so the sample size is very small.

Audiences
The results of this evaluation will be reported to the Head of School for Tourism and Hospitality at UCOL. In conjunction with staff the best ways to improve the current online resources for the Level 4 Destination unit standards will then be discussed.

Decisions
Based on the results of the questionnaire and interviews decisions will be made on the content of the new Level 4 Destination resources to be made available on Moodle.

Questions
Using the Likart scale would be the preferred method for the questionnaire, where students rank their answer from 1- 5 (eg. 1 strongly disagree to 5 strongly agree). This will be followed by a space for comments on what they liked/did not like about the online resources. This will provide an opportunity to see areas for improvement.

The two Guidelines that will be investigated from http://elg.massey.ac.nz/ are:

ST 8: Rather than just recalling facts, does the course help the student digest, reflect on and review new learning?
Sample Sub statements to be evaluated:
The basic information in each section of the Destination Units was very interesting
The review tests at the end of each section were very useful to my learning
The on-line quizzes included in the BlackBoard site helped with my learning
The BlackBoard site was confusing and hard to understand
The BlackBoard site helped me remember information on the Destination

ST9: Do the technologies employed help students successfully meet the learning outcomes?
Sample Sub statements:
The video clips made available from Getaway were very useful
The information on Blackboard was always easy to download
The links to web sites for transport and accommodation were very useful
A Facebook group set up to aid in your study of Destinations would interest me
I would be comfortable using technology such as Second Life in my study

Methods
This evaluation will be based on the Eclectic – Mixed Methods Pragmatic Paradigm using Multiple methods to conduct the evaluation. Participants will be asked to complete a questionnaire and will also be part of a small focus group to be interviewed.

Sample
As previously stated that participants of this evaluation will be the current Certificate in Global Travel Level 4 students.

Instrumentation
Students will be asked to answer the questionnaire and also will be asked to attend an interview/focus group asking for their opinions of the online resources provided on BlackBoard.

Logistics
Elaine Dittert will conduct all of the tasks for this evaluation plan.

Time line
As the Level 4 course finishes on 26th June all evaluations and interviews must be completed by this date.
Feedback and approval of draft evaluation plan
by 12 June 2009
Evaluation: Questionnaires to be completed and interviews conducted
by 19 June 2009
Data to be analysed
by 3 July 2009
Draft evaluation report to be completed
by 17 July 2009
Feedback and approval of draft final report
by 24 July 2009
Evaluation report to be completed by
by 7 August 2009

Estimated Budget
My time is charged out at $80 per hour (Estimated time to spend on evaluation: approximately 12-15 hours) Total $960 - $1200

References
eLearning guidelines for New Zealand. (2008). Retrieved May 26, 2009, from website http://elg.massey.ac.nz/index.php?title=E-Learning_Guidelines

Hegarty, B. (2003). Experimental and Multiple Methods Evaluation Models. Retrieved May 26, 2009 from website:
http://wikieducator.org/images/6/68/Evaluation_models

Reeves Thomas C., John G. Hedberg. (2003). Englewood Cliffs: Educational Technology Publications.

A Rethink...

I've recently had a very productive meeting with Heather who kindly has helped me catch up a bit, and have revised my original ideas that I had listed below.

Background
The Certificate in Global Travel Level 4 incorporates the National Certificate in Travel Level 4. Two unit standards I teach in this programme is Unit 18213 Destination UK & Eire and Unit 3727 Destination Pacific Islands. Both these units are 10 credits each, so are considerably large in their content and assessment value.

We had several students wishing to complete these papers on their own and so four years ago I was asked to begin putting together an "e-learning solution" on our current LMS BlackBoard. Over the years I've refined it to the point where not only is it used by "on-line students" but I now use it as a base for my f2f classes and I often get positive comments from my f2f students on how useful they find it in doing their assignments.

As I've already talked about, I've been off work for 3 months and as a result the current Level 4 students have had to rely on this information much more than normal to complete their work. I've also mentioned that I've been asked to transfer/create new e-Learning /Blended delivery information to be put on Moodle which is our new preferred LMS here at UCOL.

I think that I should conduct a needs analyis of my current Level 4 students asking them what they liked about the current course on BlackBoard and what they would like to see included in future courses. For instance I want to add You Tube /Getaway clips of some of the destinations they need to study. Will they find this useful? Will it be easy to download?

Earlier on I had idenitified ST 8: Rather than just recalling facts, does the course help the student digest, reflect on and review new learning?

Does the information I currently give them help with Destination units? Or is it simply a repository for information? Sometimes I think I am guilty of this - how can I make it better for them so they actually learn, not just blandly find information for regurgitation into an assignment?
I think I will ask questions in this vein to find out more about what they would think would be appropriate for future students from the new Moodle site .

I had also identified SD 3: Do students gain knowledge relevant to employment and/or current thinking in their field? However I think now I will change that to look at ST9: Do the technologies employed help students successfully meet the learning outcomes?
Is the information provided help (or hinder) the students? Should I use other technologies (such as Facebook or similar) for some areas of their learning? Again I have some ideas of questions in this vein.

More to come...

Weeks 5 & 6 Evaluation Methods

For these two weeks we were asked to address the following:
Outline some information about the type of evaluation you believe will best suit your project with your reasoning
Summarise the article(s) you have found (300 words) - full reference, evaluation design (paradigm, model, questions, methods), findings and how it aligns with the evaluation project you would like to conduct.
==============================================================

I found the article by Professor Tom Reeves The Six Facets of Instructional Product Evaluation helped me considerably in understanding more about the methods of evaluation that there are available.

I have been using Blackboard to a considerable extent in recent years using blended delivery. During this time I have conducted only informal evaluations including written student surveys, peer evaluation of site content by other staff members etc. This paper gives me a perfect platform to now perform evaluation in a more formal setting when I begin my new eLearning task of developing everything (some old, but most new) in Moodle instead.

I need to gather feedback from students and perhaps other staff members during the development and implementation process of developing the new Tourism and Travel qualification on Moodle. As I said before Blackboard is being phased out and I am meant to have Moodle up and running by the beginning of Feb 2010.

Having thought about this for a while, I think that I should base my evaluation on Formative evaluation. I think I should carry out a questionnaire for example once I have completed the first module in Moodle and get students/staff to assess it - perhaps in terms of the Key Usability Principles identified by Jakob Neilsen which I think are solid guidelines to use:

• Structure - is the online module well organised?
• Simplicity - is it user friendly?
• Visibility - is everything readily/easily available for the student on the site to complete the module?
• Feedback - are students able to contact the lecturer easily and in a timely manner?
• Tolerance - will the students be able to cancel, go back or exit without problems?
• Reuse - will the students be able to go back and find information already examined/studied easily without too much trouble?

I think this ties in well with The Project of Evaluation Toolkit which mentions some 'High value' questions that are useful to ask because they will add real value to any evaulation, for example those which will most likely highlight areas that can readily be improved.

I think that if I was able to do an evaulation on one module (perhaps before it officially goes online?) I should be able to identify problems so that I could make improvements or adjustments as I continue to implement the full extent of transferring (or let's face it completely reworking) everything I currently have on BlackBoard to Moodle. I would imagine that I would need to do this after every module is completed.

I think that the evaluation would be best done through an online survey of users and then back that up with a focus group interview and perhaps a different staff questionnaire would be required so I am not relying on just one type of evaluation as recommended by Reeves and Hedberg (2003). This would mean I would be applying the Multiple Methods Evaluation Model so that the data collected can be triangulated.


References:
Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications
Project of Evaluation Toolkit. Retrieved May 20th 2009, University of Tasmania http://www.utas.edu.au/pet/index.html

Week 4 Evaluation Paradigms and Models

OK, so things did not go according to plan... the op took much more out of me than I ever expected, and I just did not have the motivation or inclination to get back into this. Thanks to those of you who sent me get well messages etc.

However now I'm back all guns blazing so I'm going to be doing a major catch up (thanks Bron for being so understanding!) I will be steadily behind for a long time, (probably for the whole remaining time on this paper) but am determined to get there! I have Kay to bolster me along thank goodness :-)

I'm afraid like Joy reading the Educations Paradigms article confused me considerably! I don't think my brain is quite ready for language like that!

But reading Bronwyn's article made things a little clearer. As I said in Week 3 I have to find a way towards an eLearning solution which will satisfy the need of our Industry Training Organisation that students will be expected to produce a project (the word portfolio is being used a lot I notice in recent newsletters).

I had idenitified ST 8: Rather than just recalling facts, does the course help the student digest, reflect on and review new learning? So how can I use eLearning to its best ability to make sure it satisifies the need for students to use reflective methods of learning? I also want to establish information based on SD 3: Do students gain knowledge relevant to employment and/or current thinking in their field?

I think that the Multiple methods evaluation model would be the best choice for me for this. I think I could use a questionnaire to ascertain what the students think about the learning in relation to their review of learning and its relevance to their future employment.

More on this as I catch up some more...

Week 3 - eLearning guidelines

OK, I've decided to try and get this week's tasks done as well before the op. I probably won't feel like getting back to this for a couple of weeks, so trying desperately to at least get started!

Like Michelle I've been trying to figure out what area of my work I would be best looking at for evaluation. To be honest I might come back and review these ideas again but here are my thoughts so far...

Background

I teach in the tourism and travel area. In 2010 we will be starting two brand new National Tourism and Travel qualifications, based much more on the real world (thank goodness) than on manually writing tickets and other out of date information that we have been forced to teach in the old qual!

In the travel industry work based training is taking over. Numbers studying at pre-employment level at polytechs has reduced dramatically since 2001 (consider UCOL used to have 7 on the Tourism and Travel staff - last year we went down from three to two!)

My Issues:

How do we keep our students relevant and current in a constantly changing world? It is hard enough for qualifications to keep up and of course we are constantly updating our knowledge, in an eLearning environment will this be even harder to achieve?

When you add budget constraints to the mix, it becomes increasingly difficult. For example to purchase a training version of a CRS (Computer Reservation System) eg Galileo/Amadeus or Sabre is very expensive, and so students work on the one we can afford, not necessarily the best one!

I have been playing with blended delivery/eLearning for several years, and have so far developed several "destination" unit standards on the LMS that UCOL currently uses, BlackBoard. These are rather large units (10 credits each at either Level 3 or 4) and I have used it I believe to some success in the classroom, getting students to do some research which a travel consultant would normally do with a view to answering the questions in their assignment in a more practical way than just teaching in a f2f situation. In end of course evaluations, students have said they enjoyed this method of learning instead of from a book or watching videos, and I have made improvements along the way based on some comments on these evaluations.

At UCOL we have been told that by the end of 2009 (although it may be stretched to mid 2010) we will be switching to Moodle and BlackBoard will cease to be used. I have been given a sandbox to play in in Moodle, and my manager has indicated that she would like me to have everything changed over to Moodle in readiness for the brand new qualifications starting in 2010. However the content of the new qualification is still not available (it is with NZQA now) so although I have a good idea what will be in the new qual, how on earth do I use eLearning to its best ability?

2 eLearning Guidelines relevant to the Tourism and Travel field - and why I chose them

SD 3 Do students gain knowledge relevant to employment and/or current thinking in their field?

I need to make sure that any material used in eLearning/online delivery is relevant to today's travel industry. However travel changes every day, and keeping up with current practice is pretty difficult especially when budget constraints have to be considered! How will I ensure that students are always receiving a quality product?

ST 8 Rather than just recalling facts, does the course help the student digest, reflect on and review new learning?

We have been given the indication that the new qual will be based more on project work than regurgitation of content (yay) so that is another challenge for me - to use eLearning to meet this need of the student. I will have to ensure that the methods used are varied and appropriate.

As I said at the beginning, I may review these when I reflect on them in a week or two once my hospital stint is over, but any feedback would be much appreciated!

Week 2 Quality and Evaluation

First of all thanks to everyone who has sent me lovely messages either on the blog, or via Facebook. The operation has had to be rescheduled to this Thursday 19th March as I had a slight temperature so they wouldn't go ahead. For the first time in a week I feel well enough to turn on the laptop so I thought I'd try to catch up at least on a couple of week's work!

The importance of evaluation to me

To me evaluation is all about performance - how is a course doing, what can be done better. It's about asking opinions from the participants of a course, these can be in the form of questionnaires, surveys etc.

The following definition of evaluation from evaluationwiki.org to me says it in a nutshell...

evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organisations to improve their effectiveness

Looking at the strengths and weaknesses of a course or programme is the only way to assess quality, and how to make it better...

Evaluations types familiar to me and why...

Observations and Checklists
In the Tourism and Travel field I work in we have to assess many elements of the subjects through observation of computing skills (like booking an e-ticket) or a roleplay serving a customer. In our f2f situations these are recorded for moderation. For some of our students who are working in industry, as a workplace assessor I have visited them to observe these skills in their job. Checklists had to be completed for these observations, both for UCOL and for our National ITO the Aviation, Tourism and Travel Training Organisation (ATTTO).

Questionnaires
In this area I have mainly used end of course evaluation handouts to find out what students have thought of a subject in terms of its usefulness, quality, relevance etc.

Focus Groups
In the past I have been a member of a tourism consortium consisting of other NZ polytechnics where we evaluated the courses we all delivered, and worked together to have similiar assessments and teaching standards

Feedback in a Discussion Forum
I haven't used this as a tutor, but I have as an online student, doing this Graduate Certificate paper.

The Importance of Quality in eLearning

Quality is so important in education. As the mother of a 10 year old I sometimes struggle to understand some methods of how kids these days are being taught. I am probably a primary teacher's nightmare as I am often asking questions such as - "how effective is this new method of teaching?" "What will he learn from this?"

As a tertiary tutor trying to develop some of my subjects in an online environment, I have to ask those very same questions!

As Joy noted in her blog, in a f2f situation we are constantly evaluating each class, how its going, whether we need to move on, or spend more time on a particular topic, as feedback is readily available. Online however we do not have that immediate opportunity to find out how things are going so the importance of ongoing evaluation to ascertain quality is invaluable.

Students often struggle in an arena where there is reduced support, either from the facilitator or from student services etc. I know as an eLearning student I really struggled with some of the initial papers, and only for the fact that in one paper the tutor worked at UCOL I would probably have dropped out!

In a f2f situation we can often pick if a student is struggling but in an online environment, this is more difficult. If an eLearning course is not deemed good quality, or they feel they are not getting enough out of it, students will just drop off, resulting in poor success rates.

Elearning 2009!

So, I'm now enrolled in "Evaluations of eLearning for Best Practice" my fifth paper of the Graduate Certificate in Applied eLearning. (only one more to go...) Great to see Kay and Joy here, old classmates from 2008...

I'm a lecturer in tourism and travel consultancy at UCOL, a tertiary provider (polytechnic) in Palmerston North, New Zealand. I have been teaching here for 15 years (that is scary!) after doing my degree in Tourism and Hotel Management from the University of Ulster (I am originally from Northern Ireland). I have worked as a travel consultant, have run a Visitor Information Centre in N.Ireland and have also worked for the Northern Ireland Tourist Board as an intern. I became interested in eLearning as many of our students need to study National qualifications in tourism and travel but need to work as well. I wanted to see how I could meet their needs. I have developed programmes in Blackboard and am just playing with Moodle now, and look forward to developing my knowledge in that medium further.

Unfortunately the week before this course began I suffered some health problems and it turns out I have to have an operation this Thursday 12th March. I will be in hospital about 5 days but hope to catch up on the course while recuperating - with the laptop on my knee! I was all raring to go three weeks ago but have been off work since and to be honest study has been the last thing on my mind! So bear with me... I might be a week or two behind for a wee while, but I intend to catch up!

Sorry to have missed the Elluminate sessions so far, again I'll listen to them and try to join in later on once things settle down.

I hope to learn lots about evaluation and its importance in the eLearning field.

PS I began to write this post before all the medical drama, but today is 10th March even though the date at the top says Feburary!

I have had the most enjoyable 6 weeks off work that you could imagine - the combination of a lovely Xmas and New Year and fantastic weather! Being back at work is a bit of a shock to the system though...

Anyway, at the end of last year the facilitator for the course Leigh Blackall asked that we give some feedback on the FOC 08 course. Sorry its a bit late Leigh, but I deliberately stayed away from computers over the Xmas break, I needed to get away from it all for a while!

So here are some of my thoughts on FOC 08...

What were the most enjoyable aspects of this course?

The fun of discovering so many unknown technologies out there! As someone not "au fait" as such with these, I had a ball investigating Second Life, Facebook, etc. Getting to chat to people from all over the world was great too, seeing how many people from so many walks of life together in one forum was excellent.

What were the most challenging aspects of this course?

At the beginning when I felt so very lost and "amateur". I got completely overwhelmed with the email forum, finding out about RSS Feeds, and of course being new to blogs as a whole. I'm glad to see Leigh has said he is thinking of reviewing how the course begins. As I said in a previous post, I almost dropped out before it really got started through severe feelings of inadequacy!

I have to admit I did feel a little abandoned towards the end of the course when Leigh went overseas for 2 weeks and we didn't hear from him - no comments on our blogs etc. Those comments helped motivate me to continue on to the next week!

Did the course meet your expectations? How or how not?

I didn't really have many expectations to be honest. So much of this elearning qualification is new to me so I was just hoping I could cope! Eventually I did :-)

What improvements would you suggest?

I've read through Leigh's blog post giving his thoughts on how the course went (obviously many others have posted their comments already) and I like the improvements he has suggested such as changing the "intensity" at the beginning of the course and the YouTube video link to help introduce us to the topic.

It is so difficult to get people to participate in online activities such as the Elluminate sessions but I think these are really vital to see how a facilitator works. But then again Leigh very correctly said it just takes the "a ha" (I call it the lightbulb over the head switching on) for it all to start making sense. Some may take longer than others to get that lightbulb working! So maybe strongly encouraging more participation in these sessions (don't ask me how) would be advantageous.

Any other comments?

I think Leigh, you did a wonderful job. I enjoyed the course (eventually!) and was challenged by the format (away from Blackboard) and the content. I'm sure you will do an even better job this year. I will try to keep an eye on the email forum in semester 2 to see how it goes.

PS. Just to say I got an "A" for FOC 08. I was so delighted - the stress and late nights paid off. Thanks Leigh :-)