Draft Evaluation Plan

I've been working really hard this week to catch up, using every spare minute (helped by the fact a couple of my classes did not run this week) so have managed to get a Draft done of my Evaluation Plan.

I know I'm way behind most of you, but any feedback you have time to give would be most welcome!
I hope this link to Google docs work - I'm not sure what I'm doing, but just in case, a copy is also below... http://docs.google.com/Doc?docid=dhp8hmb2_0c5mfpvvg&hl=en

==============================================================
Draft Evaluation Plan

Introduction
This evaluation plan will evaluate current students’ opinion on the quality of some of the BlackBoard resources currently being used at UCOL in the tourism and travel section, as well as highlighting possible areas for improvement.

The final report of this evaluation will be forwarded to UCOL management however the main reason for this evaluation is as an assessment for the Evaluation of eLearning for Best Practice paper, which the author is completing as part of the Graduate Certificate in Applied eLearning from MIT.

The author of this report and evaluator of data is Elaine Dittert.

Background
The UCOL Certificate in Global Travel Level 4 incorporates the National Certificate in Travel Level 4. Two unit standards taught in this programme are Unit 18213 Destination United Kingdom & Eire, and Unit 3727 Destination Pacific Islands. Both these units are 10 credits each, so are considerably large in their content as well as assessment value. UCOL had previously been approached by several part time students working in industry, wishing to complete these papers on their own without attending formal classes, and so four years ago an "e-learning solution" was developed on the Learning Management System (LMS) currently used by UCOL - BlackBoard. This site has since been continually refined to the point now where not only is it used by "on-line students" but it is now also used as a basis for the face to face Destination classes conducted in the Level 4 programme. Positive comments have been received from these f2f students on how useful they find this resource in completing their assignment work.
The current Level 4 students have had to rely on this information much more than normal to complete their work this year due to the staff member being ill, so they are prime candidates for this evaluation study.

Further to this UCOL management has put in a request to the tourism and travel section to transfer any information currently on BlackBoard into Moodle which is the new preferred LMS at UCOL. This provides an ideal opportunity for tourism and travel staff to update and improve the resources available to the Level 4 students.

Purpose
The purpose of this evaluation is to assess the effectiveness of the current BlackBoard online resources used for the “Destination” unit standards and to identify areas for improvement.

Limitations
There is a tight time restriction, as the students to be targeted leave UCOL at the end of June, so evaluation will have to be conducted very quickly. Also the number in the Level 4 group has dwindled to only 8 students, so the sample size is very small.

Audiences
The results of this evaluation will be reported to the Head of School for Tourism and Hospitality at UCOL. In conjunction with staff the best ways to improve the current online resources for the Level 4 Destination unit standards will then be discussed.

Decisions
Based on the results of the questionnaire and interviews decisions will be made on the content of the new Level 4 Destination resources to be made available on Moodle.

Questions
Using the Likart scale would be the preferred method for the questionnaire, where students rank their answer from 1- 5 (eg. 1 strongly disagree to 5 strongly agree). This will be followed by a space for comments on what they liked/did not like about the online resources. This will provide an opportunity to see areas for improvement.

The two Guidelines that will be investigated from http://elg.massey.ac.nz/ are:

ST 8: Rather than just recalling facts, does the course help the student digest, reflect on and review new learning?
Sample Sub statements to be evaluated:
The basic information in each section of the Destination Units was very interesting
The review tests at the end of each section were very useful to my learning
The on-line quizzes included in the BlackBoard site helped with my learning
The BlackBoard site was confusing and hard to understand
The BlackBoard site helped me remember information on the Destination

ST9: Do the technologies employed help students successfully meet the learning outcomes?
Sample Sub statements:
The video clips made available from Getaway were very useful
The information on Blackboard was always easy to download
The links to web sites for transport and accommodation were very useful
A Facebook group set up to aid in your study of Destinations would interest me
I would be comfortable using technology such as Second Life in my study

Methods
This evaluation will be based on the Eclectic – Mixed Methods Pragmatic Paradigm using Multiple methods to conduct the evaluation. Participants will be asked to complete a questionnaire and will also be part of a small focus group to be interviewed.

Sample
As previously stated that participants of this evaluation will be the current Certificate in Global Travel Level 4 students.

Instrumentation
Students will be asked to answer the questionnaire and also will be asked to attend an interview/focus group asking for their opinions of the online resources provided on BlackBoard.

Logistics
Elaine Dittert will conduct all of the tasks for this evaluation plan.

Time line
As the Level 4 course finishes on 26th June all evaluations and interviews must be completed by this date.
Feedback and approval of draft evaluation plan
by 12 June 2009
Evaluation: Questionnaires to be completed and interviews conducted
by 19 June 2009
Data to be analysed
by 3 July 2009
Draft evaluation report to be completed
by 17 July 2009
Feedback and approval of draft final report
by 24 July 2009
Evaluation report to be completed by
by 7 August 2009

Estimated Budget
My time is charged out at $80 per hour (Estimated time to spend on evaluation: approximately 12-15 hours) Total $960 - $1200

References
eLearning guidelines for New Zealand. (2008). Retrieved May 26, 2009, from website http://elg.massey.ac.nz/index.php?title=E-Learning_Guidelines

Hegarty, B. (2003). Experimental and Multiple Methods Evaluation Models. Retrieved May 26, 2009 from website:
http://wikieducator.org/images/6/68/Evaluation_models

Reeves Thomas C., John G. Hedberg. (2003). Englewood Cliffs: Educational Technology Publications.

A Rethink...

I've recently had a very productive meeting with Heather who kindly has helped me catch up a bit, and have revised my original ideas that I had listed below.

Background
The Certificate in Global Travel Level 4 incorporates the National Certificate in Travel Level 4. Two unit standards I teach in this programme is Unit 18213 Destination UK & Eire and Unit 3727 Destination Pacific Islands. Both these units are 10 credits each, so are considerably large in their content and assessment value.

We had several students wishing to complete these papers on their own and so four years ago I was asked to begin putting together an "e-learning solution" on our current LMS BlackBoard. Over the years I've refined it to the point where not only is it used by "on-line students" but I now use it as a base for my f2f classes and I often get positive comments from my f2f students on how useful they find it in doing their assignments.

As I've already talked about, I've been off work for 3 months and as a result the current Level 4 students have had to rely on this information much more than normal to complete their work. I've also mentioned that I've been asked to transfer/create new e-Learning /Blended delivery information to be put on Moodle which is our new preferred LMS here at UCOL.

I think that I should conduct a needs analyis of my current Level 4 students asking them what they liked about the current course on BlackBoard and what they would like to see included in future courses. For instance I want to add You Tube /Getaway clips of some of the destinations they need to study. Will they find this useful? Will it be easy to download?

Earlier on I had idenitified ST 8: Rather than just recalling facts, does the course help the student digest, reflect on and review new learning?

Does the information I currently give them help with Destination units? Or is it simply a repository for information? Sometimes I think I am guilty of this - how can I make it better for them so they actually learn, not just blandly find information for regurgitation into an assignment?
I think I will ask questions in this vein to find out more about what they would think would be appropriate for future students from the new Moodle site .

I had also identified SD 3: Do students gain knowledge relevant to employment and/or current thinking in their field? However I think now I will change that to look at ST9: Do the technologies employed help students successfully meet the learning outcomes?
Is the information provided help (or hinder) the students? Should I use other technologies (such as Facebook or similar) for some areas of their learning? Again I have some ideas of questions in this vein.

More to come...

Weeks 5 & 6 Evaluation Methods

For these two weeks we were asked to address the following:
Outline some information about the type of evaluation you believe will best suit your project with your reasoning
Summarise the article(s) you have found (300 words) - full reference, evaluation design (paradigm, model, questions, methods), findings and how it aligns with the evaluation project you would like to conduct.
==============================================================

I found the article by Professor Tom Reeves The Six Facets of Instructional Product Evaluation helped me considerably in understanding more about the methods of evaluation that there are available.

I have been using Blackboard to a considerable extent in recent years using blended delivery. During this time I have conducted only informal evaluations including written student surveys, peer evaluation of site content by other staff members etc. This paper gives me a perfect platform to now perform evaluation in a more formal setting when I begin my new eLearning task of developing everything (some old, but most new) in Moodle instead.

I need to gather feedback from students and perhaps other staff members during the development and implementation process of developing the new Tourism and Travel qualification on Moodle. As I said before Blackboard is being phased out and I am meant to have Moodle up and running by the beginning of Feb 2010.

Having thought about this for a while, I think that I should base my evaluation on Formative evaluation. I think I should carry out a questionnaire for example once I have completed the first module in Moodle and get students/staff to assess it - perhaps in terms of the Key Usability Principles identified by Jakob Neilsen which I think are solid guidelines to use:

• Structure - is the online module well organised?
• Simplicity - is it user friendly?
• Visibility - is everything readily/easily available for the student on the site to complete the module?
• Feedback - are students able to contact the lecturer easily and in a timely manner?
• Tolerance - will the students be able to cancel, go back or exit without problems?
• Reuse - will the students be able to go back and find information already examined/studied easily without too much trouble?

I think this ties in well with The Project of Evaluation Toolkit which mentions some 'High value' questions that are useful to ask because they will add real value to any evaulation, for example those which will most likely highlight areas that can readily be improved.

I think that if I was able to do an evaulation on one module (perhaps before it officially goes online?) I should be able to identify problems so that I could make improvements or adjustments as I continue to implement the full extent of transferring (or let's face it completely reworking) everything I currently have on BlackBoard to Moodle. I would imagine that I would need to do this after every module is completed.

I think that the evaluation would be best done through an online survey of users and then back that up with a focus group interview and perhaps a different staff questionnaire would be required so I am not relying on just one type of evaluation as recommended by Reeves and Hedberg (2003). This would mean I would be applying the Multiple Methods Evaluation Model so that the data collected can be triangulated.


References:
Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications
Project of Evaluation Toolkit. Retrieved May 20th 2009, University of Tasmania http://www.utas.edu.au/pet/index.html

Week 4 Evaluation Paradigms and Models

OK, so things did not go according to plan... the op took much more out of me than I ever expected, and I just did not have the motivation or inclination to get back into this. Thanks to those of you who sent me get well messages etc.

However now I'm back all guns blazing so I'm going to be doing a major catch up (thanks Bron for being so understanding!) I will be steadily behind for a long time, (probably for the whole remaining time on this paper) but am determined to get there! I have Kay to bolster me along thank goodness :-)

I'm afraid like Joy reading the Educations Paradigms article confused me considerably! I don't think my brain is quite ready for language like that!

But reading Bronwyn's article made things a little clearer. As I said in Week 3 I have to find a way towards an eLearning solution which will satisfy the need of our Industry Training Organisation that students will be expected to produce a project (the word portfolio is being used a lot I notice in recent newsletters).

I had idenitified ST 8: Rather than just recalling facts, does the course help the student digest, reflect on and review new learning? So how can I use eLearning to its best ability to make sure it satisifies the need for students to use reflective methods of learning? I also want to establish information based on SD 3: Do students gain knowledge relevant to employment and/or current thinking in their field?

I think that the Multiple methods evaluation model would be the best choice for me for this. I think I could use a questionnaire to ascertain what the students think about the learning in relation to their review of learning and its relevance to their future employment.

More on this as I catch up some more...