Draft Evaluation Plan

I've been working really hard this week to catch up, using every spare minute (helped by the fact a couple of my classes did not run this week) so have managed to get a Draft done of my Evaluation Plan.

I know I'm way behind most of you, but any feedback you have time to give would be most welcome!
I hope this link to Google docs work - I'm not sure what I'm doing, but just in case, a copy is also below... http://docs.google.com/Doc?docid=dhp8hmb2_0c5mfpvvg&hl=en

==============================================================
Draft Evaluation Plan

Introduction
This evaluation plan will evaluate current students’ opinion on the quality of some of the BlackBoard resources currently being used at UCOL in the tourism and travel section, as well as highlighting possible areas for improvement.

The final report of this evaluation will be forwarded to UCOL management however the main reason for this evaluation is as an assessment for the Evaluation of eLearning for Best Practice paper, which the author is completing as part of the Graduate Certificate in Applied eLearning from MIT.

The author of this report and evaluator of data is Elaine Dittert.

Background
The UCOL Certificate in Global Travel Level 4 incorporates the National Certificate in Travel Level 4. Two unit standards taught in this programme are Unit 18213 Destination United Kingdom & Eire, and Unit 3727 Destination Pacific Islands. Both these units are 10 credits each, so are considerably large in their content as well as assessment value. UCOL had previously been approached by several part time students working in industry, wishing to complete these papers on their own without attending formal classes, and so four years ago an "e-learning solution" was developed on the Learning Management System (LMS) currently used by UCOL - BlackBoard. This site has since been continually refined to the point now where not only is it used by "on-line students" but it is now also used as a basis for the face to face Destination classes conducted in the Level 4 programme. Positive comments have been received from these f2f students on how useful they find this resource in completing their assignment work.
The current Level 4 students have had to rely on this information much more than normal to complete their work this year due to the staff member being ill, so they are prime candidates for this evaluation study.

Further to this UCOL management has put in a request to the tourism and travel section to transfer any information currently on BlackBoard into Moodle which is the new preferred LMS at UCOL. This provides an ideal opportunity for tourism and travel staff to update and improve the resources available to the Level 4 students.

Purpose
The purpose of this evaluation is to assess the effectiveness of the current BlackBoard online resources used for the “Destination” unit standards and to identify areas for improvement.

Limitations
There is a tight time restriction, as the students to be targeted leave UCOL at the end of June, so evaluation will have to be conducted very quickly. Also the number in the Level 4 group has dwindled to only 8 students, so the sample size is very small.

Audiences
The results of this evaluation will be reported to the Head of School for Tourism and Hospitality at UCOL. In conjunction with staff the best ways to improve the current online resources for the Level 4 Destination unit standards will then be discussed.

Decisions
Based on the results of the questionnaire and interviews decisions will be made on the content of the new Level 4 Destination resources to be made available on Moodle.

Questions
Using the Likart scale would be the preferred method for the questionnaire, where students rank their answer from 1- 5 (eg. 1 strongly disagree to 5 strongly agree). This will be followed by a space for comments on what they liked/did not like about the online resources. This will provide an opportunity to see areas for improvement.

The two Guidelines that will be investigated from http://elg.massey.ac.nz/ are:

ST 8: Rather than just recalling facts, does the course help the student digest, reflect on and review new learning?
Sample Sub statements to be evaluated:
The basic information in each section of the Destination Units was very interesting
The review tests at the end of each section were very useful to my learning
The on-line quizzes included in the BlackBoard site helped with my learning
The BlackBoard site was confusing and hard to understand
The BlackBoard site helped me remember information on the Destination

ST9: Do the technologies employed help students successfully meet the learning outcomes?
Sample Sub statements:
The video clips made available from Getaway were very useful
The information on Blackboard was always easy to download
The links to web sites for transport and accommodation were very useful
A Facebook group set up to aid in your study of Destinations would interest me
I would be comfortable using technology such as Second Life in my study

Methods
This evaluation will be based on the Eclectic – Mixed Methods Pragmatic Paradigm using Multiple methods to conduct the evaluation. Participants will be asked to complete a questionnaire and will also be part of a small focus group to be interviewed.

Sample
As previously stated that participants of this evaluation will be the current Certificate in Global Travel Level 4 students.

Instrumentation
Students will be asked to answer the questionnaire and also will be asked to attend an interview/focus group asking for their opinions of the online resources provided on BlackBoard.

Logistics
Elaine Dittert will conduct all of the tasks for this evaluation plan.

Time line
As the Level 4 course finishes on 26th June all evaluations and interviews must be completed by this date.
Feedback and approval of draft evaluation plan
by 12 June 2009
Evaluation: Questionnaires to be completed and interviews conducted
by 19 June 2009
Data to be analysed
by 3 July 2009
Draft evaluation report to be completed
by 17 July 2009
Feedback and approval of draft final report
by 24 July 2009
Evaluation report to be completed by
by 7 August 2009

Estimated Budget
My time is charged out at $80 per hour (Estimated time to spend on evaluation: approximately 12-15 hours) Total $960 - $1200

References
eLearning guidelines for New Zealand. (2008). Retrieved May 26, 2009, from website http://elg.massey.ac.nz/index.php?title=E-Learning_Guidelines

Hegarty, B. (2003). Experimental and Multiple Methods Evaluation Models. Retrieved May 26, 2009 from website:
http://wikieducator.org/images/6/68/Evaluation_models

Reeves Thomas C., John G. Hedberg. (2003). Englewood Cliffs: Educational Technology Publications.