Introduction
The needs assessment for the redesign of the eLearning modules was actually conducted as an evaluation of the existing online learning modules. The evaluation was conducted in conjunction with the release and beta test of the second set of modules, 7-13. Modules 1-6 were released in April 2000, and no evaluation other than a beta test was performed for those modules at that time. Modules 7-13 were released in October 2000.

Through the beta test I gathered demographics, logistics and other quantitative data. I decided that this evaluation required a cognitive approach as well, so I conducted interviews in person and over the phone. I also gathered additional quantitative data in from the existing Learning Management System in the form of course evaluations, learner profiles, error reports and list of learner completions.

Beta Test
As the beta test for the second set of modules was already in motion when I was hired to evaluate the online learning modules, I used the beta testers as the primary evaluators, as they were already a fairly captive audience. The beta testers were required to work with the Educational Systems and Documentation and Development departments to identify bugs and other possible issues with the modules. The beta test team created a level 1 evaluation in the form of an email survey. It was apparent that testers were not taking the beta test evaluation seriously and the majority of the responses came from internal eDeploy users.

Beta Test Data
Participants
The beta test participants consisted of internal eDeploy employees and outside eDeploy customers. The breakdown of participants is as follows:

• 1/3 Internal eDeploy employees
• 2/3 Customers
• 50% of Total participants had prior knowledge of the application
• 50% of Total participants had no experience with the application

Purpose
The purpose of the beta test is to:

• Allow customers advanced access into the features offered, so they are aware of what eDeploy is releasing
• Allow customers to provide feedback and be a part of the change cycle
• Flush out bugs
• Review network access

Feedback
Detailed, concrete information was not requested in the purpose statement of the beta test given to the participants, therefore the participants did not know to provide that kind of feedback. I did not feel that adequate feedback was received from the beta test level 1 evaluations, I would have liked to see more detailed, concrete and specific information about features and content. The beta test team failed to expect changes in exact content. That decision impacted feedback for the learner who may have found some aspect of the modules were confusing. I would have benefited knowing which paragraph, assignment or quiz question created confusion or concern.

Qualitative Data
As the eDeploy Beta Test Team gathered quantitative data for eLearning modules 7-13 through level 1 evaluations, I focused the first part of my evaluation on qualitative methods, interviews and one-on-one observations. The use of qualitative methods in evaluation is typically more subjective, however it allows the evaluator to gather a wider range and depth of information. Qualitative methods are more flexible and dynamic, and are not limited to a pre-conceived topic of inquiry. I feel that when combined, the quantitative results from the Beta Test Team along with the qualitative data elicited from my interviews, it was a good start for the ongoing evaluation process of the eLearning modules.

Interviewees Background and Prior Experience
Only 4 out of 20 people agreed to meet with me, and they were all eDeploy employees. The customers were each assigned an account manager and did not wish to speak with me independently of their account manager.

Amy Zeppelin, Business Services Consultant, 3 weeks with eDeploy. No prior experience with the eDeploy software. Did see a demo and had a colleague go through the software and explain the basics before she went through the eLearning modules. Completed modules 1-13.

Everton Fidel, Business Analyst, hired Oct 9th, 2000. No prior experience with the eDeploy software. Did see a demo and had a colleague go through the software and explain the basics. Everton also went to California and sat in on the classroom training before going through the eLearning modules. Completed modules 1-13.

Deb Winter, Technical Writer, contractor since October 2000. No prior experience with the eDeploy software prior to eDeploy. Did see the demo and had previously gone through modules 1-6. Deb did write some of the technical reference documentation for modules 7-13, and was familiar with the content. Completed modules 1-13.

Jeanne Callahan, Marketing Director, 1 week with eDeploy. No prior experience with the eDeploy software. Saw the demo on the web site when researching the company. Did not get past the first module.

Motivation and affective domain
All interviewees were not required to go through the eLearning modules, but it was suggested that they do so. The average time for each module varied, 10 minutes to 1 hour depending on the length of the module. Each person spaced out the training and did a couple of hours at a time. One person stated that a colleague commented on her "eLearning look", meaning that she had a glazed over look from staring at a computer and it was visibly noticeable.

The motivation to finish these modules for the analyst and consultant was fear of "baptism by fire". These employees are expected to hit the ground running and they needed to know the ins and outs of the software as fast as possible.

The other interviewees didn't have as much motivation, due to their job positions. Deb did go through modules 7-13 as a beta tester and Jeanne didn't make it through the first module. Jeanne felt that the information was preliminary sales information, and didn't give her anything she didn't already know. She stated that her patience ran out and she wasn't motivated to complete the rest of the modules.

Each person went through the modules in a chronological order. Two out of the four interviewees skipped the welcome and instructions and no one experienced any problems, the other two felt the welcome and instructions were helpful.

The interviewees that completed modules 7-13 did not complete the assignments section at the time of the interview, but did have the intention of going back and completed them when they were available.

Layout and Design

• Everyone felt that the overall layout and design of the eLearning modules was easy to interpret and didn't hinder the modules in any way.
• All interviewees felt that the navigation was clear and easy to use. The primary navigation tool used were the arrows at the bottom left, and all mentioned that the hotlinks to the module sections were also very helpful. The icons at the top right were never used (email the instructor, chat, conference) and one person commented that the icons were large and kept clicking on the one directly above the "Modules" link because they were not labeled.
• Three out of four interviewees used the sound feature. The sound feature seemed to help two out of the four focus on the content and felt that the sound helped them remember the information when it came time to take the quiz.

Content

• Each person felt that the modules were content heavy, and that the modules that exceeded ten sections were too long. The assignments section (hands-on activities) was not available at the time of the interviews, however on modules 1-6 those who took the assignments section before the quizzes scored a bit lower. The people who took the quizzes right after reading the content felt that the quizzes were easier. The participants that completed the quizzes and assignments in modules 1-6 felt prepared for the quizzes, but felt that the assignments were more challenging.
• The people that had a colleague walk them through the system prior to their eLearning exposure felt that the quizzes for modules 7-13 were too easy. Those that didn't have prior experience with the software felt that the quizzes were more difficult.
• No one used the help function, or email the facilitator. Those who took the assignments in modules 1-6 did use the "forget how" help function.
• Each person had to contact technical support at least two times for access problems.
• The general comment about the content was that it was long and dry, and at times seemed more like a sales pitch than instructional materials. Two interviewees felt that the most important modules were the last few modules 9-13, one person expressed frustration that the "meat" of the tutorial took so long to get to.

Overall

• All of the interviewees felt that the eLearning modules were helpful, but that they required additional training and or information to be able to do their job. No one had any real issues with the navigation, layout or design. They each felt that the content was too long in some modules.
• When asked what they would change about the modules, two out of the four people would make changes to the quizzes, they felt that they were too short in modules 7-13 and the quality of the questions in 7-13 was not as good as the quiz questions in 1-6. Two out of four interviewees also stated that they would add more hands-on activities with the eDeploy application.
• Three out of the four persons interviewed felt that professional services would still need to be involved in training the customers for the hands-on activities and project specific information.
  
  

Quantitative Data
The primary data source was the in house Learning Management System developed by MindRise to track the eDeploy online learner progression through the modules. The LMS also houses course evaluations, learner profiles, error reports and list of learner completions. The secondary data source was the helpdesk issue logs maintained by the customer support department.

My Independent Evaluation
General Observations
I found the layout and visual design very appealing. The navigation was easy to use and there was a status indicator, to view my placement within the tutorial, at each stage. I also found several elements of learner control, from the navigation options to setting your own curriculum based on job tasks. I felt that the length of the content was particularly long in some cases and felt ill prepared for the quizzes. I did not get a chance to actively participate in the assignments section (hands-on activities) due to technical difficulties. I was, however, able to view the questions and tasks and do feel that this is the most important part of the tutorial. There is a "forget how" feature located in the assignments section that the learner can access if they cannot remember how to perform that task, what makes this feature even better is that it displays in a separate window that remains open while the task is performed.

Overall I was impressed with the thoughtfulness and design of the tutorial. The learners were definitely in mind. I do feel that there is a lot of information here that could overwhelm a new user and that the content and assignments could somehow be more integrated to provide more practice with better retention skills.

Please see my comments in the individual sections:

• Layout and Design
• Content
• Instructional Components
• Recommendations

Layout and Design
Overall Layout
The use of frames really helps the learner focus in on the content without sacrificing navigation. With the use of frames the browser navigation becomes obsolete and you are limited to the navigation within the tutorial and can cause the learner to forget where he/she was when trying to get back to a previous page. The color scheme is aesthetically pleasing to the eye and does not interfere with the course materials. The visual elements of this tutorial are consistent throughout the modules, providing consistency for the learner.

Navigation
The Navigation is clearly explained, easy to use and consistent throughout the modules. Navigation is also available in multiple places, in the left hand frame, through a forward and back button at the bottom of the screen and at the top. The icons that are identified as navigation are explained and easily identifiable.

Readability
of text The font style, size and color allowed for high readability. There is no horizontal scrolling and limited vertical scrolling. There is an adequate amount of white space included to direct the learner's eye to the focal point and not cause eye fatigue.

Graphics
The use of graphics is appropriate to the content. The graphics used did enhance the materials and were used more in some modules than others. Visual representation of materials helps the visual learner with information processing and could be used more throughout the tutorial.

Content
Objectives
The objectives are stated at the beginning of each module as an advance organizer. The objectives do not tell the leaner what they will be able to do upon completion of the module, but rather tell the learner what to expect in the upcoming module.

Structure
The content is displayed in the "Modules" section of the tutorial. The average time to complete the content section for each module was approximately 15 minutes with the sound turned off. This section contains graphics interspersed throughout the various modules, however little graphical representation of the materials exists other than screen prints, such as matrixes and flow charts.

The sequencing of materials is displayed logically, allowing the learner to build on prior knowledge and build on higher order thinking skills. The modules are also based on the tasks a learner will need to perform, and thus provide relevancy for the learner.

Length
The chunks of information are quite long with only text and minimal graphics. The sole interaction in this section is page turning, which could lead to a bored learner and reduce information retention.

Audience
The content is easy to comprehend and appropriate for the audience. The audience will have a high level of motivation to complete this tutorial, which helps them wade through and retain the information. I did feel that certain modules had a more sales oriented sound to them than others.

Assignments
The assignments in modules 7-13 differ from the assignments in modules 1-6. In modules 1-6 the learner was asked to locate information within an existing project. In modules 7-13, the learner builds upon prior knowledge of the eDeploy application and enters information into a new project. The assignments in modules 7-13 create a folder for each participant, however these folders are not read only and the learner's actions can interfere with other participant's work. Each assignment consists of less than 10 activities that do not check to see whether or not they have been completed correctly.

The "Forget How" function is available and does help the learner navigate the task. The online help facility is also available if the learner knows what to search for.

The assignments section was not available during the beta test and as of Monday, November 20, 2000, I still do not have access to the entire section to perform the actual tasks. Access has been requested to the assignment project through the support center email.

Quizzes
The quizzes for modules 7-13 are solely multiple choice and true/false questions. There are no simulation exercises or even scavenger hunts for information. I feel that the assignments section does not adequately prepare the learner for the quizzes. This section is recorded and grades are posted. A passing score of 75% is required, however, you do not really know if the learner can apply this knowledge to their job through the use of straight multiple choice and true/false questions.

Instructional Components
Instructional Assistance
Instructional assistance is provided to the learner in several ways. The learner has the ability to email the instructor at any time, however there is not an estimated response time listed. The learner can also engage in synchronous and asynchronous chats and conferences to communicate with an expert and other learners. The learner also has email and phone contact information for technical support while completing this tutorial.

Learning Styles
This tutorial reaches learners with visual and auditory learning styles through the use of visual and audio components. The use of simulation, or showing how to perform a task and then allowing the learner to complete that task would help the kinesthetic learners. This hands-on component would also add to the overall learning experience for all learning styles.

Learner Information
Directions and expectations are clearly stated at the beginning of the tutorial. The welcome information is very comprehensive and provides the learner with enough information to start the modules. There are also various status indicators throughout the tutorial. The learner's progress is monitored while going through the tutorial; the required modules, quizzes and assignments are marked complete or incomplete. When progressing through the content ("module" section), the learner can easily see exactly where they are and how much further they have to go before the end of that module. This is done through the use of colored bullets and icons. The last page viewed before logoff is stored as a bookmark for easy access the next time the learner visits the tutorial.

Instructional Design
The instructional design of this tutorial is learner centered, the modules employ a variety of activities, reading, listening and practicing in order, to engage the learners and reach various styles. The chat and conference functionality adds a new dimension to these modules and makes good use of the technology available on the Internet, while providing a forum for learner questions and facilitator feedback.

This tutorial also provides an element of learner control, with the sequence of content. These modules are set up in a linear design, however the navigation allows the learner to pick and choose which module they wish to complete first. This tutorial also allows the learner to set his/her curriculum. I had previously thought that the learner could test out of a module, however, after revisiting the module requirements, each required module must be completed. The module completion consists of recording each page as it is read.

The learner can also set up the media elements, turn on and off the sound function as well as take personal notes on each slide. The learner can also go back and view all of the notes taken to refresh their memory before moving on to the next module, quiz or assignment.

Recommendations
After compiling and analyzing the quantitative and qualitative data, along with an independent instructional design review, my recommendations focus on the content sequencing, quizzes and assignments. I feel that each individual module is sequenced appropriately and that the information contained within is relevant in that module. I do feel that if the exercises were imbedded within the content the learner would have more interaction with the material than page turning and that it would increase the information retention rate and increase overall quiz scores.

If interweaving the content with the assignments is not a plausible option, I would suggest breaking the information into smaller chunks. Although it does not take but 15 minutes on average to read each module (without sound), it is very text heavy with no activities involved. Another option would be to have a worked example within the content, though the use of a screen cam or short movie of how to perform the tasks. The learner would not be able to perform the task, but it would provide an example of how it is done correctly.

I am unsure of the purpose of the quizzes. The quizzes seem to go before the assignments, by testing where the information is located. The quizzes consist only of multiple choice and true/false questions. If the goal is to test whether or not the learner has read the module and retained information, then the quiz is adequate. If the purpose of the quiz is to make sure that the learner can perform the tasks that are critical to their job, the quiz section is not capturing that information. With technology and reporting statistics in mind, another solution would be to model the quizzes after the assignments in modules 1-6, where the learner is asked to seek out information by performing a task, then choosing the correct answer. This is a fair workaround for simulation until it is available.

Summary
The overall impression is that eLearning is a tool that is used only partially. While there is sound layout and instructional design, the content is extremely lengthy and text-heavy, with little to no visual representation of the data. The majority of the interactivity comes from the leaner accessing the next page.

Data indicates that the course takes too long to complete, there is not enough hands-on activity, and there is no motivation to complete the course. Learners are not using eLearning immediately after they are given access and thus are requesting extensions.

There is little supporting data that learners are having issues with the content accuracy; however, as of October 2000, no measures were taken to validate the learning objectives with systems process or performance.

For additional information, please refer to the following project documents:
Formative Evaluation
Methodology Review
Cognitive Evaluation