Dabbagh, Bannan-Ritland, Silc
The move to online technologies is an evolving and dynamic process of investigation particularly in the area of examining the ability of such technologies to improve the teaching and learning process. In the case of Web-based course authoring tools (also known as Internet-based training tools), several evaluation frameworks or guidelines have been used to determine the degree of success of those tools in designing, delivering, and managing instruction. For example, Hansen and Frick (1997) suggest four areas that should be addressed by a Web-based course authoring tool for it to become a standard for the development of Web-Based Instruction (WBI): presenting information, providing human interaction, assessment of learning, and course management. Another example is the framework used by PC Week Labs (August, 97) in conjunction with its corporate partner, Wisconsin Technical College System, to evaluate courseware authoring systems for Web-based training. The products used to create the training programs and the resulting programs themselves were judged on conformity to the corporate and education landscape, exploitation of the Internet/intranet, manageability, flexibility, and ease of use. In another effort aimed at helping educators and administrators select new online delivery methods, Landon (1998) conducted an analysis of online delivery software in which he describes and compares these applications by focusing on their technical specifications, instructional design values, media capabilities, features/tools, ease of use, accessibility to persons with disabilities, and potential for collaboration and connectivity. Although useful, these frameworks (and others) have focused on the technological aspects of the features that enable the development of online instruction (Dabbagh & Schmitt, 98) and have adopted a piecemeal approach to the evaluation process which makes it difficult to form an authentic representation of the learning environment generated. As asserted by the Australian National Training Authority (ANTA), Web-based authoring tools seem to be adhering to a common pattern of development and analysis:
The Framework
The proposed framework known as The Framework for Web-Based Learning
(thereafter referred to as WBL Framework), was developed by Khan (1998) and
consists of eight dimensions: pedagogical, technological, institutional,
ethical, interface design, resource support, online support, and course
management. Each dimension has various items addressing its constituents in
relation to Web-based learning environments. For example, the pedagogical
dimension addresses issues such as instructional goals and objectives, the
overall design approach of the course, content sequencing and organization,
instructional methods and strategies, the instructional medium (the Web in this
case) and the extent to which the course uses its attributes, accuracy of
subject matter, and learner assessment. For more details about this framework
and its dimensions see: http://bookstoread.com/framework
There are three issues to consider when attempting to answer this question. First is the issue of the transformation that traditional courses experience when redesigned for Web delivery. Typically such courses undergo a "pedagogical reengineering" (Collis, 1997) because of some of the inherent features of the Web that are integral to the design of WBI (Dabbagh & Schmitt, 98). Second is the purpose of the Web-based course in the learning environment. Is the course a supplement to on-campus/on-site instruction or is it intended as a distance learning alternative in which the instruction, interactions, and feedback are delivered via the Web? Third is the type of authoring tool used to develop the course for Web-delivery and the features it affords the developer. For example, integrated application suites (authoring tools) refer to applications that provide specific tools designed for each of three user groups: learners, instructors, and technical administrators whereas component applications refer to more specialized applications that generally provide few instructor tools (Center for Curriculum Transfer & Technology, http://www.ctt.bc.ca/landonline). We will address these issues by applying the WBL Framework to six WBT sites that have been developed using six different Web-based course authoring tools (see table 1). We will then use the results of this evaluation to determine the extent to which the technology of delivery, in this case the Web-based authoring tools, can shape the pedagogy of courses and assist course developers in focusing on instructional considerations when capitalizing on the attributes of the medium.
The Courses
Selecting the courses was not an easy task. We searched for complete courses that demonstrated a full and authentic use of the application. All of the product Web sites contained at least one demonstration course that could be accessed by registering and receiving a password or a log-on ID. These product site courses, however, varied in their ability to demonstrate all of the components of the authoring tool. A few of the courses were designed and taught by instructors for accredited programs, but no longer included the student exchanges, student projects, tests, and complete content. Other courses were designed specifically to demonstrate the product and therefore were not actually used by instructors or students in a learning context. In the quest to obtain courses that reflected a more realistic use of the Web-based course authoring tool, we were able to obtain access to a few uncorrupted accredited courses which we felt were sufficiently representative of the authoring tool and therefore befitting the purpose of this chapter. Table 1 provides a list of these courses and the corresponding authoring tool that was used to deliver it.
Table 1
|
|
|
|
WebCT |
|
TopClass |
|
Virtual-U |
|
Web Course in a Box (WCB) |
|
Learning Space |
|
Web Mentor |
The Evaluation
Due to the comprehensiveness of Khan's framework, which includes eight dimensions and more than sixty sub-components, a simplified strategy was selected for our informal evaluation of these courses. Mapping the framework to the various courses revealed specific strengths and weaknesses of each course that could be aligned with many of the framework's components. To avoid unnecessary complexity, we chose to focus on the major course attributes in relation to the framework. The major attributes of each course which emerged, both positive and negative dictated our interpretive evaluative approach using the WBL Framework.
Marketing Certificate in Small Business Management
This comprehensive demonstration course highlights many features of WebCT. Most notable is the student guidance for on-line learning and navigating the course as well as excellent content organization. Developed in an on-line mode format (Brenda, is this redundant?), this course provides the student with extensive explanation of the purpose, objectives and activities involved in addition to directing students to recommended approaches for learning the material in this delivery format. The instructional content is thoroughly presented in a program-centered manner (WBL framework) providing the student with clear and concise partitioned segments of content. There is evidence that the instruction was originally interspersed with interactive elements such as forms, and assessment elements such as on-line quizzes.
On first glance, the course seems to represent a more objectivist pedagogy (WBL Framework) primarily in delivering content but inside collaboration is also included with discussion email and on-line chat mechanisms as well as conferencing bulletin board sessions that automatically create a conference topic relevant to the current content section chosen. Administration and tracking mechanisms are also included providing some metacognitive suppport for the student in resuming their place in the course and tracking progress through presenting the number and history of pages viewed and section printing capabilities (WBL Framework). Assessment of student learning through on-line quizzes and case study responses are evident; however, it is not clear how the students receive feedback on their responses. Efforts for formative evaluation on the course are elicited through various methods of response including email and forms.
SUNY TopClass Virtual Lounge & HelpDesk
WBT Systems, the distributor of TopClass, recommended a course designed and conducted by William Graziadei of the State University of New York, Plattsburgh, as a demonstration of their Web-based course delivery system. In spite of its title, "SUNY TopClass Virtual Lounge & HelpDesk (STCVL)," this is a complete course. STCVL is meant to provide a self-paced guided training for SUNY students and instructors who will participate in online teaching and learning using the TopClass application. The course overview states that Professor Graziadei hopes the discussion portion of the course will become a forum for faculty and students of different disciplines to share their experiences with online education.
The course overview encourages contributions through email (which is always available on the tool bar at the bottom of each screen), and discussions can be accessed through the course homepage, however, group interaction and collaboration (WBL Framework) are not an integral part of the course design. Only one folder, the "live interactive tutorial," appears to provide interaction between student and instructor. Unfortunately, access to this tutorial is restricted to the SUNY community. Another weakness of the course is the absence of a search engine, index, or glossary to help students who return with specific questions, sift through the profusion of information presented under each topic/unit. Also, considering the amount of information in some folders, the topic of Online Education could have provided many more outside links to resources on distance learning and instructional design strategies relevant to this delivery medium.
Effective Business Writing English 1007
The purpose of this course is to be delivered half the time in a face to face format and the other half fully online. This is known as the mixed mode compared to an adjunct or an online mode. In an adjunct mode, the online portion of the course is intended as a supplement to enhance traditional classroom instruction. The online mode is when the network serves as the primary environment for course discussions, assignments and interactions. These modes are referred to as 'modes of teaching' by Virtual-U; the tool used to develop this course, and are offered as guidelines for instructors in the Instructor Tools and Support component of the tool.
The course begins with four options to choose from; course syllabus, course overview, course calendar, and glossary. These options are not standard. They can be customized to reflect different choices based on the content of the course thus providing structural flexibility (WBL Framework). The course syllabus is an advance organizer and contains hypertext links to the course topics, units, assignments, tests, activities, resources, and online conferences and discussions. The links encompass a wide variety of electronic media utilizing virtually every attribute of the Internet and the World Wide Web. They include text files, tutorials, audio files, video clips, animations, graphics, online resources (with built-in search engines), and PowerPoint files to present course content and learning activities, as well as, synchronous and asynchronous communication forums ranging from email to a virtual café for socializing. Although intricately rich in multimodal representations and interaction levels, the variety is overwhelming and could present a level of complexity for first time users. However, it is evident that online support, resource support, and inside and outside collaboration is demonstrated through the use of these features (WBL Framework).
With each of these views there are Use features that can be accessed by the student and the instructor to perform certain activities. This Use feature promotes a student-centered approach to learning, which is one of the underlying instructional principles that Virtual-U supports. The workspace interface is a good anchor point for the user and perhaps should serve as the primary navigational structure for the course. It will improve the user friendliness of the interface.
Health Care Management
Health Care Management first conducted in the fall of 1997 provides an example of a course delivered via Web Course in a Box (WCB) authoring software. The course is structured using face to face class sessions interspersed with at least five electronic seminar discussions. This demonstration course highlights the main features and aspects of the WCB software but is currently deliberately restricted in content and resources as well as corrupted by multiple users experimenting with the authoring tool. However, in reviewing the structure and content that is available, many attributes of the course and the software are revealed.
The primary strengths of this course are the instructor's apparent attempts at management (WBL Framework) and organization of the content within the software constraints in addition to adequately addressing the on-line support issues (WBL Framework) necessary for student participation. These factors have pedagogical as well as interface usability implications. The electronic seminars focus on discussions about the off-line readings. These discussions are well directed with specific objectives and clearly outlined expectations established prior to providing a link directly to the conference on that topic. This organization facilitates the discussion of a specific topic focusing student contributions and easily directs students to the related conference session. Additional management issues are addressed in the course such as providing a help section and learning links dealing with Web development skills, however, the relationship of these skills to the required assignments could not be clearly determined. Inside collaboration (WBL Framework) between students in the course is fostered by WCB's providing space for personal biographies as well as individual and public class email capabilities.
Hospitality Industry Law and Liability
LearningSpace featured Hospitality Industry Law and Liability, a course from the University of Wisconsin-Stout, Department of Hospitality and Tourism. The course was entirely Web-based having no classroom component. The class databases were modified to remove copyright material, student discussions, and student profiles. Only a few units of the course are shown in their complete form, however, those sections demonstrate both the efficient ordering of content to help the learner achieve the objectives (Kemp, Morrison, & Ross, 1994) and clear expectations of what the student is required to do (WBL Framework). The units featured in the "Schedule" section provide reading assignments, content and process objectives, the purpose of the unit, as well as student and instructor evaluation criteria. The WBL Framework suggests that the course promote inside collaboration for working on joint projects. The instructor of this course requires collaboration with team members on many assignments, such as critiquing case studies. Collaboration is facilitated by the LearningSpace structure. It provides a shared workspace for team assignments. Teams may choose to restrict access to this space to members only. A message appears on the assignment link to indicate whether the work is "In Progress," "Ready for Grading," or ready for "Instructor Review." Collaboration is further promoted through discussion, which may be directed to the entire class or to team members.
Programming Your Avilar VCR
This course is intended as standalone instruction. It is self-contained and designed using tried and true instructional design principles. The pedagogical philosophy used in the overall design of the course actually shapes the learning environment (WBL Framework). The course starts out with clear instructional objectives and user directions on "how to take the course". Using a frame-based interface, the course is conveniently divided into six modules: lesson, exercises, questions, review, resources, and assessment. Buttons for each of these modules remain accessible throughout the entire lesson, however they are disabled and enabled based on where the learner is in the lesson. One must complete a lesson module first; then complete related exercises, answer questions, get feedback, review the lesson, and take the assessment. The progress through the lesson is therefore dictated by the organization of the content (to a large extent) and proceeds from basic to more complex learning outcomes. Although there is not much structural flexibility (WBL Framework), it is possible to jump between course modules using embedded hypertext or the Course Contents menu. It is recommended, however, that first time learners complete the course in the intended sequence. The modules can later be used as a reference or a job aid if one forgets how to perform a certain function.
The assessment and questions modules demonstrate a variety of multiple-choice question formats that can be used to evaluate learners' mastery of learning outcomes. Equally evident is the feedback feature after each response and the instant grade reporting that follows an assessment. Web Mentor supports the use of email, conferencing, chat rooms, application sharing, and other synchronous and asynchronous interactions that foster collaboration and interactivity. The course content however was not appropriately matched to the capabilities of the delivery medium (WBL Framework) and the use of a familiar methodology in designing instruction for a new technology may have once again contributed to the under-utilization of the tool.
Discussion and Implications
As is often evident in face to face courses, the Web-based courses demonstrated a mix of methods, strategies, and pedagogical approaches. Various strengths and weaknesses were noted in each case when mapped against the WBL Framework. Some courses adopted an objectivist, straightforward approach of content segmentation and delivery for mastery (see Programming VCR - Web Mentor), while others capitalized more fully on the inherent collaborative attributes of the Web in providing students with shared workspace (Hospitality Industry Law and Liability - Learning Space) and a constructivist student-centered approach (Effective Business Writing - Virtual U). Most presented a clear, logical and organized format (SUNY Top Class Virtual Lounge and Help - TopClass) and well-directed objectives and expectations (Health Care Management - WCB) while some provided exceptional guidance for learners with recommended approaches to the material (Marketing Certificate in Small Business Management - WebCT).
Primary weaknesses that emerged in the courses included confusing interface features such as indistinguishable icons and repetition of content in several simultaneous frames as well as a lack of structural flexibility in regard to multiple ways to access information. Those courses that did provide structural flexibility in content organization and representation, such as SUNY Top Class Virtual Lounge and Help, and Effective Business Writing, lacked advice for novice online learners on how to initially proceed with the course in order to take advantage of its instructional strengths. The principal caution for designers, which emerged from our investigation, is that students may need additional orientation and guidance particularly when interacting with a complex course and tool that provide a variety of Web-based delivery media. In addition, it is recommended that designers capitalize on the unique attributes of Web-delivery (such as opportunities for collaboration) when developing with this medium. Without this focus in mind, many courses may be more appropriately delivered with stand-alone multimedia software and technology.
In examining the courses and hence the Web-based authoring tools, we found that the WBL Framework is an effective tool for informal evaluation and a good guiding mechanism or checklist for the development of Web-based courses. Our position is clear in attempting to promote a more comprehensive evaluation of courses developed with these authoring tools rather than examining the tool itself. Educators do not evaluate a classroom course merely by the technology of delivery, (e.g. attributes of overheads, LCD display, teacher-directed discussion) therefore we should not discern the effectiveness of on-line learning environments merely by dissecting the delivery tool.
Our review of these courses and the attributes of the tools used to deliver them revealed several conclusions. In order for WBI to be effective it must be pedagogically driven, dynamically designed, interaction oriented, and content specific. Focus should be placed on designing a pedagogical approach appropriate for the content, inclusion of organization and interaction strategies that enhance the student's processing of the information, and integration of the medium's attributes to support the designated goals and objectives of the course. Developers of Web-based training and educational materials need to place emphasis on these tasks and view the technology in relation to its capacity to deliver the planned design. This focus is perhaps best promoted by ANTA:
Unfortunately no one tool can deliver every possible instructional approach. The software tools seem to have some common attributes as well as individual strengths and weaknesses. Most current web-based course authoring tools can also be viewed as restrictive in nature due to the visual metaphors employed by each tool that place constraints on the organization of content and the potential learning strategies employed. This factor certainly can restrict the learning environment produced and may prompt the need for web-based tools that are more open-ended, comprehensive and customizable. Hedberg & Harper make the valuable suggestion that new metaphors for authoring should be developed to match current theory. This notion seems a good start, however, a comprehensive advisement mechanism included within Web-based authoring tools providing guidance in the areas of pedagogical approach, instructional strategy, and on-line support and resources, may go further to support the developer who is lacking instructional design skills.
References
Australian National Training Authority (ANTA) (February, 1998). Teaching and Learning Styles that Facilitate Online Learning. [WWW document]. URL http://www.tafe.sa.edu.au/lsrsc/one/natproj/tal/
Barron, A. (May, 1998). Designing Web-Based Training. ITFORUM [electronic listserv]. Athens, GA: The University of Georgia. Available HTTP://itech1.coe.uga.edu/ITFORUM/home.html
Dabbagh, N.H. & Schmitt, J. (1998). Redesigning Instruction through Web-Based Course Authoring Tools. Educational Media International, v35, n2, pp. 106-110, June 98.
Center for Curriculum Transfer & Technology (May, 1998). On-line educational delivery applications: a web tool for comparative analysis. [WWW document] URL: http://www.ctt.bc.ca/landonline
Collis, B. (1997). Pedagogical reengineering: A pedagogical approach to course enrichment and redesign With the WWW. Educational Technology Review, Autumn/Winter, no. 8.
Hall, B. (1998). FAQ about Web-Based Training. Brandon Hall Resources. [WWW document]. URL http://www.multimediatraining.com/
Hansen, L. & Frick, T.W. (1997). Evaluation Guidelines for Web-Based Course Authoring Systems. In B.H. Khan (Ed.), Web-Based Instruction (pp. 299-306). Englewood Cliffs, NJ: Educational Technology Publications.
Hedberg, J. & Harper, B. (March, 1998). Visual Metaphors and Authoring. [WWW document]. URL http://itech1.coe.uga.edu/itforum/paper25/paper25.html
Jones, M.G. & Farquhar, J.D. (1997). User Interface Design for Web-Based Instruction. In B.H. Khan (Ed.), Web-Based Instruction (pp. 239-244). Englewood Cliffs, NJ: Educational Technology Publications.
Kemp, J.E., Morrison, G.R., & Ross, S.M. (1994). Designing effective instruction. New York: Macmillan College Publishing Company.
Khan, B.H. (1997). Web-based instruction (WBI): What is it and why is it? In B.H. Khan (Ed.), Web-Based Instruction (pp. 5-18). Englewood Cliffs, NJ: Educational Technology Publications.
Khan, B.H. (1998). A Framework for Web-Based Learning. [WWW document]. URL http://www.gwu.edu/~etlkhan/keynote/dimensions/framework1.html
Khan, B.H. & Vega, R. (1997). Factors to consider when evaluating web-based instruction Courses: A survey. In B.H. Khan (Ed.), Web-Based Instruction (pp. 375-378). Englewood Cliffs, NJ: Educational Technology Publications.
Lotus Corporation (n.d.) Learning Space. [WWW document] URL http://www.lotus.com/home.nsf/welcome/learnspace/
PC Week Labs (August, 1997). Evaluation of Internet Based Training Systems. [WWW document] http://www8.zdnet.com/pcweek/reviews/ibt.html
WBT Systems (n.d.) TopClass. [WWW document] URL http://www.wbtsystems.com/products/overvie1.htm
World Wide Web Course Tools (n.d.) Web-CT. [WWW document] URL http://homebrew.cs.ubc.ca/webct/webct.html
MadDuck Technologies (December, 1997). Web Course In A Box. [WWW document] URL http://www.madduck.com/
Avilar Technologies Inc. (1997) Web Mentor. [WWW document] URL http://www.avilar.com
Winn, W. (1990). Media and instructional methods. In D.R. Garrison and D. Shale (Eds.), Education at a distance: From issues to practice (pp. 53-66). Malabar, FL: Robert E. Krieger Publishing Company.
Simon Fraser University (n.d.) Virtual-U. [WWW document] URL http://virtual-u.cs.sfu.ca/vuweb/