Murray W. Goldberg, "CALOS: First Results From an Experiment in Computer-Aided Learning". Accepted for publication in the Proceedings of the ACM's 28th SIGCSE Technical Symposium on Computer Science Education (1997).

CALOS: First Results From an Experiment in Computer-Aided Learning

Abstract

This paper describes the outcome of an experiment in Computer-Aided Learning (CAL) using the World-Wide Web (WWW). The purpose of the experiment is to evaluate WWW-based course delivery both in terms of academic performance and in terms of student acceptance. A complete WWW-based offering of an existing third-year university course in Computer Science was produced. Students enrolled in the course were divided into three groups:

1. those taking the course using only the WWW-based resource (no lectures),
2. those taking the course by attending lectures (no access to the WWW-based resource), and
3. those with access to both lectures and the WWW-based resource.

Academic results were evaluated and qualitative student impressions were solicited. In general the WWW-only method of course delivery was an equal success compared to the lecture-only delivery, both in terms of academic results and in terms of student acceptance. On average, students in the combined lecture and WWW-based delivery performed better academically than students in the other groups.

Keywords

WWW Education Computer-Aided Learning

1. INTRODUCTION

The WWW provides educators with an outstanding opportunity to deliver courses, or to supplement existing courses, using a networked computer. There are clear advantages to the use of the WWW for CAL:

• courses can be developed once, by experts, and delivered at any scale. This creates the opportunity for improved course quality. It is also possible that a greater number of students can be served at a reduced cost. Finally, this ensures uniform delivery of courses; the quality will not vary with that of the instructor delivering the course.
• the pace, path and depth of the learning can be adapted by the learner to suit his or her background, level of aptitude and schedule.
• interactive Web-based exercises offer more opportunity for repeated experimentation and practice than is available with a chalk-board or pen and paper.
• the networked nature of the Web allows traditionally isolated distance-education students to be brought into a community of communicating peers through the use of bulletin-boards, chat areas and shared workspaces.
• courses can be delivered at any location. All that is required is access to a computer with a network connection or modem. This is an advantage for all students but is a special advantage for physically challenged students or those students for whom travel to a common campus is otherwise difficult.
• instant student self-evaluation is available through the use of multiple-choice questions or intelligent exercise evaluators.
• the Web is a familiar resource and Web browsers are quickly mastered even by students new to computers.

It was our intention from the beginning not to simply create the course and use it, but also to measure the effectiveness of the WWW and its level of student acceptance as a teaching resource. It was also our intention to experiment with the use of the WWW resource in varying contexts both in combination with lectures and as a stand-alone course. We have done our first round of experimentation in this respect and this paper presents our first results.

The remainder of this paper is organized as follows: section 2 presents an overview of the composition and implementation of our course. Section 3 describes the structure of the experiment in course delivery. Section 4 presents both the quantitative and qualitative results of our experiment. Finally, sections 5 and 6 present conclusions and future directions.

2. COURSE OVERVIEW

2.1 Implementation Summary

This mechanism allows for significant flexibility and interactivity, yet requires no special software or tools on the client (browser) side of the connection. The advantage is that our tools have to run only on our server; no complex porting or installation of software to client machines is necessary.

2.2 Course Contents

2.2.1 Page Layout

1. the information in the external navigation file (discussed below),
2. the student's log and preferences files, and
3. the HTML and supplementary (non-HTML) information contained in the source pages.

2.2.2 Navigation

• make the usual path through the course material obvious and direct
• facilitate unusual paths through the course material to accommodate students already familiar with certain aspects of the course material
• encourage excursions off the usual course path by allowing the student to instantly return to the point at which the path was left. A student should be able to do this using a single mouse click no matter how far off the main path they veer.
• when starting a new session a student should be placed in the same navigation context that they were in when they terminated their previous visit to the course.

In order to accomplish the latter two goals, we need to know the identity of the student making each request. Access controls, which are used to restrict course access, allow us to know the student's identity. Once the identity of the student is known, it is a simple matter to log that student's accesses to the course pages. Having a recorded history of each student's references allows us to place that student either back on the usual path (after an excursion off that path) or back at the same location from which they ended their previous session. Such a log also allows us to make inferences about the student and various course sections based on the number of accesses.

2.2.3 Communication

The chat service allows real-time communication among course participants. It presents a virtual set of rooms that can be entered. The names of the occupants of each room are displayed for all to see. The door to each room can be locked to facilitate private conversations. Each room can be labeled by its occupants in order to advertise the nature of the conversation taking place.

The bulletin board allows the posting of messages in a public forum. It can be accessed in either of two ways: either from the central bulletin-board page, or from any of the course-notes pages. When accessed from the central bulletin-board page, messages are displayed for the student to peruse. Knowing the identity of the reader allows us to refrain from presenting messages that have already been read by that student (unless he or she so desires). The usual range of bulletin-board features exist including threading and responding to messages with an embedded quote of a previous article.

When the bulletin-board is accessed from a page of course-notes, only messages pertaining to that page are presented. A message pertains to a page if it was posted from that page, or if it is a response to a message originally posted from that page. When viewed from a course-notes page, all messages, including ones previously read by the student, are presented. This essentially creates a global annotation mechanism where discussions about that page are recorded for others to see. The expectation is that questions, answers and other discussions about each page become part of the permanent record for that page. They supplement the learning experience, actually forming part of the content of the course. Shortcomings in the course-pages will usually be addressed in the related bulletin-board articles. This adds a dynamic, self-correcting nature to the course notes.

2.2.4 Interactive Exercises and Simulations

The simulations include six disk scheduling algorithms, three memory allocation algorithms, five page replacement algorithms, four process scheduling algorithms (including exponential averaging), process creation constructs, and others. Click here for an image of a disk-scheduling simulation.

These interactive exercises and simulations form an integral part of the course, adding an experience that cannot be matched in the classroom. In a typical class only one or two such examples can be presented due to the time required to set up and evaluate the problem. The WWW-based interactive examples relieve the student of having to set up the problem. As a result it is often the case that five or ten such exercises can be performed in the time required to do one corresponding pen-and-paper exercise. Click here for an image of an interactive exercise.

2.2.5 Student Evaluation

There is also a mechanism through which timed quizzes can be administered. A form with questions and text-entry boxes for the student's responses are presented. A timer is used to record the interval between when the student is presented with the quiz and when the response is given.

2.2.6 Other Course Features

• a complete, search-able course glossary containing text, images, and references to course notes
• each notes-page has an icon which, when clicked, presents the list of page and chapter references to the course texts for that page of notes
• an annotation mechanism that allows students to make private annotations of individual pages
• each page of notes includes an icon which, when clicked, presents a page containing learning goals for that page. This provides the students with topic emphasis that would otherwise be provided in lectures.

3. THE EXPERIMENT

3.1 Course Background

3.2 Experiment Setup

Those that volunteered were split into three groups. The placement was random except that an attempt was made to balance the groups so that all levels of aptitude were represented in each group. Aptitude level was measured from the student's performance in the main prerequisite course: CPSC 216; Data Structures.

The three groups were defined as follows:

• Group 1 was the WWW-only group. This group of students was not allowed to attend lectures. They took the course entirely on-line. These students were expected to keep pace with the regular offering, though initially no attempt was made to be sure they did so. This group met with the instructor once per week. This meeting was not a lecture, but was used as a time to discuss the student's perceptions and problems with the course.
• Group 2 attended lectures and had full access to the WWW-based material.
• Group 3 did not have access to the WWW-based material but received the course in the traditional format: lectures only. This group was the control group.

Two forms of results were expected. First, quantitative results could be measured by comparing the academic performance of each group. Second, qualitative results were sought through the administration of a questionnaire. The questionnaire was created by the UBC Center for Applied Studies in Evaluation. To ensure honest answers, students were assured that the questionnaire would be treated as anonymous, and that it would not be viewed until after course grades were submitted. The questionnaire was given out during the final exam and returned over the following week. Both sets of results are presented in the section that follows.

4. EXPERIMENT RESULTS

4.1 Experience Gained During the Offering of the Course

Another interesting result is that the students in the WWW-only group were divided in terms of their degree of self-motivation. One subset of WWW-only students greatly appreciated the ability to self-pace this course in order to accommodate the changing demands of their other courses. The remainder felt that without external incentive, they were unable to maintain the motivation required to keep pace with the lecture-based offering. The problem did not seem to be one of boredom with the material or the delivery method (most students seemed to enjoy the novel experience and the ability to work on the course anywhere, and at anytime). Instead, the problem was that each of these students was faced with at least four other courses being taken concurrently. Each of these other courses was lecture-based and was therefore not self-paced. As a result, when the overall load on the student was high (as it is much of the term) it was the self-paced course (CPSC 315) that tended to suffer.

Competition with other lecture-based courses became the most serious obstacle to the WWW-only student's acceptance of the course delivery method. Of the four students that opted out of the WWW-only group (and thus out of the experiment), each cited this competition with lecture-based courses as the reason. During our weekly meetings, the author and the students decided that the way to combat this problem was to institute bi-weekly quizzes. This would require that the students keep pace in order to be prepared for the quizzes. Those students that were highly self-motivated declared themselves and were exempt from the quizzes. Once the quizzes were instituted, the problem greatly diminished and no further students opted out of the experiment. We found it interesting that many students lacked the motivation to keep up on their own and were happier when the course pace was forced upon them. This may not be surprising, though, given that the bulk of most students' academic experience is not self-paced. One might speculate that if the competition for student's time was from other WWW-based courses (and not from lecture-based courses) that this would not have been a problem.

4.2 Academic Results

Instead we adopted the following approach. We decided to count the number of students in each group that performed:

• better than expected
• as expected
• worse than expected

In order to determine whether a student performed as expected, we declared a value (call it MARGIN) to be used as follows. If the grade in CPSC 315 was within MARGIN (either way) of the grade in CPSC 216, the student was said to have performed as expected. If the grade was less than the 216 grade - MARGIN, then the student was said to have performed worse than expected. Finally, if the grade in 315 was higher than the 216 grade + MARGIN, the student was said to have performed better than expected.

Determining the value of MARGIN was somewhat arbitrary. Even so the choice did have an effect. If we chose the value too small, a small difference in performance could move the student from better than expected to worse than expected (or the other way). If we chose the value too large, then most of the class would appear to have performed as expected. We experimented with many values for MARGIN. As expected, changing the value of MARGIN moved students either into or out of the performed as expected group, but did not move them between the better than expected and worse than expected groups. Thus, all values for margin gave the same qualitative outcome. For the purposes of the following results, a MARGIN value of 5% was chosen. The results are as follows (each number represents the percentage of the students in that group that performed better than, worse than, or as expected):

While these results can only be interpreted as anecdotal, they do correspond with general intuition. Those students that had access to lectures and the WWW-based material performed better than either of the other groups. Students that had access to only the WWW-based material or the lectures performed roughly the same. It is encouraging that it seems possible for a WWW-based offering to be as effective as a traditional lecture-based course.

4.3 Student Acceptance

• All groups of students were generally satisfied with their course offering. The WWW & lecture group was the most satisfied, followed by the WWW-only group and then the lecture-only group.
• All groups felt that their method of course delivery was effective. The WWW-only group and the WWW & lecture group felt equally strongly that their course delivery method was effective. The lecture-only group felt less strongly that their delivery method was effective than did the other two groups.
• When asked which components of the WWW-based course were most useful, students overwhelmingly cited the interactive exercises, bulletin-board, goal statements and quizzes.
• In all groups there was a large range in the number of hours students spent working on the course (including time spent attending lectures). However, the WWW-only group had the largest proportion of students that reported spending under five hours per week. Also, as expected, the WWW-only group spent more time on-line than did the WWW & lecture group.
• WWW-based students guessed that they had learned roughly the same (or perhaps a bit less) than they would have in the lecture-based course. On average, they felt that they had to commit a bit less time than they would have to a lecture-based course. Also, most agreed that taking WWW-based courses requires more personal responsibility that a lecture-based course.
• WWW-based students felt that access to the instructor was important, and on average felt that having this access once per week was sufficient. Preferred methods of meeting with the instructor were either one-on-one or in a classroom setting with other students. The use of a bulletin-board was also a reasonably popular choice for instructor contact. When asked how often lectures should be held (if at all) in conjunction with a WWW-based course, all agreed that lectures should indeed be held, and most said once or twice per week was preferred.
• When asked whether there was anything missing from their education because they took the course on-line (and could not attend lectures), most students said no. Those that said yes cited in-class discussions and interaction.
• On average, students taking the WWW-based course preferred that mode of delivery over lecture-based courses. Roughly one third far preferred WWW-based courses, and another 40% somewhat preferred WWW-based courses. When asked whether they would take another WWW-based course, almost 90% said yes.
• Given the choice between a WWW-based course and a traditional distance-education course (with paper-based notes and telephone contact with the instructor), almost all of the WWW-based students said they would prefer the WWW-based course.
• When asked what they liked best about the WWW-based course and taking WWW-based courses in general, the following points were mentioned (roughly in order of decreasing frequency):
  • The ability to self pace in response to the availability of time and the difficulty of the topic
  • Not having to attend classes
  • It was easy to refer back to previous topics
  • The interactive exercises and computer simulations
  • The whole course is available at once
  • There was no arbitrary 1-hour division in learning sessions (some topics required more time, some less)
  • The ability to learn anytime and anywhere
  • The novelty of the experience
• Finally, when asked what they liked least about the WWW-based course and taking WWW-based courses in general, the following points were mentioned (roughly in order of decreasing frequency):
  • The extra motivation and responsibility required
  • The inability to get immediate answers to questions
  • They didn't meet and interact with other students
  • They disliked being in the first offering as they helped debug the course pages

• How satisfied were you with the method by which you took CPSC 315?
• How effective do you feel the method used to deliver the course to you was?
• On average, how much time did you spend working on the course each week (including lectures)?
• On average, how often did you sign on to the on-line course?
• Compared to a lecture-based course, I (in the WWW-based course) have learned (more, less):
• Compared to a lecture-based course, I (in the WWW-based course) have spent (more time, less time):
• Compared to a lecture-based course, the WWW-based course requires (more or less personal responsibility):
• When taking an on-line course, how important is it for you to have contact with the instructor?
• When taking an on-line course, how much access to the instructor do you feel you need?
• How often should lectures be held in conjunction with an on-line course?
• Did you feel there was anything missing from your education because you did not attend lectures?
• Given the choice between lecture-only courses and WWW-based courses such as CPSC 315, I would prefer:
• Would you take another course on-line?
• Given the choice between a traditional distance-education course (paper-based notes, textbook & telephone contact with TA) and WWW-based courses such as CPSC 315, I would prefer:

5. CONCLUSIONS

Some of the most interesting results of this experiment are summarized below:

• The group of students in the WWW-only offering achieved roughly the same level of academic performance as did the group of students in the lecture-only offering. As one might expect, students with access to both sources of information achieved a higher level of performance.
• The level of student acceptance of the WWW-based offering was high. Most students preferred the WWW-based offering to lecture-based offerings. Almost 90% of the WWW-only students said they would be willing to take another WWW-based course.
• When taken concurrently with other lecture-based courses, the most serious impediment to success in a WWW-based offering seems to be self motivation. Yet other students seemed to thrive on the self-paced nature of the course.

6. FUTURE WORK

First, now that we have some student opinion as to the best combination of WWW-access and lectures, we plan to follow this up by experimenting with a WWW-based course offering supplemented by one or two lectures per week (as opposed to the traditional three lectures per week). Perhaps this would provide the missing human contact and motivation, yet still yield most of the advantages of the WWW-based offering.

Next, WWW-based distance offerings are scheduled for the Spring and Summer of 1996. It will be interesting to review the opinions of those students that are accustomed to taking traditional distance-education courses. It will also be interesting to observe the interaction on the bulletin-board and chat areas between the local and distance students. It is our hope that the distance students will experience a far greater connection with the distributed community of students than is possible in traditional distance-education.

Finally, new technologies (for example, Java) being incorporated into popular WWW browsers enable us to create interactive exercises that are more attractive and responsive than our current cgi-based exercises. We plan to take advantage of this opportunity.

References

[1] E. Bilota, M. Fiorito, D. Iovane and P. Pantano, "An Educational Environment Using WWW." Computer Networks and ISDN Systems 27 (1995) 905-909.

[2] J. Campbell, S. Hurley, S. Jones, N. Stephens, "Constructing Educational Courseware Using NCSA Mosaic and the World-Wide Web," Computer Networks and ISDN Systems 27 (1995) 887-896.

[3] The World Lecture Hall, http://wwwhost.cc.utexas.edu/world/instruction/index.html

[4] Introduction to Object Oriented Programming Using C++, http://info.desy.de/gna/html/cc/index.html

[5] B. Ibrahim, S. Franklin, "Advanced Educational Uses of the World-Wide Web," Computer Networks and ISDN Systems 27 (1995) 871-877.

[6] D. Nicol, C. Smeaton, A. Falconer Slater, "Footsteps: Trail-blazing the Web," Computer Networks and ISDN Systems 27 (1995) 879-885.

[7] D. Dimitroyannis, "Virtual Classroom: A Case Study," http://www1.cern.ch/PapersWWW94/ddimitri.ps

[8] Murray W. Goldberg, "CALOS: An Experiment with Computer-Aided Learning for Operating Systems," To appear in Proceedings of the 1996 SIGCSE Technical Symposium, February, 1996.

About The Author

Back to List of Papers
Back to Home Page