Papers Presented at the 1st National NADEOSA Conference
Held 11-13 August 1999

Author:
Dr Liz (ESG) Greyling Bureau for University Education Rand Afrikaans University

Title:
How To Determine The Instructional Quality Of Online Courses: A Case Study
Sub-theme: Courseware Design and Development & Technology

Abstract:
In answer to the powerful argument of why South African higher education institutions should continue to pay the high costs of traditional instructional methods when new technologies can make education available in better and less costly ways, the Rand Afrikaans University (RAU), like many other South African universities, has introduced a number of online courses at undergraduate as well as postgraduate levels. These courses are offered in addition to reduced contact and distance courses in all faculties. The primary task of the Bureau for University Education (BUE) is teaching development. This being the case, the BUE is involved in assessing the teaching and learning in all study programmes and courses offered by the university. Since the introduction of online courses, the BUE was therefore also faced with assessing such courses to determine if they were instructionally sound and if they met institutional, faculty, departmental, lecturers’ and students’ needs. This paper deals specifically with a case study on the assessment outcomes of two Engineering courses. It aims to highlight the assessment procedure and compare specific outcomes with trends emerging from research done for the context of the paper.

About the Author:
Liz Greyling, a Senior Researcher at the Bureau for University Education, RAU, is currently responsible for the assessment of Web-based, reduced contact and distance courses; teaching-learning development; editing; and lecturing Hons and M-students in applied Linguistics. She holds a BA Languages and Senior Teaching Diploma (STD) from the University of Stellenbosch, and a BA (Hons), B.Ed (Hons), M.Ed (Educational Management) and D.Ed (Tertiary and Adult Education) from the RAU. Special fields of interest: Developing and assessing study material for technology-based distance education.

INTRODUCTION

Students and lecturers in South African higher education institutions are increasingly being exposed to the virtual world’s potential as a teaching, learning and research resource. It is widely believed that virtual education will have major advantages for and will fit into the policy framework for South African higher education. Technologies spearhead serious institutional reform because they create real change, especially in the area of course content and teaching. However, their successful use depends on the quality of the teaching staff and the excellence of the teaching and learning programme (Wydeman, 1996:10; Privateer, 1999:62). Across all sectors of the higher education landscape, institutions continue to struggle with key aspects of IT planning and infrastructure: developing a strategic and financial plan for it, planning curriculum integration, and providing adequate user support. According to Ehrman (1999:27) and Green (1997:1), the cost of putting a course online is affordable but requires advanced faculty skills. Whether this be true or not, universities worldwide, as well as in South Africa, have timeously embarked upon introducing online courses. Such courses will cater for larger and more diverse student populations, thus affording an increasing number of students access to higher education.

The BUE of the RAU assessed the online courses offered at the university. The assessment outcomes of two Engineering courses, namely Electromagnetics 3A and Electrotechnics 2A, will be discussed in this paper. Both courses are structured in the form of a combined teaching-learning programme, consisting of 80% contact and 20% computer-based learning, thus combining working with the learning content with e-mail responses. The computer-based part of the course therefore amounts to 20% of the students’ total formal assessment marks.

ASSESSMENT RATIONALE

The main objective of this assessment was to determine the usefulness of the computer component of the courses. The lecturers were responsible for formulating the critical and specific outcomes of their courses, ensuring that learning takes place and for designing adequate assessment strategies. They needed to formulate learning outcomes that would be consistent with how teaching technology could be integrated throughout the course curricula, preparing students for what they would need to demonstrate on completion of the courses. What was needed, was re-engineering of what was being done and re-inventing of what was produced. Both lecturers agreed that student competence with technology had become essential for the courses (compare Ehrman, 1999:26/27; Privateer, 1999:63-66).

ASSESSMENT PROCEDURE

At the request of both lecturers, discussions were held to determine the format of a proposed questionnaire. BUE staff provided a clear understanding of the aim of the proposed questionnaire, namely to obtain important information on how the Engineering student experienced the computer-based part of the course. It was also explained that

honest, anonymous student opinions would be of great value for course development in the department and that feedback should be viewed as perceptions of students, being the "clients" doing the course. Subsequently, a data bank of 50 questions was compiled and structured in three sections:

• Section A included 39 questions on the design of the programme, with 5 response options: strongly agree, agree, neither agree nor disagree, disagree, strongly disagree.
• Section B consisted of 8 questions on the use of the computer-based study guide, with 5 response options: very important, important, neither important nor unimportant, unimportant, definitely unimportant.
• Section C dealt with 3 open-ended questions on the organisation of the course (i.e. available computer time), printing a hard copy of the study guide, and comments on the course in general, with 2 response options: yes/no, and motivation.

Responses anticipated for Section C of the questionnaire included answers to the following questions (compare Ehrman, 1999:29):

• Do students have access to computers?
• Would ‘naive’ computer users (Heese, 1998:60) be able to navigate through the programme with ease?
• Would they persist?
• How would students’ attitudes change towards the course content?
• What would the students’ attitudes be towards the computer section of the course?
• Would their academic skills improve?

The lecturers were advised to select no more than 25 questions, to alter where they deemed necessary or add questions they wanted to include. The two lecturers use different teaching strategies and styles and therefore selected different (though still 25) questions. Electromagnetics 3A is strongly based on a computer study guide, while Electrotechnics 2A is strongly based on practical exercises. Both lecturers use WebCT software. Questionnaires were handed out to students during their usual contact session. A total number of 72 students completed the questionnaires.

ASSESSMENT OUTCOMES

The assessment produced a mix of good and bad results. Outcomes were reported in writing, and subsequently discussed with both lecturers (Van Tonder, 1999).

Electromagnetics 3A

Outcomes

Only half the respondents (50%) agreed with the following statements regarding the design of the programme (compare figure A):

Q1: Sufficient training was provided for using the computer programme.

Q2: The self-assessment questions in the programme helps students to understand the learning content better.

Q11: The volume of learning content is manageable.

Q14: The students gain basic computer competency through using the programme.

Section B: The lecturer selected 5 questions from the databank of 8 questions.

Outcomes

The students rated the statements from very important to definitely unimportant, as follows (compare figure B):

Q19: Only half the students (50%) rated the study guide as important to prepare for contact sessions.

Q20: Students did not regard the study guide as important when preparing for practicals.

Q21: Students regard the computer-based study guide as important when preparing for tests.

Q23: Students feel that because of the reduced contact time, the study guide is very important.

Section C: Students completed all 3 open-ended questions from the databank.

Q24: A total number of 8 (23,5%) students indicated that they had printed a hard copy of the computer-based study guide, while 26 (76,5%) did not print a copy of the study guide.

Commentary was as follows:

• The hard copy was needed to follow lectures.
• Online study can be expensive.
• It is difficult to save the web page on disk.
• Information is too abstract - insufficient down-to-the-point information is provided on practicals.
• At the computer centre we may only print 10 pages.
• It is a waste of paper and toner.

Q25: A total number of 17 (50%) students commented as follows on the course in general:

• 6 of the 17 students (33%) had computer access at their residences.
• More detailed summaries of previous work should be provided - the text book is very complicated.
• 3 of the 17 students (17%) would like more practical examples.
• Students would like advance information on practical assignments.
• The linking of the work done during contact sessions and the computer assignments is not clear.
• The students would like memoranda of problems and exam papers.
• They would like to submit their pre- and post-reports via e-mail.
• Students would like more contact time - it is too much like distance learning.

Electrotechnics 2A

Section A: The lecturer selected 17 questions from the databank of 39 questions. The lecturer altered Q2 to include tests and added a question on the linking of class and computer work (Q18).

Outcomes

The students did not agree with the following statements regarding the design of the programme (compare figure C):

Q3: Sufficient computer access is available.

Q14: Students can control their learning pace (with reference to the completion of assignments).

Section B: The lecturer selected 6 questions from the databank of 8 questions.

Outcomes

The students rated the statements on the use of the computer-based study guide from very important to definitely unimportant, as follows (compare figure D):

Q19: Students rate the study guide as important to prepare for contact sessions.

Q20: Students rate the study guide as very important when preparing for tests.

Q24: Students feel that because of the reduced contact time, the study guide is very important.

Section C: Students completed all 3 open-ended questions from the databank.

Q25: A total number of 23 (60,5%) students indicated that they made use of the allocated computer time in the computer centre, while 15 (39,5%) did not use the allocated time. Commentary was as follows (compare figure E):

• Students used the time to classify and categorise work to be studied.
• They did not have time to go to the computer centre.
• Students had Internet facilities at home.
• They would like to move the session to another day that suits them better.

Q26: A total number of 22 (57,9%) students indicated that they had printed a copy of the computer-based study guide, while 16 (42,1%) did not print a copy of the study guide. Commentary was as follows:

• 2 of the 16 students (12,5%) did not know they were allowed to do this, but thought it was a good idea.
• It helped students to understand the learning content better when they had a hard copy of the notes.
• When a computer was not available, they could still work on their assignments.

Q27: Students commented as follows on the course in general:

• "Computer-based sessions are very important, and I have noticed a difference in my understanding of the subject."
• The lecturer does not provide any computer support and is not available to help with problems.

TRENDS EMERGING FROM RESEARCH COMPARED TO SPECIFIC OUTCOMES OF THE ASSESSMENT

When studying the outcomes of this assessment, useful trends to consider when planning to assess online courses, emerged. These trends echo research findings that were studied for the context of this paper (compare Green, 1998:1; Van Ryneveld, 1998:49/50; Ehrman, 1999:26-29); Privateer, 1999:61-75). Computer-based courses:

• reduce teaching and learning time;
• provide self-paced and just-in-time learning;
• allow students to learn faster by ‘doing’ rather than reading or simply observing;
• foster motivation to learn;
• exploit interactivity;
• allow students to learn from mistakes;
• do not meet important student needs or provide adequate user support: students with low technical skills are not identified early and provided with assistance often enough;
• often focus on the storage and recall of random information;
• expect similar learning outcomes from all students, regardless of what computer programmes they use;
• result in tests often being echoes from lectures: the testing of a student’s knowledge by requiring him/her to "fill in the blanks" exemplifies reproduction-based learning;
• use online portfolios as a performance measurement tool;
• help students to organise information in personally meaningful ways;
• leads to slight addiction to and overconfidence in using computers; and
• distract students from the learning content covered in class.

CONCLUSION

The computer extends the traditional face-to-face classroom to include a virtual classroom. Real education is to excite students’ interest and Virtual Reality has the possibility to foster better learning outcomes (Van Ryneveld, 1998:54; also compare Brown, 1998:7 and Ehrman, 1999:29). In addition, interactive classrooms also offer ‘telepresence’ - the feeling of being present in a classroom with a teacher and other students where there is considerable opportunity for human intellectual and social interaction during the teaching and learning process (Hegarty, Phelan & Kilbride, 1998:3). I would like to conclude with two quotations:

"There are ‘... three areas where computers are more skilled than human minds: [they] are evaluations of computations, storing massive amounts of data, and remembering things without forgetting’" (Privateer, 1999:76).

"Technology - with its unpredictable changes and vast expenditures - has made flying blind much more dangerous than it used to be. The good news is that [some] institutions are learning to see" (Ehrman, 1999:29).

REFERENCES

BROWN, S. 1998. Use the Internet to teach! Who? Me? Pro Technida, 15(2): 5-15.

EHRMAN, S.C. 1999. Asking the hard questions about technology use and education. Change, 31(2): 25-29.

GREEN, K.C. 1998. Colleges struggle with IT Planning. The Campus Computing Project: 1998 National survey of information technology in higher education [http://www.compuscomputing.net.ericir.syr.edu/Projects].

HEESE, M. 1998. Developing an interactive multimedia academic reading skills course for naive computer users. Progressio, 20(2): 50-64.

HEGARTY, M., PHELAN, A. & KILBRIDE, L. 1998. Classrooms for Distance Teaching and Learning: A Blueprint [http://www.kuleuven.ac.be/upers].

PRIVATEER, P.M. 1999. Academic technology and the future of higher education. The Journal of Higher Education, 70(1): 60-79.

VAN RYNEVELD, L. 1998. Virtual reality in the adult learning environment. Vital, 12(1): 47-54.

VAN TONDER, A.H.D. 1999: Kursusterugvoer: Internetkursus in Elektromagnetika 3A en Elektrotegniek 2A (Departement Elektriese en Elektroniese Ingenieurs-wese). Randse Afrikaanse Universiteit: Buro vir Universiteitsonderwys.

WYDEMAN, J.L. 1996. Evaluation of a real-time broadcasting model for distance education. A NETECH report on a pilot project. Pretoria: HSRC.

NADEOSA Homepage |  Ist NADEOSA Conference Papers Index  |
Global Distance Education Network : South African Resources

Send comments on the web site to the web designer