Executive Summary

This evaluation concentrates on the effectiveness of the distance education practices in the part-time Mpumalanga Master of Management degree delivered over two years from 8 July 1997.

It is important to see the Mpumalanga Master of Management Programme in its context in the Mpumalanga Management Programme (MMP) as a whole. Although described by the programme manager as the most important project, the Master of Management is only one of several projects in the MMP. The key objective of the MMP is not to develop a satellite campus but to extend its existing academic programmes to Mpumalanga via distance education for a five year period so as to provide management capacity building, training and technical assistance in the region during a period when indigenous institutions are establishing and building more permanent capacities.

It was clear from the documentation as well as from interviews, that there is a resistance to distance education, as it is perceived to be a strategy for mass education, rather than for intensive postgraduate programmes. At the same time it was regarded as essential for the programme to be offered at a centre other than Parktown – for maximum strategic impact, as well as for the sake of the Mpumalanga students, who would be able to study while remaining in close contact with their workplace. Therefore, the programme uses certain distance education methods (such as videoconferencing technology), but every effort is made to retain the ‘customized personal relationships’ of a face-to-face programme.

SCOPE AND FOCUS OF THE EVALUATION

The major question addressed by the evaluation was: ‘What are the key lessons of experience regarding distance education practices, modes of delivery, and areas of improvement when compared to similar initiatives?’ A subsidiary but important question was: ‘Are the Mpumalanga students getting a comparable learning experience to the students in Parktown?’

In attempting to answer these questions, SAIDE’s evaluation team undertook an evaluation of the Mpumalanga Master of Management programme from July to October 1998. Data was collected from a range of sources, including:
- Teaching and learning policy documentation;
- Interviews with academics, support staff, MMP managers, and student representatives;
- Focus group discussions with Mpumalanga first year students;
- Survey questionnaires distributed to learning group leaders of first and second year groups;
- Observations of contact and video conferenced sessions;
- Review of course packs of the Information-based Policy Analysis and Decision-making course ;
- Review of a sample of individual and group assignments from the Information-based Policy Analysis and        Decision-making course; and
- Reviews of first and second year student profiles and assessment results.

In terms of analysis of the data collected, we focused on the effectiveness of the teaching and learning and assessment practices, as well as the appropriateness of the blend of methods for the students at a distance in Mpumalanga. The practices and methods were evaluated in terms of:

- Access and equity;
- Interactivity; and
- Support.

KEY FINDINGS OF THE EVALUATION

The key findings of this evaluation can be summarized as follows:

- Technology such as PictureTel can be used to provide access to students at a distance, but the interactive possibilities of video-conferenced lecture sessions should not be overestimated. It is important that other modes of delivery more suited to interactive and supportive teaching and learning be used in conjunction with lecture sessions conveyed via PictureTel. These include locally based tutoring, local face-to-face classes, self-managed learning groups, comprehensive and supporting course packs, e-mail and telephonic support, and work-related assessment.
-  Promotion of distance learning, and orientation and training of staff in the use of distance education methods is   crucial.
- The MMP confirms the more general experience that the move to providing for distance education learners poses a challenge to conventional management and administration processes that have been established for full-time learners.
- When considering result-based criteria, the evaluation shows that senior participants in the Mpumalanga programme are of the opinion that they are now empowered and that their knowledge and skills have improved in the work environment. In addition, examination results and throughput rates are comparable with those of the Parktown students.
- These key findings are expanded upon in the comments which follow and which have been structured around the two questions the evaluation was designed to address.

THE PROGRAMME AS A WHOLE

What is clear from the evaluation is that there is a commitment among the management and staff responsible for the programme to the continuous improvement of the service provided, guided by the criteria set out in the P&DM Teaching and Learning Policy document.
In evaluating a programme as a whole, however, we consider all the factors which give the programme coherence and which help to ensure that the overall purpose is achieved. This leads to consideration of factors such as the following:

- A clear strategic direction
- A clear purpose and outcomes related to content and assessment
- Effective management and administration
- An appropriate and effective blend of methods.

In terms of these requirements, what is still needed is for final clarification on the overall outcomes of the programme as well as guidelines on how these outcomes could best be assessed.

There is also a need to assess current practice in terms of the extent to which MMP provides a good educational experience addressed to the needs of a particular group of students, rather than focussing on concerns relating to the extent to which MMP manages to replicate a particular teaching model in a second centre.

Apart from lessons regarding the use of the technology (which are explored more fully in Chapter 4), the MMP experience offers a number of other useful lessons for future potential service providers:

- Time and resources need to be committed to formulating a strategic direction for the programme and to sharing and committing to the overall vision.

- A fundamental aspect of the MMP approach has been to manage the distance education programme from the point of delivery and to support distance education with locally based tutoring and local management. This has resulted in a blend of different methods being used (explored more fully in Chapters 4, 5 and 6).

- There is a need for distance education initiatives of this nature to focus on a joint vision and working programme between centres, especially the need for agreement on programme objectives and learning outcomes.

- The MMP experience indicates that a full-time senior academic at the point of delivery may have advantages of direct support in various areas as this allows for hands-on management and continuity and the counselling of students in outlying regions.

- The importance of establishing a local resource centre is an important lesson arising from the MMP experience and should be noted by potential new service providers. Linked to this is the need to consider sharing resources with other service providers in order to further reduce the distances students have to travel to access support..

FORMAL LECURE AND CONTACT TIME

The main means for communication of the curriculum is the 40 hours of lectures offered per module. The motivation for running the lecture sessions as frequently as they are run is clearly a result of the vision for the programme (the importance of ‘customized personal relationships’ with students), as well as the desire for equity of provision between Parktown and Mpumalanga.

- If the programme runs in the same way next year, attendance should be monitored so that programme staff can understand more clearly the reasons for lack of attendance in some sessions, and whether or not this can be linked to particular forms of presentation.
- Although staff-student contact is an important part of P&DM’s vision, we argue that long lecture sessions – which insufficient numbers of students attend – is not a way to ensure effective personal contact with lecturers, particularly if the lecture sessions do not make very good use of time. Replicating the same model of delivery as that used in Parktown may not result in effective (and equal) provision of opportunities for teaching and learning.
- It may be more effective to have fewer lecture sessions, and provide carefully structured ways in which students can prepare for these sessions – through independent learning from the course packs and other resources either individually or in learning groups.
- Thought should be given to how students can be encouraged to prepare adequately for the contact sessions. It may be necessary in the early stages to build in some kind of assessed reportback or test linked to the readings at the start of a contact session in order to establish a habit of adequate preparation. Linked to this is the need for sensitivity on the part of lecturers to the fact that students are studying whilst working. Fewer high quality readings may result in a greater commitment to preparation than supplying many nice to have but not essential resources.
- In short, it is necessary to interpret the P&DM Teaching and Learning guidelines within the context of the Mpumalanga experience rather than assuming that the approach which works for Parktown students will necessarily be equally applicable in Mpumalanga. If there is some divergence of approach in this way, it becomes even more of an imperative to have agreement on the vision, programme objectives and learning outcomes between the centres.

COMPREHENSIVE AND SUPPORTING COURSE PACKS

The P&DM materials development brief states that the course packs should be able to stand alone. However, this goal has not been met with all the course packs, and we therefore make the following recommendations:
          - Particularly if students are expected to prepare more thoroughly for sessions, the quality and administration of the             course packs will have to be improved.
          - Course packs will also have to be made more comprehensive. At present, even the best example reviewed mainly             communicated information rather than actually teaching through the provision of exercises with worked solutions or              feedback on questions, or summaries.
          - A greater emphasis on independent learning also requires more thought about how students at a distance can access             resources not contained in the course packs.
          - It is suggested that P&DM should set standards for the development of course packs more along the lines of       traditional distance education materials.

PICTURETEL VIDEO-CONFERENCING

The PictureTel Video-conferencing technology offers some advantages over traditional correspondence type approaches to distance education or the use of one-way communication satellite-based television in that it allows for a certain amount of real-time interaction between lecturers and learners operating in two different venues. MMP’s use of the system is an important study case for other service providers contemplating the use of the technology in their own distance education initiatives.

Internationally, use of video conferencing technologies appears to be significant among firstly, the corporate sector, and secondly, among distance education providers in the United States and Australia. Of critical importance is the fact that there appears to be an assumption that video conferencing technologies can be used to support mass lecture formats, typical of higher education institutions which must deal with larger numbers of learners and fewer resources. In this sense, decisions and technological choices seem to be incompatible with the educational or teaching and learning needs and potentials. According to one study on video conferencing and mass lectures,
        students and staff felt the lecturing, learning activities and interactions were not improved. They were also slower. Other disadvantages were time lost through technical difficulties and the greater likelihood for distractions at the remote campus. Students at the remote campus felt disadvantaged despite various preventative strategies…[Hence], the for which the medium was first really designed, namely administrative meetings, small group teaching and research seminars,appear to offer greater and significant productivity improvements.(Freeman, M. op cit. p209)

In South Africa, it appears as if some institutions have developed a similar form of use, in the sense that the technology is used as a mechanism for ‘democratising’ specific educational experiences among groups of learners, some of whom are distantly located. Use of the technology appears to be localised to institutions located in major metropolitan areas in South Africa, in part, due to the lack of technological infrastructure and support services available in rural areas. It also appears as if there is cautious use of the technology, in educational contexts, and hence, it is used to support teaching and learning experiences that occur outside of the typical academic, lecture-hall environment.

As noted in Appendix One, compressed video such as the PictureTel system allows can be used when limited moving video input is needed, such as show case-study material; to conduct role plays; to demonstrate interview techniques; to present information requiring high quality graphics or limited movement. It is not very successful if used to simply replicate transmission mode lecture sessions and use should be limited to group learning, discussion based sessions in which the presenter has made explicit in planning how to use collaborative learning techniques to ensure interactive exchanges with and between learners in both centres. It is recommended that in the MMP project, use of PictureTel is limited to these types of sessions for shorter periods, and that more time and resources are devoted to strengthening other forms of learning and teaching, in particular strengthening the learning group processes in contact sessions.

The lack of comparably extensive uses of the technology in South Africa, as well as the dearth of research in this area, suggest that the WITS P&DM MMP project should be the subject of a more in-depth study related to the impact the technology can have on a distance education programme. It is encouraging to note that the MMP staff are currently involved in documenting experiences of effective and non-effective use, and it is believed that the guidelines which emerge from this will be a valuable resource for other potential users of PictureTel and similar technology.

What follows is an outline of the more specific findings and lessons of experience organized around the three dimensions of the evaluation.

ACCESS AND EQUITY

The programme has succeeded in making the P&DM Master of Management programme more accessible to Mpumalanga students, and PitcureTel has played a part in this. However, students and staff feel that Technology Enhanced Learning is second best to face-to-face contact with lecturers. What the evidence in this chapter suggests is that if you try to provide the same experience for two groups of students, you immediately set yourself up for accusations (particularly from students) of unequal provision from the students perceived to be disadvantaged. The technical difficulties at the beginning which seriously affected student access, could probably have been avoided to some extent by more thorough research prior to implementation. The amount of staff time involved even currently when the systems are working well should not be underestimated.

A lesson of experience is that MMP facilitators attempt to fly lecturers to Nelspruit early during a course, so that students get to know them personally and also during concluding sessions, when students may need personal assurance and to feel that quality support is being received. A further lesson of experience is that professional, regular and committed technical support is vital. Major problems have occurred around Telkom ISDN usage.

INTERACTIVITY AND EASE OF USE

PictureTel technology offers an opportunity to promote real-time contact between students and lecturers. This serves the interests of both educational and social interaction. However, the extent of the interactivity which is possible with the use of the video-conferencing technology should not be over-rated. The entire video-conferenced session needs to be based on cooperative and collaborative learning strategies if the possibilities for interaction are to be optimized. It is clear that some lecturers are better able to do this than others and the experience of these people needs to be harnessed in the initial training and as well as the ongoing support process. The implications of the 0,7 second transmission delay, as well as potential problems related to the aural and visual quality of the presentation need to be considered in the planning of each session.

As the research in appendix one indicates, it is likely that given the current nature of use of the technology that video conferencing technologies are not specifically ‘better’ at achieving interactivity or effective delivery than other conferencing technologies.

Audiographic conferencing technologies are one option, but here, too, convergence between audiographic conferencing technologies and Internet technologies, suggests that the latter option would be preferable.

Lessons of experience for the MMP team have been that present technology has limitations in terms of its capacity to stimulate interactivity. Other users are encouraged to experiment, especially during training sessions and to adopt an open learning attitude towards what the situation may require. Specific lessons concern the use of camera shots, zooming, lapel microphones, using the window facility, muting for specific purposes, set camera positions, maintenance on equipment and so on. Guidelines are at present recorded and documented for future use.

SUPPORT AND FACILITATION

It is clear from the MMP experience that the use of experienced facilitators at the remote site fulfils an important need expressed by students. It is also clear that presenters and facilitators need ongoing support and guidance in the use of the technology. The appointment of technical support staff as well as the emerging guidelines on effective use of the technology, coupled with ongoing monitoring of use, can be seen as key lessons of experience.

The MMP team has facilitated the use of video conferencing equipment within certain capacity and financial constraints which have forced them to consolidate and improve on basic video lecturing before attempting to use the Internet, international conferences etc. in conjunction with their classes. In comparison with established distance education providers in the field, the MMP team does not have access to the following types of support which may be considered as lessons of experience for other users in the field.

- Some universities have fully equipped video editing studios where all lectures are taped, edited and distributed.
- Established distance education service providers may have access to considerable capacity in administrative and technical support e.g. IT personnel, video cameramen, printing, distribution and organising staff that form the conventional academic support typically found at a university/ technikon.
- A consortium approach in which several institutions make use of the same resources may be one way of ensuring the optimal use of the technology.

In initiatives which start off from scratch in outlying areas, it is regarded as an important lesson of experience to consider a computer facility and access to or the establishment of a resource centre or library with books, literature, videos and teaching equipment such as overhead projectors, and so on.

LEARNER SUPPORT SYSTEM

P&DM’s MMP argues that an integral part of its learning and teaching approach is the provision of appropriate support to students that enables them to achieve the course outcomes and develop management competence. From programme documentation made available to us, we identified the following forms of academic support, which have been envisaged for students in addition to the normal lecture sessions:

- Syndicate or learning groups;
- Support by lecturers on request;
- Workshops to assist students to develop specific skills (for example, use of Internet) or to bridge skills gaps (for example, statistical analysis);
- Comments by lecturers on assignments; and
- Supervision support on the research project.

A more comprehensive list of suggested vehicles and modes for learner support which arose from a discussion with Christo de Coning are summarised in a table on pages 75 and 76. We believe that this forms the skeleton of an integrated and flexible system of support that provides students with enough support to ensure successful attainment of the learning outcomes of the MMP, and the ability to be actively involved in their own learning processes. It is also critical that the support system be managed effectively and resourced properly.

The use of learning groups is an integral part of the MMP programme design and should be strengthened. P&DM approaches in using learning groups in distance education (compared to other practices of correspondence, one way TV, the attendance of lectures, etc.) holds many lessons of experience for application by other distance education practitioners. Some specific uses of learning groups include:

- Academic preparation/ learning/ organising for lectures.
- The use as a social support group (e.g. with the death of a member) in times of personal difficulty.
- Logistical support e.g. travel arrangements and the distribution of material.
- The development of skills to handle conflict in the group, to facilitate relationships and learning groups as a vehicle for motivation (e.g. peer pressure), discipline and the execution of work programmes.
- The use of the learning group leader, in improving communication with other learning groups, class representatives, the MMP staff and P&DM. Learning groups handle a wide variety of issues such as seminars, complaints and social events.
- Learning groups also have potential for life long learning and individual tutoring as these relationships are often transcended to the work place and beyond the duration of the MM course.

ASSESSMENT STRATEGY

It could be argued that unless an effective assessment strategy is in place, successful teaching and the use of expensive technology can have little impact on student learning. Assessment tasks compel students to engage with what has been taught, and the way assessment tasks are set encourages learning processes that are either helpful or unhelpful, that contribute to the goals of the programme or that distract students from achieving them. In consideration of this, we note the following:

- It is clear from the features of assessment design of the P&DM MMP that the assessment is work related and that every attempt is made by lecturers to integrate theory and practice. The assignment tasks and examination questions do by and large approximate activities/skills that form an integral part of the student’s work situation. The assessment of these tasks could be improved by including assessment criteria which are more explicitly linked to the overall outcomes of the course and which reflect a good balance between theory and practice. Group assignments lend themselves to the evaluation of actual management competencies required by students in their work environment.
- Ongoing attention will have to be paid to ensure that students develop ‘transformative’ competencies which are crucial if they are to become change agents in their own environment. This implies a more rigorous approach to workplace assessment.
- A strong feature of the assessment is the group assignments, which are structured into the assessment design. There is a potential for students to acquire team process skills by interacting with group members and co-operating on group assignments. This is an area, which needs further attention in terms of assessment. Developing criteria for group processes and integrating them in the assessment of group assignments would encourage students to engage more enthusiastically and meaningfully in the group activities.
- Detailed and useful comments on the assignments make feedback an important teaching mechanism on the MMP programme and a way of establishing dialogue with and motivating students. Feedback is also a useful tool to promote the goals of the programme and to encourage students to improve their academic and managerial practice by linking what they write in assignments to their own contexts.
- While oral feedback is also advocated, it may actually be far more effective to provide students with detailed written feedback. This type of feedback does not demand that lecturer and student be in the same location at the same time. Written feedback is a flexible form of individual and group teaching which may be well suited for students in Mpumalanga who are in demanding managerial positions.
- The present feedback schedule could be used quite effectively by lecturers to keep a record of the student’s progress. Such records provide a useful insight into the development profile of each student.
- Lecturers could make available regularly to the groups examples of good practice drawn from individual and group assignments.
- A more streamlined administrative system is necessary for staff in Mpumalanga to monitor the turnaround time of assignments. This makes it easier for them to interact with students and to iron out any problems immediately. The feedback schedule form could easily be used to record the necessary information and to monitor this process.
- The current marking scheme could be revised to link it more clearly with the overall learning outcomes of the programme. Impact on managerial practice and critical evaluation and qualification of theories and conceptual models are two additional categories for which criteria should be developed. Clearly defined criteria should enable students and staff to identify the standard expected at each level of competence. Such an assessment criteria schedule is a powerful means of integrating the different courses in the programme.
- It is useful for students if there is an attempt by staff to work towards more coherent and consistent assessment practices. Students need to know what is expected of them and how they are progressing across the different courses. The tension between compliance to basic assessment standards and practices and the need for flexibility and diversity of approach and practice needs to be managed.

Despite some initial start-up problems, and the fact that in the MMP experience there may still be too much use of the PictureTel technology at the expense of other forms of teaching and learning experience, it would appear that learners in Mpumalanga are getting a comparable experience to their Parktown counterparts.

This is borne out by the comparable examination results between the centres. In fact, the MMP project may have lessons for the enrichment of the Parktown experience in the improvement of the resource packs, the move to more interactive, collaborative learning approaches in contact sessions, the strengthening of learner groups and the slight reorientation to more emphasis on linking theory to practice in assessment procedures.

Feedback from students themselves (Appendix 6) also reflects a positive view of the programme overall:

- Impact of the programme on practice

Students have remarked that the course has had an impact on their management practice. Examples put forward include: improved ability to engage in strategic planning, greater confidence and ability to grapple with issues, greater motivation, a clearer understanding of processes, an ability to see the link between theory and practice and how to implement theory. Students also commented that the course keeps them abreast of new developments in the public service. One person cited that participation in the course had led to promotion. A student noted that his/her accountability and responsibility had improved and that there was no more late coming, absenteeism, loafing, postponement of submissions.

- What has been the most positive experience on this programme?

Students highlighted the following benefits: exposure to managerial practices and issues on a wider scale, debating current issues and linking them with theories, critical analysis of issues, exposure to computers.

They also regarded as very important their interactions with the lecturers and fellow students on topical issues about government and learning to work as members of a group. One student indicated that the course has helped him/her to improve relations with managers in the work environment.