Evaluation of the NAMCOL-Bath Partnership Project

Many people contributed to this evaluation report and we hope that it is a true reflection of the experiences and insights of those who participated in the NAMCOL-Bath Partnership Project over the past three years.

A vote of thanks to DIFD for generously funding the evaluation. We also thank NAMCOL and Bath for inviting SAIDE to become involved in this important phase of the Project. .

We could not have achieved our goals without the efficient support of Alison Mead. Thank you Alison for scheduling and rescheduling interviews and for compiling the documentation so accessibly. We also appreciate Tris Bartlett’s efforts to get the evaluation process moving forward.

The evaluation was driven by interviews that we conducted with a range of people. We would like to thank them all for their valuable insights and suggestions, which have contributed substantially to the report. These are:

vSome learners from Rundu and Oluno.
vSome tutor markers and tutors from Head Office and the North East region.
vFull-time staff at NAMCOL some of who were interviewed several times as members of different teams.
vMrs Frances van Wyk, the NAMCOL Director and the Senior Management Team.
vAlison Mead, the Technical Adviser.
vAndrea Taylor of the Bath Management Team.
vMary Hayden, Director of Studies for Advanced Courses at the Education Department of the University of Bath.
vEd du Vivier, the APSO Adviser.
vTony Dodds, Director of the Centre for External Studies at the University of Namibia.

The purpose of this evaluation was to review the NAMCOL-Bath Partnership Project and asses its impact on strengthening the capacity of NAMCOL to deliver sound open and distance learning programmes. We hope that this report will offer valuable insights for all those who were involved in the Project.

The SAIDE Evaluation Team

Tessa Welch and Christine Randell

31 October 1999.

***Back to Contents***

The introductory section provides direction and overall focus for the evaluation report and gives an indication of how the different parts are linked together. In this section we examine the following:

vPurpose and focus of the evaluation

vEvaluation approach and method

vPurpose of the report

vStructure of the report.

1.1 Purpose and focus of the evaluation

The Terms of Reference specify that the evaluation is required to assess the effectiveness, in terms of impact, sustainability and cost benefit, of the inputs designed by the Project to meet the objectives and outputs listed in the Project Logical Framework for the period 1996 – 1999.

In a discussion held with the Bath Project Manager prior to our visit to Namibia it was agreed that SAIDE would not carry forward the financial analysis but would restrict its focus to an appraisal of the Project in terms of technical and institutional impact and sustainability. The Project Manager indicated a willingness to assist the process by looking for another consultant who may be able to carry out an economic analysis.

The documentation has revealed that there are two distinct phases of the Project: the period before April 1998 when NAMCOL was established as a parastatal and the period afterwards. This is also reflected in the two versions of the Project Logical Framework (Logframe). The original version, which had been based on the Project Memorandum dated December 1995, was in place up to March 1998, and an amended version was used to guide the Project from April 1998 to September 1999. NAMCOL staff were only involved in contributing to the amended version. The changes made to the original Logframe are significant and an important focus of the assessment is how effectively the Project responded to the changing needs of NAMCOL, and how this affected the design of the Project.

The challenge in writing this report is to keep our focus on the Project and its impact on the staff and the organization without digressing into NAMCOL areas and issues that may be linked but that are outside the scope of this evaluation. The Logframe, its purpose, outputs and objective verifiable indicators, is the thread that we will pull through the fabric of the report.

1.2 Evaluation approach and methods

The evaluation is organized at three distinct yet interrelating levels as illustrated by the overlapping circles in the diagram below.

At level A, our focus is primarily on the implementation of the Project plan in order to establish what it was supposed to achieve, what changes were made and why, how many staff participated and what their reaction was to the Project.

At level B we probe deeper in order to assess what knowledge and skills were learned by staff participating in the Project and which interventions staff found to be particularly useful.

At level C we investigate the impact of the Project on staff’s work practices and on the organization. A range of work-related evidence such as samples of course materials, feedback on assignments, tutorial letters, handbooks and manuals, reports and plans are examined in order to establish what improvements have taken place since the start of the Project.

The cascade model of training is integral to the Project’s sustainability and we will examine and assess how effectively it was in building capacity. We also want to establish the level of self-sufficiency that staff have achieved and determine to what extent NAMCOL’s reliance on external expertise has diminished as a result of the Project.

Information gathered at all three levels in the critical areas should enable us to construct a multi-faceted picture of what was achieved and how effective the Project was for NAMCOL staff.

The philosophy, which underpins the research approaches and methods we used are those that stress the importance of the experience of individuals and their reflection on their experience as ways of constructing social reality. The search for understanding such reality is by its very nature qualitative rather than quantitative. Our choice of illuminative and interpretive approaches to the evaluation has enabled us to get an understanding of the issues from the perspectives of a range of people: those who planned and implemented the programme, those who participated in it, and some who viewed it from the outside.

In terms of interpretive research we explored a range of emerging themes and issues with selected core individuals and teams and tested their views, values and understanding. The discussions became an important mechanism through which people were able to refine their own view of things and in some instances one could see their awareness growing. The transcripts of these interactions have become a rich source of information from which we have been able to create a picture of the Project as experienced by the people in it. This has made it much easier for us to distil the most important issues and findings.

We used two types of interviews to collect the data. The more question driven interview with some of the part-time people and the in-depth issue driven interviews with full time staff. We held focus group interviews with a range of teams and we interviewed individuals within NAMCOL and outside the organization.

We conducted interviews with NAMCOL staff at the Head Office in Katutura, Nambia from 18 – 31 August 1999. One evaluator, who was accompanied by the research and evaluation officer, travelled to the Rundu and Ondwanga region to interview area coordinators, a learner support officer, Heads of Centres and tutors. The research and evaluation officer interviewed two groups of learners from Oluno and Rundu. Another evaluator travelled to Otjiwarongo to interview the senior management team at their strategic review retreat.

A full list of people who were interviewed can be found in Annex One.

We held two in-depth focus group interviews with both the Programme and Materials Development Team and the Learner Support Team on separate days. During the first interview we focused mainly on the impact of the Project, whereas in a follow up interview we explored issues of sustainability. By having both evaluators present for three of the four interviews it was possible to extract a wealth of information. During the second interview with the PMD team we involved the team in an evaluation activity of their own materials. This was done as a reality check.

We conducted a separate focus group interview with the Project Team and the MAEC Participants. Both these teams provided useful information about their experience and views of the approach and specific methods of staff development used in the Project.

We were able to interview the Director on two separate occasions. This gave us the opportunity of checking and testing our findings.

A number of staff members were selected for individual interview sessions because they had experience of a specific type of training, for example the regional placements, or because of their role.

We held focus group interviews with tutors, tutor markers and heads of centres (HOC’s). Our intention was to check whether the training is cascading to the part-time staff at head office and the regions.

We decided to involve one of the NAMCOL staff members in interviewing a few groups of learners as a reality check. The learners are the beneficiaries of improved systems and processes. We wanted to check whether any improvements have filtered to them.

We interviewed the APSO Adviser and Tony Dodds who have been associated with NAMCOL for a long time. They provided insights into NAMCOL prior to the Project. They were also able to offer impressions of the impact of the Project from an outsider’s viewpoint.

Three separate interviews were held with the TA. She holds a unique position in the Project and her views and insights have been particularly useful in helping us gain a sense of the progress of the Project over time. Her written reports have helped to fill in the gaps.

We conducted a telephonic interview with two members of the Bath Team to explore their views on the key issues of the Project. We had a separate interview with the Director of Studies for Advanced Courses to explore specific issues around the Modular Advanced Education Courses (MAEC) programme.

A questionnaire was distributed to 15 staff members who had participated in the Project. We wanted to get an individual assessment from them about the main components of the Project. This information complements our findings in the interview sessions.

The Project is very well documented. A full list of the documents made available to us is included in Annex B. We scanned the documents and identified issues and themes. The documents were a rich source of information and we have used them extensively in writing the report.

One of the evaluators has used key criteria for distance learning materials to assess selected materials developed by NAMCOL staff since the start of the Project. As a check she has also done the same evaluation of a pre-Project set of materials. These evaluations have been incorporated as part of section three of the report.

1.3 Purpose of the report

The evaluation is summative in that it is being conducted at the end of the Project. However, its purpose is essentially formative because it is hoped that the evaluation report will be used as a tool to improve future practice for the three key stakeholders: NAMCOL, Bath, and DFID. NAMCOL and Bath are considered the primary audience. NAMCOL will be looking to the evaluation for information on how the institution can be strengthened in future; Bath will expect to get information on how it might improve its Project management for similar Projects in future. DFID considers itself a secondary audience for the report, and will look to it to provide information on how to improve its future plans.

In this report we aim to present a critical review of:

vthe level of implementation of the inputs planned by the Project and the staff’s reaction to them;

vthe appropriateness of the staff training and development approaches and methods;

vthe impact on the quality of distance education materials developed and the staff’s ability to continue on their own;

vthe impact on the quality of learner support services for open and distance learners, and the ability of staff to continue on their own;

vthe impact on senior management’s ability to provide an improved strategic focus for NAMCOL.

1.4 Structure of the report

Figure 1.1 on page two is a useful organizing framework for the report. The Project is extensively documented and has been monitored and evaluated internally on an ongoing basis by the Bath Team. We do not intend to reproduce what already exists but see our role rather as providing a helicopter view on the Project. Our approach is to put forward the main themes and issues, to reflect the views of the different participants, to critique such views and present, wherever feasible, suggestions for the way forward. We reserve the concluding section for an analysis of the main findings and for making final recommendations.

In section two we present a level one assessment of the Project. At this level we present a summary of the inputs made over the three year period. We establish what progress was made in achieving the Project’s purpose and whether staff are satisfied with what has been achieved.

In sections three, four and five a level two and three assessment is presented in the three critical areas: programme and materials development and production, learner support and management. In all three sections we investigate what capacity building has taken place and t the Project’s impact on the work practices of staff and on the organization. Sustainability of the initiative is probed in order to determine whether staff in these areas are able to function independently and carry forward the work of NAMCOL with their newly found skills.

In section six we examine the appropriateness of the approach and staff training and development methods used. We want to know how they have enabled or hindered the capacity building initiative and identify what lessons can be learned.

The annexures at the end of the report include the research instruments, a record of interviews held, a critical review of own materials conducted with the PMD team on 20 August 1999 and the evaluator’s review of NAMCOL course materials.

***Back to Contents***

At a first level evaluation the main questions are:

vWhat was the Project supposed to achieve? Why?

vWhat progress was made in achieving the main purpose and outputs?

vHow did the Project cope with the constraints and threats to implementation?

vWas a ‘critical mass’ of people trained in key areas?

vAre the participants satisfied with the programme? If not, why not?

We examine these issues in the following parts of this section:

vPurpose of the Project

vProgress of implementation

vConstraints and opportunities

vFacts and figures

vLevel of satisfaction

vConclusion.

2.1 Purpose of the Project

There were two versions of the Project Logical Framework document and the purpose was expressed differently in each one.

The Project Memorandum dated December 1995 states that:

The overall aim of this Project is to build up the capacity of NAMCOL staff to train its own personnel on an ongoing basis and to produce quality distance education materials to meet the needs of specific target groups. In so doing, the Project will assist NAMCOL in the development of sustainable programmes of continuing education that will not be reliant upon donor funding or expatriate expertise.

The purpose may have been stated in different ways but in essence the thrust of the Project was the same. From the outset the intention was a capacity building initiative to empower NAMCOL staff with the competencies required to deliver open and distance education programmes of an acceptable standard.

The NAMCOL-Bath Partnership Project was conceived of as part of ODA’s Development Policy and Programme to provide assistance to the education in Namibia.

A full programme of assistance to the education sector was agreed with the Minister of Education in early 1992. This included support for the reform of the examinations system, English language teacher development, primary education literacy development, and continuing education policy development strategic planning. An Education Sector Review in August 1994 confirmed the thrust of the original strategy and identified support for NAMCOL, as a worthwhile initiative to increase the accessibility and cost effectiveness of the Namibian education system.

From the beginning the Project was seen as supportive of the Government’s own development objectives and priorities, a key one being the production of specially prepared learning materials in established programmes with a view to reducing dependence on foreign correspondence colleges.

2.2 Progress of implementation

The challenge for the NAMCOL-Bath Team was to ensure that inputs and activities would result in the achievement of the pre-determined purpose. The Team monitored this aspect on a quarterly basis and rated the Project’s progress towards achieving it by using a 1-5 scale, 1 being most likely. A scrutiny of the seven quarterly DFID monitoring and progress reports reveals that there were two ratings of grade 3 (likely to be partially achieved), and five ratings of grade 2 (likely to be largely achieved). The first grade 3 rating was made in December 1997 when the College’s move to autonomy impacted severely on the work of the Project. During the period April 1998 to January 1999 the Team was more confident that they would largely fulfil the Project’s purpose. The second grade 3 rating was made in April 1999 when issues of sustainability became a concern. However, according to the minutes of a Project Steering Committee Meeting held in June, Mr Richard Arden suggested regrading the 3 to a 2 because he felt that the Project management had done exceptionally well and he was of the opinion that a grade 2 was more appropriate. Whether this regrading was justified should become clear after examining the progress of the Project.

Tables1 and 2 below summarize the progress made towards achieving the outputs detailed in the Logframe before and after the transition of the College to parastatal status. The time frames for the two phases of the Project are 18 months. The tables make it possible to compare the two time periods both in terms of the level of implementation and the change in focus. Information for the summaries was extracted from the Project’s Quarterly Monitoring reports, the Mid 1997 Report, the Project Annual Review Report, and the Technical Adviser’s quarterly reports.

The inputs and activities were verified as having been completed by the auditor/verifier team (Interim Evaluation Audit/Verification Report, August 1999)

The mid 1997 report indicates that after ten months: About 38% of the inputs were delivered to contract. This despite the huge constraints experienced as a result of a normal settling in period but more especially due to the instability experienced in the organization as a result of its transition to parastatal institution. Given the level of staff attrition at this time the capacity of the organization to achieve its objectives was severely affected. In response to the new emerging needs an amendment was made to the Logframe and a Project extension proposal was submitted and approved in April 1998.

The inputs and activities were verified as having been completed by the auditor/verifier team (Interim Evaluation Audit/Verification Report, August 1999)

When we compare the inputs made and outputs achieved during the two time periods as reflected in the previous tables and we check this with the Project Logical Framework of April 1998 and the Interim Audit/Verification Report of August 1999, we observe:

There is strong evidence that Bath largely delivered the programme according to the intentions expressed in the Project Logframe;

Some variations to the original intentions were made in response to institutional changes at NAMCOL. These were necessary for the Project to make progress towards achieving its purpose;

Delays in implementing some inputs, such as the learner support activities and the evaluation workshop and taught module, were due to the uncertainty and upheaval of the transition to parastatal;

With the exception of a few outstanding inputs that are scheduled for later in the year, the Project has delivered what it promised in the areas of management, programme and materials development and production, learner support, monitoring and evaluation;

More was achieved during the second period when the organizational context was more focused and provided a more conducive and receptive environment than in the prior period of turbulence and uncertainty.

Two important planning issues emerge from the above. One concerns the timing of the Project and the second is linked to the flexibility of Logframe design.

Even though the Project delivered what it promised during the three year period, we pose the question: Should the Project have started only after the transition to parastatal? One could argue that in actual fact the project really started for most staff with the implementation of the amended version of the Project plan in April 1998. Some staff who were with NAMCOL at the start of the Project were of the opinion that the timing of the Project was unfortunate because they felt that they could not participate fully in activities due to preoccupation with their own future within the organization. Others, on the other hand, mentioned the benefits of having the Project’s input and support during the transition phase. The Director, in particular, was very positive about the Project’s support during the restructuring period. It seems that the advantages of having the Project’s input during the restructuring of the organization outweigh the disadvantages of lost opportunities. The amendments made to the Logframe indicate that the NAMCOL – Bath team achieved a common vision of the new organization and its needs. Economically it was probably a costly exercise, adding as it did, another year to the Project. It is futile to speculate about the timing of the Project at this stage. However, a valuable lesson to be learned from this experience is that organizational dynamics exert significant influence on capacity building programmes. This aspect should be considered carefully during the compilation of a Project Logframe. Furthermore the timing of a project in an institution’s life cycle has a bearing on the kind of development that is necessary or that can take place.

Another implication for future planning of projects stems from the modifications made to the Logframe in April 1998. The dilemma is that logframes are usually drawn up based on needs that have been identified in relation to a specific context at a particular time. Professor Tony Dodds provided a useful insight into the reason for the materials development focus in the Project Memorandum dated 1995. When he developed the memorandum he recognized that the APSO Adviser was already working at NAMCOL and that he would mainly be involved in management and learner support activities. Therefore he built the project very much around the development of materials. By the time the NAMCOL- Bath Project was implemented the organization was in a state of flux and embarking on massive restructuring. This demanded a fresh response and it became clear to the NAMCOL-Bath Team that management areas such as change management, strategic planning, marketing and market research would be helpful to the College after the transition. One of the Bath people said that the team would have specified the Project differently if they had known in advance about the impact of transition. In such an instance they would probably have chosen an organizational design option. There can be no doubt that the flexibility shown by DFID, Bath and NAMCOL in repositioning the Project helped to strengthen the likelihood of achieving its purpose. The Project has demonstrated amply the importance of using the Logframe flexibly and of establishing monitoring and evaluation mechanisms to ensure that the plan does not end up being imposed as a straitjacket.

Based on the inputs made and outputs achieved thus far we are satisfied at a first level assessment that the Project largely achieved its purpose and concur with Mr Richard Arden in awarding a Grade 2 for delivery and immediate impact. Nevertheless the success of the Project is dependent on how well staff can continue the work started by the Project. It seems there is some uncertainty at this stage about the dimension of sustainability.

In the April 1999 Progress Report, the TA flagged her concern about the staff’s level of development and ability to continue on their own, in terms of managing course teams, course writing and editing skills, academic learner support, evaluation and monitoring. The Deputy Director of PMD shared this sentiment. He remarked during the interviews conducted in August that he was not convinced that the staff in his division could go it alone now even though they had expressed optimism about their ability to develop courses unaided. This has obviously to be seen in context. Many staff have only been exposed to the Project for 18 months and this shorter period has severely hindered their ability to implement fully what they have learned. It is still too early to gauge adequately how well staff will be able to cope. This opinion is echoed by the auditor/verifier who remarked in her August 1999 report:

…that the introduction of new strategies and techniques can only be expected to have their effect after the formal lifetime of the Project itself, and it will be for NAMCOL to continue to develop its work in the evaluation of impact on the system.

In subsequent sections we examine the Project’s impact and sustainability in areas such as course materials development, learner support and management. We also analyze methods used by the Project to build in sustainability in section six.

2.3 Constraints and opportunities

The main constrains and threats to implementation and achievement of the Project’s purpose that emerge from the various reports and the interviews with people from Bath and NAMCOL staff have been identified. They are:

The majority of staff identified the transition to autonomy as the single most disruptive event in the life span of the Project. They experienced the preoccupation with organization-related issues and their own roles within the emerging institution as a dislocation and distraction that prevented them from participating fully in the inputs organized. As a result, the Project experienced a hiatus of some six months. This affected the momentum of the Project and necessitated a revival strategy to put it back on track. In the process the time scale of the Project was seriously compromised. When the Project restarted in April 1998, there were significant staff changes. A number of staff trained during the first phase had left NAMCOL, others were appointed to different positions and new staff joined to fill the vacancies created by these moves. The scale of these changes was such, a staff member commented, that the Project was virtually starting off anew.

Prior conceptions of the infrastructure at the College and its state of readiness for embarking on the planned programme proved to be inaccurate. A visit by members of the Bath Team and the newly appointed Technical Adviser in September 1996 revealed a rather less cohesive organization structure than envisaged. The fact that the regional learning centres were not under the direct control of NAMCOL and that the HOC’s reported directly to the Ministry was very problematic for achieving output 5: Improved and expanded Resource Centres to support continuing education learners. It also affected Output 2: A cadre of trained continuing education staff with relevant skills, methods and techniques for the training of trainers, because the HOC’s and tutors were all on one year contracts.

Many staff indicated that with increased workload it was difficult to focus attention on the programme, to follow through after workshops and consolidate what they learned. This applied particularly to participants of the MAEC programme. The reasons cited for increased workload were: the dramatic 100% increase in student numbers in 1996 and continuing annual growth; the fact that NAMCOL did not have a full complement of staff; the internal reorganization in respect of roles and responsibilities; inadequate equipment and resources; the move to a new building; getting used to a new job.

The learning approaches and methods used in the programme were quite new to many colleagues and initially they struggled to get used to them because, as the Director put it,

They come from a spoon feeding school culture that encouraged a dependent attitude to learning and people did not necessarily learn to think for themselves.

Most staff who enrolled for the MAEC programme embarked on a new experience as distance learners and found it difficult to balance the demands of work, home and study.

The weight of development work happening in the organization results in professional staff becoming ‘workshopped out’ according to the TA in her report dated August 1998. The menu of workshops delivered in quick succession includes not only the Project’s workshops but also internal and external training events. In section six we investigate the ‘weight’ of training and development in order to establish whether a critical point of ‘too much too quickly’ was reached at any time when people were just too tired to extract optimum benefit.

The full implementation and effective use of the cascade model of training at head office and in the regions proved to be problematic and seriously hampered the transfer of skills especially in the regions. This is examined in sections four and six.

From the outset the Bath Team put in place monitoring and internal evaluation mechanisms to ensure that the Project would fulfil the purpose outlined in the Logframe. The Bath Team kept abreast of what was happening at NAMCOL so that they would be in a position to respond appropriately. For example the TA kept her finger on the pulse of emerging issues recording them in her reports to key players. Her regular contact with members of the Bath Team in England by e-mail, telephone and fax kept them in touch with day to day happenings. Members of the Bath Team paid frequent visits to NAMCOL. Such visits provided them with opportunities to become familiar with the situation on the ground. Apart from internal communication channels to keep in touch with the reality, the Project also conducted two separate Learning Needs Analyses (LNAs) to ensure that people and the organization would receive the training they needed at two main periods of the Project. One LNA was conducted when the Project started, the second after the transition to autonomy. As a result the training events were modified to suit the needs of the participants and the organization more closely. The Director commented favourably on Bath’s flexibility in responding to the needs of staff and the organization, Those people were very flexible because it was important to them that we get what we need.

The hiatus in the first phase of the Project brought about by the upheaval of transition was in the words of a Bath Team member an opportunity to sit back and reflect on what we were about, and to refocus and devise a workable strategy. The modified Logframe was part of that strategy. Other examples that show flexibility and responsiveness to changing needs are evident in reports and plans:

the workshop content was refined as a result of the learning needs analyses done;

the workshop-consultancy idea was derived from determining what works best in practice, and based on the TA’s feedback after the first couple of workshops, more time was allocated to consultancy and less to the workshops as it appeared more useful in the context;

the senior management team visited the UK to work alongside Bath colleagues on selected organizational development projects and to observe a selection of best practice at other UK institutions;

due to the large number of resource centres that NAMCOL uses but which are not under its control, it was decided to use the funds allocated to equipping resource centres more optimally by fully equipping the resource centre at Yetu Yama in Katutura;

regional consultants from Southern African distance education organizations/institutions were invited to facilitate some workshops and consultancies in learner support and course design at NAMCOL’s request.

The Project has shown how good monitoring and evaluation of critical aspects of a project, ongoing internal communication with key people, flexibility and prompt response to emerging issues and needs can lead to more relevant and satisfactory training and development for staff and the organization.

2.4 Facts and figures

If the Project is to achieve its original purpose of building significant capacity in critical areas, then it becomes vital to establish how many of the original staff who were trained are still with the Project now. A high level of staff turnover could hinder the achievement of a critical mass of trained personnel who would be able to continue with the work of NAMCOL. Critical mass can be illustrated by applying it to the PMD.

Course Design and Development Workshops were run in June 1997 by Sally Jefferies of the Bath DCDE. The first, in June, was for staff with prior experience. Of the 7 staff who participated, only 2 are still with NAMCOL. They are the current DD PMD and the Regional Manager of the North East. A second workshop was run in July 1997. 8 of the 17 participants subsequently left and 1 has died (July 1998). Of the current PMD staff, three (Cornelius Hess, Sonja van Rooy, Daniel Dausab) and the current DD PMD participated.

Clearly that was insufficient to provide critical mass for the PMD to become an effectively functioning division. Since NAMCOL became a parastatal, the situation has changed dramatically.

The table below shows an increase in workshop activity focused on developing critical skills for course writers and programme developers.

Over the past 18 months the current team has received enough exposure in areas that are critical to the functioning of the division. The Materials Production Manager has also benefited from a regional placement at Technikon SA to gain more experience of the materials development and production process. Although many of the current staff have thus only had 18 months of development, they have participated in enough training to make the assumption that they have reached an adequate level of knowledge, skill, expertise and confidence to provide a foundation for further development.

A key aspect of this development was their development of a real team spirit, under the guidance of the Deputy Director of PMD and supported by the TA. This included a reciprocal expectation that they would edit each other’s work critically. Reaching that point was a significant breakthrough because previously their editing work had not been of an acceptable standard mainly because they lacked the skill and experience. Critical mass in this situation refers to retention of sufficient members of this team to sustain this effort and train new staff coming into the department in the disciplines and practices. If too many leave, or if too many new staff members are introduced too rapidly, this effective group learning could be undermined.

PMD has been used as an example to illustrate why the departure of a number of staff made the extension of the contract critical, what the basis is for current confidence in their capacity to cope on their own, and the fragility of the current situation if the team is depleted or too many new, untrained people are introduced too soon. We can now speak of critical mass having been achieved in the PMD. Although they have come a long way, they are still relatively inexperienced and will need to continue their development, building on the foundation that has been established, especially if NAMCOL has plans to expand its materials and introduce innovations of which they have no direct experience.

2.5 Level of satisfaction

The Director responded with a definite yes when asked whether she thought NAMCOL is now more ready to deliver sound and effective open and distance learning programmes than it was three years ago. When she looks at her colleagues she can say that they have grown in confidence and in their ability to be more critical. The general impression of the Project is very favourable and most staff are satisfied that the Project has largely met their needs and expectations and has succeeded in strengthening the capacity of NAMCOL to carry out its work. This is borne out by the internal evaluation report dated May 1999 and the questionnaire completed by staff as part of the external evaluation

The following is a selection of critical comments made by staff. Comments have been drawn from the questionnaire, interviews and the internal evaluation.

vThe Project succeeded in its purpose to lay the foundation for the strengthening of NAMCOL’s capacity to accomplish its mission;

vIn a way NAMCOL has succeeded, but maybe not 100% because NAMCOL learners still fail examinations, although there is an increase in pass rate;

vNAMCOL staff are equipped with the tools to write materials and carry out learner support services in a reflective way;

vCertain skills, e.g. writing and editing, have been improved to a certain extent but there is room for improvement;

vPersonally I gained more knowledge on learner support, especially when I was coordinating the learner support guide;

vIt did not address the real issue of empowerment. I think it is probably only with regard to the structure that in most cases we did not move over to real empowerment;

vThe Project was implemented correctly but the timing was not right;

v.NAMCOL should now be given the opportunity to ‘think’ and work on its own.

A range of opinions on the Project effectiveness in building capacity is to be expected. Individuals’ experiences vary and are coloured by their unique perceptions with some being more critically aware and sensitive than others. The comment about empowerment must be seen in context. It arose out of a concern that the cross-functional team appointed to shadow the work of the TA was not as successful in developing staff training and development coordination skills as hoped. At a first level assessment we are aware that staff have different perceptions of the level of capacity building achieved, and that most staff were satisfied that a good foundation had been laid.

The following selected comments from staff reflect a fair cross-section of how they perceive the impact of the Project on the institution.

vThe level of support we are giving to our learners now differs considerably from the level of support some 2-3 years ago, e.g. quality of tutorial letters, quality of DE material, efficiency of dispatch, turnaround time of assignments, improved learner support guide, etc;

vWe have developed a more systematic approach to course development, e.g. The House Style Manual;

vProgress has been made on institutional development;

vThe management tour to Bath improved effectiveness through observation and the subsequent restructuring work;

vStaff ability in planning and organizing has increased through staff development activities.

Many staff have indicated that their confidence level has improved as a result of the training and development activities organized by the Project. The cumulative effect seems to be a much higher level of motivation, commitment and professionalism in some individuals than before. It is difficult to ascribe this only to the impact of the Project because there have been other positive influences as well. However, it is probably safe to conclude that the Project has contributed largely to a more professional way of doing things at NAMCOL.

2.6 Conclusion

The purpose of level one assessment is to establish whether the Bath Team has succeeded in delivering what it promised and the level of satisfaction with the achievement of the Project’s purpose.

The NAMCOL-Bath team must be commended for achieving a high level of inputs during a very turbulent time. They managed to regroup after the turmoil brought about by the transition and came up with an amended plan that was tailored more to the actual needs than the original. They turned a threat into an opportunity and re-launched the Project with a new focus. This is due in no small measure to their professionalism, their monitoring and evaluation mechanisms, and their determination to make the Project work for NAMCOL. Despite the resignation of a number of staff who benefited from the Project before transition, the NAMCOL-Bath Team was successful in achieving a critical mass of trained people in PMD who can take forward the current materials development work with ongoing support and development.

Most staff are satisfied that there is now a better equipped group of people to meet the challenges facing NAMCOL than there was three years ago. Some are not convinced that the Project was able to achieve full empowerment in all areas but most feel confident enough to build on the foundation laid to try their wings.

The purpose of this section is to examine the impact of the Bath NAMCOL Partnership Project on course materials development and production in terms of the following verifiable indicators in the revised Project Logical Framework (April 1998):

vIncreased capacity of professional staff

vImproved quality of DE materials

vProduction schedules operational

vIncreased achievement of course development deadlines

vBudget analysis for course costing operational.

As has been mentioned elsewhere, it cannot be said with absolute certainty whether it is the interventions of the Bath Project that have resulted in the impact or whether the impact is attributable to other conditions, forces and interventions at NAMCOL. However, since the focus of the Project was primarily on course materials production and development, and since none of the other consultancies available to NAMCOL at that time had that focus, the impact can with more certainty be attributed to Bath.

This section will seek to establish impact in the following way. The situation prior to the project will be analysed in order to give a sense of the baseline – what the Project started with in terms of staff capacity and the state of materials development, production and distribution. Secondly, the inputs provided by the Project will be outlined, in order to see if the relevant areas were addressed. On the basis of this, the impact of the Project will be established – on staff capacity, course production and distribution, processes of course production and quality of courses.

The first point to be made about the staff is that most of them are recent appointments in a newly created unit within NAMCOL. When the Project started, the staff had very little experience in distance education. They came mostly from a school teaching background (except for Jerry Beukes – who had materials development experience at the University of Namibia). The only way staff found out what they should be doing was to look at what others were doing and do it themselves. As the Director commented:

There was no formal training be it short-term or long-term in the field of open and distance learning that you could fall back on

As has been mentioned elsewhere, for nearly two years of the Project there was instability in the organization, which affected materials development probably most severely, because staff tend to ignore that if other more pressing duties intrude.

The following table presents the current position and previous position of the various programme and materials development team members within NAMCOL.

A second point to note is that several staff left when NAMCOL became a parastatal or thereafter. Kavena Shimhopileni, Deputy Director: PMD, left in early 1999, and Vaughen Steenkamp, a programme manager who had a hand in the Physical Science course, passed away in July 1998.

Another important fact was that the decision to form a Programme and Materials Development Team was taken by management only at the end of 1998. During 1998, most of the team were involved in programme coordination as well as development and tended to neglect the development aspect of their jobs because of the pressure of delivery issues, such as getting the continuous assessment results in on time.

According to Ed du Vivier, when NAMCOL started in 1995,

NAMCOL was offering a Grade 10 and a standard 10 course. Some of the standard 10 course was produced pre-independence by the Department of National Education. They were type scripts with pasted illustrations. If found, they could be given to someone to reproduce. They were not on computers and there were many problems because of inability to find a number of originals so copies were made from the best copy. Results were sometimes illegible.

They offered quite a few [courses] because they had to do a language for some subjects offered in English or Afrikaans. In addition they bought some of the subjects from Turret of SACHED and they had an arrangement with them which I never fully understood. Ordering was sporadic, when they had money, and no one quite knew who had a full set of materials and what was available. There were boxes of materials that were unused and other stuff they couldn’t get enough of. The Grade 10 materials were still being finalised that year so there were a few subjects still coming out, again in typescript with pasted illustrations, although the material was on computer somewhere.

Learners’ views on materials (collected informally as preparation for the Janet Jenkins Adult Learning workshop in March 1997) were mixed. Some commented that the language was easy to understand and that the self-mark activities were helpful, but the more negative comments were that there were many typographical errors, they were not learner-friendly, and they often arrived late.

When the Project started towards the end of 1996, there was very little actually being done in course materials development, even though it was recognized that the materials were not adequate. The only completely new course being prepared was Physical Science for Grade 12 and this had many difficulties.

The TA comments that shortly after the Project started,

The Head of Materials Development .. went on extended maternity leave – Vaughan Steenkamp took over her role. He also was the programme coordinator responsible for the IGCSE Physical Science, so the course suffered. It was supposed to be offered in 1998, but is not quite ready yet.

In conclusion, at the beginning of the Project, there was a need to improve the quality, range and number of courses, as well as their production and distribution.

Although individuals were exposed to earlier workshops in the programme, and Isabella Katjerungu and Sonja van Rooy are likely to complete their MA qualification, and all the other team members participated in more than one MAEC module, the team only experienced training as a team from August 1998 (with the exception of Jerry Beukes, who was only appointed as DD in early 1999).

The relevant workshop material for course material development for the team is therefore to be found in:

Course Design and Delivery 111 (Brian Sayer) – August 1998

Writing and Editing (Sayer) – November 1998

Delivering Course Material by Radio (Campbell) – January 1998

Instructional Design (Kamau) – May 1999.

However, there has, throughout the Project, been a healthy linking between materials development and learner support, and materials development and management, and many of the PMD staff mention in the comments on the workshops what they learned from:

Learner Support – July 1998 – Veronica McKay

Change Management and Team Building – May 1998 - Paul Walker

Learner Support – November 1998 – Brian Sayer

In addition, there is the assistance that the TA gave to the PMD team. In the beginning of the project, this took the form of group training sessions (each usually half a day) – on issues such as planning of course development and running a course team. But in the final year of the Project the assistance was more individualized because the team were doing a great deal more writing themselves, and hence needed one to one assistance. As the TA commented:

In terms of materials development, if we had just had the workshops they would be miles behind where they are now. My being there has been the follow-up. It’s been needs driven by each individual and all that is based on what they know, on what they want to do, on what they’re tasked with. That whole continuum of different variables. I have six or seven of them and they’re all different. They’re all at different stages. At different levels and all approach the same things in different ways. But they’re finding their own way and that’s what I think is most important. They’re not operating the Bath system of materials development. They’re developing the NAMCOL system of materials development.

The accumulated learning from the various inputs of the project contextualised by the team as the ‘NAMCOL system of materials development’ is perhaps best summarized in the House Style Manual, which contains units on Planning (including course specifications, cost analysis and production schedules), on Development (including working in course teams), on Writing, (including writing objectives, ITQs and SMAs), and on Production (including templates, signing off procedures), as well as Post Production, Audio and Radio.

In this section the views of the director, the TA, and PMD team themselves will be reflected as well as the impressions of the evaluator from discussions with the team. These will be used to draw conclusions about the extent to which staff capacity has increased.

According to the Director,

If she looks at her colleagues mostly the PMD staff, she can say that they are more confident. They won’t just take something and do it but they will now ask questions.

The TA indicated that she was happy with the development of individual people:

The fact that they have taken what we’ve taught them and be able to apply it in a real situation and come up with something that they’re happy with and something that I’m happy with. …

I think the cumulative effect has been greatest upon the programme and materials development people. They were participants in the MAEC, and in a sense they are ones most concerned with education.

In more detail, the TA commented on the different groups within the team.

The TA is happy with the progress of the five programme developers. The work of all of them has improved a great deal. Confidence has increased, even of the two who have had least experience. They could be given any course to write or rewrite and they could do it in pretty much the right way and come up with something good.

However,

They are still not ready to run on their own, and I’m glad we’ve got these additional days, and I would hope that in any time I have with them, that they will be asked to do another course.

The TA also commented that English language skills and editing skills are still a difficulty, but even these have improved. With regard to the programme managers, the TA is of the opinion that development is still necessary in communication and ways of dealing with course writers.

Generally, her assessment is that, with a little more support, they would be equipped to manage the writing or revision of secondary school courses. However, they are not sufficiently skilled to do curriculum development for a different sort of course. She comments:

We’ve done the ‘this is how we get a course out’ bit. They can all get a course out. You need another two reruns to get into that whole content and sequencing. They have objectives ad infinitum in various different workshops and modules. … My philosophy is ‘Start simple, do what you do well’. If you look at our learner profile, what is actually needed? The current curriculum is about mastering the facts. But for vocational courses will need more than this. Hope to get involved in at least one of those with them. They need to be applying for another technical assistance programme in technical and vocational course writing - developing the capacity of the PMD team to do more than the alternative secondary school programme; professional as well as vocational and technical. They could do it, but they would struggle.

The comments of the PMD team on their increased capacity echo some of these points:

The workshops on course development by Brian Sayer gave me a very clear idea about the steps of course development.

Can now give direction to colleagues working on radio programmes

Management systems extremely important – from then on we started thinking of areas of work as processes.

I can now do course specification.

[I have] implemented many of the things I learnt about the layout of materials, such as using white space, organizing the headings, giving space between the lines for more readability … helped me design the layout for the Geography Guide and the Learner Support Guide for HOC’s and tutors.

We have the confidence now to be critical or our own work – let’s revise it, we can do it better.

However, although the team recognized the importance of developing new courses for the sustainability of NAMCOL, the relevant programme manager said that he had no clear idea of how to start. This indicates that the assessment of the TA, that the team is not ready to do curriculum development, could be accurate.

In the initial interview the team gave the following description of their understanding of instructional design:

It allows the materials to perform all functions that a teacher performs in the classroom. We want to see in each unit objectives, self-marked activities, in-text questions, key words to be highlighted and explained, style of writing to be personal and conversational, summaries at the end of the unit linked with the next unit, and engage the learner with digestible chunks. Instructional design is part of the total package - makes the materials user friendly. Learner should be motivated to go through the materials and use it. Layout - white space, font size, pictures, bold/emphasis, icons, etc. - as aspect of instructional design as well.

This indicates that the team have a sound grasp of instructional design, but it is a particular view of instructional design: it foregrounds features of design, rather than learning processes assisted by those features. Again, this echoes the TA’s assessment of the ability of the team that they can ‘get a course out’, but have not been required to think very deeply about learning processes, because the externally set curriculum concentrates on mastering the facts.

When the evaluators engaged the team in critical comment on their own work, the discussion illustrated that the team is indeed able to be critical of its own work. In a very short time, they were able efficiently to pull out several necessary criteria, and make several accurate and pertinent comments on the material.

As can be seen in detail in Annex Fifteen, the comments of the team on their own materials focused on accessibility in terms of how the learners find their way around the materials. For example, that diagrams and columns are providers for the learners to answer, the absence of a glossary in one of the guides, the difficulty of tracing feedback in some of the materials.

The exception was the pair that commented on the Physical Science module. Even though they were not as good as the other pairs in terms of drawing criteria, they made interesting comments on the actual material. It was clear that they were tackling the material as people currently engaged in materials development rather than as people who know about materials development. Their first interesting comment was on the objectives in the Physical Science module – they actually tracked that some of the stated objectives were not realized in identifiable pieces of content in the material. The discussion that ensued from this comment touched one of the tricky issues in materials development – at what level of specificity the objectives should be written. Their second comment was also a substantive one – how time allocations for particular parts of modules should be indicated. The discussion that ensued from this point caught the main points of the debate neatly – importance of time allocation as a motivation for learners, the complexity of estimating time, the importance of writers breaking material up into chunks that are clearly manageable for learners.

In conclusion, it is clear that the current team know about materials development, have engaged with what makes materials accessible to learners, and can apply this knowledge to their own materials. Although the team’s confidence in their ability to develop alternative secondary school courses is probably well-founded, there is a need for further capacity building in the curriculum development skills necessary for the expansion into technical, vocational and professional courses.

The Project assisted with the production of materials in a range of ways, as will be illustrated in the table in this section: training of writers, in editing layout and design, assistance with course specifications, editing and layout. The assistance with distribution of materials was indirect - mainly through the workshops on management systems and learner support. The improvements in materials distribution can therefore not be attributed with the same degree of certainty to the project.

There has been considerable improvement in the distribution of materials: According to Tony Dodds:

They’ve improved no end in getting materials out to students. Two years ago materials didn’t reach the students. For a Namibian institution for whom 20 000 students is an incredible number, credit must be given for an immense improvement.

A more detailed view on improvement in dispatch is that of the Materials Production and Dispatch Manager. As a result of his own initiatives as well as those of the then Deputy Director Marketing and Support Services, he has improved the distribution of materials quite a lot even if there are still a few loopholes. He has introduced a new stock count system that allows him to have more control over the movement of stock. He is now able to locate where excess stock is lying, whether at Head Office or in the regions, and he has put in place a procedure that surplus stock must be returned to Head Office. He is busy installing a computerized stock control system that is linked to finance.

The Senior Management team provided some numbers to back this up:

In 1996 only about 50% of learners received their materials and we were still printing materials after the period of enrolment so that learners only received them in June. There was a slight improvement in 1997, and in 1998 we were able to provide materials to about 80% of learners. The Materials Production Manager is now in a position to make a more accurate projection of the materials needed for next year. We are confident that by November we’ll be able to supply the regions with the materials so that the learners will get them when they enrol at the beginning of 2000.

It was not within the scope of this evaluation to verify these opinions about improved delivery by tracking actual delivery and learner response. It is suggested that a monitoring mechanism be set up by the management team to check whether the distribution targets for 2000 are met.

In terms of materials production, the table on the next page, drawn from the quarterly reports of the TA gives an indication of the progress, as well as the kind of courses developed – new, wrap around, or revision. All the courses produced were for the alternative secondary school curriculum. (NAMCOL’s Certificate of Education for Development offered in conjunction with UNISA’s ABET Institute is the only distance course offered by NAMCOL which is not for secondary school, and it was outside the scope of the Bath project because additional development work was funded by another funding agency).

In addition to these programmes, the production of other materials was facilitated by the TA during the period:

NAMCOL Good Study Guide - prepared by TA in 1997 with two colleagues – compilation of study skills information being used in various forms in NAMCOL.

Learners’ Handbook (rule book) - TA assisted with editing, layout and DTP in 1997.

Learner Support Guide (tutor and HOC self-instructional training manual) was prepared between November 1998 and February 1999. Raw material provided by participants in a Learner Support workshop was edited and laid out by TA and the programme developers.

House Style Manual (the ‘way we do programme development at NAMCOL’) – the Deputy Director, PMD prioritized this in the first half of 1999. A contents list was drawn up, and individual sections are being written by PMD team members.

In terms of the wrap around modules, there appears to have been good progress. The Accounting production schedule indicates that the writing of the first module will be complete on 24 March 1999 and that all modules will be complete and checked by 24 May. This target date was not reached for the reason suggested in Table 6 - unexpected work on the Learner Support Guide. However, as at the time of interview in mid August, it was clear that the course would be completed in time for delivery in 2000. Also of interest is the Business Management course, which has gone to the extent of being piloted, and has incorporated audio material in a limited way.

In terms of the new self-contained courses, the picture is less clear, probably because development was started prior to the 1998 change. The Physical Science course started in 1996, and is nearing completion this year – which means that it has taken four years to complete. This is a long period – particularly since the curriculum was already determined. The two History courses also seem to be taking a long time to materialize – work was started on them in 1997, and in May 1999, they were only at the first draft stage. Mitigating factors include the staff changes and organizational stress during the period of transition to autonomy. Both the History and Physical Science courses suffered from loss of writers as well as lack of continuity in the management of the team. However, the TA also reflects critically on her approach, particularly with regard to the Physical Science course:

The one thing I could have done earlier on was to be more pushy about the Physical Science course. … [But my] approach has always been if people come to me for help I will offer it, but I will not go to them and say ‘What can I do to help you?’ With hindsight, perhaps I could have done more. But I was finding my feet, and I had to be very careful at the beginning not to tread on people’s toes, not to take over. .. I still today think that if they don’t make these things happen, I shouldn’t make it happen for them.

The Director, who was critical of the TA’s lack of pushiness with regard to the Physical Science course, nevertheless endorses the TA’s approach with this comment:

The Physical Science study guide (for Grade 12), which will be ready for the learners in 2000, is an example of the staff’s capacity to develop materials on their own. This study guide is their own product from start to finish. NAMCOL staff did it without support from external editors. The way the staff approached the development of the course also shows how they have grown in confidence. They are now much more critical of their own and each other’s writing, and can identify what it is that they need to improve on. They have learned to review and critique each other’s work and have learned the importance of critical analysis and feedback in the materials development process. It probably took them so long to develop the Physical Science course because they reviewed it and were not satisfied that is was ready.

In conclusion, if materials production at NAMCOL prior to the Project is compared to the current situation, the difference in quantity and range is notable. Despite the difficulties of the organizational restructuring, during the period of the project NAMCOL produced four wrap around courses, and will have a completely new course as well as another wrap around ready for 2000. In addition, four in house course development /learner support documents have been produced, and two further new courses should be completed during 2000. Although the new courses appear to be taking a long time, this allows for the slow process of professional development. Course production can easily be fast tracked by external input, but at the risk of taking ownership away from the people whose ultimate responsibility it is.

The second level of impact on work is the development of materials according to effective processes that build the long term capacity of the organization to develop quality courses.

Aside from two early courses (Life Science and Natural Economy) which made use of a single external author, all courses listed in the table have been developed in ways that build the capacity of the PMD team and employ a team approach. Of particular interest is the fact that at least four programme coordinators (now programme developers) have been involved in actually writing materials, and that many are involved in the internal editing of each other’s work. Also, according to the TA’s reports, the team were also involved in the actual training of external course writers. The TA supported them, but PMD team members actually did the training.

An essential element of sustainable materials development is a proper costing and course specification process. The House Style Manual contains several pages of guidance on how to do this, and the table of course production indicates that course specifications were prepared for each of the new courses. However, the TA is not sure that ability of the staff to do course costing and specification is sufficiently well established. The team acknowledged the valuable role that Ed du Vivier had played in developing cost analysis procedures which were later incorporated in a simplified and modified form in the House Style Manual. However, one of the team members indicated that even after the workshop to discuss Ed du Vivier’s cost analysis process, he ‘needed the help of the TA in implementing it’. The evaluator would need to examine the cost analyses done by various team members and interview them further about their work in order to determine how much help the various team members still need with doing cost analyses.

The fact that the House Style Manual is being developed by the team themselves and driven by the DD PMD will go a long way to ensuring that staff discuss materials development processes, review them, and induct new staff in how to do the work. The TA comments:

The whole development process of the House Style Manual came from doing process review. From getting started on producing materials and .. [analysing] Where do we start and where do we finish? What’s the next stage? Where are the loops? …

Because of the House Style Manual they have confidence, and if we got new people in, they would be able to start from somewhere.

The PMD team commented that the development of the House Style Manual is an important team building exercise, as well as a forum in which policy decisions can be made about how programme development should take place within NAMCOL.

What is noticeable is the extent to which the House Style Manual contains material directly attributable to the workshops on course design and development conducted as part of the Bath project – particularly the Brian Sayer and Judith Kamau workshops.

However, the PMD staff indicated that their difficulties with writing the House Style Manual were Finding enough right information about what to put into the manual.

This could indicate that the team themselves have not seen sufficiently clearly the link between the House Style Manual and the material they received in workshops.

In conclusion, even if the team is not fully aware of the process, and even if some of the material has not been fully internalized, writing the Manual is a way of taking what has been learned and transforming it into a usable form both to inform newcomers and to enable the continued review of their own processes. A way of helping the team further would be to analyze with them the contents of their existing Manual, compare it with similar documents from other institutions, and together work on areas which need further development. As yet unexplored is how the Manual will actually be used – for process review, for induction of new staff, for reference by existing staff.

The third level of impact on work is the quality of the material produced. What needs to be considered is the extent to which the products are attributable to the work of the PMD team as opposed merely to the TA. Although this cannot be determined until some time after the departure of the TA, certain comments both of the TA and of the Director indicate that the TA’s approach to assisting has been responsive rather than directive. In fact, she reports that very often the team did not implement what she suggested, or she did not see whether or not they had implemented it because they did not necessarily send the materials back to her for approval.

She is critical both of the fact that the NAMCOL staff did not follow through on the learning from the workshops, and of herself for insufficient follow up:

I distinguish between follow up and follow through. The project people should follow up, but the recipients should follow through. I don’t think there was enough of either for all sorts of valid reasons. Too often there was a feeling of ‘ the workshop’s over’ now we can get on with our work. It has got better, but could have been better sooner.

It is very difficult for me to follow up on someone else’s training because inevitably they don’t do things exactly as I would have done them. There was a lot more follow up on my part with the materials but only on a day to day basis. It was certainly very needs driven and that is how I think it should be. I wished they’d come to me more than they did. I don’t see how they could have done because of the sheer impact on my time but I’ve rarely gone to them and said we must do this or we must do that.

This seems to suggest that whatever was done on the materials was largely done by the staff themselves.

However, the Learner Support Guide appears to be an exception. The TA admitted that she herself wrote far too much of the Guide, even though she based it on the contributions of the head office and regional staff.

As far as the TA’s comments on the quality of the materials is concerned:

Physical Science:

The course as it stands is no way near an acceptable standard. It’s a start. It will be re-edited in 2000. Teaching a science course is incredibly difficult because of the practical element. What I have tried to encourage the writer to do is to describe experiments so that the learner will think about it as if they going to do it.

Business Management and Accounting:

- layout and style are excellent … They have learned about levels of headings, and pictures, etc. They have made the decision about what features they want in all their courses and that all their courses can look different. The Accounting textbook is not good, poorly organised, so the job has been difficult

The comments from the PMD team recorded in a previous section indicate that they share this assessment of the materials. They are critical of the Physical Science and seem generally satisfied with Business Management.

In Annex Sixteen is the evaluator’s review of a sample of three modules - one in existence at the start of NAMCOL, one worked on throughout the Bath project, and one produced in the last year of the project.

The general conclusions to be drawn from the evaluator’s review are as follows.

There has been considerable progress in imagining the difficulties of the learners and responding to them. This can be seen in the improvement in the introductions to the guides and the individual units in the guides, as well as in the provision of feedback. It can also be seen in the much improved ways in which difficult concepts and vocabulary are dealt with. A final point is that there is much more discussion with learners about their experience of learning, and strategies for learning.

The teaching approach is much more interactive, and less encouraging of rote learning. There are many more activities, much more feedback. However, care needs to be taken that interactivity is not interpreted simply as putting in activities and feedback. As is illustrated in the review of the 1994 module, the style of writing can draw the reader into the ideas and engage him – he doesn’t necessarily have to do a specified activity.

There appears to be a shift towards helping the learners develop skills, rather than merely mastering content. This is partly a result of tightening the task words in the objectives, and providing activities that will help learners master the skills needed. However, the science module needs a much clearer approach to dealing with practical work. In addition, the main way in which learners develop skills is through particular types of assignments, and individualized feedback on those assignments. At present, because tutors do not mark the work of the same learners over a period of time, there is no possibility for engagement with the developing competence of those learners.

There has been considerable improvement in layout and accessibility. It appears that NAMCOL is now maintaining a healthy tension between the need for consistency and the need for variety in its layout formulae. However, attractive appearance cannot replace the need for cohesive and coherent text in which the different parts are logically arranged, properly connected, and clearly cross-referenced. In the Business Management guide, coherence and cohesion has been achieved, but in the new Science module, there is still work to be done in this regard.

The impact of the project on the quality of course materials has been significant. It would appear that the improvement is not simply a result of overwriting done by the TA, but is the work of the team themselves, with assistance. It is also important that it is not only the TA who is still critical even of those materials currently being produced – the team themselves have learned to be critical, and this will help them continually improve the quality of the materials.

Probably the most important dimension of impact is the extent to which what has been achieved is sustainable after the end of the project.

The Senior Management Team made an unequivocal assessment of the impact of the project on NAMCOL as a whole, and on materials development in particular:

vAn overall tangible effect of the Project is that the institution is moving towards a more professional staff complement, which has enhanced capacity in critical areas such as course materials development, learner support and management. As a result NAMCOL is able to portray a more professional image.

vThe quality of materials, both in terms of revising and developing new materials, has improved considerably;

vEstablished a system of evaluating existing materials with tools specifically designed for this purpose;

vImproved processes and systems for the development, production and distribution of materials.

In our opinion, this represents a good summary of the impact of the project on programme and materials development. It is a notable achievement considering the relatively recent establishment of the PMD team, their lack of previous experience in materials development, and the considerable instability that characterized the first 18 months of the project.

The question to be asked with regard to sustainability is whether the team will be able after the end of the project to:

vContinue to improve their own programme development processes.

vContinue to improve the quality of the courses they produce.

vImprove the range and number of courses they offer.

The team themselves expressed somewhat contradictory views on this:

vThere is a question about whether we can stand on our own two feet at the end of the project…

vAlison tried to encourage people to do things – Go do it! Believe in yourself – I can’t be here forever. This made her work quite successful. …

vPeople have a dependency and are afraid to make decisions for themselves.

vWe don’t need a TA anymore. Our team is more than able to cope.

The fact that there are contradictory comments indicates that there is still a need for support of the team. The deputy director PMD shares this view. He feels strongly that the present team still needs ongoing support even though they have indicated that they feel confident enough to cope on their own.

With regard to areas of assistance necessary, the TA believes that the following specific assistance is still necessary:

vReinforcement of the very recently attained skills in materials development and production.

vProgramme development skills for NAMCOL accredited technical/vocational courses.

vLanguage editing.

From these comments and the analysis in this section, it seems clear:

vThat the team can continue to improve its own programme development processes – the thinking behind these processes is recorded in the House Style Manual. However, the House Style Manual is not yet complete, and precise ways in which it will be used have not yet been determined.

vThat, because the team have built in a peer editing system, and have developed their own critical commenting skills, they will be able to continue to improve the quality of the course materials they develop.

vThat the team have increased the number of courses produced, and this trend will continue as the team is now stabilizing and their capacity is more firmly built.

vThat the team understand the need to diversify the courses they offer, but as yet do not feel confident in determining what courses to develop or how to develop them.

What could not be established from this evaluation, and what should possibly be investigated by NAMCOL in future are the following:

vThe extent to which the improved materials distribution processes are actually working.

vHow learners engage with the course materials (this is being done to some extent in the piloting of the Business Management course, but it could be done more thoroughly).

vTo what extent learners in the practical subjects actually do the practical experiments/tasks that the materials seem to require.

vTo what extent the course specifications and course production schedules actually guide the development of the courses (this could be established by interviewing programme managers, programme developers and contracted writers on how they work with the written course specification documents).

The purpose of this section is to establish the impact of the project on learner support in terms of the following verifiable indicators from the Revised Project Logical Framework:

vIncreased capacity of professional staff (in the area of learner support)

vImproved quality of student support services.

To work towards this, two of the outputs of the project are relevant:

vAppropriate staff trained in student support services

vResource centre equipped at Yetu Yama.

Also strongly present in the original phrasing of the project outputs is the notion of training the trainers:

In this section, we investigate not only the increased capacity of head office or senior regional staff involved in learner support and therefore the direct beneficiaries of professional development through the project, but also the way in which part time staff (Heads of Centres, tutors, and tutor markers) might have benefited indirectly from the project through ‘cascade’ training or the use of support materials. If the part time staff have not benefited from the project, it is unlikely that the quality of student support services would have improved, as the part-time staff are the interface with the learners.

In this section it is particularly difficult to determine which developments were a result of the Project and which were natural improvements as a result of NAMCOL becoming a parastatal and hence coming under new management. As the learner support team pointed out:

It is difficult to say exactly who has been instrumental in achieving all the changes that have taken place. Quite a lot of change can be ascribed to the movement out of the government ministry.

Also, learner support is about systems and systems involve a range of people across the country who are subject to a range of influences.

This evaluation will assume that if the Project addressed an issue, it is likely that improvements with regard to that issue are at least partly attributable to the work of the Project.

A first step in assessing the impact of the project on learner support is to examine the context in which efforts to build capacity were conducted.

The origins of NAMCOL in a government department have had a considerable continuing effect on learner support. Within the government department there was a divide between the distance education and the face to face (continuing education) provision that NAMCOL offers. Distance education was organized centrally, but continuing education was organized regionally.

This had an effect on the running of the project in a number of ways. One of the project goals was ‘improved and expanded resource centres to support continuing education learners’ with ‘greater utilization of Teaching and Resource Centres by continuing education staff and students’. However, because NAMCOL does not own or control the resource centres in the regions, the project had to focus on the one resource centre that NAMCOL does own – the Yetu Yama centre. Secondly, as will be pointed out below, some of the difficulties in the implementation of improved learner support strategies (for example, the use of the Learner Support Guide) are a result of the continuing effects of this initial division between regional control of continuing education and head office control of distance education.

In the early days of NAMCOL, there were inadequate systems in place for handling assignments, materials distribution, and handling enrolments. The Learner Support Team commented that there was also very little effort in the past to stick to any deadlines about returning of assignments to learners. During the interviews with the regional support staff, the difficulties of enrolment were mentioned several times. One of the tasks of the new management of NAMCOL, assisted by inputs from the Project, was to devise appropriate systems to handle the large numbers of students spread out across country in the various regions.

Another factor to consider is the effect of the need to employ large numbers of part-time staff. The centres in the regions are mainly located in schools, and are managed by part time HOC’s (usually senior teachers or principals), who employ teachers as part time tutors. All in all NAMCOL employs over 900 part time staff in 80 centres. There are problems with recruitment of part time staff (often they are inadequately qualified), as well as with monitoring (attendance, as well as quality of work). In addition the part-time tutors need training to handle NAMCOL learners in a programme where the time for teaching is far less than in the formal schools in which they teach during the day.

With regard to the staff, even though there has been restructuring at the central office, the regional staff have stayed constant for some time.

This is both a good and a bad thing. On the one hand, the regional staff know their jobs well because they have been in them a long time. On the other hand, as the TA says, they

weren’t very well trained. They were just government officials. They were form fillers. We’ve still got pretty much the same team – their hearts were in the right place. They want to do a good learner support job, but nobody has told them how….

Building the capacity of these staff therefore entails developing their understanding of learner support as an educational rather than merely a bureaucratic job.

In conclusion, in order to address the needs of learner support in NAMCOL, the Project would have to:

vhelp the staff devise appropriate systems to handle the needs of the large numbers of students spread out across country in the various regions;

vhelp the staff train and monitor the part-time staff in the employ of NAMCOL;

vdevelop the understanding of the learner support staff that their work is educational rather than being merely an administrative, bureaucratic job.

If the Project addressed these issues in the support it offered, and if there has been improvement in these areas, it would be fair to say that the Project is at least partly responsible for some of the improvements.

The first point to be made is that though the focus of the Project was meant to be on materials development and management, learner support received considerable attention. This happened because the Project was responsive to needs and especially the needs of the regional learner support staff. It is interesting to note this in the TA’s reports:

August 1998: I have spent more time with regional staff during this period and find a new spirit of optimism there. …[staff are] finding it hard to come to terms with their new roles and responsibilities within the organization .

November 1998: I am pleased that during this period the project has been able to focus more closely on the needs of the regional staff and particularly on learner support.

The TA visited the Northern and Southern regions in 19978, the North Eastern region in July 1998 as well as in 1999, and gained a sense of the difficulties in the regions.

Eleven regional staff participated in one or more of the MAEC modules, and four central learner support staff, but only one learner support person participated sufficiently to have really benefited from the programme.

A number of workshops addressed learner support issues. The 1997 workshop on management systems started the thinking about processes and systems, and the one run by Veronica McKay and Elijah Sekgobela assisted the staff to see the need for an understanding of adult learning. Another result of this workshop was that staff saw the need for assistance in training heads of centres and tutors. This led to the Brain Sayer workshop in which the basis for the Learner Support Guide was established. This guide, and to a lesser extent its companion, the Tutor Markers’ Guide, provide stand alone materials which regional and central learner support staff can use as a basis for training part-time staff. Further work on process review and assessment systems as well as the overall model of learner support was carried out in the final learner support workshop in June 1999. Further work on face to face training will be carried out in a Training Skills workshop in October.

The TA comments on how participation in various of the workshops assisted learner support staff to develop the management skills necessary for their work:

The style of our workshops has been very much trying to ask the right questions. … So for us to help them look at some of the qualitative issues has been helpful to them. Issues include: change management and team building, how they worked in teams, and the things that motivated them. Looking at their views on communication, and on delegation, important challenges the organisation was going to face. We did a series of one day workshops for all the different groups – three northern regions together and the southern regions. That’s what started them off, thinking about being a team and what teams should be.

In addition, of the three out of country placements, two were directed at learner support staff – the chief Registry Officer visited UNISA’s ABET Institute, and the Head of the Yetu Yama Centre visited BOCODOL and the Department of Vocational Education and Training in Botswana.

In can be seen from this brief account that the project did attempt to address each of the three needs identified in the contextual analysis – development and review of learner support processes; assistance with training of part-time staff; and assistance to learner support management staff in understanding their jobs better.

Impact will be considered through comparing the views of the various stakeholders – management, part-time staff, and learners. Even though insufficient learners were consulted to form a conclusive picture of the services offered, some of their comments will be used as a reality check for the views of the other stakeholders.

The general comment by the Senior Management Team indicates a high degree of satisfaction with the impact, which needs to be examined further. Their view is that the learner support services are better organized. There are improved processes for enrolment, acquisition of materials, assignment handling; there is an improved Handbook for Learners; a well-equipped resource centre at Yetu Yama has been established, and tutor marking has improved. In addition, the training of part-time staff is of a higher and better standard than before. Tools such as the Learner Support Guide, Tutor Markers’ Manual, Course Writers’ Manual, which were developed with the support of the Technical Advisor, have been useful in the training of part-time staff such as HOC’s, tutors and tutor markers.

The process for materials distribution has already been commented on in the section on course materials development. This section will concentrate on the impact of the Project on the assignment management system. As has been indicated, this was the subject of at least three of the formal inputs provided through the Project.

Assignment management is the responsibility of the registry, the Distance Education Coordinators, and the Senior Tutor Markers. The aim is a minimal turnaround time. Add to this is maximum accuracy in marking, and helpful and timely feedback to learners.

According to one of the Distance Education Coordinators, in the past the main delay has been getting the assignments back from the tutor markers.

The improved return by tutor markers can be attributed to better contact between them and the Distance Education Coordinators as well as an incentive scheme: tutor markers receive 25% more if they return their marked assignments within 7 days. However, a problem with an incentive that rewards quantity is that the quality of marking and feedback suffers.

Even though this incentive could result in tutors rushing through scripts rather than marking carefully, it must be noted that there are several measures in place to ensure quality – for example, the Tutor Markers’ Handbook as well as the requirement of moderation by the Senior Tutor Markers. One of the Distance Education Coordinators also reported an improvement in feedback to learners.

In 1998 tutor markers did not manage to write tutorial feedback letters. Now they meet after each assignment, discuss the problems learners experienced, and compile a tutorial letter for each assignment.

According to the Chief Registry Officer, the system is improving. He cites the following improvements he has made, as a result of his exposure to the UNISA system during his out of country placement in 1998:

Assignment covers are now standard with all assignments. The learners just have to write their names and student numbers on the cover and make sure that the right cover is used for the right assignment.

They are now receiving standard envelopes with their assignments so that they do not have to buy their own .The standard A4 envelopes make it easier for them to send the assignments which can now be sent without having to be folded.

We are now parcelling assignments into bundles whereas before they received them loose.

They are also receiving printouts of learners’ names, which they can tick. This means they do not have to write them manually.

We have introduced a plastic covering for dispatching learning materials. This makes it easier for people in dispatch to check that the right materials are going to the right learners.

He also mentioned the improvements in the method of filing. Whereas information was previously all manually filed, it is now in digital form, and with the new database software that NAMCOL is acquiring, relevant staff (for example, distance education coordinators and programme developers) will be able to view information. This will reduce the number of queries the registry staff will have to respond to.

However, he also notes problems:

vHold-ups beyond the control of the registry office.

vErrors and omissions on assignment covers.

vFluctuating workload for the registry office.

What is notable, though, is that for two of these problems, he can envisage solutions. For example, he suggests that

assignment covers should be part of the assignment and not a loose cover. The grade 10 and 12 assignments should be colour coded. There should also be space for a name and student number on the first and last page of the assignment.

With regard to the fluctuating workload, NAMCOL has adopted a policy of employment of casuals during times of additional pressure.

The staff present a picture of considerable improvement, and their confidence as well as creativity in suggesting solutions to existing problems need to be commended. However, there are a number of signs that there is still room for further improvement.

Although none of the information that follows proves conclusively that the assignment management system is not effective, it serves to indicate that there are problems which should be further investigated.

Firstly, Tony Dodds highlights the problems of huge numbers of assignments being handled centrally. Although for reasons of monitoring and control, assignments need to be handled centrally, the strain of processing the numbers centrally is telling.

Secondly, not one of the 16 learners interviewed in the last week of August in the Northern region had received any assignment back in any subject. Is this simply a problem with those particular students? Is it a problem with the students in the Northern region? Is it a problem only with certain of the centres? Have the assignments been sent back but haven’t actually arrived? These are the kinds of questions that need to be asked.

Thirdly, one group of tutor markers said the assignments that need marking ‘are lying around the office too long’. One person produced an example of an assignment that had a date stamp of the 5 July 1999, but he only received it for marking on the 19 August. It is likely that from the time the learner had sent off the assignment for marking to the time he received the marked assignment some three months would have elapsed. Another group of tutor markers reported:

NAMCOL not sending feedback letters in time – we see this because the same mistakes recur. Often, NAMCOL staff are telling the tutors to hurry up and mark the third assignment when the first and the second are hanging around the office unreturned to the learners.

There could be many reasons for these perceptions. However, even if they are inaccurate, they should be engaged.

Part of the process of assignment management is the setting of assignments and the writing of memos. The senior tutor markers, who have responsibility for this, reported that a particular problem is being solved. Last year, memos and assignments were drawn up, but not sent back to tutors to read, so a whole host of inaccuracies crept in eg no spaces left for learners to write, lots of spelling and grammatical mistakes. This year, there is provision for proofreading, so that these mistakes do not occur again.

Other problems cited by the tutor markers were:

vThe time for marking is often too short – tutors can manage up to 50 scripts in a week, but sometimes some markers have had to mark up to 123 per week.

vThe fact that Distance Education Coordinators are not subject specialists is a problem – too much falls on part time senior tutor markers.

vModeration guidelines not clear. In any case, senior tutor markers should not only moderate, they should have marking meetings to establish standards so that the need for moderation is reduced.

vThere is a problem with not marking the work of the same group of learners- in order to give effective feedback, markers need to develop a relationship with specific learners.

Although the reasons for the difficulties provided by tutor markers sometimes differ from those given by the learner support team (for example, each ‘side’ blames the other for the delay in assignment return), there is often agreement; for example, the Chief Registry Officer mentioned that one of the things he had learned from his visit to UNISA was the importance of tutors consistently marking the same students’ work.

The conclusion to be drawn from this is that although there has been improvement in the systems, there are certain indications of actual or perceived problems from three sources other than the full time staff. These should be followed up.

As has been indicated in the section on context above, an effective learner support system in NAMCOL is very dependent on training and monitoring of the part time staff who deal with the learners.

The area coordinators interviewed all saw monitoring as central to their role, which is a positive sign. Various concrete examples of monitoring were provided:

The area coordinator in the Southern region indicated a recent improvement in the recruitment of tutors. Instead of allowing HOC’s to employ tutors (which often led to the employment of friends, rather than suitably qualified tutors), the area coordinator and centre management committee now approves the CVs of applicants.

One of the area coordinators in the Northern Region described his experience of monitoring HOC and tutor attendance:

In the course of monitoring, I visit a centre and may find that the HOC is not there (sometimes there are not even tutors). I check and see if they mark themselves present on that day, and if they do, will take disciplinary action – usually in the form of a warning letter. Often tutors absent themselves from class on pay day because they have gone to the bank. I deal with this by having a meeting with the HOCs and the tutors, and also by calling the learners and telling them that they are paying money for tutors and HOCs to be there and they must complain if they have problems.

This shows quite considerable understanding of the situation, and an ability to find ways of acting on the information obtained through monitoring.

An area coordinator in the North Eastern Region said:

I have to deal with HOCs very carefully. For example, some HOCs are not careful with recording absence or presence of tutors. I discover this through keeping eyes open – I see tutors walking about town when they are supposed to be in class. I check whether the HOC knows about and has recorded this, and then call in the HOC and talk to him.

This is further evidence of alertness and ability to act on information.

There is an awareness of monitoring attendance, but not very much evidence of monitoring the effectiveness of the teaching provided by tutors, even though there is an official monitoring form provided for this purpose.

The HOCs claim to visit classes, but the following emerged from discussion with tutors. Five tutors in the Ondangwa area reported that HOCs visit but don’t give feedback or provide copies of the evaluation forms. Two of the tutors in the Rundu area said that they had not been visited at all in 1999, and two said they had been observed once, but the HOC had not discussed their lessons or provided a report.

There is quite a marked level of dissatisfaction with the training that NAMCOL staff are able to give to part-time staff.

The TA worked quite consistently with staff and commented that the ability of the staff to plan and implement training has improved. Initially :

They didn’t know how to go about planning for it, they didn’t even know what to think about

But when she started working with a small group and identifying how they should go about planning and delivering the training, it was much better:

I started working with a small team and we identified who was going to do different parts. Once you’d done that, once you start guiding them in the right direction, they pick it up very quickly.

However, the TA says that the staff tend repeat the training given to them, and cannot look at it critically or attempt something different for new purposes, or simply for variety. The tutor markers also raised the problem of the repetitive character of the training they receive.

This problem exists with the centrally trained tutor markers, but it is even more severe in the regions, as the Director elaborates:

… the people in the regions were taken through that document as to how to train the tutors in the region. They went to the training sessions and they handed the files to the tutors and said, ‘You teach yourself’. I only found out about that through my own evaluation when I went to the centres.

The area coordinator for the Southern region says that the training of HOCs tends to be repetitive and not very helpful, particularly for HOCs who stay in their jobs year after year. The area coordinator in Rundu says that the main thing in training is to try to help tutors understand that they are teaching adults, but another area coordinator says that that doesn’t work, because tutors don’t want to move to a facilitation role because it is time consuming and learners are predominantly passive.

The HOC’s say they receive training, and give training to their tutors. However, the tutors say they don’t receive training. Two of the groups of tutors interviewed said they just have an information-sharing meeting, and the third group said they are ‘just given a textbook and a class and told to teach’.

A solution to these problems could be the provision of training materials. This would mean that the tutor markers and regional part-time staff could access the information directly, and not rely on the face-to-face training provided by the HOCs or the central learner support staff. Two main documents have been produced for this purpose, the Learner Support Guide and the Tutor Markers’ Guide.

The Learner Support Team see the production of the Learner Support Guide as a notable contribution of the Project. As one team member reports:

The one thing I have gained from the Project is the Learner Support Guide that we developed last year. We used to develop our own materials for training tutors but then we came up with the idea of putting everything into one guide which can be used in the regions. The idea came from us that we need support in learner support. We then worked through Alison who organized a workshop. After the workshop we came together to write pieces ourselves. The guide is our own product.

Efraim Davids outlines how the Guide is meant to be a self-instructional manual that could eventually lead to a certified NAMCOL course:

Each tutor has the Learner Support Guide, and the training engages them with the Guide during the first term. Each tutor has to go through it individually, do activities, and then the HOC will take each tutor’s file and assess the tutor and recommend certification to Head Office.

However, interviews with the tutors indicated that not all of them had even received the Guide. Those that had received it, had had a meeting about it, but the purpose of the Guide appears not to have been adequately grasped.

The tutors in the Southern Region, for example, said that they thought their views and suggestions were being elicited by means of the evaluation questions in the Learner Support Guide. They thought that the guide was part of a research project to find out what tutors think of the various sections in the guide. They were not sure what they should do about returning the completed evaluation sheets. The HOC’s thought that they would have to collect the guides from the tutors at the end of the year and give them to the NAMCOL staff who would then capture the responses to the various sections in the guide from the tutors. This they felt was part of a review and revision exercise. When asked how they had used the guide, these tutors reported that four of the five had not read the whole guide before they started their tutoring task this year. One person did all the activities but the others only did those they found relevant. No one seemed to be aware that the activities are part of a self-instructional method of learning.

According to the TA, the Tutor Marker’s Guide is a start, but is not adequate as yet. As she says:

The Tutor Markers’ Guide needs developing enormously. It needs to follow the same route as the Learner Support Guide. It needs to be a NAMCOL qualification. It needs to go into higher level things. There should be much more in there about marking assignments. There is a problem with the tutor markers. Need to get them in to build the team – could concentrate on that, rather than spend time on direct repetitive training.

The tutor markers echoed this impression. They complained that the information they receive in the orientation workshop at the beginning of the year is the same as the information in the Tutor Markers Handbook. They are not sure that this handbook is in fact so useful. Most of them read through it at the beginning of the year but do not refer to it again. They say that the main purpose of the Handbook is for new people.

In conclusion, it appears that the effort to remedy the ineffectiveness of cascade training of part-time staff through materials based training is not yet successful. Part of the problem with this could be that materials are centrally developed, and the regional staff do not feel that they own them. Also, a culture needs to be developed of learning from materials. It appears that this has not happened in the regions.

In addition, it appears that the central learner support team do not see it as their responsibility to see that the materials are properly used. In the interview with the Learner Support Team, there was considerable vagueness about how the evaluation questionnaires on the Learner Support Guide were to be collected so that the Guide could be revised; there was also no evidence of a plan for the evaluation of the Tutor Markers’ Guide. Unless the central Learner Support Team take responsibility for driving the use and the evaluation of the Learner Support Guide, a considerable training resource will be wasted.

Even though the systems may not be perfect, and even though the training may not be completely effective, the kinds of activities and input through a sustained professional development effort can have an effect on how the staff understand their work. In the long term, this can have an effect on the ways that work is done.

Central to effective systems is an awareness amongst staff that they need to work as a team. With regard to the assessment system, the Registry, Distance Education Coordinators and the tutor markers, particularly the senior tutor markers, need to work together. As has been mentioned above, there appears to be insufficient communication between the Distance Education Coordinators and the senior tutor markers – according to the Learner Support Team only three meetings a year. The impression of the evaluators is that the senior tutor markers are committed, have considerable expertise in their subjects and in the processes likely to lead to quality marking, and would like to do a good job. They have suggestions for how things can be improved and could be consulted more frequently. The TA remarked that what the senior tutor markers need is not training in how to do the job but opportunities for team building.

The Chief Registry Officer says that he knows he should work closely with the Distance Education Coordinators to ensure that the coordination of the assignment handling process runs smoothly, but that he is unable to do this effectively at the moment ‘because the workload in the registry office is too heavy’. He also says that his monitoring role extends to finding out what the learners’ problems are and streamlining processes within the registry office to make the handling of assignments much more efficient.

The challenge in the regions is to shift from a purely bureaucratic view of the work to one which places the learners’ needs at the centre. Although the area coordinators have an increasingly broad view of their work, this does not appear to have filtered down to the HOCs.

The area coordinators see their role as monitoring, evaluation and training as well as administrative support and provisioning. In addition, they realise the importance of interfacing with and motivating learners and their parents – and responsibility for changing the image of NAMCOL.

The HOC’s tended to emphasize the bureaucratic side of their job - filling in all the forms (claim form, tutor control summaries, tutor attendance, learner attendance), and the difficulties of managing facilities which don’t belong to NAMCOL. Some reported that they were unable to do training because they ‘haven’t got an OHP or photocopier’, or said that tutors ‘don’t have problems because they already teachers’. Several HOCs complained about facilities. It was also interesting when asked to state what they had learned from the monitoring visits of area coordinators, they produced the following list:

vShould have a contents page in our files;

vShould be up to date with recording;

vShould not be behind with monthly reports.

However, when they talked about further training needs they did mention the importance of learning about how to manage people, understanding open learning, and exploring the role of parents.

The system depends heavily and perhaps rather unfairly on the HOC’s. What could be explored is the possibility of using other staff for training and monitoring of the teaching effectiveness of tutors, rather than insisting that HOC’s perform these tasks which they seem reluctant and/or not able to perform.

With regard to the tutors, there is an awareness that NAMCOL learners are ‘different’ and should be taught differently. This ‘difference’ is usually expressed negatively – the learners don’t attend or come late or under the influence of alcohol, they don’t buy books even though special rates are offered, and so on. But some tutors gave reasons for this behaviour; for example, learners don’t attend because they don’t want to be associated with the poor public image of NAMCOL. Certain of the tutors distinguished between two broad categories of learners – school drop outs and adults wanting to reenter the formal system; others said that certain of the sections of the Learner Support Guide had alerted them to the disadvantage of the learners and that they should treat the learners as adults. However, generally there does not appear to be an understanding that the way you teach should be different for the different mode of delivery (reduced contact time, mainly in the evenings). Many tutors referred to how rushed they were to get through the work in the reduced time.

Across the staff, there is an awareness of the importance of learner centredness and teaching adults in a different way from the way that young people in formal schools are taught. This awareness has come through the workshops, as well as through the Learner Support Guide. However, it appears to be a little at the level of the slogan at the moment – many staff talk about it, but perhaps the assumption that learners are adults might be blinding the staff to really analysing how the learning needs of NAMCOL learners might best be met, and how the tutors might best be helped with how to handle these learners. The TA makes a very interesting comment:

There is a difficulty with treating our learners as adults. Predominantly they are school push outs, and I can see that the tutors have a much greater problem with it than I do. Three months ago, they were teaching those same children in that same classroom in that same subject. How on earth can we expect them to change from November in one year to March the next year in the same classroom with the same learners? If a teacher leaves a school and goes to a college you make that transition. But this is not the case with these tutors. So you need a happy balance – much more effective research about our learners. … My gut feeling is that they don’t want to be treated like adults. In certain aspects they do, but in educational terms, I don’t think they do. They don’t want to take responsibility for their own learning. They want to look at the teacher that they looked at last year and be taught. It doesn’t matter how much we tell the tutors that they must use adult learning methods, if the other side of the equation is not playing to the same tune, then it isn’t going to work.

This insight could provide the basis for further investigation into what the most appropriate ways might be to train tutors to deal with the kinds of learners in the NAMCOL programmes. Simply exposing tutors to adult learning theory is not adequate.

In conclusion, there has been an impact on most of the staff in terms of a deeper understanding of their work and their learners, but this still needs to translate into different teaching approaches. There is also an understanding of the need to work together to run an efficient system for the benefit of the learners, but as yet there are some difficulties with communication and co-ordination.

Although originally the project intended to assist resource centres in all the regions, the decision was made to concentrate on the only resource centre that NAMCOL actually owns – Yetu Yama in Katutura. Very tangible results have been achieved at this centre, as stated by Efraim Davids, the permanently appointed Head of Centre:

The resource centre was opened in March 1999 to provide learners with access to textbooks, study guides, exam papers - to improve results further. He started work on collecting donated books from other resource centres and libraries when NAMCOL was still under the ministry. The Bath Project helped by providing donations to purchase new textbooks and stock the resource centre properly.

10 computers were donated by the City Council and he is negotiating with Utech to upgrade the computers. The idea is to run computer courses for a broader market than the traditional NAMCOL market.

The Head of Centre indicated that there has been an impact on student numbers at the Centre, which have risen sharply to 1130 in 1999. In addition, there have been outstanding students results, probably a result also of running mock exams each term and providing rewards to best performers.

However, besides serving the needs of the students doing the alternative secondary school curriculum, the vision is of a full-scale technical and vocational centre. Through his out-of-country placement the Head of Centre became aware of what a techvoc centre is, but also of the fact that without considerable assistance, he does not personally have the skills and knowledge to develop Yetu Yama into such a centre.

With regard to the development of centres in the regions, there is a proposal prepared by Ed du Vivier (25 June 1999) for:

The implementation of a project to establish a number of Open Learning Centres for use by students registered with publicly funded open and distance learning institutions in Nambia, Funding for equipment and limited start-up costs for this project have been allocated by the European Union under its programme of assistance for human resources development of Namibia.

Although there is evidence that the first of the verifiable indicators - increased capacity of professional staff in terms of learner support - has been achieved, it is not has been as easy to determine impact in terms of the second indicator - improved quality of student support services. This is because of the difficulty of attributing impact, as well as the limitations of the evaluation in tracking improvement.

From the analysis in this section, the following general conclusions can be drawn.

The project has had some impact on the development of more effective processes for learner support, particularly assessment management. However, to ensure sustainability, there needs to be much tighter monitoring and coordination of the process to ensure successful delivery to the students in the regions.

The project has not assisted NAMCOL sufficiently in developing effective training of part-time staff. The part-time staff tend to perceive the training as ‘meetings’, or as repetitive and, by implication, boring. The training skills workshop planned for October will help to equip staff better to carry out training, but this would need to be followed up with support if the training is to have a lasting effect.

The training strategy with the greatest potential for continued impact is self-access training material. This has been developed, and, particularly the Learner Support Guide is of extremely high quality. However, unless the use and review of these materials are effectively driven by the central learner support staff, they will not have the desired effect.

Both full-time staff and part-time staff in the regions have a deeper understanding of the work. The project has contributed to an increased focus on understanding the needs of the learners and an increased awareness of the critical importance of monitoring tutor attendance. However, there has to be a greater emphasis on support of the tutors in changing their teaching approach to accommodate the different mode of delivery demanded by NAMCOL courses. Currently HOCs, who are relatively insecure staff on one year contracts, are expected to take considerable responsibility for training and support of tutors. They are not able to perform these functions effectively, and simply requiring them to do this is unlikely to make them able to do so.

With regard to the distance learners, the Distance Education Coordinators need to understand much more clearly how crucial their role is in ensuring the proper functioning of the assignment management process as well as the improved quality of the teaching on assignments. Generally, however, head office staff are aware of the problems that exist, and in some cases can develop solutions.

Our main aim is to determine how the Project has enabled the senior management team to sharpen its strategic focus and prepared the team to carry forward the work of NAMCOL. We look at two fundamental areas: NAMCOL’s strategic planning and review capacity and NAMCOL’s readiness to deliver sound and effective open and distance learning programmes. In each area we explore how the team can sustain the initiative of the Project and build on the foundation established.

This section has the following parts:

vBackground

vCriteria used in assessing the impact of the Project on senior management

vExposure to the Project

vPerceptions of what NAMCOL’s senior management has gained through the Project

vConclusions about sustainability

According to the Director most managers lacked formal training for managing their areas before the Project started. Her own development as a manager had been through her experience in previous positions, e.g. as a school principal. Ed Du Vivier, an Agent Personal Service Overseas (APSO) consultant, who worked in NAMCOL prior to and during the early part of the Project, confirmed this lack of formal management training in the organization. In addition, many staff had come to NAMCOL from different government departments and had no formal training in open learning or distance education.

Assessment of the impact of this Project on senior management is complicated by the fact that it had virtually two starting points. The first, when the Project started, and a virtual restart when NAMCOL became a parastatal. Senior management played a key role in rebuilding momentum after the virtual suspension of the Project from November 1997 to April 1998, and the departure of a significant number of staff involved with the Project during that period. It also resulted in the Project being extended by a further 12 months.

A further factor was the resignation of the Deputy Director PMD, Kavenna Shimhopileni, with effect from the end of February 1999. NAMCOL lost part of the development effort, especially the three week management visit to Bath in October 1998, invested in a member of its senior team. The Deputy Director MSS took over responsibility for PMD and one of the Regional Managers, Harold Murangi, who had been with NAMCOL since 1992, became Deputy Director MSS. The management team has shown resilience in adjusting to the change.

Another factor to be acknowledged is the contribution of the initiative, in the person of Ed du Vivier, which was focused on management and organization structures, and was operating within NAMCOL prior to and during part of the Project. It is important to note that it was complementary to the Project and that Senior Management benefited from both initiatives. It will not be useful to try and untangle the precise impact of one from the other.

A final point is the number of other learning experiences the Director, in particular, was exposed to during the Project including participation in an international Quality Assurance workshop in Zambia, a visit to the United States, and attendance at the DFID Regional Education Workshop in Johannesburg.

Mrs Frances van Wyk was appointed Director in May 1996. The transition process to a parastatal began in October 1997 and the senior management positions in the new structure were filled in 1998.

In the interests of establishing a common understanding, the criteria used in assessing the impact of the Project on senior management are listed below. They reflect the assumptions of the evaluators about what the NAMCOL Senior Management Team needs to increase the probability of sustaining and building on what the Project has achieved.

The criteria are:

vAn understanding of open learning and distance education informing senior NAMCOL management that is giving their leadership direction and providing a clear picture of the type of institution they want to be.

vIndications of more demanding professional standards within NAMCOL.

vNew knowledge and skills acquired by senior management that can strengthen organizational capability, e.g. managing change, managing systems, decision making, delegation.

vConfidence levels, firstly of senior management, and secondly of other staff, in their individual and collective capability effectively to build on the foundation provided by the Project.

vAwareness of a range of development methods and ability to set attainable benchmarks and planning how to use them to sustain the gains already achieved.

vFocus on the purpose of providing continuously improving quality of service to the learners.

5.3 Exposure to the Project

Before we can assess senior management’s ability to carry forward the activities of NAMCOL it is useful to look at some of the tools and techniques used during the Project to build their capacity.

* The DD MSS attended these workshops when Regional Manager South.

Table 7 is not a complete list of training and development activities. It shows some of the formal activities but does not reflect consultancies, on the job training activities, informal contacts of the TA with members of the Senior Management Team. No. 10 just shows who enrolled for MAEC modules, not which modules they participated in. What is clear from the above is that, after the second needs analysis conducted by Bath, there was a shift of emphasis leading to greater inclusion of management areas such as strategic planning, management of change and team building.

By far the most powerful event for the top management team, the Director and her two Deputies, one of whom unfortunately resigned a few months later, was no. 6, the visit to the UK. The visit enabled them to observe best practice in open and distance learning by doing a detailed review of the distance learning operations in the Department of Continuing and Distance Education at Bath. They participated in one workshop and had some ‘protected time’ for reflection and reading. Each worked on a NAMCOL-related project and each person was assigned a Project Adviser to support and guide them in their project work and a Counterpart to provide general support and guidance. They were also encouraged to use Personal Reflective Diaries. During their time in England they visited other distance education institutions such as the National Extension College and the Open University. One of the most important outputs of the visit was the teambuilding that happened. Not only was the individual capacity strengthened but the three members emerged as a united team. In the words of the Director, If the top team is strong you have a strong organization.

5.4 Perceptions of what NAMCOL senior management has gained through the Project

This part is organized around two fundamental areas relevant to assessing the impact of the Project on the senior management team. Compared to three years ago we wanted to establish what improvement has taken place in regard to:

NAMCOL’s strategic planning and review capacity

NAMCOL’s readiness to deliver sound and effective Open and Distance Learning Programmes

 

We know that senior management’s strategic capacity forms part of NAMCOL’s overall readiness to continue with its work. The specific focus we have given it stems from its inclusion in the Project Logframe as a separate output, namely output 5 that states: NAMCOL strategic focus improved.

Information is drawn principally from:

vInterviews with the Director on 19 and 30 August 1999.

vA group focus interview with the Senior Management Team on 25 August 1999.

vThe Deputy Directors were each involved in three or four group interviews.

vA telephone interview with two members of the Bath Management Team on 3 September 1999.

vInterviews with the Technical Adviser on 21, 27 and 31 August 1999

vRelevant documentation.

The verifiable indicator in the Logframe that pertains to strategic planning is described as: Increased achievement of NAMCOL strategic plan. In order to establish what has been achieved we start with what was in place before the Project and then provide a short overview of the activities and inputs organized by the Project in this area. We also take note of contributions made by the APSO consultant.

A strategic development plan for 1996 – 2001 was compiled by Ed Du Vivier and Dalene van der Westhuizen before the Director joined NAMCOL and before it became a parastatal. The change in status and the appointment of the Director demanded a fresh strategic plan. At this time the NAMCOL-Bath Team collaborated to develop an amended version of the Project Logframe that was more suited to the needs brought about by the change. As a result, a number of important management areas were included such as strategic planning, change management, market research.

In April 1998 a strategic planning workshop was run by Paul Wallace and Tim Bilham with all members of the senior team participating and Ed Du Vivier attending at the Director’s request. Participants found the workshop very useful. They used the original plan as a resource and started planning afresh. With follow up support from the Bath Team and Ed Du Vivier the new plan was drawn up and accepted by the Board.

There is nothing to be achieved by comparing the two strategic plans. What we have observed, however, is that the new plan has some features that go a long way to providing the senior management team with a clear vision for the college up to and including 2003. The following elements contribute to making the plan a very practical monitoring tool that encompasses the main areas of the institution.

vThe ten aims are stated simply and unambiguously;

vThe objectives identified to achieve the aims are presented as progressive steps for each annual period;

vIndicators of success are useful benchmarks that enable the team to monitor whether objectives have been achieved;

vA useful reminder to the team is the part that shows what resources they need to achieve their objectives;

vThe landscape layout makes it possible to show the aim, objectives, indicators of success and resources required on one or in some cases two pages. This makes it easy for the team to see the plan at a glance.

vOther useful information that is included is the year themes, enrolment figures (actual and projected), recurrent income and expenditure, development budget and project development budgets.

A measure of the increasing confidence of the management team is their proactive decision to organize a review of the strategic plan from 23-25 August this year, with some assistance at a distance from Andrea Taylor. As the Director said in one of her interviews, one of the tasks they set themselves was to check the viability of aims and objectives set last year, given the reality inside and outside NAMCOL. The review is an important step in keeping the college on track. The team has shown that they are sensitive to the changing conditions and are using the strategic plan to move the college forward in a direction that is in tune with emerging realities.

The outcome of their review was not available to the evaluators. As with several other aspects of the Project’s impact, its value will be revealed through the quality of the review and the subsequent actions senior management takes to implement the revised plan.

During the market research consultancy conducted by Andrea Taylor in 1998 a working team consisting of the APSO consultant and the research and evaluation team drew up a research action plan for the annual statistical digest. This plan is contained as annex four of the market research consultancy report. The development of the NAMCOL Statistical Digest 1998 was completed by the APSO consultant with input from the research and evaluation and the registry teams. It was published in April 1999 and the management team now has access to crucial quantitative information. The following comments made by a member of the management team during the group interview shows that management is using this type of information to establish progress of strategic targets, e.g. printing and dispatch:

We are confident that by November we’ll be able to supply the regions with the materials so that the learners will get them when they enrol at the beginning of 2000. We have made steady progress since 1996 when only only 60% of learners received their materials and 1998 when the percentage increased to 80%.

A suggestion was made in Section 3.4 that the management team set up a monitoring mechanism to check whether the targets for 2000 are actually met.

Our conclusion is that the Senior Management Team’s level of strategic planning and review capacity has increased and this shift can largely be attributed to the inputs and support provided by the Project and the APSO consultant.

The Senior Management Team has taken the strategic plan through one annual cycle ending with the strategic review in August this year. They have shown confidence and determination to take responsibility for driving the Strategic Plan during 1998-1999. The Director is confident that they can now continue to manage the strategic planning process on their own. The needs-focused objectives approach they have adopted will require rigorous follow-up to monitor achievement, to take corrective action and to formulate new objectives to meet changing needs. The onus is on senior management to ensure that different staff in the organization are also involved suitably in the ongoing monitoring of identified areas to bring about continuous improvement in the quality of services to learners.

The team is encouraged to pay attention to the following:

Use the report to identify priority areas that need attention next year;

Members of the Bath Team suggested that the team develop a strategy to get funding for new projects (ie identify the needs, establish the support required, develop a proposal, engage with the support received).

The readiness of an institution to provide quality open and distance learning programmes depends largely on three interrelated functions: the ability to make available to learners sound teaching and learning resources, the ability to provide effective support to learners, the ability to establish and sustain efficient management systems. In this part we offer some overall impressions of NAMCOL’s capacity in these three critical areas. In our conclusions about sustainability in each of the areas we highlight points for the attention of management.

The senior team admitted that understanding of open and distance learning was very limited when they first started. As mentioned earlier, the visit to the UK was a significant watershed in deepening their understanding and giving them a much clearer idea of what open and distance learning means and what organization capacity is required to run an effective organization. Their subsequent participation in a range of workshops and consultancies as is evident from table 7 has enabled the team to increase their understanding of what is entailed in managing an open and distance learning institution.

We have used the term ‘teaching and learning resources’ because it is an inclusive term that can refer to a range of learning resources not necessarily only course-based materials. In resource based learning even tutors are considered to be learning resources. We refer in this part to the course materials development and production work of the PMD division.

The director and other members of the management team have shown a healthy criticism of what has been achieved in materials development and production. Even though they recognize that increased professionalism has taken place they are also aware of gaps in the present PMD Team’s ability to cope with curriculum based technical and vocational courses that have been identified in the strategic plan.

Three important factors have contributed to strengthening PMD:

As outlined in section three the PMD Team has the capacity and confidence to continue to review and develop the alternative secondary school courses and this is a good foundation to build on;

The Materials Production Manager has the necessary expertise to continue to improve on the production of materials;

The Deputy Director has a useful track record for leading the division. He comes from a background of working with materials at pre-NAMCOL initially as a sub-editor. He joined the organization in 1992 and since then has been exposed to a range of relevant management training and development events as is evident from table 7. His stint as DDMSS has enabled him to get a useful overview of the link between materials development and learner support.

We are confident that this division has enough collective capacity to continue its work. The DD PMD is now faced with a number of important decisions about building on what has been achieved and further strengthening the capability of the PMD division in order to:

vproduce new courses for existing and new markets;

vrevise existing courses;

vcreate markets for existing courses;

vmove into new areas of development which may involve additional skills which the present team does not yet have;

vinvestigate the use of new technologies in developing course materials.

For the division to progress in the above areas he has to find ways to:

vabsorb new staff without harming the PMD’s capacity to produce;

vdevise strategies to train part-time course writers;

vstrengthen the Programme Managers to manage the course development process more effectively ;

vstrengthen the language editing capability within the division;

vdevise new training and development strategies for full-time staff..

In the short-term the DD PMD, in consultation with the management team, has to decide how best to use Alison Mead’s additional consultancy time of 35 hours over the next six months. We have indicated in section three that a priority is to complete the House Style Manual and to develop suitable approaches to use the manual to train part-time writers and new full-time staff.

Longer-term staff development strategies have to be developed now that the Project is not providing this type of support anymore. The Director indicated that a new staff training and development plan for the college would have to integrate the work done by the Project Team. The thinking is that the Human Resource Division will take over the responsibility for coordinating all staff training. We have indicated elsewhere in the report (in section six) that the Project Team as a mechanism should be allowed to continue to play a role perhaps as a sub-team of the central staff development committee. It would be a pity if the capacity currently located in the Project Team would be lost through restructuring. The ongoing training and development of the professional staff working in PMD is of vital importance if NAMCOL is to continue improving the quality of its course materials.

Learner Support was a central theme chosen by management as part of its efforts to create awareness in the college about its commitment to improve its services and support to learners. The Director and other members of the Management Team seem satisfied that there have been some important improvements in this area especially over the past year. They mentioned the improved distribution of materials to distance learners, a better handling of assignments, the availability of tools like the Learner Support Guide and The Tutor Marker’s Guide for use in the training part-time staff, the opening of a well equipped resource centre at Yetu Yama in Katutura, the improved understanding of learner support issues by a range of staff at Head Office and in the regions. The Management Team acknowledges the substantial role played by the Project but they also remarked that the college’s independent status since April 1998 made it possible for management take decisions and move things forward more quickly than before.

Through exposure to institutions in the UK during their visit in 1998 the top management team has a much clearer understanding of the standards for learner support that NAMCOL must strive to attain. They have shown a healthy self criticism and the team as a whole is aware of the enormous challenges that face them especially in the regions. They recognize that one of the most daunting tasks is the training of some 900 part-time staff who, in the words of one person, are the backbone of the college. Everyone agrees that up to now the attempts to cascade skills through the Area Coordinators to the HOC’s and ultimately the Tutors has not worked. They are still optimistic, however, that with revised manuals they should be able to improve the training of part-time staff.

We view the improvements indicated by management and staff as important indicators that a process has begun at NAMCOL to improve its services to students. We also recognize the role that the Project has played to enable staff to become more critical and that a greater synergy has been achieved between the registry staff and the professional staff. Although we are not as confident about the level of capacity achieved in learner support as we are about PMD we do recognize that the College is better off now than it was three years ago. We base this on the fact that:

the present DD MSS has been exposed to a range of pertinent management training and development as indicated in table 7. His experience as a Regional Manager provides him with insight into the practical demands of meeting the needs of learners. This experience and management expertise he has acquired are strengths on which he can build to lead the learner support initiatives planned for next year;

the Regional Managers are part of the Senior Management Team .Through exposure to the Project inputs as detailed in table 7 and the ongoing interactions as members of the team they have gained a better overall view of learner support and the implications for the regions;

one Distance Education Coordinator has had experience as a programme developer in the PMD division and has been exposed to all the Learner Support inputs. He is also the MAEC student who has already completed the six modules and who is busy with his dissertation. He can make a sizeable contribution in taking forward the work in learner support;

the Chief Registry Officer has gained confidence and experience through the Project inputs he has participated in. He sees the Registry as an integral part of learner support;

senior management understands the issues and are committed to offering quality learner support;

internal processes such as the assignment handling and distribution of materials, are in place.

The above are strengths on which the institution can build. However, as indicated in section four the capacity in learner support is still fragile and much more development is needed both at Head Office and in the regions.

What complicates the learner support area is that the responsibility for support to learners is shared by the MSS division and the regional managers. Furthermore NAMCOL is reliant on some 900 part-time tutors, tutor markers and HOC’s to offer tuition and support to the learners in the different centres and regions of the country.

We highlight three priority areas for attention by management so that the gains made can be used to improve further.

A more holistic and integrated vision of learner support is necessary to avoid unnecessary fragmentation into learner support for distance learners and that for face-to-face learners.

The present practice of relying on the HOC’s to be responsible for the training of tutors has to be reviewed because it has not worked well.

An innovative training strategy must be devised to train the large numbers of part-time staff who are scattered throughout the country because the way the cascade methods have been used have largely been unsuccessful.

We have some thoughts on the above that may help management.

It is unrealistic to expect the HOC’s who are employed on a yearly contract basis and who are in a part-time position to do more than the administrative work required at centres.

One strategy that we could think consists of longer term and short term components.

In the longer term we believe the Area Coordinators can play a more effective role in training and supporting the HOC’s and tutors in the centres. For them to take on this expanded role it is necessary to review their workload and in some of the regions it may be necessary to employ more people. They also need to be trained well, particularly in the use of new learning and teaching approaches that are appropriate for NAMCOL learners.

In the shorter-term it may be necessary to train a roving team to lay the foundation. The length of time this team stays in a region is determined by the needs in the region but it should be long enough to make some visible improvement. The team can then set up peer tutor teams that can continue to help and support one another when the team leaves.

Once the initial round of training has taken place, the Area Coordinators can continue with their efforts to improve the quality of teaching and learning at the centres.

It may be possible to treat the training of part-time staff as a project for which funding is sought.

We endorse Professor Tony Dodd’s suggestion that management keeps in touch with the thrust to build cooperation between the various distance education services in the regions. NAMCOL can benefit from such a collaborative venture.

One of the first workshops organized for staff in December 1996 dealt with management systems. The Director and two other members of the present senior management team participated in this initial workshop which resulted in the development of some key processes such as student enrolment, materials development, materials distribution, assignment handling.

The Senior Management Team has indicated that they are now more critical than they were before because… they have the knowledge of how things should be done. They ascribe the improvements in learner support and course materials development and production mentioned earlier to the systems and processes that are in place as well as to their improved management practices. Since the Change Management and Teambuilding workshop in May 1998 there has been a growing awareness among staff of how the different parts of the organization should work together to achieve their objectives. The restructuring that the management team embarked on with the assistance of the TA, members of the Bath Team and the APSO consultant led to a streamlined organization in which people now have a clearer understanding of what they are expected to do. The job specifications that were developed are a contributory factor.

The management team is critical of how well the different processes and management systems are functioning:

There is currently a lack of good coordination of the different processes and systems in the college. The staff responsible for managing these systems need intensive training to equip them to coordinate effectively.

Our own findings, detailed in sections three and four, support this view. The TA believes that, Senior Management is able to cope but middle managers are still a weakness.

In our opinion certain key people such as the distance education coordinators and the area coordinators are key positions and people need to be adequately trained in coordination as well as training skills. We also think that critical processes like assignment handling and materials distribution should be monitored closely to track improvements and identify problems that require immediate attention.

An important observation was made by the TA

The different inputs have helped to develop managers who come from an educational background but I think a more formal and structured development is needed to build on the foundation laid. Key management skills that could be developed further include delegation, planning, setting objectives.

We endorse this suggestion because the adequate development of management capacity is often left to chance.

We have identified some strengths in terms of management systems:

The Senior Management Team shows a commitment to standards of excellence, a critical attitude and the will to improve;

Essential processes and systems are in place in PMD and Learner Support;

There is a realization that middle management needs intensive training to coordinate these processes and systems;

A number of staff working in PMD, Learner Support and the Registry have had some basic training in establishing processes and have been involved in several process reviews;

Some monitoring and evaluation mechanisms are in place, for example, tools have been developed and are used to evaluate existing materials;

A Statistical Digest provides data about the learners and their performance during the examinations, which can be useful for planning more effectively.

There is still much work that needs to be done to ensure that systems are actually working as intended. Every staff member has to become involved in ongoing monitoring and evaluation and middle management needs to be strengthened. Senior management also have to play a part in monitoring key indicators that show that the various systems are working and improving to benefit the learners.

Some areas that need attention so that gains made are not lost.

A priority area is the training of middle managers in coordination skills. This could be a short project for which funding could be found.

Attention should be paid to the ongoing and systematic development of managers at all levels to ensure that they continue to improve their management skills such as coordination, delegation, decision making.

We suggest that the senior team use this report to reflect on the decisions they made when revising the strategic plan to check whether or not their decisions could be improved through a deeper understanding of the issues and the emergence of new priorities.

There is clear evidence of vision being deepened , strategic focus being improved and higher standards being adopted, with an improved understanding of what this entails. The senior management team has shown confidence and resilience during a difficult period of change and is showing clearly the impact that the Project and other factors have had on their ability to lead NAMCOL. There are still a number of significant challenges facing them in all areas. How they respond to them will require that they draw on the skills they have learned and the insights they have acquired.

***Back to Contents***

Our aim in this section is to assess how effective the approach and methods were that the Bath Team used to achieve the purpose and outputs listed in the Project Logframe. We confine ourselves to an analysis of information derived mainly from three sources: the views and reflections shared by the people we interviewed, namely, selected NAMCOL staff, two members of the Bath Management Team, the Technical Adviser and the Director of Studies at the University of Bath; responses to a questionnaire, and evidence from available documentation.

We wanted answers to three key questions in order to appraise the effectiveness of the methods: What aided learning and acquisition of relevant knowledge and skills? What was a hindrance to learning and acquisition of relevant knowledge and skills? Can NAMCOL sustain and build on the work they are currently doing?

This section is divided into the following parts:

Approach and Methods of Programme Delivery

vIn-country workshops and consultancies
vMAEC Programme
vResident Technical Adviser
vRegional placements
vGeneral conclusions

Methods for sustainability

vMentoring: Counterpart vs Project Team
vCascade model
vGeneral conclusions

6.1 Approach and Methods of Programme Delivery

The information for this particular part is derived mainly from Bath University’s Project Proposal Document, Appendix A – General and Technical, and complemented with inputs from two members of the Bath Team and the TA.

Flexibility, continuous review and ongoing consultation with NAMCOL staff was regarded as critical in order to facilitate the most flexible response as specific needs and requirements were identified. The methods of support included.

An in-country Technical Adviser whose role it was to work closely with employees of NAMCOL to develop their skills and competencies in a variety of open, distance and continuing education techniques.

A series of in-country workshops and consultancies to develop specific areas of knowledge and expertise and to enable staff to participate by cascading this knowledge to colleagues within NAMCOL and to staff in the regions.

Places for selected NAMCOL staff on modules of the University of Bath’s School of Education Advanced Modular Programme, leading to the qualifications of Advanced Certificate, Advanced Diploma or MA in Education. In addition, staff would be able to participate in modules of this programme on an INSET basis.

Out-of-country placement opportunities for selected members of staff, to enable them to observe current and groundbreaking practices at other institutions within Southern Africa.

From the outset the menu of inputs was designed as an integrated staff development programme made up of several components, each linked to the others and working in concert to achieve maximum effect. The menu of workshops identified for NAMCOL was compiled from the University of Bath’s portfolio of specialist training programmes.

One of the Bath Team members commented that the resident TA played a role much like a conductor of the orchestra. The TA would also provide ongoing training and support to staff particularly in the area of course materials development. It was hoped that the practical focus of the workshops would enable easy transfer to the work situation. The consultancies and assistance of the TA would be work-related support to help staff improve their work practices and in the process consolidate their skills. Short-term attachments with distance education institutions in the Southern African region would introduce a contextual aspect in the delivery of distance education courses.

Participation in Bath’s School of Education’s Post-graduate Modular Advanced Education Course (MAEC) would offer NAMCOL staff with suitable background the opportunity to obtain an Advanced Certificate, Advanced Diploma, or MA in Education. Ten of the 32 modules in the MAEC programme were considered relevant for the NAMCOL Project.

It was decided to broaden participation in this programme by delivering the course in Namibia to allow equal opportunities for as many of the staff as possible. Three modes of delivery were identified: taught modules in Namibia, distance learning modules, a summer school module in Bath. It was hoped that diverse delivery modes would give students the opportunity of experiencing first hand the learning and teaching approaches and methods providers of continuing education such as NAMCOL can use.

Fourteen of the fifteen evaluation reports at the end of the workshops/consultancies gave very positive feedback. Subsequently these ratings were supported by the responses on the questionnaire which 13 out of 15 respondents completed during the evaluation. The statements to which they responded did not focus exclusively on the methods used but also included the content aspect. All respondents but one were very positive about the impact of the workshops on their work as borne out by the results of the questionnaire. Details about the impact of specific workshops have been covered in sections three, four and five.

The sustained and regular schedule of workshops seemed to have made an important contribution to staff’s understanding of open and distance learning issues and more importantly resulted in improved work practices. There were, however, some comments about too many workshops and too little time to integrate what was learned.

An analysis of the workshop evaluations and responses by staff shows that they regarded the following factors as having been most beneficial to learning, acquiring new skills and improving work practices:

vThe timing of the workshops to coincide with the immediate needs of staff. The greater the congruence, the bigger the impact recorded by people;

vDirect work relevance made the transfer to real work situations easier.

vWorkshops that were followed up by immediate on the job practice resulted in consolidation of skills;

vThe handouts and supporting materials which, according to one person remain an information bank or backup for reference purposes, when one needs to refresh one’s mind on a specific issue;

vThe workshops that resulted in tangible products, such as the Learner Support Guide;

vWorkshops that introduced new approaches and practices and that were not refresher courses of existing knowledge;

vHaving a global view of the system was cited by many as helping them to understand their own role and place within the system;

vDedicated and uninterrupted learning time during the workshops;

vWorkshops where more direction and guidance were given rather than participants drawing on their own experience, especially in course materials writing workshops;

vEnough time during the workshops to grapple with the issues properly, as someone commented, we do many things in a short time;

vThe use of regional facilitators to provide contextual reality.

A member of the Bath Management Team was satisfied that some significant learning took place as a result of the workshops. The first two inputs were not as good as the later ones which employed a consultancy style format. The evaluations made by the TA on what worked and did not work became part of an evolutionary process that resulted in ongoing modification and led to the concept of training consultancies. These became much more focused and geared towards practical outcomes. NAMCOL staff liked this format because it proved to be very effective in linking the workshop learning with the actual work situation.

The TA observed that the inputs actually changed dramatically over time and they became far more focused on helping staff to cope with their day to day work. She believes that flexibility, even within a workshop, to meet the needs of people was the strength of the programme.

We used four criteria to assess the quality of the workshop/consultancies:

vthe workshop/consultancies met the identified needs of staff;

vthe range of subject areas covered was comprehensive enough for staff to get a good understanding of the basics of open and distance learning;

vthe workshops/consultancies were well planned and structured;

vthe workshops/consultancies were directly relevant to the participants’ work.

The subjects covered by the workshop/consultancies and the MAEC Programme are presented in the diagrams below. This overview reflects the range of areas covered by both types of inputs.

We are satisfied that the Project covered the range of areas that were identified for staff in the first needs analysis documented in the Inception Report. Training skills is the only topic that was identified as a need for some individuals that was not delivered during the period under review.

Although the training of trainers workshop is scheduled for later this year we believe that a strategy could have been devised to train a core of people as trainers much earlier in the Project in order to build sustainability. We would also have included a topic that deals specifically with Learning and Teaching Approaches for Open and Distance Learners in the Southern African context because tutors, in particular, are still locked into a traditional style of teaching that is inappropriate for NAMCOL learners.

We attribute the high level of success attained through the workshop/consultancy method to the following factors:

vclear terms of reference were developed for each input;

vfacilitators were briefed by the TA before the workshops;

vparticipants were actively involved in diverse activities during the workshops;

vworkshop notes were often in the form of practical and useful follow up actions plans, checklists and tools, e.g. Research Plan for the Annual NAMCOL Statistical Digest;

vparticipant evaluations were completed for each event;

vfacilitators submitted a report in which they gave their views and perceptions and made suggestions;

vthe TA compiled a report which gave an overview of the event and put forward useful recommendations.

Using the above criteria we must conclude that the workshop/consultancy inputs were of a very high standard.

The lack of follow through by staff after many of the workshops in order to consolidate learning was identified as the single biggest obstacle. Several reasons were identified: the heavy workload; the ongoing stream of training events; lack of reflective work behaviour on the part of many staff. The latter drew some diverse comments. A Project team member recognizes that there are some individuals who have developed as reflective practitioners. However, many colleagues still view participation in workshops as discrete events. After the workshops they file away the documentation and forget about it. Another team member felt that there should be structured mechanisms in place so that people cannot shelf away the ideas and skills that they have learned during the workshop.

The TA’s view is that people cannot be made to be reflective practitioners because it is self driven. In our experience people can learn to become more reflective especially if they can see the benefits of this practice. We have found that peer group review is an excellent method of helping people to learn reflective behaviour.

The workshops were extensively used in the Project and from the evidence collected it is clear that they proved to be very popular and successful. There are two important issues that we want to raise for careful consideration in any future projects and internal initiatives that use the workshop method as an integral element of their capacity building technique.

Exposure to too much training in a short time can lead to ‘training fatigue’ which becomes a deterrent to learning. What constitutes ‘too much’ will vary but, from comments made by some staff and the TA, there was certainly an indication that they had reached saturation point, especially towards the end of 1998 and the middle of 1999. For example in 1998 there was a taught module or workshop every month from February to August. Some staff members attended most of these training sessions and had to travel long distances in order to attend the sessions, which were mainly held in Windhoek and environs. In short some people viewed the workshops as being ‘disruptive’ at times, especially when the immediate demands of the job clashed with the timing of the Project inputs. Such conflicting priorities were sometimes resolved in favour of the job. This was a dilemma, particularly for some MAEC participants.

While we think the regular schedule of training inputs helped to create a learning climate at NAMCOL, nevertheless we feel that at times the training load was too heavy and could not be sustained at that level for an extended period. People had insufficient time to integrate fully into their work what they had learned. A lighter schedule, which would have created more opportunity for reflective practice and more thorough on-site follow up and training, may have been more effective.

NAMCOL should find it easier to manage the pacing of its own initiatives in future as it will not be faced by the same dilemma facing an external agent of balancing the delivery of sufficient input on the one hand against providing opportunities to assimilate the input thoroughly on the other within a fixed period.

The Project grappled with this problem and came up with a workable option in developing the workshop-consultancy approach. Structured learning from the workshops flowed to the work situation

In the PMD the TA built on this approach by offering ongoing support and help to staff to consolidate their skills which led to the development of the House Style Manual. The TA assisted learner support staff to develop the Learner Support Guide but there is no evidence to suggest that the same intensive type of support was available to them on an ongoing basis. From these two examples and our own experience of capacity building we conclude that the strategy used in PMD for building capacity works well and that it could have been used to greater effect in the learner support area.

The consultancy style workshop that evolved in the Project with its more practical on the job follow-up proved to be effective in building skills particularly where staff themselves followed up with activities that consolidated their learning and where they received ongoing support.

This practical orientation can also increase learners’ capacity to withstand the onset of workshop fatigue.

Getting regional consultants to conduct some of the workshops provided staff with contextual learning opportunities, which they found particularly useful.

We interviewed the remaining eight people who are still enrolled with the programme. They also completed a section in the questionnaire that dealt specifically with this programme. The majority agreed that what they learned in the modules was sometimes immediately relevant to their work and that they are now more confident as a result.

As with the workshops, the modules that were most closely associated with their own area of work were perceived to be most relevant.

There was great interest among staff who were not enrolled in the programme to attend the taught modules. Throughout the delivery of the modules, people attended for their own development purposes and they accounted for more than half the participants in the last three modules. They reacted very favourably to these learning opportunities.

Education Technology was the first taught module in 1997 and followed the Management Systems Workshop and Consultancy that was held at the end of 1996. Both these training and development events helped to introduce staff and, in particular management, to learning and teaching approaches and open and distance education systems and processes.

The Director of Studies for Advanced Courses at the University of Bath is very pleased with the progress made by some of the students. It is of course disappointing that a relatively small number will eventually progress to an MA. qualification. Realistically, given the circumstances, she is pleased with what has been achieved and is confident that most of those who are still enrolled will complete a qualification be it the MA, the diploma or the advanced certificate

Respondents identified five forms of support they found particularly useful:

vface-to-face tuition because they get immediate response to their queries;

vimmediate and constructive criticism and directional guidance from the tutors on the assignments. There were different views about the quality of the comments from different tutors. Rita Johnston was identified as the most helpful and with her direction they could improve on their work;

va visit in October by the Professional Development Coordinator, Rita Johnston, from Bath who held workshops in Windhoek and Ondangwa to assist students with assignment planning and writing. This proved very successful for students and has renewed motivation to submit assignments. Feedback from the visit was very positive on both sides;

vthe course materials were very readable and useful. The students seemed to be aware that the materials were a limited source for a masters level course and that they would need to complement them with additional reading and research;

va full set of basic texts for each module was provided by Bath. Some texts are held by the TA in Windhoek and others are held in Ondangwa where colleagues from the north can gain access.

One respondent, who is the only one to have completed the six required modules, found the two-day a month study leave very useful and used it to progress his studies. He also attended the summer school at Bath this year.

The MAEC programme was very vulnerable to the disruptive effects of the organization’s transition to parastatal. The Director of Studies at the University of Bath commented that it took people’s thoughts and energy away from their studies and the staff at Bath experienced difficulty in supporting them during this time. It also complicated the logistics of the programme.

The main challenge for participants was to find time to study. Balancing the demands of work, home and studies was probably the most difficult aspect of studying part-time. Sometimes they managed, at other times when the pressures of work mounted they did not.

There were different views on the level and quality of support received from the tutors at Bath. The students complained that in many instances they had to wait too long for marked assignments and the feedback was not always detailed enough to be helpful. The Director of Studies expressed her satisfaction that they had done whatever they could to support the NAMCOL students. Tutors supported them at a distance by telephone, fax, e-mail. Those tutors who taught modules would make themselves available to the students afterwards for queries and additional support. Some made more use of these visits than others.

Students who have been used to a more teacher dependent learning approach, find it difficult to take full responsibility for their learning. This has been our experience of students in South African institutions. For the most part we work on the assumption that students need additional support and encouragement to change from dependent to more independent learning behaviour, even at the post graduate level. This type of support was indeed offered to students in October 1997 with Rita Johnston’s visit to Namibia.

Many students have had difficulty getting used to the academic approach to writing the assignments. The Director of Studies said that this was a common difficulty that students experience when they start off. Students who participate in the advanced courses are generally mature people who have had work experience but who have been removed from the academic environment for some time. They usually need a lot of handholding and encouragement at the beginning until they have produced their first assignment. The School of Education’s experience with students who participate in this programme from all over the world is that they gain in confidence after completing their first assignment.

A number of staff from different areas participated in the taught modules, but the actual in-depth development could only really be done with a handful of staff members. Of those who are expected to complete their MA, four work in the PMD division, one in Learner Support and one person is from the Research and Evaluation Unit. A senior manager will probably complete the Advanced Certificate.

From the point of view of the institution, the distribution of students likely to attain their MA is uneven. What are the reasons for this? We think that the nature of the work in PMD lends itself more readily to what the MAEC programme offered. In addition, their central location combined with the consultancies/workshops, support from the TA, and work relevance of a number of modules provided a more stable and stimulating environment for the learners. The in-depth approach offered by MAEC modules provided an additional dimension to the other development work being done. MAEC has contributed significantly to PMD, and together with the other methods, has strengthened the institution’s capacity in a critical area. The impact of the Project would have been weakened in this area if MAEC had not been included, but this may not have been critical in the short term.

One of the Bath Team when interviewed about a change of approach to the Project if Bath had known about the transition said that it would have been wiser to.

Invest the money not on training people to write distance education learning materials but to do that in concert with training people about organizational design, etc to ease the transition process…I think I would have gone for an organizational design option.

The above comment is in line with our reservations about the timing of the MAEC programme. We also felt that focus on the more pressing fundamental issues would have been more relevant and that the introduction of MAEC at a later time might have been more useful. However, NAMCOL now has a more developed PMD than it would otherwise have had. A priority now is to come to grips with the outstanding learner support and middle management challenges.

The chosen mode of delivery for the taught modules with tutors visiting NAMCOL was preferable because it was felt that more staff could benefit from the inputs made by the programme. It also meant that tutors could focus more on supporting students in adapting the materials to the NAMCOL environment. This type of dedicated focus would not have been possible to the same extent had the students participated in the course at Bath. From an institutional capacity point of view the help they received in adapting the materials was far more useful at that time.

We believe that the dissertation, in particular, can make a substantial contribution to developing the institution and the current group of students must be encouraged to research priority areas from which NAMCOL can benefit

Through their own experience of open and distance learning methods the group feels that they now have more understanding for their own learners and the difficulties they are experiencing. They believe that this experiential insight should enable them to write better course materials and plan more effective support for their learners.

Namibian students like their counterparts in South Africa have come from an educational system that has rewarded dependent learning behaviour. The organizers of programmes like the MAEC, must take this into account and build in more effective support especially at the beginning.

Restructuring an institution places enormous demands on staff. The additional demands on individual staff need to be carefully estimated in advance to determine whether they will be able to cope successfully with both sets of demands. a new potential benefits of a programme such as MAEC for the institution more likely to be medium to long-term for the institution. and some of them are likely to be indirect. is more likesThe institution should INSInmtHEMACE students should be encouraged to make more explicit contributions to the institutional development of Nionay For the institution to benefit more from the participation of MAEC gain more

 

At this point the prediction is that at least five people will complete the MA qualification and probably two will receive the Advanced Certificate. Most students have completed the taught modules and are busy with the distance education modules. According to the Director of Studies the late starters who missed the early taught modules still have three options to complete them.

They can get distance learning packs for those modules which they have not yet completed and a supervisor from Bath would liaise with them and support them;

They could contact the study centre in Botswana and find out when they are conducting the taught modules they have missed;

If there are funds available and they get the necessary approval, they could attend summer school at Bath and do the outstanding modules.

The students were not aware of the above options.

A suggestion was put forward by the group that the student who attended the summer school at Bath, where he participated in the workshops on how to approach and plan the writing of the dissertation, should run a workshop for them and transfer his knowledge and skills.

The Technical Adviser’s role was primarily seen as a staff development role in the area of course design, development and production at NAMCOL. This was her main function and is captured in the terms of reference document. An integral part of this role was the mentoring relationship with a NAMCOL counterpart.

The TA was also expected to be the Project’s ‘eyes and ears’ in Namibia to ensure that the programme delivery was implemented and, where necessary, modified to cater for particular needs as they emerged. This demanded ongoing communication between NAMCOL management and the Bath Team. In the words of a Bath Team member she was also the ‘conductor’ of the orchestra of ‘soloist’ consultants who came to NAMCOL to facilitate the workshops and consultancies. As the coordinator of these efforts she had to keep in touch with everyone to make sure the arrangements ran smoothly and the training event could take place as planned.

As part of the ongoing monitoring and evaluation of the Project she collected relevant data, analyzed it, made suitable recommendations, compiled reports, and communicated to the relevant stakeholders to keep everyone abreast of what was happening on the ground.

The PMD staff who worked closely with the TA on a daily basis said they benefited enormously from the regular interaction and rated her support highly. One person thought she was excellent. Initially they found it difficult to get used to her approach because she made it clear that she was not going to do things for them. She would be available to assist them but they would have to do it for themselves. A few felt nervous at the beginning because they were unsure of what was expected. However, she would encourage and nudge them along and eventually they became used to her approach. They now value her critical input and have learned to be much more critical themselves. The remarks of one person showsa shift towards a more independent way of working towards the end of the Project, I was in a position to bounce ideas off and to consider her advice before taking crucial decisions.

More generally staff agreed that the support they received from the TA was very valuable as borne out by the responses to such a statement in the questionnaire. The people in learner support had fewer interactions with the TA than the PMD. This may account for two respondents indicating that they disagreed with the statement.

The Director praises the work done by the TA. She has benefited personally from regular interactions with the TA especially during the first year. She knows that people appreciate the work the TA has done even though they may not always have agreed with her. She thinks the TA has also learned to become more relaxed with the staff.

The availability of an in house TA in an area such as programme and materials development and production has been recognized as an extremely useful resource to have. Without the TA people doubt that the Project’s other inputs would have been so successful. Even though the TA regrets that she could often not follow up as she would have liked, the fact that she was there and that people could pop in as they needed her gave people confidence to work on their own.

The ongoing working relationship between the TA and the Project Team has helped to keep the TA in touch with the actual training and development needs of people in the organization. The TA felt that it was particularly the cross-functional nature of the team that was useful because it helped to highlight concerns and needs from the different areas in the organization. The Project Team members commented that a broader representation resulted in a much more thorough refinement of the terms of reference for the different interventions. They do not believe the final terms of reference would have been as satisfactory if the TA only relied on input from a counterpart.

The TA’s professional way of doing things and the high standards that she set for the staff are visible in a number of documents we scanned, for example the regional placement reports, the course materials, the Learner Support Guide, the Tutor Marker’s Handbook.

Members of the Bath Team expressed the view that the TA was crucial to the Project. Without her regular communication to keep them informed about what was actually happening, they would have found it extremely difficult to take proactive action, as they did significantly during the period of transition, in order to keep the Project on track. She also played a vital role in sensitizing and supporting the stream of consultants that came to NAMCOL to facilitate the various workshops and consultancies.

The disruption of the transition dealt a heavy blow to the time frame of developing people and of ensuring sustainability. The challenge for the TA was to get people up and running in a relatively short period of time, in fact in 18 months. The more practical focus of the workshops was an attempt to bring this about. The training of trainers workshop planned for October is another example. This was not originally part of the inputs that had been planned but emerged later as a need. The TA feels that people are only now ready for this type of workshop. They first had to do it themselves before being ready to cascade the ‘do how’ to others.

The extension to the Project due to the upheaval of the transition phase of the organization has meant that the TA stayed longer than originally intended. The staff’s response to the extended period was varied. Some people regarded it as enabling and others as an obstacle to independent working behaviour. Some believed that the TA should stay on longer because they still feel insecure and were not fully convinced that they could cope on their own. Others felt that the longer she stayed the more the staff would rely on her being there and the less inclined they would be to take the plunge and fend for themselves. A few thought that the three years was long enough even though they thought there would be a vacuum when she went. A discussion about how long a TA should stay in an organization to ensure that optimum use was made of the input and support, resulted in some interesting views.

The length of a TA’s involvement should depend on the skills level of the people who need her support;

A weaning period should be built in from the start;

It is good to have a TA for one year. When the advisory role is for longer than a year it is probably better to have two or even three TA’s involved at different times who can focus on different areas of need.

From the start, the focus of the TA’s role was on the programme and materials development area. At the request of staff she did become involved in a number of other activities, e.g. assisting with the Learner Support Guide, Tutor Marker’s Handbook, assisting with the training of tutor markers, advising on the establishment of a pilot Project in the north, editing job descriptions, advising on management issues, supporting staff of the Research and Evaluation Unit. However, her actual involvement in the learner support area has not been as intensive as in PMD.

The TA felt that the administrative tasks got in the way of her actual role of supporting and advising. She was doing massive amounts of admin: getting terms of reference drawn up, compiling lists and lists of who was doing what and why, and doing an extensive amount of reporting. She believes strongly that a NAMCOL counterpart could have done a lot of this work.

Some valuable lessons have been learned:

vIn future the partner institution should drive the actual Project in the country with assistance from the TA. This would free the TA up to concentrate more on the job of advising and supporting. By taking ownership of the Project at this level from the start, it is more likely that the initiative would be sustained at the day to day coordination level.

vIt may be more effective for the organization to have more than one TA for different areas of need and they could be involved on a rotation basis.

vAs indicated under the heading, Enabling Factors, a Project Team can be of invaluable assistance to a TA. Such a cross-functional team can help to make the interventions more relevant for staff and the organization.

vA structured weaning period built into the TA’s plan of action would allow for a more conscious transfer to an independent level of performing.

It has already been agreed that the TA will provide 35 days of support to NAMCOL over the next six months. The working relationship will be different in that the TA will not be functioning as a member of the Bath Team but rather in her capacity as an independent consultant. The staff will determine in consultation with her how best to use her input during this time. Many see this period as the actual weaning period and are looking forward to consolidating their skills.

The purpose of the regional placements is described in the Terms of Reference for Regional Placements. These are to:

vobserve best practice in distance education in organizations that operate in the Southern African context;

vdevelop skills in the practical development and/or delivery of distance education;

vpractice new techniques observed in these organizations to consolidate this learning;

vimplement some of the practices observed, or participated in, within NAMCOL;

vact as ambassador for NAMCOL whilst participating in the placement.

The reports scanned show that in all cases there was thorough preparation to ensure that candidates would get maximum benefit from this type of exposure. One person’s report contained an evaluation sheet. It provides insight into the factors that he considered enabling:

vHaving objectives for the visit that were directly relevant to the work context and that were focused and achievable;

vHaving a prepared programme of activities that was well designed and organized;

vThe support provided by the host institution;

The Chief Registry Officer cited the following as contributing to his learning during his placement at the ABET Institute at UNISA:

vThe experience of ‘seeing things with your own eyes’ and actually doing helped him to understand ‘how it is done’. He now feels more mature and confident because he can talk from experience;

vHis involvement in areas other than his own area helped him to get an insight into the whole process. He now has an idea of how the different parts work together and influence each other;

vHe could benefit indirectly from the ABET staff’s exposure to UK training and expertise.

The reports that candidates were required to write and submit on their return pushed them to document their learning. The TA provided the candidates with support to plan their reports. Each report is unique but there were some common parts: Planned outcomes, programme of activities, detailed comments on the activities, unplanned outcomes of the visit, evaluation, recommendations, follow-up action plan. One person had added samples of documents used by the host institution. One report contained a detailed description of the various materials production processes. Two of the reports had photographs. The Chief Registry Officer explained that he had decided to buy a camera to capture his experience in the different departments in picture form so that he could share it more effectively with his colleagues.

A strong expectation is created that this method will facilitate effective delivery when participants return to their own situation. The following table shows some key activities/benefits that resulted from the placements.

Some constraints were identified by the candidates.

Sending only one person limited the benefit and impact of the experience.

The time frame was considered to be too short by the candidates who visited the DVET and ABET centres. The Head of the Yetu Yama centre indicated that while he learned a lot about the administration and management of Technical Vocational Centres he still feels that he needs more experience and skills to set up a centre.

The timing of the placement is important. The Chief Registry Officer said he could not get the full experience of the registration time because he was only there for part of the time.

The biggest difficulty that the TA experienced in organizing the placements was that she had to rely on verbal agreements with the people in the host institution to design a suitable programme of activities for the candidates. She only discovered after an agreement had already been entered into with one of the institutions that they charged a daily fee for their hosting service. With hindsight she believes it would be more beneficial and cost effective for the staff developer at NAMCOL to visit the intended institution for a few days, get a guided tour and then draw up a programme tailored to the needs of the organization.

The three candidates all said that the two week placement was an eye-opening experience and it is evident from their reports and the interviews we had with them that they have all acquired valuable insights and skills. One outcome that candidates did not mention but that can be implied is that they have learned to take responsibility for their own learning. They have also discovered the importance of documenting learning as part of the action learning cycle because the placement gave them an experience of reflective practice. It is possible that they will only fully understand the domino effect of this method of learning much later. Factors such as realistic expectations, the timing of the placement, duration, an appropriate programme of activities and support for candidates when they return are all factors that affect the impact of these learning opportunities.

The flexible approach the Bath Team used, with its sensitivity to the local context once the Project got under way, contributed significantly to the relevance of the delivery methods. The overall impression is that the different methods used by the Project worked well in combination to achieve a fairly high level of capacity building especially in top management and PMD. Despite various inputs made by the Project to strengthen learner support staff there is still much more development work that needs to be done in this area. The complexity of dealing with large numbers of part-time staff in the farflung regions of the country makes this a particularly demanding area.

From an analysis of the information in this part we arrive at the following conclusions.

Some significant learning took place as a result of the workshops that evolved over time to more of a workshop/consultancy format. The practical focus enabled a number of staff to consolidate their learning in the work context, acquire new skills and develop materials and tools of a higher standard than before. The House Style Manual, Learner Support Guide and Tutor Marker’s Handbook were developed through workshops and on-site training and support.

The Project component that was most severely affected by the disruption of the transition period at NAMCOL was the MAEC Programme. At the time of the evaluation the small remaining group of participants had not yet completed their qualification. It is therefore difficult to make a more in-depth assessment of the value of this type of method of development for the organization. The benefits for individuals are more apparent and, based on the views of those who were involved with the programme, it seems that it did provide people with the opportunity of gaining a greater understanding of open and distance learning issues. One of the most significant experiences for the participants has been that of being open and distance learners themselves.

The Project could not have delivered the inputs it did without the ongoing support of the TA who played a pivotal role in coordinating the various inputs. The level of capacity building attained in PMD is due largely to the continuous assistance and support provided by the TA within the work context. People learned while they were doing their work.

The regional placements were intensive learning experiences for the three staff members who were exposed to other institutions in South Africa and Botswana. All three participants came back with renewed confidence and were immediately able to apply their newly acquired knowledge to improve some of their work practices. This type of learning experience is especially useful in building capacity quickly in a specific area and promises much potential benefit for the institution.

The combination of the different methods worked best with staff in PMD where the intensive and ongoing support of the TA was crucial in helping staff to benefit from the workshops and consultancies. The choice of a workshop/consultancy approach as the main thrust of the Project had advantages and disadvantages. It was possible to give staff a useful overview of the main components of open and distance learning in a short time frame and it allowed them to benefit from exposure to different consultants. Inherent in this approach was also the danger of overload, inadequate transfer of skills to the work situation and interference with staff’s work demands. The Bath Team was sensitive to this dilemma and responded by evolving a more practical focus in the workshops and linked the workshops to consultancies that assisted staff to achieve work related outputs.

A decision was made to use a cascade method of training that would facilitate ongoing transfer of skills. The expectation was that, the continuing repetition of one simple model of transfer for many transactions and at all levels promotes quick ownership of both process and content.’

It was also expected that the mentoring relationship between the Technical Adviser and a NAMCOL counterpart would result in the transfer of essential skills necessary to organize training and development for professional staff especially in the area of course materials development and production.

It would also play a crucial part in the training of some 900 part-time tutors who are working in the regions. It was hoped that through the skills transfer process the Project would leave behind a trained and qualified staff with the tools to continue the work they have started and to keep up with their own professional development both individually and as an organization. Such a level of competence would enable staff to continue the work of NAMCOL.

The original idea of having a single person as a counterpart who would shadow the work of the Technical Adviser and eventually take over the full staff development coordination role was modified at the start of the Project. The Director felt that it would be more beneficial to the organization for a representative team drawn from different areas, including the regions, to work closely with the TA. The notion was that the benefits would be more widely distributed. The team approach would also allow for more diversified input.

The Terms of Reference for the Staff Development Team, as it was initially called, are detailed in the Inception Report as:

vWork with the Bath Team to develop a staff development strategy for NAMCOL

vPresent the staff development strategy to the Management Committee

vDevelop procedures for sustained staff development within NAMCOL

vAdvise the Bath Team on customization of workshops to be most appropriate for the needs of NAMCOL staff.

Regular progress updates of the team’s activities appear in the TA’s quarterly reports. The December 1997 report notes a change in name to the Project Team. This was done to avoid confusion with the Staff Development Committee that has been set up as part of the organization’s human resource division and that has responsibility for the development of all staff. The Project Team focuses largely on the needs of professional staff.

From interactions with the Project team it appears that in practice they performed an advisory function. They would help to identify training needs of staff, make suggestions about who should participate in particular workshops and consultancies, discuss the terms of reference for interventions, and help to modify and refine the draft terms of reference prepared by the TA. In the December 1997 report the TA acknowledged the valuable input made by the team that resulted in the preparation of ‘a very focused extension plan’. According to the reports some key Project Team activities were:

Advising the TA in many areas of the Project, particularly the organizational needs of NAMCOL, the content and structure of workshops, the detail of the NAMCOL staff learning agreement, objective selection criteria for progression of students on the MAEC, important cultural and organizational information;

Helping to finalize the Agreement between NAMCOL and staff who are participating in the MAEC programme;

Meeting with the MAEC candidates to discuss their progress.

The team expressed regret that something was lost in opting for a team to work with the TA. They acknowledged the importance and value of the broad representation and view that the team brought. However, they also identified that there is now no single person who has the capacity to carry out the tasks and activities done by the TA, like compiling the terms of reference from scratch, identifying suitable consultants, interacting with consultants and agreeing rates. The Director is disappointed that individual members of the team did not shadow the TA in different areas and become skilled.

One member of the team, who was a training officer at the time, had the opportunity of working closely with the TA for some months. She regarded him as a possible counterpart but unfortunately his job changed due to restructuring. However, he did continue as a member of the Project Team. He maintains that the involvement was too brief to develop his skills in areas other than coordinating the local logistics of training events.

The Director is of the opinion that the onus was on the team members to be more proactive in acquiring skills but recognizes that workload and different perceptions of the team’s role got in the way. She is convinced that the team approach is the best way to empower people and would organize it differently in future. After intensive discussion the Project Team concluded that they would opt for a combination counterpart and Project team model. The TA’s view is that she would have needed a counterpart, someone with clout in the organization, someone who would have ended up running the Project. She would have felt more comfortable with a greater buy-in from NAMCOL at the staff development level. She agrees that the Project team worked quite well because it was cross-functional.

The Project Team has successfully been used in an advisory function. The biggest regret is that not more hands-on and in depth capacity building of the staff development coordinator’s role has taken place. In fact some commented that real empowerment had not taken place in this area. Even though the team now understands what is involved in the training process, they did not feel that they had all the necessary skills to coordinate training events. As one person pointed out, We haven’t managed to build the house, but we’ve laid the foundation.

Despite this the Project Team seems confident about taking over the TA’s role during the forthcoming period of extended support granted by DFID. The Project Team will be responsible until March 2000 for coordinating the support provided by the TA in her new capacity as consultant. How well they carry out this task should provide insight into their actual capacity. Projections for the Project Team’ involvement in staff development beyond March next year are still speculative. One person thinks the team will become absorbed into the existing Staff Development Committee. His view is that this committee could in part not fully get off the ground as expected because the Project Team was in place and most of the attention was focussed on the training interventions of the Project.

There is still a strong feeling among a number of people that in future projects the development of a counterpart is desirable and complements the Project Team.

Reactions to the question whether the cascade method of training has worked at NAMCOL were varied and tended to be more pessimistic than optimistic. Most people agreed that the method did not work in the regions and a few even said they thought it did not really work for the institution. Some disagreed with this view and provided examples of how they had personally benefited from cascade type training experiences. The Director said that she profited enormously during her UK visit from the mentoring interactions she had with the Bath Team and the subsequent support she received from them. As a result she was able to plan the strategic review meeting with minimal support from a Bath Team member. She herself is committed to cascading information, knowledge and skills. When she returns from any training event she identifies who could benefit most and then conducts training session for her colleagues.

The transfer of skills seems to have been most successful in the Production and Materials Development Division from the TA to the PMD team. The intensive interactions and support have borne fruit and are visible in the improved materials that they are developing. The TA’s comments in her February 1999 report shows that they are making progress in doing things for themselves. For example the TA is now only checking the editing of the person responsible for the IGCSE Physical Science Course. PMD team members who are responsible for The Grade 10 Business Management Course and the Accounting Course are doing it on their own with minimal support from the TA. Colleagues are also editing each other’s work

The TA recognizes the progress they have made but is unsure about everyone’s ability to cascade what they know and can do. So far the team has trained the part-time course writers with her assistance but in future they will have to organize this type of training themselves.

The cascading effect of knowledge and skills is also noticeable in those staff members who participated in regional placements. They came back from their two-week learning experience able to transfer many of the things they had learned to improve their work practices and this has also affected the people they work with. As a result of the expertise he had gained during his placement at UNISA the chief registry officer was used as a resource person for the learner support workshop/consultancy held in June 1999.

The cascade model worked least effectively in the regions. Here the HOCs are supposed to cascade the training they receive from the area coordinators to the tutors. One area coordinator could not say how good this training is because no proper evaluation has been carried out. Some learner support staff suggested that the problem with the regions is that you have to rely on part-time staff to do the training. Some HOCs have the natural ability to train whilst others struggle. An example was given from the pilot Project in the South. Here training was given to staff who were specifically appointed to coordinate the Project and to train tutors. However, the results were disappointing. In the words of a learner support team member,

The problem at NAMCOL is that maybe some people are not trainable, for cascading the information to tutors. Because if you get training you don’t get training for yourself but for training other people. Sometimes it’s a waste of resources to train, retrain people but people don’t cascade.

Despite the difficulties experienced with the method quite a few people recognize that the model itself makes sense in their context. No one could think of an alternative method to train the large numbers of part-time staff. A Project Team member commented that they have in fact come up with an adaptation to the model. When it became clear that the regional staff were not coping with the training of HOCs and Tutors, a request was made by the learner support staff for a tool that would enable staff to improve the training.

The result was the Learner Support Guide, which was developed by staff with the assistance of the TA. This is a self-instructional guide that was distributed in the regions at the beginning of the year. A similar manual was developed for tutor markers. A House Style Manual is another in the portfolio of tools that can be used to assist NAMCOL staff with the training of new and part-time staff. People are generally very proud of these materials and are convinced that they will definitely help to improve the training of staff in future.

During interviews with a group of HOCs, tutors and tutor markers we investigated the use of the Learning Support Guide and the Tutor Markers’ Handbook. One group’s responses are pertinent to the concern that those on the edge of the cascade get wet.

The interviewees believed that the guide was part of a research project to find out what they think of the various sections in the guide. They were not sure what they should do about returning the completed evaluation sheets.

The HOC’s present thought that they would have to collect the guides from the tutors at the end of the year and give them to the NAMCOL staff who would then capture the responses. This they felt was part of a review and revision exercise. The interviewees used the guide in different ways. Four of the five had not read the whole guide before they started their tutoring task. The one who did so felt it was his duty to read if from cover to cover. The others dipped into the guide as the need arose. One person did all the activities but the others only did those they thought were relevant. No one seemed to be aware that the activities are part of a self-instructional method of learning.

This is of course only an example and not a representative sample. Much more research needs to be conducted to investigate how the manuals are used in the different areas. Nevertheless it illustrated the point that the manuals are no guarantee that the transfer of information and knowledge will necessarily take place.

There seem to be different perceptions on what it means to cascade. People use the terminology quite loosely and we often had the impression that simple communication of information was equated with the more intensive transfer of knowledge and skills that requires a more sustained mentoring and coaching input to be successful.

Factors that have enabled successful transfer of knowledge and skills in the deeper sense are:

vSustained interaction between the mentor and the trainee in a time frame not shorter than two weeks;

vOn the job learning;

vCommitment on the part of the trainee to follow through and do what has been learned;

vContinuous reinforcement of acquired skills;

vSupport that encourages independence and gradually nudges people to become self motivated to continue learning and improving.

The final assessment on the effectiveness of the training and development for NAMCOL staff can only be made after the Project. Improved organizational capacity will be demonstrated by the ongoing quality of outputs that NAMCOL staff produce on their own. All training inputs were geared to achieve this level of capacity. However, in this part we have reflected on two specific methods and mechanisms to achieve sustainability: the use of the Project Team as counterpart to the TA and the cascade method of training.

An analysis of the information we gleaned leads us to make the following conclusions.

The general view that the combination of Project Team and individual counterpart would be better in future indicates that the team on its own was not an entirely satisfactory mechanism. The cross-functional team worked with the TA mainly in an advisory capacity. While their input was particularly valuable in modifying planned training interventions in order to make them more relevant for staff, the team did not achieve the expected level of ownership of staff training and development.

The Project Team plans to coordinate the inputs made by the TA in her new capacity as consultant during the forthcoming six months. This short-term follow-up strategy provides the Project Team with an ideal opportunity to test its ability to drive the training for professional staff. Our concern is that the team does not seem to have a clear-cut vision for the long term of how it will operate within the organization’s centrally driven staff development strategy. We strongly believe that the Project Team, while becoming part of the organization’s staff development initiative, must continue to function as a separate team with an explicit focus on the development of professional staff working in the PMD and learner support divisions.

The cascade model rests on very optimistic assumptions about the skills and commitment of participants. At head office, where the skills of full-time staff are more developed, the method was effective in PMD and at more senior levels. In the regions, drawing largely on part-time HOCs and tutors who were exposed to less sustained training and support, the model was too demanding and less effective. A general assessment of this method must be that it had limited effect.

Even though a number of supporting resources for training part-time staff were developed these have as yet not been used optimally. A note of caution must be sounded at the current over optimistic expectations of the role manuals can play in promoting development because there does not seem to be a culture of self learning from written materials. Much more in depth reflection is necessary to devise workable strategies that use the manuals in conjunction with relevant training methods.

This is the concluding section of the report. A brief outline of the constraints and limitations within which we conducted the evaluation should help to view the conclusions and recommendations we make in context. We present our conclusions and recommendations per section and also indicate areas that need further exploration and research.

The special challenge we faced during this evaluation was to remain focused on evaluating the impact and sustainability of the staff development programme delivered by the Project over a period of three years and not become side tracked by NAMCOL related issues. Furthermore we had to restrict ourselves to assessing the Project’s achievements against the Logframe dated April 1998. We hope we have succeeded in drawing out the main issues from the overwhelming amount of detail we had to process in a relatively short time. It has been our aim to reflect the different experiences and perceptions of people and to construct a balanced view of the Project. We accept that it is not possible to fully satisfy everyone’s expectations but we hope the report will be used to further improve the work done by the Project.

The following issues had an influence on what we could and could not achieve in the evaluation.

In the time available to carry out the evaluation and develop the report it was not possible to track sufficiently every piece of information. We were frequently forced to rely on the views and perceptions of the people in the Project and in a number of cases our conclusions were more tentative than we would have liked. It was not possible to carry out a survey with a representative sample of the final recipients, namely tutors and learners, to establish adequately whether and how they have been affected by the Project and in which areas NAMCOL has to focus on in future. It was also not possible to make an in-depth study of some components, for example tracking the course specification and course materials schedules.

Determining impact and sustainability of a capacity building programme is always a challenge because it takes time for people to develop their new skills and techniques and the effects of these are often only visible much later. We agree strongly with the Auditor/Verifier that many effects of the Project will only really become apparent after the contractual time frame of the Project.

It was relatively easy for us to make an assessment of how the course materials changed as a result of the Project because we had access to some pre and post Project materials. Frequently we were forced to rely on the reports, views and perceptions of people in the Project in order to arrive at some insight into the shifts, if any, that have taken place as a result of the Project.

We acknowledge that there were a number of other influences that contributed to the progress made at NAMCOL. For example, the APSO Adviser assisted and supported management with the restructuring of the organization and the development of the strategic plan. The Director was exposed to a number of outside training and development events, such as the Quality Assurance Workshop arranged by the Commonwealth of Learning. Staff’s own prior expertise and experiences cannot be ignored.

Despite other influences that have contributed to the progress made at NAMCOL, there is an overwhelming body of evidence that points to the considerable impact made by the Project. Staff now have a much better understanding of open and distance learning principles, standards and issues than before. Individual staff members have acquired a range of technical knowledge and skills particularly in the course materials development and production area and they have grown in confidence. The Project has contributed substantially to developing the strategic planning and managing capacity of the senior management team. With a sharpened focus provided by the revised strategic plan and increased confidence this team is ready to lead NAMCOL into the future.

The Project has been less successful in assisting staff to establish workable and sustainable methods and techniques for the training of part-time staff because the cascade model has not been as successful as expected in the regions. The training of part-time staff to provide quality services to learners in the regions is an area that requires further exploration and research. Now that the Project has ended NAMCOL has to find appropriate methods of incorporating the coordination of professional staff training and development previously done by the TA and the Project Team into its organizational staff development initiative. There is no evidence that there is a clear vision or plan to do so.

It is too early to make an informed assessment of the level of empowerment achieved. In some areas this is more developed than in others. What can be said with certainty is that a foundation has been laid in the crucial areas on which staff can build.

Before the inception of the Project, there was very little materials development taking place at NAMCOL. Courses were of poor quality, and were very limited in type. In addition, the distribution was not at all effective. During the course of the Project the programme and materials development staff changed considerably – there were changes in personnel, as well as in the conception of their role. Initially programme developers were also responsible for coordination of delivery, which left them very little time for development work. It was only in the last eighteen months of the Project that the staff were settled enough to develop into a team.

The criteria for establishing the impact of the Project on programme and materials development were taken as the verifiable indicators in the Project Logical Framework. The impact in terms of each of these follows.

Largely as a result of the considerable input from the Project, the current team know about materials development, have engaged with what makes materials accessible to learners, and can apply this knowledge to their own materials. Although the team’s confidence in their ability to develop alternative secondary school courses is probably well-founded, there is a need for further capacity building in the curriculum development skills necessary for the expansion into technical, vocational and professional courses.

When materials prior to the inception of the Project were compared with those developed as a result of the Project, the following improvements were noted:

vThere has been considerable progress in imagining the difficulties of the learners and responding to them through the provision of more helpful introductions, feedback, and discussion of strategies for learning.

vThe teaching approach is much more interactive, and less encouraging of rote learning.

vThere appears to be a shift towards helping the learners develop skills, rather than merely mastering content.

vThere has been considerable improvement in layout and accessibility.

The improvement is not simply a result of overwriting done by the TA, but is the work of the team themselves, with assistance. There is still considerable work that needs to be done on the materials, but the staff demonstrate the capacity for self-criticism.

The Project clearly made the team aware of the importance of developing production schedules, but more noticeable is the increase in number and range of types of courses developed during the period of the Project. Despite the difficulties of the organizational restructuring, during the period of the Project, NAMCOL produced four wrap around courses, and will have a completely new course as well as another wrap around ready for 2000. In addition, four in-house course development /learner support documents have been produced, and two further new courses should be completed during 2000. Although the new courses appear to be taking a long time, this allows for the slow process of professional development. Course production can easily be fast tracked by external input, but at the risk of taking ownership away from the people whose ultimate responsibility it is.

This verifiable indicator needs to be interpreted more broadly as referring to course development processes in general, rather than merely budget analysis, which is one of those processes. The main way in which staff awareness of the processes of course development have been consolidated is through the ongoing process of writing the House Style Manual. The extent to which the staff have internalised all the processes they have described in the Manual will only be evident over time.

It is suggested that the following could be done to ensure that the gains made during the period of the Project are strengthened.

Do further work on the House Style Manual. Once it is complete, the contents of the Manual could be analyzed and compared with similar documents from other institutions. Areas for further development of staff or of processes could be identified. What the staff also need to do is to develop structured ways in which the Manual will be used – for process review, for induction of new staff, for reference by existing staff. The Manual could possibly be developed further (along the lines of the Learner Support Guide) into a self-instructional manual which could be used formally for induction purposes.

Build the capacity of the team in curriculum development. It is necessary for NAMCOL to move beyond the alternative secondary education market into the technical and vocational field, and also to offer courses which NAMCOL itself accredits. There is very little indication that the team currently have the ability to do this. Possibly the process could start with curriculum review of a range of courses being offered in Southern African region. It would be important to select for review not only ‘pure’ distance courses, and not only courses in the technical and vocational area, but courses which adopt a variety of flexible approaches to delivery whatever their content. A particular area of focus could be how flexible courses handle the development of practical skills.

What could not be established from this evaluation, and what should possibly be investigated by NAMCOL in future are the following:

The extent to which the improved materials distribution processes are actually working.

How learners engage with the course materials (this is being done to some extent in the piloting of the Business Management course, but it could be done more thoroughly).

To what extent learners in the practical subjects actually do the practical experiments/tasks that the materials seem to require.

To what extent the course specifications and course production schedules actually guide the development of the courses (this could be established by interviewing programme managers, programme developers and contracted writers on how they work with the written course specification documents).

In the early days of NAMCOL, there were inadequate systems in place for handling assignments, materials distribution, and handling enrolments. Large numbers of part time staff are still employed and there have been problems with recruitment as well as with monitoring. In addition, the full time staff in the regions were appointed as government officials during the time when NAMCOL was part of a government department, and they were not trained for the different roles they now have to perform.

The verifiable indicators for learner support developed as part of the Project Logframe refer rather vaguely to development of staff capacity and improvement of learner support systems. The following more precise criteria arising from an understanding of the context have therefore been developed.

Effective systems to handle the needs of the large numbers of students

Effective training and monitoring of part-time staff

Development of staff understanding of their work.

It must be noted that even with these more precise criteria, it is difficult to determine whether or not improvements can be attributed to the input of the Project.

The learner support staff at head office talk confidently and in detail about the improvements in learner support systems, particularly the assignment management system, which was dealt with in some detail in the learner support workshops provided by the Project, as well as in one of the out-of-country placements. They are also able to be critical of the processes and suggest ways to improve them. However, there are certain indications of actual or perceived problems from three sources other than the full time staff. These should be followed up.

The Project spent considerable time and energy supporting the development of the Yetu Yama Centre, with tangible results:

a resource centre at Yetu Yama was opened in March 1999 to provide learners with access to textbooks, study guides, exam papers;

10 computers were donated by the City Council and the head of centre is negotiating with Utech to upgrade the computers. The plan is to run computer courses for a broader market than the traditional NAMCOL market.

One of the important points made in the workshops and materials developed with the assistance of the Project is the need for monitoring. Regional staff are aware of the importance of monitoring the attendance of tutors, learners and heads of centres, and in many cases provided evidence of ways in which they used information gained from monitoring, but it is not clear how thoroughly monitoring can be done given the large number of centres and the distances between them.

The Project has not assisted NAMCOL sufficiently in developing effective training of part-time staff. The part-time staff tend to perceive the training as ‘meetings’, or as repetitive and, by implication, boring. The training skills workshop planned for October will help to equip staff better to carry out training, but this would need to be followed up with support if the training is to have a lasting effect.

The training strategy with the greatest potential for continued impact is self-access training material. This has been developed, and, particularly the Learner Support Guide is of extremely high quality. However, there are problems in that the role of the Guide has not been understood by the people who are meant to use it.

Even though the systems may not be perfect, and even though the training may not be completely effective, the kinds of activities and input through a sustained professional development effort can have an effect on how the staff understand their work. In the long term, this can have an effect on the ways that work is done.

In the regions, the Project has contributed to an increased focus on understanding the needs of the learners and an increased awareness of the critical importance of monitoring tutor attendance. However, there has to be a greater emphasis on support of the tutors in changing their teaching approach to accommodate the different mode of delivery demanded by NAMCOL courses.

With regard to the distance learners, the Distance Education Coordinators need to understand much more clearly how crucial their role is in ensuring the proper functioning of the assignment management process as well as the improved quality of the teaching on assignments. Generally, however, head office staff are aware of the problems that exist, and in some cases can develop solutions.

To ensure the continued development of learner support staff and the improvement of learner support systems, we would recommend the following.

vNAMCOL learner support staff need to find ways of engaging various stakeholders to provide systematic feedback on how processes are working. These monitoring processes should not be left to the research and evaluation unit and should also include mechanisms for using feedback for improvement of the processes.

vAppropriate ways of training tutors to deal with NAMCOL students need to be developed. Training should focus on teaching methodology specific to the alternative mode of delivery and to the particular learners, rather than merely on processes or adult learning in general.

vThe limitation of the cascade training method is that staff by virtue of their position (rather than their interest or aptitude) are expected to train others. What could be done in future is to identify certain staff who have an interest in and aptitude for training. These staff could then be developed into a training team who would operate directly in the regions as well as centrally.

vA strategy needs to be found to develop staff capacity in how to learn from materials. This is necessary to prevent the excellent work done in preparing the Learner Support Guide from being wasted. But it is would also help staff understand the difficulties their own learners experience in working with materials.

vAlthough what has been done at Yetu Yama could not be replicated at the centres currently used by NAMCOL students, lessons of experience could possibly be used in the broader Open Learning Centres Project that Ed du Vivier describes.

Before the start of the Project none of the senior managers had received formal management training and most had little or no understanding of open and distance learning organizations. The strategic plan that existed had been compiled by people other than the senior management team for a different organizational context. The transition to parastatal caused considerable disruption with many trained staff members resigning. The new status required drastic organizational restructuring. An amended version of the Project Logframe was developed to meet the emerging needs of NAMCOL senior management. An APSO consultant was also working with management during the time of the Project.

The criteria we have used for assessing the impact on senior management was the Project Logframe verifiable indicator, ‘increased achievement of NAMCOL strategic plan. We have confined our focus to the strategic planning and reviewing aspect. In addition we have compiled our own criteria arising out of the context to establish senior management’s readiness to continue the work of NAMCOL in three critical areas: course materials development and production, learner support, management systems.

NAMCOL’s New Strategic Plan dated 15 February was adopted by the board and used by the senior management team during the current year. The plan is the culmination of inputs made by members of the Bath Team and the support given to management by the APSO adviser. It is a well-structured and practical tool that enables senior management to monitor the progress of the institution. The NAMCOL Statistical Digest 1998 complements the strategic plan and it is the joint product of the Project, the APSO Adviser and selected NAMCOL staff. A measure of increasing confidence of the senior management team is their proactive decision to review the strategic plan in August this year, which they did with minimal assistance from a member of the Bath Team. The outcome of the review was not available to the evaluators. Its quality can only be revealed through the subsequent actions that senior management adopt to implement the revised plan. The senior management team has taken the Strategic Plan through one annual cycle. They seem to have developed enough confidence to continue implementing and adjusting it to move the college forward in a direction that is in tune with emerging realities.

There is clear evidence that the senior management team has deepened its understanding of the implications of managing an open and distance learning institution. They have shown confidence and resilience during a difficult period of change. Three factors have contributed to their growing ability to manage the organization: the Project inputs and support, the assistance of the APSO Adviser, the independent status of the institution has enabled the team to make decisions more quickly than before.

Under the team’s direction NAMCOL has begun to adopt improved standards in course materials development, production and distribution and in learner support. This has contributed to building a more professional image for the college. The management team’s critical and analytical ability is crucial for the ongoing improvement of the college. Management is able to determine what needs to be done, what has been achieved and identify what the gaps are. There are still significant challenges facing management in all areas. Although a foundation has been established particularly in the PMD division careful attention must be paid that in the time ahead a deliberate effort is made to consolidate capacity. The area of Learner Support requires particular attention. Apart from the daunting task of training the large number of part-time staff it is also necessary to find ways of strengthening the regions to ensure that learners are actually experiencing improved support. How management responds to these demands will require that they draw on the skills they have learned and the insights they have acquired and that they do not hesitate to seek the necessary assistance where it is required.

We encourage the senior management team to consider the following.

For the quality assurance theme for 2000 identify key indicators for monitoring NAMCOL’s progress in each of the three critical areas (course materials development and production, learner support, management systems). Set up suitable monitoring mechanisms to keep track of actual progress in achieving specific targets that have been set in the Strategic Plan, for example in distribution of materials check that the targets have actually been met.

Arrange for an appraisal towards the end of the next annual cycle of the senior management’s ability to implement the NAMCOL Strategic Plan. In the interests of building on the foundation that has been established and given that the implementation ability was not assessed during this evaluation, we urge the management team to arrange for an external consultant to evaluate their new found ability to plan, review and implement the Strategic Plan. This does not remove the responsibility of self monitoring and evaluating.

Get funding for new priority projects based on needs emerging from this report.

We support the suggestion made by members of the Bath Team that a strategy be devised to get funding for new projects. We link this idea to the priority needs that have been identified in this evaluation. A small project would be the training of middle managers, including area coordinators and distance education coordinators, to acquire good coordination skills. A bigger project is the training of part-time staff, HOC’s and tutors in the regions and the tutor markers.

Strengthen middle management in all areas. Middle managers like the Distance Education Coordinators and Area Coordinators play a pivotal role in ensuring that NAMCOL delivers quality services to the learners. These are the people who have the responsibility for interfacing with part-time staff and for providing them with the necessary training and support. It has become apparent that the capacity of this level of management needs to be strengthened if NAMCOL is to provide its learners with quality support services.

Involve the Project Team in coordinating the inputs made by Alison Mead in her new capacity as independent consultant. We support the suggestion put forward by the DD PMD and the Project Team that this team coordinate the support and inputs made by Alison Mead. This type of involvement builds on the capacity already established in the Project Team and gives the members an opportunity to consolidate their staff training and development coordination skills.

Develop an overall staff training and development strategy that incorporates the Project Team. For the sake of continuity and to build on the work of the Project Team that assisted the TA with the training and development events organized for professional staff, we put forward a strong suggestion that this Team be incorporated in any staff training and development initiative undertaken by the Human Resource Division. We feel that the Project Team has a good insight into the needs of professional staff who work in PMD and Learner Support.

Develop a more holistic and integrated vision of learner support to avoid fragmentation. Even though different parts of the organization handle different aspects of learner support it is important that one team or unit or division take responsibility for ensuring that a unified set of standards is established and applied that governs all forms and methods of learner support offered by NAMCOL. This team or unit should monitor whether in fact improvements are happening across the board. The principle of a unifying vision and strategy but a diversification of tasks is an important one in a complex area such as learner support. .

Before the Project most NAMCOL staff had little or no experience of the diverse methods of training and development that the Project offered. Here are some factors that had an influence on the way staff used the different methods: most staff have had an experience of an educational system that rewarded and encouraged dependent learning behaviour; only a few people had previously experienced distance learning; during the time of the Project the organization was in a transition phase that caused enormous upheaval and uncertainty and that made heavy demands on staff’s time.

The effectiveness of the capacity building efforts made by the Bath Team can only really be made after the Project when staff are faced with the challenges on their own. However, we have reflected on the appropriate nature of the approach and methods that were used in order to achieve the outputs indicated in the Project Logframe. Our analysis is based on the perceptions and views of the different people in the Project and on the evidence we have seen of products that staff have produced under the guidance of the consultants that came to Namibia to facilitate workshops and conduct consultancies and through the ongoing support of the TA.

We did question the Project design’s reliance on its principal component, the schedule of workshops and consultancies. In our own experience workshops tend to more useful in mapping the terrain than in building solid capacity in the work situation. We weighed up the advantages of introducing NAMCOL to the essential areas of open and distance learning. The workshop approach does allow for a broad exposure in a relatively short time. We are generally satisfied that the subjects areas chosen corresponded with the training needs analysis conducted when the Project started and in our view they represent some essential areas that need to be covered as part of an introduction to open and distance learning.

We reflected on the real dangers that were apparent of overload and training fatigue as well as the demands on staff’s time especially when the frequency of inputs was high and when some staff viewed the workshops as being ‘disruptive’. We came to the conclusion that our response would have been negative had the workshops not evolved into the workshop/consultancy style learning events with their more practical focus. We are impressed with the depth of learning that was achieved in some cases.

A contributing to our positive view about the workshop/consultancy component as the main thrust is the support by inputs from the other three components. Having a resident technical adviser has been pivotal to the success of this four-pronged approach. We think that the Project would not have achieved its purpose had it not been for someone who was on the spot playing a vital coordination.role. The TA’s valuable capacity building role is most apparent in the PMD Team even though she made very useful inputs in other areas.

The top management team’s visit to the UK and the regional placements have been eye-opening experiences for the participants and all have reported that they could not have learned as much as they did in such a short time in any other way. Experience of the regional context has been particularly useful in terms of strengthening the institution’s capacity. This element is not immediately noticeable with the MAEC programme at this stage because the people who have perservered have as not yet completed their qualification and have been too busy studying and meeting their work deadlines. From a superficial perspective the impression may be that this was a totally inappropriate method given the more urgent need for organization building. However, we have weighed the pros and cons and have come to the conclusion that there many potential benefits for the institution. The way the programme was delivered to NAMCOL staff has meant that more people could participate and it has also given them the experience of being open and distance learners themselves. The institution must capitalize on the investment made by encouraging and supporting the present staff to complete their qualification. The institution can benefit enormously from their dissertations if staff base them on areas of need within NAMCOL.

Overall our impression is favourable that in concert the different components added dimensions to the capacity building initiative that resulted in a diverse and stimulating experience for staff. In our view the level of capacity building was probably most intense within PMD because the four components were most efficiently used here. The cascade methods intended for the transfer of knowledge and skills did not work as well as expected. While we understand the complexities of training large numbers of part-time staff we are disappointed that the Project did not adequately help staff to develop a suitable strategy. The exposure to a range of capacity building approaches and methods has given staff the additional experience of how different methods can be used in staff training and development initiatives.

Review the role of the TA. We make two points: one is that more emphasis should be placed on the follow-up training role of the TA and two that the workload should be checked carefully to ensure that the TA is able to do the first. From the experience at NAMCOL it is clearly evident that where the TA was more intensively involved with follow-up training better results were achieved. During our interview the two members of the Bath Team informed us that they were already beginning to implement a more pro-active training role for the TA at BOCODOL. The experience in Botswana should be monitored in order to draw out important lessons about capacity building. This could be shared more widely with other institutions in the region. Effective ways must be found to lighten the TA’s administrative role. A suitably trained person of the host institution who does not require intensive training and who can take over many of the administrative responsibilities would be the ideal.

Investigate different options for using the TA to build capacity. We support the suggestions made by some NAMCOL staff members that different TA’s could be used for shorter periods to focus on the areas in which they have the expertise. It might also be conceivable to have two TA’s working in an institution at the same time for a shorter period. What the staff members argued is that the needs of the institution should be taken into consideration before decisions are made about the most suitable type of TA input.

Investigate the need for a structured weaning time for sustainability. The additional input of the TA in her capacity as independent consultant over the next six months can be considered a weaning time during which the staff have limited access to the TA’s support. The expectation is that the Project Team will coordinate this short-term training and in the process consolidate their coordinating skills. It might be well to monitor this process and to identify what works and what not. The findings may show that this is an important part of the design of a capacity building a capacity project.

Include regional consultants as part of the schedule of workshop/consultancies. NAMCOL staff were very appreciative of the inputs made by the three regional consultants. Is it possible to include this dimension into the workshop component from the outset?

Although these areas are placed for NAMCOL’s attention the Bath Team could also be interested in some of the research indicated below.

Investigate a range of different training approaches and methods that can be used to train large numbers of part-time staff who work in the regions. The cascade method of training part-time staff has not worked in the regions nor at Head Office with the tutor markers. We believe more research is necessary to investigate why it has not worked and what other approaches and methods have worked elsewhere that could be useful in Nambia.

Areas in course materials development.

How learners engage with the course materials (this is being done to some extent in the piloting of the Business Management course, but it could be done more thoroughly).

To what extent learners in the practical subjects actually do the practical experiments/tasks that the materials seem to require.

To what extent the course specifications and course production schedules actually guide the development of the courses (this could be established by interviewing programme managers, programme developers and contracted writers on how they work with the written course specification documents).

The usefulness of self-access guides and manuals in training staff. Much is expected from the Learner Support Guide, the Tutor Marker’s Handbook and the House Style Manual. However, there is no currently little understanding of how the different groups of staff are using them and what the implications are for training. Much more investigation has to be done to find the best ways to use the manuals to build capacity.

This capacity building project has made great demands on all those who were part of it. It is thanks to their resilience, cooperation and determination to make it work that it has largely been a success. The next few years will show how deep the roots have gone and how well NAMCOL is able to convert the benefits of this training and development programme into benefits for the learners of Namibia.