Papers Presented at the 1st National NADEOSA Conference
Held 11-13 August 1999

Author:
Tessa Welch, South African Institute for Distance Education, August 1999

Title:
Evaluating the quality of assessment in teacher development programmes

Introduction

It could be argued that the best way to influence any education system is through reform of the assessment policies and practices. In an outcomes-based educational system, this is even more clearly the case. Outcomes-based unit standards and qualifications give guidelines on only two things – outcomes and ways to judge the achievement of outcomes. The assumption is that learners can achieve the same outcomes through a range of different learning programmes, or even through life and/or work experience. The way to change the learning is not through changing the syllabus, but through changing the outcomes. Assessment then becomes crucial as a means not of measuring the mastery of the syllabus but of measuring the achievement of the outcomes.

It follows then that providers wishing to move into the new paradigm need to focus first and foremost on investigating and reforming their assessment practices. It also follows that evaluators wishing to investigate the quality of educational programmes in an outcomes-based system should focus on the evaluation of assessment practices.

This paper outlines current policy positions on the design, implementation and management of assessment. It then illustrates how to analyse the assessment strategy in a given programme through providing a worked example and finally discusses this example in terms of some of the important values and principles emerging.

ways to describe and evaluate an assessment strategy: emerging criteria

Aside from general documents outlining the role of assessment in outcomes-based education programmes and formats and guidelines for the generation of standards and qualifications on the NQF, the major documents impacting on assessment in teacher development programmes are:

1. Norms and Standards for Educators, now moving into a 22 page policy statement entitled, A Framework for Norms and Standards for Educators in Schooling. This latter document is likely to contain a set of key strategic objectives relating to assessment which are as follows:

• The assessment strategy is clearly related to the purpose and exit level outcomes of the programme.
• The assessment strategy assesses the extent to which learners have achieved horizontal integration, that is, the integration of roles and the knowledge and skills delivered through the different courses/modules which make up the educator development programme. It is designed in such a way that the seven roles are assessed through the subject specialism.
• The assessment strategy also assesses the extent to which learners have achieved the vertical integration of foundational, practical and reflexive competence. In other words, it assesses whether learners are able to integrate the ability to perform important teaching actions competently (a practical competence), understand the theoretical basis for these actions (foundational competence), and reflect on and make changes to teaching practices (reflective competence) so that they can be described as achieving an applied and integrated competence.
• The assessment strategy assesses the extent to which learners have the ability to teach in authentic and changing South African contexts.
• The assessment is ongoing and developmental.
• There are detailed diagnostic records of learners’ progress.
   

1. Criteria and Guidelines: Providers (Draft 1), a SAQA document incorporating Criteria for Quality Distance Education in South Africa (developed by the Centre of Educational Technology and Distance Education in the Department of Education). The section on Policies and Practices for the Management of Assessment reads:

• The assessment strategies must be in keeping with the aims and outcomes of the learning programme or course and as these relate to the outcomes specified in the standard or qualification.
• A range of parties is involved in the assessment of learners. This can be designed appropriate to context and outcomes and can include self-, peer and other forms of group assessment. Moderators for assessment and even assessment monitors can be included in this grouping.
• Assessment information, including learning outcomes, assessment criteria as well as assessment procedures and dates, should be provided to all learners and assessors.
• Records of assessments must be kept and learners must receive detailed and accurate feedback on their progress and performance.
• The processes and results of assessment must fulfil the requirements of the NQF standards and qualifications for which the provider has been accredited and must meet the requirements of the ETQA

A critical issue supporting assessment systems design and management is that of appeals. This involves ensuring that learners have access to appeal an assessment outcome either to the practitioner or assessor and if unsatisfied to the management of the provider and, in the final instance, to the ETQA.

Two other documents related to assessment are being produced by SAQA – Guidelines for the NQF Assessment System, and Criteria for the Recognition of Assessors. These will refine but will not substantially affect the thrust of existing policy documents.

Criteria such as the above not only list the elements to be considered in assessment systems, but also assign value to particular ways of carrying out assessment. In attempting to get to grips with these criteria, it might be useful first to list the elements to consider when designing, implementing and managing assessment, and only then consider the values element. This helps us to work out what the business of assessment involves in any system, before we work out the values and principles that inform the new directions in assessment.

 Design and implementation

• design at a macro level – the strategy as a whole (purposes of assessment, weighting, different methods, people involved in assessment)
• design of individual assessment tasks (methods, people involved, instruments and evidence requirements)
• information and guidance to students
• methods of making judgements about student achievement (outcomes, assessment criteria)
• feedback to students

• assignment management

• marking procedures
• moderation

• appeals.

What is interesting is that the principles that emerge as policy directions informing criteria are the generally accepted principles of validity, reliability, feasibility, and fairness. However, there are new emphases, the most notable being

• the importance of integrated (and applied) assessment,
• the assessment of competence (not merely isolated outcomes),
• the primary purpose of assessment being for student learning,
• transparency and accountability,
• quality assurance rather than merely quality control, and
• recognition of prior learning.

There is a danger that policy statements become checklists against which atomised bits of practice are evaluated. What this paper suggests is that evaluation is about values – discerning the values orientation of a particular programme, rather than its superficial adherence to requirements.

In order to illustrate how these elements relate to each other and how they are informed by principles, one worked example will be given of the assessment strategy in the Further Diploma in Education (English Language Teaching) offered by the University of the Witwatersrand. In the discussion, reference will be made to other programmes studied during the Educator Development Support Project.

Background information about programme

The Further Diploma in Education of the University of the Witwatersrand was initiated in 1995 by the Department of Education (Faculty of Education) in association with Applied English Language Studies (Faculty of Arts), Mathematics, Chemistry, Botany and Zoology (Faculty of Science). The purpose of the programme is to provide practising teachers with subject and subject teaching knowledge and skills over two years, part-time.

The programme is aimed at secondary and senior primary teachers who have a Std 10 certificate and three year teaching qualification (M+3). In addition, the programme accepts NGO facilitators and teachers with higher qualifications who similarly wish to specialise in either Mathematics, Science or English. After the Diploma, students who have done well can move into a Bachelor of Education, without first having to complete a degree.

The Further Diploma in Education specializing in Mathematics, Science or English Language Teaching was launched with 141 students in February 1996. In 1997, a further 120 were enrolled. The numbers of students taking the specialization course in English Language Teaching was 36 in 1996, and 66 in 1997.

The programme embraces an open learning philosophy and uses distance learning methods together with face to face sessions. Within the programme, there is a chronological sequencing of courses, designed to be completed over two years. In addition to materials and ‘homestudy’ sessions, there are four residential sessions of four days each in the first year and three sessions in the second. There are also two one-day workshops during the two-year period, in May and February respectively. The programme is therefore described as mixed mode - distance materials, combined with residential workshops, some tutor/lecturer support between workshops, teaching on assignments and collaborative learning with study partners. Mixed mode delivery allows students not resident in Gauteng to participate in the programme, and currently students are enrolled from Northern Province, KwaZulu Natal, Mpumalanga, Free State, North West Province and Eastern Cape.

From 1999, the programme will not be delivered by Wits, but by an associate college of Wits, Promat College at two of their centres - in Cullinan (north of Pretoria), and in Pinetown (KwaZulu Natal).

The main ways in which learners are supported are: course materials, lecturers and tutors, and compulsory residential blocks. Other forms of support include tutorial letters, assignment feedback sheets, telephone, study partners, individual visits to the university by learners and mid-cycle workshops.

Description of assessment design at the programme level

At a macro level, the assessment strategy, consists of the following:

• self-assessment activities
• tutor-marked assignments
• course portfolio
• examination or examination equivalent project.

Each course in the programme has different versions of the above outline, according to the demands of the specific course. According to Wits regulations, the year mark is worth 50% and the exam mark is worth 50%. The exam can be an examination equivalent project.

When examined in more depth – at the level of the various courses that make up the English Language Teaching specialization of the FDE – additional information emerges.

Students are required to do three full courses (two in English Teaching and one in Education Studies and four half courses (two in Education Studies and two in English). Typically, full courses demand four tutor-marked assignments and half courses require between two and four tutor marked assignments. There is an examination for each of the courses, but in certain options, this takes the form of an examination equivalent assignment – for example, a research report, a portfolio of creative and objective writing, or a set of classroom materials. The grammar course, requiring the mastery of discrete items of content, is atypical and includes more assignments and even a test.

Description at the level of individual courses in the programme

When examined at the level of individual assignments in individual courses within the specialisation of English Language Teaching, the following more textured information emerges:

1. There are patterns in the ways in which the assignments are set in the individual courses. For example, in the Theory and Practice course, every assignment demands that students work directly in the classroom — either teaching or doing research on their learners. Whereas the detailed content of the assignments and examination questions differs from unit to unit, the basic processes through which students are required to work are broadly similar. These processes of design (or adaptation of lessons or approaches provided), implementation and reflection are shown diagrammatically below. Students on this course are required to go through the same processes four times in each of the assignments over a period of a year, and in the end of year examination as well.

Figure One: Assignment Design in the Theory and Practice Course

2. Even though the programme does not state up front what the outcomes are, the implicit outcomes fall broadly into four categories. Students must be able to design, implement and reflect on teaching, learning and assessment strategies in their English teaching. They must also be able to interact with each other as teachers, and with the broader teaching and academic community about their teaching. In all of this, they need to demonstrate an understanding of and ability to apply and relate to practice the content knowledge, as well as the theoretical and methodological information they have been exposed to.

1. In two of the courses, individual reflection and the development of students’ ability to assess their own progress is encouraged through the requirement that students should submit all or a selection of their assignments at the end of the year in the form of a portfolio with a reflective comment in which they record what they think they have learned through the process. In one of the courses specific criteria are provided for each category of writing, so that students have a basis for selection of their ‘best’ pieces for inclusion in the portfolio.
2. Another feature of the assignments is that they usually structure in self and peer assessment processes (even though these do not contribute directly to the students’ result). In one assignment, study partners are required to observe each other in the classroom and then share their reflections before submitting individually written assignments. In another, study partners are asked to work on an assignment together and then submit one jointly written or two separately written accounts of their work. In other assignments, students are required to try out a lesson or series of lessons in the classroom and then use various means to reflect by themselves on the success and lack of success.
3. Evidence requirements are in written mode, though the Education studies did experiment to some extent with assessing oral presentations. The kinds of written evidence vary, however. Some are typical academic essays. But others are lesson plans, reflections on lessons, lesson materials, work produced by pupils in the teacher’s class, questionnaires and other research instruments, creative writing by the teacher himself, personal biographical accounts, full blown research reports.
4. Students are assisted in a variety of ways to achieve the requirements of the assessment. One is building peer support into the completion of assignments. Another is dividing the assignments carefully into sections, and giving the students targets and products for individual sections. Another is discussing the requirements of the assignments carefully during residential sessions. A third example is the division of the final examination equivalent assignment into stages with interim reporting and formative feedback from both lecturer and peers. This allows students to see clearly where they are going wrong and what strengths they can build on, so that they can adjust their work to perform well in the final report.
5. The assignments are spaced in such a way that students receive lecture/tutor feedback on an assignment before being required to submit the next assignment.
6. Assignments are carefully worded. Explanations are typically at least a page in length and often up to three pages. In the Education Studies courses, students receive an assignment booklet, as well as a special guide on Reading and Writing.
7. Feedback comes in two forms – general points made in tutorial letters, and individual comments (both in-text and at the end of each assignment) on each student’s assignment. In one of the courses, students are provided with specific criteria and different percentages of the marks are allocated to different criteria. In others, the criteria are implied only in the detailed description of the assignment tasks. In the Education Studies courses, general criteria are provided for each assignment, but when each assignment is marked more specific criteria emerge and are tabulated on a grid so that the student can get an accurate picture of the rationale behind mark awarded.

 Description of assignment management system

Assignments from learners are sent to the university, and are received by the administrative office of the FDE. Assignments must reflect the following details: course/specialization, Further Diploma in Education, Faculty of Education and Wits University. They are then recorded and sent to the lecturers for marking. When assignments have been marked and individual comments made, feedback or tutorial letters are prepared. The marked assignments together with feedback/tutorial letters are then sent to the administrative office for mailing back to the learners. The turnaround period for assignments is estimated at four to six weeks.

Description of marking procedures

The programme has the usual practice of moderation of examinations and examination equivalent assignments by an external moderator. There has been an effort to extend the role of the external moderator to include critical comment on the programme and course materials.

However, it is only in the Education Studies courses that the same batch of assignments is divided between different tutors, necessitating the development of set of common procedures for marking to ensure a common approach. These are as follows:

• Tutors individually sample mark some assignments to get a sense of criteria to use.
• Tutors as a group workshop the criteria.
• The coordinator works out the criteria on a grid.
• The tutors group mark and discuss several assignments together in order to get a sense of how to use the grid.
• They then mark their own set of 30 assignments.
• Sometimes the coordinator moderates the assignments, but the benefits of this must be measured against the disadvantage of increasing the turn around time.

Discussion of the Wits FDE Assessment Strategy

An attempt has been made (not altogether successfully!) to keep to description and not allow evaluation to intrude in the above outline of the assessment strategy.

However, an assessment strategy is informed by certain principles, whether explicitly acknowledged or not. If there is to be effective evaluation, the principles governing the choices within the different elements of the description need to be made explicit. The way to improve an assessment strategy is not to tinker with bits and pieces here and there, but to evaluate the broad values thrust, and how the different elements contribute to the values and could be changed to promote a better set of values. What this section of the paper does is discuss some of the principles emerging in relation to the Wits FDE assessment strategy with reference to policy, literature on assessment and distance education and other case studies conducted in the course of the Educator Development Support Project.

Assessment purpose – student learning

In the literature as well as emerging criteria around assessment, there is an emphasis on using assessment formatively – as part of the process of teaching and learning – rather than as an afterthought or merely for summative purposes. The Norms and Standards for Educators talks about the importance of ongoing developmental assessment, and the Criteria for Quality Distance Education in South Africa say:

‘assessment is integral to and integrated into every learning and teaching strategy adopted, and includes formative as well as summative processes’.

In a distance course, integration of assessment into teaching and learning needs to be emphasized a great deal. Although self-assessment is important, it should not be the only kind of formative assessment used, as distance learners often can’t monitor themselves adequately to use self-assessment effectively. Very often it is only when faced with an assessment task that is going to be marked that the students actually engage with the material. When the Maths 101 Open University Foundation Course, was designed, special permission had to be obtained to require students to submit assignments fortnightly. The strategy was disputed by the university authorities because they thought it was too onerous and students would drop out. However, it had the reverse effect. It motivated students not to drop out, because they could see themselves making demonstrable progress through the results obtained from each assignment, and they felt a growing sense of mastery as the course proceeded.

In research done for the President’s Education Initiative, SAIDE field workers visited teachers in their classrooms and interviewed them about the course materials and the assessment. It was clear that, although the more capable students were stimulated by reading the course materials, all students – whether academically capable or not – were galvanised into productive engagement with the materials through the assessment. The conclusion reached in the PEI research was:

It is clear both from the features of assessment design recorded above and from the processes students record moving through to complete assignments for the Theory and Practice course that the assessment is creative, motivating and classroom based, and that it assists students to design lessons and research, implement these and reflect on their own practice in collaboration with others and on their own. It is also clear that the students appreciate the opportunities created by the assessment. It must also be noted that it is not only the high achievers or the secondary school teachers that express this appreciation. All the students — those that are struggling with the theoretical parts of the programme, and those that are relishing the intellectual stimulation — find that the emphasis in assessment on classrooms and learners is invaluable to them.

What emerges from this is the importance of enough assignments, and assignments of a kind that integrate what has been read with the work that teachers actually do in the classroom.

But a further essential part of the effectiveness of the use of assessment for teaching and learning is the quality and timing of feedback on assessment. As David Sewart has pointed out, distance education methods offer a way to overcome the depersonalization of the industrialised models of higher education which have been adopted in response to the massification of higher education. Because the teaching of is done mainly through materials, money and time can be spent on individualised attention to students – through counselling, but also through individualised comment on assignments.. When this is done well it links assessment to learning in a very powerful way.

In the PEI research into the Wits FDE, SAIDE researchers found that ‘detailed and useful comments on the assignments make feedback an important teaching mechanism on the FDE programme and a way of establishing dialogue with and motivating teachers’. For example, one student commented:

I have gained confidence that at least I am able to cope. After not having studied for a couple of years, I was not sure how I would cope with this distance course at a university. But through the assignments I have realized that I am capable. I feel that I am communicating with [name of tutor] and she has come to know me a lot through my assignments though we do not have much physical contact. The comment I liked very much was the one where she commented about my writing style. 'I really enjoy your writing style', 'I like this term chronological status'. My wife looked at it and she was thrilled.

In the case studies conducted for the Educator Development Support Project, a worrying trend in large scale distance education programmes were the paucity of assignments (in one case, assignments were voluntary, in another there was only one assignment per course) leading to:

reliance on summative assessment practices to determine a final result,

the lack of opportunities for students to present draft assignments,

the lack of systematic feedback to learners on examinations and assignments.

This is clearly not using the potential of distance education for individualised instruction even within a mass-based system.

Concluding comment

If distance education is going to exploit its potential for individualised teaching and learning for large numbers of students, a key strategy is extensive formative assessment with individualised feedback. The requirement of the Norms and Standards for Educators for ‘ongoing developmental assessment’ should be seen as a significant part of the teaching and learning strategy.

Applied competence

If a primary purpose of assessment is student learning, what kind of student learning? In an outcomes-based system, the learning is defined by outcomes. However, there are different ways of thinking about outcomes. What the policy documents make clear is that outcomes cannot be tied to particular syllabi, because the intention is that learners who have been exposed to different learning programmes with different content and methods can nevertheless achieve the same outcomes. Moreover, there is an understanding of competence as integrating a range of outcomes - performance outcomes as well as outcomes relating to underpinning knowledge. SAQA documents refer to the notion of applied competence as:

The ability to put into practice in the relevant context the learning outcomes acquired in obtaining a qualification.

The Norms and Standards for Educators follow the ETD Practices Project in a more elaborate definition of applied competence:

Applied competence is the overarching term for three interconnected kinds of competence. Practical competence is the demonstrated ability, in an authentic context, to consider a range of possibilities for action, make considered decisions about which possibility to follow, and to perform the chosen action. It is grounded in foundational competence where the learner demonstrates an understanding of the knowledge and thinking that underpins the action taken; and integrated through reflexive competence in which the learner demonstrates ability to integrate or connect performances and decision-making with understanding and with an ability to adapt to change and unforeseen circumstances and to explain the reasons behind these adaptations.

The Wits FDE set of exit level outcomes with its categories of design, implementation, reflection and interaction as well as underpinning knowledge (see appendix one), provide an interpretation of applied competence which is congruent with the understanding in the Norms and Standards document. Furthermore the way in which design, implementation and reflection are integrated within each assignment for the major course in the programme indicates that students are encouraged to apply foundational knowledge in practice, and reflect on that practice as well as on theory in order to ‘make considered decisions about which possibility to follow’ in future practice.

Concluding comment

The way in which exit level outcomes are expressed needs to reflect the particular way in which that programme interprets applied competence. However, as will become clearer in the next section of this paper, if the statement of exit level outcomes is not actioned in the assessment strategy, it is unlikely that students will develop the required applied competence.

Validity and integrated assessment

Validity of assessment has two dimensions – whether you are assessing the right things, and whether the ways you have chosen to assess these things will actually assess those things. In terms of the comments made in the previous section, the exit level outcomes indicate that, at least in terms of policy statements, the programme aims to assess ‘the right thing’ – applied competence. However, the second dimension of validity is whether or not it is being assessed in the right way. Both the key strategic objectives emerging from the Norms and Standards for Educators and SAQA policy call for integrated assessment: the assessment of applied competence, rather than individual outcome. The Norms and Standards calls for ‘vertical’ integration – i.e. the integration of foundational, practical and reflexive competence; and ‘horizontal’ integration – across modules/courses within a programme. In addition, it is implied in both policy documents that the complexity of the assessment of applied competence demands that a range of methods and instruments be used to assess it.

As regards the Wits FDE, figure 1 provides a picture of the assessment process in one of the courses – a fairly typical one for those courses in the FDE concerned with improvement of English language teaching skills. It is clear from this diagram that applied competence is being assessed in each assignment in an integrated way. Students are also expected to do a range of kinds of tasks – observation, discussion with partner, reflection on course materials, reflection on their own practice, design of teaching and learning strategies. If one looks more broadly across the English courses, this list expands to include development of English teaching and learning materials, and personal and professional writing. Content is not separate from practice or from reflection in the assessment process. The tasks include the three dimensions of competence – foundational, practical and reflexive. Moreover, the fact that the same process is followed in several assignments means that there is an understanding of competence as something acquired over time. As was commented on in the PEI research into the Wits FDE:

It is important that students are required to go through the same processes again and again as it gives them an opportunity to develop broader abilities over time rather than merely master the content of individual units one by one. Furthermore these broad abilities are central to the successful teaching of reflective practitioners.

Although the Education Studies assignments are more heavily weighted in favour of the development of foundational understanding of educational theories, they require the development of applied competence in a sense through the structuring in of a variety of methods of research rather than merely book-based research. These include interviews with educational stakeholders, and documentation of personal opinion drawn from classroom and school-based experience. This encourages reflective practice, even though it does not close the loop back into the classroom in the truly reflexive sense.

However, if integrated assessment implies that there needs to be assessment tasks that integrate all courses on a particular programme (the so-called ‘horizontal’ integration), then the assessment strategy for the Wits FDE is not fully integrated.

In this, the Wits FDE programme is not alone. In the Educator Development Support Project, there was acknowledgement of the importance of integration of foundational, practical and reflexive competence (even though there were many different understandings of what this meant), but some resistance to integration across courses/modules in a programme. One of the face to face programmes argued that applied and integrated assessment is possible

at the level of the research essay or in teaching practice but not across the modules … integration may take place across modules as a result of lecturer and student comments and through assessment task design, but it may not be possible to explicitly assess this.

Concluding comment

It is important, particularly in those courses that deal directly with teaching, to encourage the development of applied competence, and not assess individual elements of competence in a fragmented way. Teachers must be able to use their foundational knowledge in the classroom, and they must be taught how to do this by the way that they are assessed. It is also important that they are developed as professionals with the tools to carry on learning from experience and reading and other research once they have completed their formal course. So assessment in programmes that develop professional practice must demand that students display mastery of the kinds of processes that will enable them to do this. However, this does not mean that every single course has to do this in the same way, or that there should be integrated assessment across all courses. By all means, course writers should work together in terms of a set of coherent goals; by all means they should cross-reference and build on each other’s work; by all means they should avoid the repetition that dogged preset curricula in the past. But natural opportunities (as suggested by the comments of the face-to-face programme above) should be taken for integration across courses, rather than forced plans which could distort the specific nature of the learning in individual courses. In other words, integrated assessment should be seen as a value, rather than a rigid bureaucratic requirement.

Assessment in authentic contexts and feasibility

There is a further challenge to validity of assessment in teacher development programmes, particularly those that are offered at a distance to students separated geographically from each other and from the delivering institution. The Norms and Standards for Educators pick up the notion of validity in the statement: ‘The assessment strategy assesses the extent to which learners have the ability to teach in authentic and changing South African contexts’. Combined with the obvious fact that the development of practical competence for teacher education programmes involves work in schools and classrooms, this means that some assessment should involve direct observation of teachers in schools and classrooms (and possibly even in other authentic contexts such as union gatherings!).

It could be argued that if the Wits FDE aims to improve classroom practice but does not assess the teaching competence of the students directly in the authentic context, then the assessment cannot be regarded as valid. It might be able through written forms of assessment to assess the students’ ability to design teaching and learning strategies, and it can to some extent assess the students’ ability to interact with the broader teaching community and write in acceptable ways. But the reality of implementation, and the real effectiveness of reflexive action (using what you have reflected upon to inform planning and further implementation) cannot be measured by pen and paper means. One of the programmes that was part of the EDS Project would probably dispute this. They said that:

Problem-solving assignments constitute a sound approach which could be implemented, [but] .. argued that facilitators would have to have access to the classroom in order to assess the degree to which the problem had actually been solved.

Another programme reflects on the difference between the assessment of outputs and the assessment of outcomes:

competence should be measured both as outputs and outcomes. Outputs could refer to things like written documents, whilst outcomes could refer to behavioural changes. The latter, which are in a sense equivalent to practical competence, are extremely difficult to measure.

However, the Wits FDE did not deliberately avoid assessment in authentic contexts. Support in schools was intended, but there was insufficient funding. In other words ‘assessment in authentic contexts’ is not done because it is not feasible or practicable.

In the Educator Development Support Project research, this was a not unexpected theme. Even face-to-face programmes refer to ‘financial and human resource constraints’ with regard to classroom-based assessment.

I would like to argue that in a situation of resource constraint, it might be inadvisable for Government policy to insist upon classroom observation. Even in preset face-to-face colleges where students do teaching practice in local schools, the organization of teaching practice is often not effective. It is difficult to ensure that tutors interpret criteria similarly. Tutors usually observe a proportion of lessons outside their area of expertise, and often see particular students for no more than a single lesson. Tutors are often not involved in the planning process giving rise to the lesson they observe. They are also usually unable to see whether the student has been able to make use of their comments in the improvement of subsequent lessons. In other words, even though students’ practical competence is being observed in an authentic context, the assessment is not integrated. Tutors cannot judge teaching as a process of developing competence – it is usually observed seen as a once off display.

Rather than waste time and money on ineffective assessment of classroom practice, I think it would be much better to put resources into ensuring that the kinds of tasks that students do are likely to develop applied competence. If the purpose of assessment is the promotion of effective student learning rather than merely the measurement of that learning, then one has to consider how the assessment is designed, what it makes the students do, what experiences it exposes the students to, and not merely where the assessor and student are when the assessment takes place. If the assessment requires work in the classroom, discussion with colleagues, observation of each other’s lessons, interviews and classroom based research, it will develop the students’ practical competence. The fact that assessment methods will only be able to infer the degree of practical competence achieved rather than observing it directly is less important than that the quality of students’ learning experience.

Concluding comment

In view of the often ineffective and piecemeal observation of classroom practice, and in a context of resource constraint, providers should concentrate more on creating opportunities for students to acquire competence than on creating opportunities to assess it in authentic contexts.

Validity and reliability

A related issue is the extent to which our measurement whether of outputs or outcomes is reliable – i.e. whether assessors grade students in similar ways, but also whether students can falsify the kinds of responses they provide. If students falsify the kinds of responses they provide, this means not only that the assessor is coming to the wrong kinds of conclusions about what the students are worth, but the student is not taking advantage of the opportunities for learning which the completion of assessment provides.

There are two simple answers to this problem. One is to favour means of assessment which allow for the maximum control both of learners and of markers – the written examination. However, this solution will mean that valid assessment methods are being sacrificed for methods that are more easily managed.

The second simple answer to this problem is that it is easier to avoid student ‘cheating’ if there is assessment in authentic contexts. However, the reliability of assessment made in ‘authentic contexts’ is difficult to control. One face to face programme in the EDS Project argued

That assessment in an authentic school context creates difficulties at the level of reliability and comparability, and that ‘most colleges and schools do not have the management infrastructure to sustain a sufficiently high degree of internal communication to achieve reliability across individual staff and departments responsible for assessment.

A distance education programme which involved a partnership between an NGO and a university to ensure that there was a school-based practical component to the learning and assessment similarly expressed problems with assessment in authentic school contexts:

Given the fact that in South Africa we come from a context where successful completion of a course is crucial to accessing a range of opportunities which are not otherwise available, there is a tendency for South African learners to get through the assessment process at all costs. Hence implementing assessment in an authentic context implies that there has to be a strong element of trust in the relationship between the learners and the "system". Programme team members believe that the element of trust is missing in the relationship between programme providers and learners, and hence it is difficult to creatively combine summative assessment and formative development approaches in an authentic context.

These two points make it clear that difficulties of reliability are not simply solved by assessing in authentic contexts.

In terms of the Wits FDE programme, reliability is addressed in several ways. Firstly the assignments are designed in ways that require individualised responses from the students’ own context. Secondly, students are required to do a range of types of activities in their assignments – design lessons, conduct interviews/research, write their own poetry, write academic essays, observe each other, write reflections on their own teaching. In some activities, it is easy for students to pass off other people’s work as their own (eg personal writing), but in others it is much less easy. Thirdly, assignments are divided up into different stages, with feedback from lecturers along the way. This makes it difficult to ‘cheat’ because what is required is evidence of planning and process. Fourthly, marker reliability and fairness is encouraged by marking procedures that involve processes for arriving at consensus amongst markers around criteria and standards.

Concluding comment

As Kathy Luckett has pointed out current trends in assessment, and particularly in outcomes-based assessment, emphasize validity over reliability. It is more important that the assessment encourage the right kind student learning, than that exactly the right judgements are made about the nature of that learning. What this means for providers is that written examinations can no longer be the only means of assessment because they cannot adequately encourage the development of applied competence. Even if they are less reliable, other methods should be used. If a range of methods is used, then the possible lack of reliability in one method can be balanced by greater reliability of other methods.

Transparency and accountability

Unit standards and outcomes-based qualifications require statements not only of outcomes, but of assessment criteria. The use of assessment criteria could be seen as a means of ensuring marker reliability, but in my opinion, their chief function is increased transparency.

Alison Wolf, draws attention to the fallacy that tightly defined criteria ensure reliability:

The assumption is that, once you have the specifications right, and generate, on the basis of them, a good (‘valid’) test, the process of actually making a judgement about a candidate is unproblematic. However, it is not obvious that this is necessarily the case. Whether the assessor has been responsible for constructing the assessment, or inherits a ready-made instrument, the actual process of judging whether someone has ‘reached criterion’ and can be described as able to do something, can be very problematic.

She makes the point that:

While assessment systems may vary in the degree to which these complex judgements come into play, such judgements are universal to all assessment.

While a set of criteria cannot be interpreted without professional judgement, it is also true that criteria can assist professionals to make their judgements well – particularly if they create or at least participate in the setting of criteria. In a sense, this process is making transparent to themselves the basis on which they are making judgements. If one looks at the Wits FDE marking procedures, tutors share in the process of interpreting the broad criteria set for each assignment by the course coordinator, but they also participate in drawing up a grid of more specific criteria after the assignments have been received (see appendix two).

However, the reason for providing criteria is not mainly for markers to be transparent to themselves, but for course writers to be transparent to the learners about what is required of them in meeting the outcomes. Part of this transparency is to do with the importance of accountability – there is an increasing imperative for educators to be accountable to their learners about the basis on which marks are awarded. In this respect the Wits FDE assessment strategy could do with a little tightening up – not all courses provide assessment criteria, even though all courses clear instructions about how to do the assignments.

Another mechanism to improve accountability is to involve a range of parties in the assessment – not only the external examiner and the tutor, but other students, perhaps, or the student himself, or perhaps involvement of people outside the delivering institution, in schools or in the community. At the moment the Wits FDE uses a limited form of self and peer assessment. Self-assessment is achieved through self-assessed activities in the materials and through the preparation of a course portfolio, which involves the learner in some reflection on his/her own work. However, if students are really going to learn to assess themselves and each other, there would need to be much more carefully worked out assessment criteria across the various courses.

Concluding comment

Through ensuring transparency and accountability, educators can give students ways to take responsibility for their own learning. This is crucial in distance education programmes where independent learning is a necessity rather than merely being desirable. Assessment criteria provide students with tools to assess their own and each other’s learning, as well as understand the basis for judgements made by their assessors.

Conclusion

It is impossible in the space of a short paper to discuss all the important aspects of an assessment strategy. There are notable gaps in this discussion – recognition of prior learning has not been referred to at all; and there has been no full discussion of quality assurance. However, certain of the critical issues for teacher education programmes offered at a distance have been dealt with.

In summary, it has been the contention of this paper that providers should not interpret the emerging policy criteria as requirements to be followed slavishly, but should step back and understand the values behind these criteria. Providers need to be able to describe their assessment strategy in more detail than they are accustomed – both in terms of the number of elements of the description and in terms of principles underpinning the strategy. Finally, in evaluating an assessment strategy in a teacher development programme, the following key pointers have emerged from the discussion:

• It is competence we should be assessing, not atomised bits of content or individual outcomes. What matters is how you think about and use knowledge, rather than how much of it you possess.

• We shouldn’t worry so much about reliability that we forget about valid and varied forms of assessment of that competence.
• We also shouldn’t rush into assessment of classroom practice in authentic contexts before considering whether we couldn’t spend our limited resources better on increasing the number and kinds of assignments we offer to our students and the quality of support and feedback we give.
• We should be concerned about transparency and accountability, not because the policy says that we should be, but because it can enhance the quality of student learning.

 References

Centre for Educational Technology and Distance Education, May 1998, Criteria for Quality Distance Education in South Africa: A Statement of Policy (a draft document).

Coates, Bob, 1997, comments made at a workshop on the Open University Maths 101 Foundation course held at SAIDE in November 1997

Department of Education, 1998, Norms and Standards for Educators, prepared by the Technical Task Team on Norms and Standards for the Department of Education, Pretoria.

Luckett, Kathy, 1998, material provided at an Assessment Workshop conducted at the University of Natal, 15-16 April 1998

Musker, Paul, 1999, Educator Development Support Project: Final Report, Paul Musker and Associates for the Joint Education Trust and the Department of Education

SAIDE, 1998, Strategies for the Design and Delivery of Quality Teacher Education at a Distance: A Case Study of the Further Diploma, English Language Teaching, University of the Witwatersrand, President’s Education Initiative and Joint Education Trust

SAQA, (undated) Criteria and Guidelines: Providers (Draft 1)

Sewart, David, 1996, comments at ‘Quality Assurance for Distance Learning’, a conference held 24-26 September 1996, at the Grosvenor House Hotel, Sheffield.

Wolf, Alison, 1993, ‘Assessment Issues and Problems in a Criterion-Based System’, A Further Education Unit Occasional Paper, Institute of Education, University of London.

 Appendix One

Design

Teachers should be able to:

1. Design teaching, learning and assessment strategies with the needs, interests and contexts of the learners in mind.
2. Design teaching, learning and assessment strategies with a clear purpose, and methods appropriate for the achievement of that purpose.
3. Design a range of activities that engage the learners.
4. Develop appropriate teaching and learning materials.
5. Develop appropriate ways to assess the learners.

Implementation

Teachers should be able to:

6. Facilitate learning through the organization of the physical environment in the classroom and the effective manipulation of teaching aids and learning materials.
7. Ensure learner participation in the lesson through skilful questioning and an appropriate blend of individual, pair and group work.
8. Facilitate the learners’ grasp of key concepts and processes through a range of methods which facilitate the construction of knowledge and provide sufficient scaffolding and reinforcement.
9. Manage the discussion in the lesson in such a way as to encourage critical enquiry and sensitive response to differing viewpoints.
10. Respond constructively to learners’ varying needs, interests and difficulties.

Reflection

Teachers should be able to:

11. Refer to and use current concepts in education in South Africa and the rest of the world in order to reflect on their own practice.
12. Gather sufficient and relevant evidence to form the basis for reflection on their own practice.
13. Reflect constructively on the success or lack of success of the teaching and learning strategies they implement.
14. Plan improvements to teaching, learning and assessment strategies they use as a result of critical reflection on experience as well as in relation to concepts they have learned.

Interaction

Teachers should be able to:

15. Conduct both literature and empirical research.
16. Read a variety of texts critically.
17. Write coherently in the appropriate academic style.
18. Use appropriate technology for research, communication and teaching purposes.
19. Talk about education and schooling appropriately to a range of audiences, including parents and colleagues.
20. Interact with a range of formal and non-formal educational providers and use the services and resources appropriately in their teaching.
21. Contribute professionally within their schools and the broader teaching and learning communities of which they are part.

KNOWLEDGE

The programme aims to develop the teachers’ knowledge which naturally is given expression through the various abilities described above. However, the extent of the knowledge in the English courses in year one (relevant to the sample of teachers selected who are only now entering year two) needs to be specified.

Teachers should have knowledge and understanding of :

• theories and processes of language learning and acquisition in the multilingual South African contexts;
• literacy events and literacy practices in the social contexts of the home and the wider community as well as school literacy;
• a range of strategies to develop oral language in the English class based on understanding of classroom interaction — the value of talk for learning, the contribution to learning and to language acquisition of effective group work, the contribution of students’ main language in talking to learn English activities, the various roles of the teacher in stimulating and supporting classroom talk, and the importance of questions and tasks which encourage the development of listening and speaking skills;
• a range of strategies to develop the reading and writing competence of their learners —approaches to the teaching of reading, learning to read in an additional language, reading to learn, becoming a critical reader, the challenges all writers face, ways of supporting learner writers, and the genre-process debate in regard to writing development;
• the basics of English grammar, and an ability to design lessons in which grammar is understood and used for meaning and in meaningful contexts.

Wits University, Further Diplomas in Education: "Curriculum and Classrooms": Assignment 3 - Final Marking Grid

© Karin Brodie

NADEOSA Homepage |  Ist NADEOSA Conference Papers Index  |
Global Distance Education Network : South African Resources

Send comments on the web site to the web designer