CHAPTER SIX
Assessment in the Information-Based Policy Analysis and Decision-Making Course

INTRODUCTION (This framework for the analysis of assessment design, support and quality assurance comes from previous research conducted by SAIDE. (1998).: Strategies for the design and delivery of quality teacher education at a distance: a case study of the Further Diploma in Education (English Language Teaching), University of the Witwatersrand.)
It could be argued that unless an effective assessment strategy is in place successful teaching and the use of expensive technology can have little impact on student learning. Assessment tasks compel students to engage with what has been taught, and the way assessment tasks are set encourages learning processes that are either helpful or unhelpful, that contribute to the goals of the programme or that distract students from achieving them.

Usually, an assessment strategy is evaluated in terms of its validity, reliability, fairness, and feasibility. While we will not be commenting in depth on all these criteria we would like to place our reflections within this broader framework, and where appropriate expand on some of them.

Validity has to do with setting meaningful goals for student learning and then designing effective ways to assess student achievement of those goals. Kathy Luckett outlines the major questions around validity: (Luckett, K. From an overhead transparency displayed at a workshop on Assessment, at the University of Natal in April 1998.)
    'Are we assessing the right things? In other words, are our learning outcomes (our construct) the right things to be assessing?'
    'Are we assessing the thing right? In other words, does the assessment assess (the construct) what it claims to assess? '

Our assumption is that the learning outcomes, which have been identified for the two modules we have chosen to evaluate, are the right ones. Our focus will be on the second question in evaluating the assessment design.

Reliability and fairness are about quality assurance measures set in place to ensure that standards are maintained from year to year and across the numbers of students enrolled in the programme. Very often there is a tension between reliability and validity in assessment design. More valid ways of assessing complex goals (e.g. sharpen analytical capabilities rather than a more simple outcome such as design, set up and conduct a personal interview) are often less reliable than standardized objective tests which enable one to generalize about student achievement at different times and places and with a range of assessors.

We will review the mechanisms, which are in place to ensure that there is a measure of reliability and fairness in assessing the students’ submissions.

Feasibility has two dimensions – designing assessment strategies that are manageable in terms of workload for staff and complexity and expense of implementation; and designing assessment strategies that are feasible for the students – is there enough support to enable them to succeed in completing the assessment successfully? Again, as with reliability, there is often a tension between validity and feasibility. In terms of this programme which aims ‘to enhance the capacity of experienced managers to: expand their managerial knowledge, test and explore new ideas, sharpen analytical capabilities, broaden awareness of the complex issues that influence decision making, it would be desirable to incorporate assessment of work performance’(P&DM Brochure. (undated). The Master of Management Degree in the Field of Public and Development Management MM (P&DM).). However, this may not be feasible for the programme at the moment, even though there is evidence that some of it is happening in an ad hoc way in Mpumalanga. A question, which needs to be explored, is whether the present assessment is able to achieve the development of functional management competency without an explicit work place assessment component.

In programmes which use distance education methods, there is a further dimension to the evaluation of an assessment strategy – teaching through assessment. Because of the limited contact between student and lecturer or tutor, it becomes important to use assignments not just to grade students, but as a major form of teaching. As is pointed out in the Tutor’s Guide prepared by SAIDE, (SAIDE. (1998). Supporting Distance Learners: A Tutor’s Guide, Cape Town: Francolin)
You can think of teaching through assignments as a learner-led dialogue between yourself and the individual learner. This dialogue starts with the learner’s work, and continues with your feedback, and their response to your comments.

As pointed out elsewhere in this report, the Master of Management in Mpumalanga is not strictly speaking a distance education programme. In theory, there is opportunity in the lecture sessions to give feedback on assessment. However, in practice many of the assignments tend to be marked just before the examination for the course, and students need feedback to inform their writing of the examination. Although, the use of individual feedback on assignments is a distance education practice, it can usefully be incorporated in the teaching support offered to students in this programme. This type of dialogue with individual students is of particular relevance in a programme which seeks to provide a small number of students with a stimulating and rich learning environment.

To summarize: if you do not design assessment properly, students will not achieve what you want them to achieve. If you do not support students adequately, they will not achieve what you want them to achieve. If you do not have effective quality assurance procedures in assignment marking and management, some students will achieve what you want them to achieve, and some will not.

In this chapter we will examine the assessment design and support in two modules of the Information-Based Policy Analysis and Decision-Making Course. The decision to concentrate on these two modules was made on the basis of availability not only of the assignment and examination topics, but also of a sample of the actual assignments produced by the students. The comments made on quality assurance refer to the programme as a whole.

Assessment design, support and quality assurance were investigated in terms of the following criteria:

- Design
Assessment needs to designed in such a way that it contributes to the achievement of the programme goals, mastery of the course content, and development of broad abilities both in terms of the product expected and in terms of the process students go through to achieve that product. Another aspect of assessment design is the extent to which it supports students to achieve assessment requirements by providing clear instructions and criteria, and scaffolding and support in the achievement of the tasks.

- Support – teaching on assignments
The commenting on assignments needs to be an effective means of extending the teaching on the course and providing supportive formative feedback.

- Quality assurance
There need to be mechanisms in place to ensure that the marking is fair and reliable, particularly if there are different markers, and there also need to be mechanisms in place to ensure that the turnaround time on assignments is kept to a minimum.

These aspects will be evaluated using information from assessment documentation, interviews with course convenors and lecturers, and then by analyzing student opinion collated from the focus group interviews held. For the section on feedback to students a sample of individual and group assignments will be examined.

According to the course convenor, the Information-Based Policy Analysis and Decision-Making Course consists of three interrelated modules which are weighted as follows:

The university’s assessment structure applies to all courses in the programme.

The following guidelines have been set up by the Faculty of Management for each course: (P&DM. (1997). Teaching and Learning Policy. Johannesburg: P&DM)

- The examination must constitute a minimum of 50% of the overall course mark and must be an individual examination.
- The Faculty Board has placed an upper limit of 25% of the total course mark on syndicate work (presentations, assignments, etc.).
- The balance of the work for the course will make up the remaining marks to be allocated. This must be made up of individually evaluated work such as the examination, tests, or individual assignments.

The assessment activities designed for the three modules of the course are as follows:

As is described in chapter two, the most reliable statement of programme purpose and objectives is presented in the brochure to students:

Public and development managers are confronted with rapidly changing political, economic, technological and social challenges that test skills, knowledge and decision making capacity. The purpose of this degree is to enhance the capacity of experienced managers to: expand their managerial knowledge, test and explore new ideas, sharpen analytical capabilities, broaden awareness of the complex issues that influence decision making, interact with peers and leading academics and practitioners and reflect on their personal goals, values and careers. Graduates of this degree will be managers who can contribute to, and act within, a democratic, accountable and responsive public and development environment. Course participants will also complete an in-depth policy research project. (P&DM Publicity Brochure)

However, interpretations of this general framework vary, as do the understandings of what is implied by the requirement to state learning outcomes for a module.

An overarching goal for this module is to equip the students with the theoretical and practical knowledge and experience, skills and ethos to act as a professional in advising clients on the facilitation of public policy approaches and processes. The learning outcomes for the module are as follows.

By the end of this module, participants will be able to:
- use policy relevant concepts and terminology
- critically engage in strategic debates relevant to public policy and information management
- facilitate policy processes in which adequate attention is given to the interplay between public policy, analytical process and decision-making
- facilitate institutional arrangements necessary for managing policy processes
- balance the need for rationalism and a normative approach to public policy
- conceptualize and understand the nature of policy analysis. (Course Pack compiled by Christo de Coning, August 1998, Introduction to Public Policy: The role of information, p. 4.)

The assessment design contributes to the achievement of the above outcomes in the following positive ways:

1. The most important point to be made is that the assignment and examination provide students with an opportunity to explore work related issues and problems. Grappling with real issues not only makes the learning experience more meaningful, it also enables students to use the learning environment to deepen their understanding of the issues. Such an approach respects students both as adults and experienced managers. For their assignment, students chose topics such as ‘Evaluation Report On The Envisaged Policy Process For The Development Of An Information Management Policy For The Mpumalanga Province’; ‘ A Policy Management Initiative of the Maputo Corridor’; ‘Disaster Management Policy’. The assignment brief required students to present a good understanding of their chosen case, to discuss the issues within relevant theoretical frameworks, arrive at specific conclusions, put forward options and recommendations on how public policy processes could be improved. The focus on work place relevance is echoed in the examination paper where two of the three questions are linked directly to the work environment and demand that students demonstrate a number of abilities which they require as part of their management role, for example, analyze context critically, identify key issues, prepare and present written presentation of reflections and suggestions in the form of a proposal or evaluation report.

2. Linked to work place relevance is the theme of integrating theory and practice. Students are encouraged to analyse and understand their own context and apply theoretical frameworks critically and selectively. The assignment and examination questions demand of students that they examine the strengths and weaknesses of chosen frameworks for application within specific contexts. The open book examination strongly supports the practical application approach adopted in the assessment strategy.

3. What is interesting about the assessment design for this module is that it encourages students to integrate the knowledge and skills they acquire in the other two modules. Students are expressly urged to consider integrating useful frameworks from the other modules in the course. Both the assignment and the examination questions are structured in such a way as to compel students to include the most relevant information gathering techniques as an integral part of policy development and analysis processes.

4. Another feature of the assessment in this module is the linkage with the research project which students are expected to do in year two. In the assignment brief students are advised to consider assignment themes that will assist them with background material for their research projects.

5. Students have access to the lecturer and their learning groups to help them identify a case and theme for their assignment. The lecturer advises that students do not immediately choose a case at the beginning of the module, but that they wait until they have some insight into the policy development and analysis processes. He suggests that students make their selection by the 4^th session. (There are six sessions in this module.)

1. Two of the learning outcomes for this module have not been adequately covered by the assessment strategy described above. These two outcomes refer to actual facilitation of processes:
- facilitate policy processes in which adequate attention is given to the interplay between public policy, analytical process and decision-making
- facilitate institutional arrangements necessary for managing policy processes.

A criticism of the assessment design which focuses entirely on written submissions is that though students may develop excellent ways of describing the processes and indicating comprehensively what is involved in planning and implementing them, it is by no means assured that they will actually learn the competencies required for facilitating them within their own work environment.

3. In addition, the provision of three choices in the examination paper, one of which is a theoretical discussion, could mean that students can avoid engaging with the practical dimension. If it is regarded as important for students to engage with real issues within their own context in order to extend their managerial expertise, then the assessment design does not fully succeed for those who choose the more theoretical option.

3. A shortcoming which was apparent from reading three of the assignments submitted was the students’ lack of ability to integrate fully theoretical frameworks and models within real contexts and practical realities. Theoretical components were described but not assessed critically, and more importantly the rationale for including them was not fully argued. In order to be guided to perform the assessment tasks satisfactorily, students need not only clear instructions and a range of means to support them in achieving the requirements: they also need clear statements of criteria in terms of which they will be judged. These are noticeably absent. Students only learn after the fact from the kinds of comments made by the marker what the criteria are against which they were judged.

4.Communicating clearly and unambiguously and presenting ideas in a logical, convincing and persuasive manner are skills which students need to acquire as part of their repertoire of management competencies. On reading some of the students’ assignments it is obvious that there is a real danger that they fall into the ‘policy speak’ trap. In some cases the over use of terminology obscured the message and one was left with an uneasy feeling of confusion.

5. In the absence of an assessment criteria marking schedule, it is difficult to criticise this aspect. However, attention needs to be paid to it when assessing students’ submissions.

The overall learning outcome for the above module is described in the course pack as:
    'By the end of the course participants would have been introduced to managing the techniques of data collection, which are most often used in policy analysis and policy research.'

The assessment design contributes to the achievement of the above goals in the following positive ways:

1. The task set for the learning group in the assignment is placed within a context with which students are familiar and with which they have to deal as managers. They are asked to commend to a group of senior civil servants in a local council methods of gathering information to feed into their planning process to achieve an integrated development plan. Simulation of the demands of the work place is particularly meaningful and useful for students in this programme.

2. The choice of a group assignment is appropriate for this module because it mirrors the interactive nature of collecting information. As students engage with one another in a collaborative process to plan, collect, and present information for this assignment they are learning valuable skills inherent in managing the information gathering process. This aspect is emphasized in the document outlining the learning and teaching approach envisaged for the master’s programme: ‘Syndicates enable students to share experiences and deal with some or many of the problems of real life management in the context of group dynamics. This means that they are able to experience the frustrations of real management in a simulated context.’ However, specific criteria and techniques for assessing collaborative work should be developed. Students should also know in advance that the collaborative component will be assessed.

3. The integration of theory and practice is central to both the assignment and the examination. Students are asked to demonstrate understanding of the different information gathering techniques and are challenged to apply them selectively within different contexts. As in the previous module, the open book examination strongly supports the practical application approach adopted in the assessment strategy.
The inclusion of the newspaper article from Business Day in the examination task helps to locate the task in the ‘current reality’ in which students are working. Why not include the date on the extract? The article highlights the conditions in which local councils have to operate to provide their services. If students are to expand their managerial expertise within a climate of change they have to keep abreast of current developments and respond appropriately. This links to one of the overall learning outcomes of the programme.

1. Generic comments about the group assignment made by the lecturer who marked the assignments reveals that:
- In general the assignments show that there is an understanding of the concepts introduced in the module.
- There is little substantiation for why the two/three methods have been selected. How do they complement each other? How will they yield the most pertinent and relevant information?
- There was also a general lack of application of the information gathering techniques.

When there are shortcomings that are fairly widespread, it is necessary to view things from the students’ perspective and ask oneself: Was it clear to the students that they had to provide a rationale for the methods they selected? Were they aware that they had to explain how these tools would generate the necessary information within the given context? There may be two reasons why students did not provide this type of detail. One is that they were unprepared for the task, and the second is that the assignment brief could be faulty. Assuming that the students were adequately prepared, let’s focus on the assignment brief. Was it comprehensive enough? Did it make the application to context sufficiently clear?

From a reading of the assignment brief it seems that there may be a problem here.

Students were informed that they needed to advocate a local council of their choice about at least two information gathering methods that they had been exposed to. They were to discuss these methods according to specific criteria, including appropriateness of method. However, they were also told that ‘this exercise does not require finding information but instead requires an analysis of the various tools/methods that can be used to find the information.’ Herein lies some ambiguity. It is obvious that one can only determine the appropriateness and effectiveness of the methods one proposes if one has a clear understanding of the actual context in which they will be used. This was not explicitly stated. It can be argued that it should not have to be spelled out because one assumes that at this level students are able to understand that knowledge of the context has to be considered when making these kind of recommendations. As it is obvious that students have struggled to apply knowledge about information gathering techniques sufficiently, further support in this area is indicated.

2. The present assessment methods are helping students to understand the concepts introduced in the module but are they sufficient to enable students to achieve the overall goal of the module which is to manage the techniques of data collection which are most often used in policy analysis and policy research? Matching the right information gathering techniques and methods with the right context is a crucial management competency. The assessment methods used presently do not seem to assess the student’s ability to manage information gathering adequately.

It is clear from the features of assessment design recorded above that the assessment is work related and that every attempt is made by lecturers to integrate theory and practice. The assignment tasks and examination questions do by and large approximate activities/skills that form an integral part of the student’s work situation. The assessment of these tasks could be improved by including assessment criteria which are more explicitly linked to the overall outcomes of the course and which reflect a good balance between theory and practice. Group assignments lend themselves to the evaluation of actual management competencies required by students in their work environment. The collaborative process should be included as an assessment component and creative techniques should be designed to assess the skills involved in the group process.

The MMP embraces a constructivist approach to learning with a focus on ‘interactive, discussion based methodology designed to achieve specific learning objectives and develop independent thinking and problem solving skills. ( P&DM. Teaching and Learning Policy. op cit.) As such, feedback from lecturers is central as a teaching mechanism:

Learning is far more than simply ‘being taught’. To learn, we need to plan what we’re going to do; attempt to do it; and then receive feedback on our work. We then use this feedback to improve the work we have just done, or, more often in education, to ensure that the next work we do embraces what we have learned. (Baume, D. and Baume, C. (1996). Learning to Teach: Assessing Students’ Work. Oxford: The Oxford Centre for Staff Development. p.10. )

Rowntree states that:
    'The assessor may respond as a teacher, using the knowledge gained to interact with the student in helping him grow; or he may respond as a reporter, classifying, labelling, or describing the student for the benefit of others who have an interest in the student.^' (Rowntree, D. (1993). Assessing Students: How Shall we Know Them?. London: Kogan Page. pp. 199-200)

P&DM’s learning and teaching policy document advises that: (P&DM. Teaching and Learning Policy. op cit)
    ‘All (individual) assignments and exercises must be returned with appropriate written comments and verbal feedback which will enable the student to learn from the assessment process and develop their skills.’

    ‘All syndicate assignments and exercises must be returned with a list of written comments which will enable syndicates to learn from the assessment process and develop their skills.’

Comments on assignments can be grouped into two broad categories: comments which are written in the margin, paragraph by paragraph; and comments which are made on the assignment as a whole. Whatever the category of the comment we cannot underestimate the importance of written comments in helping students to grow and develop.

In distance education, in the absence of face-to-face contact, written feedback should be regarded as a teaching opportunity. Rowntree says about feedback in a distance education context:
    'It has to be more considered and articulate because it must carry all the weight of the teaching that other teachers are able to share between written comments and spoken amplification. ^' (Ibid. p. 209.)

While we recognize that students have opportunities for face-to-face contact with lecturers in the MMP, the practice of providing students with detailed comments on assessment is good educational practice. It can be used in the programme as a whole but especially for students in Mpumalanga as a mechanism for helping individuals to develop their academic and managerial competence.

To analyse the feedback on assignments, we looked at the comments made on seven group assignments of the module Information Gathering for the Policy Process, and three assignments from Introduction to Public Policy. (Assignments from the other two modules in the third compulsory course were either not yet available or had not yet been submitted and/or marked.) We fitted them into categories of what is commonly regarded as good feedback practice. The comments and categories appear in the boxes overleaf.