Techniques of Quality Assurance

 Overview 

These materials support a discussion on the topic of quality assurance techniques that are appropriate to educational organisations. First, the materials review the steps that comprise the continuous cycle of quality assurance implementation, and then present five tools which you may find useful for improving quality.
Second, the materials discuss the advantages and limitations of using formal, published standards in implementing your quality assurance scheme. The materials then discuss the concept of 'performance indicators', which are part of many quality assurance schemes in education. 
The materials close with two practice exercises, one on performance indicators and the other on applying quality assurance tools.

Source materials for this topic

Davis, D. The real world of performance indicators: a review of their use in selected Commonwealth countries. London: Commonwealth Higher Education Management Services, 1996.
Guri-Rosenblit, S. Quality assurance procedures at the Open University of Israel. In A. Tait (ed.) Quality assurance in higher education: selected case studies, p. 32. Vancouver: The Commonwealth of Learning, 1997.
Robinson, B. Assuring quality in open and distance learning. In F. Lockwood (ed.), Materials production in open and distance learning, pp. 185-94. London: Paul Chapman, 1994.
Tait, A. (ed.) Quality assurance in open and distance learning: European and international perspectives. Cambridge: Open University, 1993.

 Tools for improving quality 

The quality assurance cycle

At a practical level, actually doing quality assurance involves a continuous cycle that comprises the following steps:

· setting standards for a key activity;
· carrying out the activity;
· judging achievements against the standards;
· planning for improvement; and
· taking action to implement desired changes.

Some quality assurance tools

Here are five tools that you may find useful in implementing this cycle. They were developed by Bernadette Robinson for workshops delivered for IEC, and later published (Robinson 1994).
Tracking the source using the five whys
The first reason given for a failure of quality may not get to the heart of a problem, yet typically this is where most questioning stops at this point. The reasons need to be tracked back to source. One simple technique is to ask 'why?' five times (more if necessary) in answer to each response. This gets to a deeper level of understanding and analysis.

Example: The institution has a system of 'second marking' for all assignments submitted by students enrolled in its M.A. course. This means that part-time tutors mark an assignment and then send it to central office for second marking by a member of the full-time teaching staff before it is returned to the student. In one instance a full-time faculty member
reduced the grade the tutor/first marker gave to an assignment by only one percentage point, without consulting the tutor. This upset the tutor, making her feel demeaned and incompetent.
Why did the faculty member do this?
Because she wanted to make it appear that she was more than a 'rubber stamp'.

Why?

Because she came from a university system that does not routinely involve second marking, and did not understand the role of second marker.

Why?

Because this role had never been discussed with her or thoroughly explained.

Why?

Because the director of the programme took it for granted that all universities operated this way and that hence all faculty members would understand the role of second marker and the procedures they were to go through (for example, the obligation to discuss any differences of opinion with the first marker).

Why?

Because assuming something to be the case is easier than checking it out when one is exceedingly busy with this and other duties, including a great deal of travelling.
And so on.

Flow diagrams

Flow diagrams are useful for mapping processes so they can be understood. Flow diagrams provide a way of tracking and displaying how quality is shaped through several stages or phases. As Robinson (1994:190) points out, flow diagrams are useful for:

· systematically recording steps, decisions, and activities required in a sequence;
· providing a clear diagrammatic representation of a process as a way of sharing information

  about what happens;
· identifying critical points or bottlenecks;
· displaying the consequences of planned change;
· standardising practice;
· training; and
· tracking and diagnosing the sources of failure.

The following flow diagram presents the process of course approval at the Open University of Israel (Guri-Rosenblit 1997:32).

A flow diagram: Course approval procedures at the Open University of Israel 

 

An economist called Pareto is quoted by Robinson (1994:191) as suggesting that 80 percent of problems arise from 20 percent of causes. If you track the causes that create most failures of quality, you can concentrate your efforts on those areas that pay the most dividends. To apply this, you need to:

· identify the problems;
· try to quantify them; and
· use this information to make a simple bar chart which ranks the categories.

Example: An educational organisation tracks the nature of complaints received on a student 'hot line' from learners enrolled on their courses, categorises them, and records their frequency. The results of this tracking over the first month of classes are set out below. The fact that the top five complaints all relate to the bookstore operation gives managers a major clue as to where to focus their efforts in order to improve the quality of service to students.

A Pareto Analysis Bar Chart

Top Five Complaints	Frequency 50 100 150 200 250 300 350
Set texts not available at start of course.
Not enough set texts available for number of students in class.
Hours of bookstore inconvenient for students who work part-time.
Far too many set texts prescribed at too high a cost.
No used-book sales scheme in place.

 

Fishbone diagrams

A ‘fishbone’ diagram maps in a structured way:

·     the causes of a problem, both major and minor;

·     the effects; and

·     the factors involved.

It is useful in helping to diagnose and analyse problems with colleagues, especially when there is more than one cause. To use this you need to:

·    identify the problem in a few words and put this as the head of the fishbone;

·    identify the main issues or causes (four is a manageable number);

·    under each of these four, explore the component factors or sub-causes (using a ‘brainstorming’ technique or the ‘five whys’);

·    discuss the completed diagram, trying to distinguish between symptoms and causes; and

·    agree on the one main cause and highlight it so that plans for remedies are focused on it.

 Here is a sample fishbone diagram. The problem under discussion is put into the box at centre right. 

A Fishbone Diagram

Milestones and barriers

This technique charts the learner’s progress through the institution or programme to identify key milestones and barriers. Once identified, the processes that converge on these milestones and barriers are mapped and examined in detail. They are then reviewed and evaluated. This highlights critical points in systems, from the learner’s point of view. The same can be done for other key players, for example,

·     corporate clients;

·     tutors; and

·     support staff.

The points of hand-over or junction between stages or players are often critical barriers or milestones. Following is an example.

A Milestone Map

 

 

 Using formal standards 

What are formal standards?

Formal standards are nationally or internationally agreed rules for judging quality and excellence. They

· aim to state clearly the way in which an activity is to be performed and how it is to be

  measured, verified, or evaluated;
· encompass basically the cycle: plan, do, check, and act;
· aim to bring consistency to products, services, and processes; and
· ensure that what is specified is what you get but say nothing about the quality of a product

  itself.

Example: BS (British Standards Institution Quality Systems) 5750 is a series of British standards for quality management. The standards also exist as EN 29000 (European Norm) or in international version (ISO 9000). Some of the standards in the series provide definitions and guidelines; others are used for external quality assurance purposes, such as certification. All versions are more or less equivalent.

What advantages do formal standards offer?

Because formal standards originate in manufacturing, their application to education and training is not straightforward. Nonetheless, they can prove to be useful in:

· providing a framework for an organisation's own system for managing quality;
· providing an external reference point;
· enabling comparisons to be drawn between stated policy and actual practice;
· facilitating communication in transactions;
· regularising procedures;
· clarifying expectations and contracts;
· assisting in the definition of roles and responsibilities;
· identifying ownership of processes and the accountability of individuals for them;
· reducing uncertainty;
· offering a framework for audit and review; and
· carrying external validation and recognition (formal certification).

What are the limitations of formal standards?

Unfortunately formal standards also have limitations in that they

· may fail to provide a holistic view of an organisation's quality assurance management;
· may be mechanistically implemented, resulting in a reductionist view of quality;
· can involve too much paperwork so that documentation becomes an end in itself;
· may omit some important aspects of the product;
· are no guarantee of the quality of the product;
· may fail to take sufficient account of professional judgement and expertise;
· are static while needs and the environment are not;
· involve a heavy investment of staff time; and
· can create a 'checklist culture' .

 Performance indicators 

Terminology

Distance educators increasingly are being required by their funding agencies, governments in particular, to develop sets of indicators by which institutional performance can be measured. 
This exercise is closely tied to the process of quality assurance. Two important distinctions must be made between these processes, however: 

· quality assurance is primarily focussed on quality, whereas performance indicators tend to

  focus on quantitative measures; and
· quality assurance tends to be primarily an internal activity, whereas performance indicators

  tend to be externally driven and mandated.

The term performance indicator is rather loosely and variously used. There is general agreement, however, that performance indicators, although based on the same data as management information, are clearly evaluative measures, clearly related to institutional or sector goals.

Here is a working definition:

Performance indicators provide a measurement for assessing the quantitative performance of a system.

Discussion: Again, you may wish to begin this discussion by drawing out participants' own definitions of what constitutes a 'performance indicator' and using those as a basis for what follows.

Audiences and purposes

There are many audiences for performance indicators, each with particular purposes and needs. Some of these audiences and their purposes are set out in the following table.

Discussion: You might wish to first draw out participants' ideas of who constitutes the audiences for performance indicators and the purpose these indicators serve, in the context of their own programmes.

Performance Indicators of Various Audiences 

Audience	Purpose
Units and institutions	·      Internal management
	·      Comparison with other units and institutions
	·      Marketing, image building
	·      Evaluation of teaching and research activities of individuals and departments
Government	·      Accountability
	·      Policy and planning
	·      Allocation of resources
	·      Funding
	·      Value of investment in research
	·      Human resources planning
Public	·      Accountability
Learners	·      Institution choice
Industry	·      Research funding
	·      Graduate employment
Research councils	·      Selective distribution of research funds

 

As you can see, the purpose of quality assurance exercises and the development of performance indicators overlap a great deal. But, again, the performance indicator exercise is most often performed in response to pressure from external agencies. Performance indicators become the object of controversy when they are used as a ranking device to allocate esteem and resources differentially.

Categories

A number of different approaches can be taken to the development of performance indicators that provide a basis for their categorisation. One common approach is based on a production model. Another is based on internal and external measures.

Discussion: What categories do your participants use now?
Measurement of input, process, and output

· Input: resources used (for example, financial and physical facilities, learners and staff);
· Process: management of teaching, research, and services; and
· Output: products of teaching, research, and services.

Internal, external, and operational indicators

· Internal: market share of undergraduate applications, and graduation rates;
· External: first destination of graduates, publications, and citations; and
· Operational: unit costs, staff-to-learner ratio. 

Key programme indicators

The following table lists some key programme indicators that are used in these approaches.

Key Programme Indicators

Indicator	Purpose
Learner indicators	·      Population
	·      Entry qualifications
	·      Progression and completion rates
	·      Destination
	·      Learner satisfaction
Staff indicators	·     Qualifications (for example, staff with Ph.D.)
	·      Gender balance
	·      Age ratios
	·      Academic-to-support staff ratios
	·      Value of investment in research
	·      Manpower planning
Resources and finance statistics indicators	·      Operating funds
	·     Research funding
	·     Other income
	·     Staff-to-learner ratio
	·     Expenditures on academic centres, central administration, and library
	·     Different ratios of income-to-expenditure
	·     Other selected financial ratios
Research indicators	·      Number of research students
	·      Research funding
	public sector research funding
	industry research funding
	total research income per academic staff member
	research expenditure per academic staff member
	ratios of research expenditure and income
	·      Publications
	number of journal articles
	number of books
	other publications and conference papers
	·     Patents and licences
	income earned
Estate management and physical resources indicators	·      Space utilisation
	·      Performance in maintenance, improvement, and capital expenditure

 Characteristics of 'good' indicators 

There is general agreement on the criteria for good indicators. They should be:

· relevant;
· able to be updated;
· based on reliable figures;
· understandable; and
· valid, that is, measure what is supposed to be measured.

Indicators should be clearly related to the defined functions, objectives, and mission of the institution. Where there are common indicators across a system, however, different values might be put on low or high figures by different institutions, reflecting different objectives. 

Example: High learner-to-staff ratios are desirable in open and distance learning, since one of the advantages of open and distance learning provision is its ability to make a relatively small number of teachers and other experts available to a wide population. Conventional institutions, however, may well pride themselves on low learner-to-staff ratios, as an indicator of small class sizes and hence a personalised and individualised approach to learners.

Concerns about indicators

There are also many concerns about performance indicators. Among these are:

· the costs of providing additional data when the data required cannot be extracted from

  existing information;
· an emphasis on one particular aspect of performance (e.g. financial performance) at the

  expense of others;
· the inappropriateness of trying to rank institutions on the basis of performance indicators

  when institutions have different objectives;
· the tendency to use performance indicators in isolation rather than in conjunction with

  measures using professional judgement such as peer review;
· the fear that diversity among higher education institutions might be lost as institutions seek

  to maximise performance on the same set of indicators;
· the imposition of indicators may be used as an instrument of control by government;
· the limited value of indicators in measuring 'quality' including the quality of teaching and

  learner outcomes; and
· the fear that the use of performance indicators will stress efficiency and economy rather than

  quality.

Discussion: What concerns do your participants have about the use of performance indicators in their programmes? What indicators of performance seem to be important to the institutions profiled in the case studies?

 Practice exercises 

Listing performance indicators

Instructions: Divide participants into small working groups. Ask each group to

· review the list of key institution indicators, and the list of concerns about performance

  indicators;
· draw up their own list of concerns focusing on the difficulties that distance education

  programmes are likely to encounter when they are compared with conventional programmes

  in terms of the same set of indicators; and
· present their lists to the group as a whole.

Timeframe: Approximately three-quarters of an hour, including a half hour for discussion and fifteen minutes for presentations, depending on the size of the group.

Materials required: Flip chart paper and marker pens.

Applying quality assurance tools

Instructions: Divide participants into five working groups. Assign one of the 'tools' discussed in this section to each group, and ask them to use it to analyse and map or diagram a problem that is characteristic of the institution or institutions from which they come. Have them present their diagrams to the group as a whole when they have completed them. Use the diagrams as a springboard for discussing the strengths and limitations of each approach.

Alternatively, after you have discussed each tool, ask participants to pair off and try their hand at using the tool to identify more clearly some problem in their organisation which they would like to solve. A few or all of the pairs can then present their findings to the group as a whole, time permitting.

Timeframe: Approximately one hour.

Materials required: Flip chart paper and marker pens or overhead transparency slides.