Coert Loock
DEd
Vice-Dean and Associated Professor, Faculty of Education, University Of Johannesburg, Member of the SAELA Executive Committee, Deputy Director of the Board: Centre for Education Law and Policy
46 Volume 1 2013 pp 251
Download Article in PDF
OPSOMMING
Statistiese Aanpassing van Matriekpunte: Die Reg op Toegang tot Inligting
Hierdie artikel neem die historiese rol van Umalusi, ’n liggaam wat wetlik daargestel is om as kwaliteitversekeraar en kurator van die Nasionale Senior-Sertifikaat (Graad 12) te funksioneer, onder die loep. Die doel is om vas te stel in welke mate die Bevordering van Toegang tot Inligting Wet 2 van 2000 gebruik kan word om toegang tot sensitiewe inligting aangaande die statistiese verstelling of “standaardisering” van matriekpunte te verkry.
Die artikel verskaf in-diepte agtergrond aangaande twee aspekte, naamlik: die legitieme mandaat van Umalusi om punteverstellings te mag doen en hulle aanvanklike weiering in 2011 om die uitkomste van sulke verstellings aan die media en breë publiek bekend te stel, gegewe die historiese konteks; tweedens ontleed die artikel die doel en bepalings van die Wet ten einde uitvoering te gee aan artikel 32(2) van die Grondwet van die Republiek van Suid-Afrika, 1996.
Die uitgang van hierdie studie is die betoog van die media en publiek in 2011 dat hulle die grondwetlike reg het om ingelig te word aangaande die proses en uitkoms van die statistiese verstelling van matriekpunte (Graad 12). Volgens regsvertolking is Umalusi deur parlementêre wetgewing daargestel as ’n staatsliggaam wat in die openbare belang binne ’n bepaalde politieke en juridiese raamwerk funksioneer en sy aksies moet as sodanig gemoniteer kan word.
1 Introduction
Learner assessment and standard setting have always been an issue of discussion, not only among professionals, but also in public. Broadfoot1 indicates that the search for an unambiguous and dependable way of measuring “ability” is indeed one of the enduring themes of assessment research in the 20th century. In addition, the tension between the scientific aspirations of assessment technologies to represent an objective reality, and the unavoidable subjectivities contaminated by the human focus of these technologies is very much in evidence in most countries. 2
The production and setting of standards becomes even more of an issue in countries where exit level examinations and assessments mediate university entrance. Access to institutions that cater to a mass market, entry to which is very important to people’s development in life, is bound to be a vexed, and political, issue.3
Umalusi4 is required to approve the release of results, once it is satisfied that the examinations have been conducted in a credible manner.5 The final step in determining the validity and credibility of the examination, before Umalusi releases the results, is called “standardisation” or “normalisation” of marks.6 Until February 2011, the standardisation process and outcomes were treated as confidential. On the advice of the Umalusi Council and its Assessment Standards Committee, Umalusi does not, as a matter of principle and practice, disclose the individual subject standardisation decisions.7
2 The Historical Context and Current Contextual Factors.
Assessment in South Africa has been dominated by the Senior Certificate (Matric) examination, which doubles as a school exit certificate and a university entrance qualification.8 From 1921 up to 1953, the Joint Matriculation Board (JMB) granted permission to various provincial departments of education to run school leaving examinations, and, thus, to become examination bodies. The JMB’s reason for being was to ensure that these exams were of a comparable standard to that set by the board itself. As far back as 1933, a standing committee of the JMB supervised examination statistics, because of the considerable variation in the failure rate of various examinations in the course of time, “... the only conclusion one can come to is that the variation must be in the standard
of the examination.”9 The argument was that an obligatory standard distribution curve ought to be applied per subject in order to adjust the marks to a standard score before the comparative process could be applied.
In order to improve the application of the standard distribution curve, a further sub-committee for standard distribution was appointed in 1975. They immediately attempted to structure the issue concerning standard distributions and formulate an equation and demanded rectifications from time to time. Concern was voiced about the propriety of the adjustment of examination marks on the basis of standards and norms.10 The year 1992 was marked by the demise of the JMB. Matriculation was to be controlled by the Matriculation Board, which was to be a sub-committee of the Committee for University Principals (CUP). A new statutory body, the South African Certification Council (SAFCERT), was established in 1986.11 Until 2002, SAFCERT would be responsible for the moderation and quality control of all school leaving examinations. SAFCERT was transformed by the end of 2002, to cater for a new education and training system, currently known as the General and Further Education and Training Quality Assurance Council (Umalusi).12 Umalusi continued to build on both SAFCERT and the JMB’s approaches to controlling the standard of the Senior Certificate Examination. However, it was becoming increasingly evident that the context in which the examination was being written was continuously changing.13
While most of the examination bodies in countries, other than South Africa, utilise statistical data to standardise results, few of them apply pre-determined statistical norms or desired distributions. Not only is the standardisation of raw examination records recognised as an educationally sound practice, it proved to be a cost effective, reliable and appropriate process for the South African scenario.14 The National Senior Certificate (NSC) requires candidates to pass minimum levels of a prescribed combination of subjects. These subject packages not only differ in terms of the subject content, they also differ in the level of difficulty. For the purposes of Grade 12, prior to 2008, candidates could offer subjects at Lower, Standard and Higher Grades.15 The problem that faced SAFCERT at the time, and currently faces Umalusi, is to ensure that candidates with equal ability, who write different examination question papers, under different circumstances and marked at different marking venues, will obtain equivalent results, in order to comply with the requirements for the issuing of a single certificate, as indicated above. SAFCERT concluded that: “Statistical moderation is necessary to take care of the variation in the standards of marking that may occur from year to year, from one subject to another, or from one examining body to another”.16 The conclusion since 1933 was, that statistical data could be utilised much more effectively to bring greater reliability to the Senior Certificate Examinations, an approach that is still adopted today.
2 1 Statistical Adjustment of External Examination and Assessment results
In its report to the Minister of Education SAFCERT reiterated that there is an abundance of evidence, both in South Africa and elsewhere in the world, that despite the careful attention, and diligence, of competent and experienced examiners, moderators and markers, it is impossible to determine whether a question paper is actually of the required standard until it has been written and marked. It is essential, for this reason, to review the raw examination marks. These raw results should be adjusted, or “standardised”, if evidence indicates that the question paper did not produce a fair result.17 If the examination papers of the examinations in one year are the same standard as those of the previous year, the results should theoretically compare closely with the norms, as calculated by the quality assurance body (Umalusi). If this is not the case, adjustments will be applied. The suggested adjustments are analysed and evaluated by the National and Inter-provincial Standardisation Committees.18
2 2 Statistical Adjustment of School-based Continuous Assessment Results.
While moderation is a process of ensuring that the same assessment standards are applied to learners from every school, statistical moderation is a process of adjusting a particular school’s continuous assessment marks to a set standard, while maintaining the rank order of learners as reflected by the standardised external examination marks.19 School-based assessment is an important component for the calculation of a learner’s promotion mark.20 It is important to ensure that the continuous school-based assessment results of all schools throughout the country are comparable. Statistical moderation is a process of adjusting the level, and spread, of each school’s assessment results for a specific subject, to match the level and spread of the same learners’ scores on a common external examination.21 Following these processes of standardisation, and the statistical moderation of the 2010 Grade 12 results, Umalusi declared the results valid, by indicating that as a matter of principle and historical practice, it cannot disclose the individual subject adjustment decisions. The media statement released by Umalusi on the occasion of the release of the 2010 NSC examination results,22 sparked a public and media outcry on the legitimacy of the 2010 NSC examination results, due to the lack of transparency, and Umalusi’s seemingly non-compliance with the Bill of Rights, and access to information that is deemed to be in the public interest.
2 3 The Issue of Non-disclosure of Information
Umalusi believes that disclosing standardisation decisions in respect of particular subjects will be prejudicial to the learners, as Umalusi and the assessment bodies are “... dealing with the entire system in the standardisation processes and not with individual learners”.23
The media and general public did not accept this announcement on face-value:
The message is clear: Umalusi was the authority, bursting at the seams with experts and why should they explain themselves to anyone - let alone us poor mushrooms whose children are subjected to the state education system?24
The media continued to voice the public frustration against Umalusi’s position, stating:
Umalusi admits that there were statistical moderations to the outcome of the exams and that such moderations are normal. But what they refuse to answer are questions relating to how these adjustments were made and what the 2010 matric pass rate would have been without the adjustments.25
Readers also expressed their opinions, calling on the constitutional right to information in the public interest: “the Constitution guarantees every citizen the right to know what is going on in the country. Any violation of this hallowed tenet in our Bill of Rights must be challenged in the highest courts”. 26
In an effort to defend its position not to disclose the details of the adjustments applied to the 2010 NSC-results, Umalusi indicated that the non-disclosure of raw mark adjustments is a universal practice, because examinations cannot be field tested despite being moderated by Umalusi, before they were written: “Standardisation of marks involves a sophisticated statistical model, which increases or decreases candidates marks by a proportion of their total”.27
In this context the question can be asked, whether the initial argument posed by Umalusi with regard to understanding the complexity of the process is valid and not in the public interest?
Is this an issue that Umalusi is in the position to resolved unilaterally? Section 32 of the Constitution of the Republic of South Africa, 1996 (the Constitution), clearly states that everyone has the right of access to information held by either the state, or another person that is required for the exercise of protection of any rights.28 The Promotion of Access to Information Act29 (PAIA) gives effect to the right. The South African Human Rights Commission (SAHRC) indicates that the PAIA represents a landmark in South African history, by addressing, for the first time, the pre-1994 culture of secrecy in state and private institutions, seeking instead to foster a culture of transparency and accountability in South Africa.
Although access to information regimes are fast gaining momentum around the world, South Africa’s freedom of information legislation remains unique, since it is the only such law that permits access to records held by private, as well as public bodies.30 It is within this broader context that one needs to consider the initial position adopted by Umalusi, in light of the provisions of PAIA.
3 The Promotion of Access to Information31
The Constitution provides that every person has the right of access to information. Legislation, in the form of the PAIA, was promulgated with the aim to foster a culture of transparency and accountability in public and private bodies, by giving effect to the right of access to information, and, possibly more importantly, to promote a society in which South African citizens have access to information to enable them to exercise and protect all of their rights more fully,32 as well as give effect to the constitutional right of access to any information held by the State, and any information that is held by another person, which is required for the exercise of the protection of any rights, and to provide for matters connected therewith.33
In the preamble to PAIA, one is reminded of the fact that the:
system of government in South Africa, before 27 April 1994, amongst others resulted in a secretive and unresponsive culture in public and private bodies which often led to an abuse of power and human rights violations.
This seems to be core of the media and public demand for transparency and openness during the release of the standardised “matric” results, and actually challenges Umalusi’s claim that the detail of subject adjustments was never revealed, since 1918. According to Roberts, the mere existence of a legislative framework regulating aspects, such as the nature of the right of access to information, administrative matters, legitimate limitations and enforcement mechanisms, does not mean that the right of access to information will automatically be fulfilled in the way that the drafters of PAIA envisaged.34
The general provisions of PAIA stipulate that it applies to a record of a public body, to the exclusion of other legislation that:
(a) Prohibits or restricts the disclosure of a record of a public body or private body; and
(b) Is materially inconsistent with an object, or a specific provision of PAIA.
The right of access to any information held by a public, or private body may be limited to the extent that the limitations are reasonable and justifiable in an open and democratic society based on dignity, equality and freedom.35 The Umalusi Council decided not to disclose the full record on the marks adjustments applied to particular subjects during the standardisation process, because they believed that learners could be prejudiced as the general public might not understand the sophistication and complexity of the process. They also argued that in terms of the General and Further Education and Training Quality Assurance Act,36 Umalusi is permitted to adjust the raw marks when necessary.37
Considering the media and public reaction discussed in the introduction to this article, it would be prudent to establish what recourse was available to the general public, and media, in terms of the right of
access to information.38 PAIA states that a “requester” must be given access to a record of a public body if the requester complies with all the procedural requirements in PAIA relating to a request for access to that record in accordance with section 18(1) of PAIA.39 The decision on a request, and notice thereof, must be taken and the “requester” notified, as soon as is reasonably possible, but in any event within 30 days after the request has been received.40 The information officer may extend the period of 30 days (the “original period”) once, for a further period of not more than 30 days, if the request is for a large number of records and compliance with the original period would unreasonably interfere with the activities of the public body concerned. The “requester” must be notified accordingly. In the case of Umalusi, public access to, and/or the publication of, the large scale NSC examination and standardisation records within a short period of time might be problematic, even challenging, as it needs to be presented in a different format than the computerised data, which are used by the Umalusi Technical (standardisation) Committee for the purposes of adjustment and resulting.41 An interesting dimension to the Umalusi debacle was that initially no person, organisation or individual formally submitted an official application to Umalusi in terms of section 18(1) of PAIA. Calls and demands for more information on the standardisation of the NSC results have been mostly verbal, while public interest was initially mostly articulated by the media. Only after Umalusi’s continued persistence not to disclose the details of the marks adjustments did Media24 file a formal application.42 Other media from the Avusa-stable (City Press and Rapport) opted to threaten with litigation. On the 23 February 2011, Umalusi called a press conference and summarily disclosed the extent of the individual subject adjustments applied during the resulting process stating that Umalusi is taking this, “unprecedented step because of the intense interest in the standardisation process and after intensive
consultation with stakeholders and other interested parties”.43 The media had a field day and the news was carried in the tabloids the following day.44 As far as Media24’s application was concerned, full records were released to them by Umalusi on the 30 day deadline.45
The fact that Umalusi disclosed the sum total of the individual subject adjustments under public and media pressure, will not make the issue disappear and it will be prudent for the Umalusi Council to consider its position in view of the provisions, and prerequisites, of the PAIA, as well as, lessons to be learned from emerging case law.
In Nextcom (Pty) Ltd v Funde46 the court held that the respondent is an organ of State, conceived and born in Parliament. It is also financed by the State and its activities prescribed or determined by statute.47 This is equally valid for and applicable to Umalusi. The Court further held that the justification of administrative and executive decisions is only truly possible if there is transparency. A free flow of information is the very essence of justification. It was held that the applicant had a right open to protection by section 32 of the Constitution. The respondent was in possession of all the information, which would enable the applicant to establish whether its rights had been compromised.48
In CCII Systems (Pty) Ltd v Fakie49 the court held the number of documents requested by the applicant was too vast, and the work involved in processing the request would substantially and unreasonably have diverted the respondent’s resources from its core business. The court held that the respondents approach made it impossible to evaluate its entitlement to privilege in respect of the records was justified.50 Considering the media coverage the matter had enjoyed and the prominence of the members, the court held that maximum access was necessary to dispel any suspicion of a cover-up.51 In addition, the applicant had alluded to conflicts of interest and political pressure. The court held that if at all feasible such suspicions had to be put to rest.52 The court held further that section 44 did not deal with historical situations. 53
4 Conclusion
The Constitution stipulates that every person has the right of access to information held by the government. To give effect to this right, legislation in the form of the PAIA was promulgated. Up to 2011, the non-disclosure of individual subject mark adjustments relating to the Grade 12 exit level examinations has never been challenged. The Umalusi Council for Quality Assurance in General and Further Education and Training, was established by an Act of Parliament in 2001, to moderate, standardise and certify exit level exams in the country, with a mandate to adjust raw marks if necessary. Umalusi maintained the historical position of the previous quality assuring bodies on the non-disclosure of information pertaining to individual subject raw mark adjustments prior to resulting, and informed the media of its intended persistence in this approach during a press release in January 2011. Over and above media and public outrage, Umalusi was threatened with litigation. In terms of the Act, Media24 subsequently applied, formally, to the Umalusi Council to release all relevant records to enable them to determine whether, and to what extent, public and individual rights have been compromised by the non-disclosure of the standardisation and mark adjustment records of the Grade 12 examinations.
This article explored the historical context and role of Umalusi, as an organ of state established by an Act of Parliament with the view to being the custodian and quality assurer of all exit level examinations and assessment in South Africa, within the relatively new access to information regime. The article explored the scope of the Promotion of Access to Information Act with reference to the nature of information that may be requested, the public institutions to which the Act applies and who is allowed to request information in terms of this legislation. These aspects are analysed by drawing on some case law and the corresponding provisions of the Act, as it relates to the challenge to access the Umalusi-records on subject marks adjustments for the first time since 1918.
1. Broadfoot “Dynamic Versus Arbitrary Standards: recognising the human factor in assessment” 2003 Assessment in Education: Principle, Policy and Practice 12.
2. Davies “Levels of Attainment in Geography” 2001 Assessment in Education: Principle, Policy and Practice 8.
3. Bakker & Wolf “Upper-Secondary Examinations and Entry to University: The School University transition in an age of mass higher Education” 2001 Assessment in Education: Principles, Policy and Practice 26.
4. “Umalusi” is the Council for Quality Assurance in General and Further Education and Training, established by an Act of Parliament, the General and Further Education and Training Quality Assurance Act 58 of 2001 (GENFETQA).
6. Loock The dilemma of equating examinations and assessment standards for the National Senior Certificate in South Africa 2003.
8. Lubisi & Murphy “Assessment in South African Schools” 2001 Assessment in Education: Principle, Policy and Practice 34-46.
9. Trümpelmann The Joint Matriculation Board. Seventy Five Years: Achievement in Perspective (1991).
10. Trümpelmann (1991). Such views were based on the feeling that examinations itself should be the measure of success - the examination paper, set by competent examiners and moderated by experienced moderators should be the final criterion. The fact is that practice proved different, as it has been repeatedly demonstrated, by having several examiners mark the same script independently, that the marks given by different examiners to the same answer to the same question may differ widely and a variation of 10% is common, even in the so-called objective subjects. It was true then and it is still true today.
11. Under the South African Certification Council Act 85 of 1986 with school leaving certification as its primary objective.
13. Lolwana Umalusi: a historical perspective (2006).
14 See SAFCERT (2002) 18. It is clear that the concept of high stakes standardised testing and the issue of equating examinations and scores, using one common scale is not a simplistic one and needs retrospective research. Equated exams enable us to compare the performance of learners over a number of years. The equivalence of marks obtained in the
14. matriculation examinations in South Africa prior to the 2008 changes in the curriculum always required careful consideration, however, one cannot simply assume that the traditional reference norm or reference instrument can still be applied, or that established equating procedures, such as those currently adopted in South Africa, will still provide meaningful information used to control the performance level of certified learners and their consequent progression.
15. National Education Policy Initiative (1992) 16. All these subjects were (and still are) internally assessed through school-based, continuous assessment procedures, and externally benchmarked by an exit level external written examination.
17. Standardisation, of examination results is the process whereby the results of the examination in a particular year is compared with established or calculated norms and adjusted if necessary. The standardisation process consists of a set of computer programs by way of which examination raw marks are compared with the norms as calculated and provided by Umalusi.
18. During the standardisation process a graph and statistical data are produced separately for each subject - the graph and statistical data reflect the outcome of the examination accurately in comparison to the norm.
19. At present we assume that the only common measure of standard in the grade 12 schooling system is the examination even though the external written examination may assess different skills and competencies from school-based assessment, the assumption is that a group of learners that does well in the examination should also do well in the school-based assessment. Since school-based assessment provides more opportunities and also a less stressful environment, the assumption is that school-based marks of a group should in general be better than the external common examination.
20. The unadjusted continuous assessment mark (CASS mark) of a learner for a specific subject, at a specific school, is the cumulative result of the educator/learner’s efforts during the year.
21. This is done after the achieved results of the common external examination have been standardised against the expected national norm, as previously discussed. Because the common examination is written by all learners in the country, it is regarded as the common standard against which schools’ results can be compared. As the standardised examination mark is used as the norm, the CASS mark moderation can only be executed once that process is completed. The statistical moderation is done according to the formulae and procedures as determined by a statistics committee of the General and Further Education and Training Quality Assurance Body, Umalusi.
23. See Umalusi press release (2011-01-06): “... standardisation decisions are made behind closed doors because the work is highly complex, technical and qualitative, and because the welfare of many hundreds of thousands of candidates depends on that work. Disclosing the outcome of this rich discussion to the general public without the benefit of the plentiful debate that informed a particular outcome in respect of a particular subject is simply inappropriate.”
24. Trench http:www.trench.com.2011-03-23/bust-open-secrets-southafricas-wonderful-access-to-information-laws.htm (accessed 2011-08-05).
26. Business Day (2011-02-01) 5 lead with the mainline: Questions remain over Umalusi data and further stated that, “Umalusi, the body responsible for certifying SA’s matric results are up to standard, has done itself no favours in the manner in which it has handled widespread scepticism over the credibility of the 2010 pass rate”. Similar reports appeared in most of the newspapers in the Media24-stable, including City Press, Rapport and Beeld (2011-01-05); The Sowetan (2011-01-25) even went as far as to declare: “Public trust in Umalusi wiped out: Body should not take citizens on a guilt trip over the right to ask questions”.
28. S 31(1)(a) Constitution entrenches the right that everyone has access to any information held by the state.
30. See http://www.acts.co.za/promotion_of_access_to_information_act_2000. htm (accessed 2011-08-05) on-laws.htm.emaintext.efootnotes. (accessed 2011-08-05).
31. The Act has been amended several times since 2000, the last of which was the amendment by the Judicial Matters Amendment Act 66 of 2008. See also Notice R 1185 GG 23806 (2009-12-18), as well as Notice R 1094 dealing with the Form D.
32. When interpreting the provisions of PAIA, a court must prefer any reasonable interpretation of the provision that is consistent with the objectives of PAIA as articulated above.
33. See http://www.acts.co.za/promotion_of_access_to_information_act_2000. htm (accessed 2011-02-20).
34. Roberts “Prerequisites for the successful implementation of the access to Information Act 2 of 2000” 2006 JPA 1.
35. Consider Umalusi’ s decision to withhold information in terms of the justifiable limitations articulated in s9(b)(i) PAIA.
37. S 9(b) PAIA states that the objectives of PAIA is subject to justifiable limitations, including but not limited to, limitations aimed at the reasonable protection of privacy, commercial confidentiality and effective, efficient and good governance; and in a manner which balances that right with any other rights, including the rights in the Bill of Rights, Ch 2 Constitution.
39. A request for access must be made in the prescribed form to the information officer of a body and the requester must provide sufficient particulars to enable the official of the public body to adhere to the request.
40. s 24(1)(a) PAIA stipulates: “If the information officer of a public body decides to grant a request for access to a record, but that record- is to be published within 90 days after the receipt or transfer of the request or such further period as is reasonably necessary for printing and translating the record for the purpose of publishing it; ... the information officer may defer giving access to the record for a reasonable period”.
41. The standardisation records, per subject, per province and, in the extreme cases, per learner are very comprehensive. Such records will typically include norm calculation tables, marks-distribution tables, marks-adjustment tables and school based assessment (SBA) moderation records.
42. See http://andrewtrench.com/2011/02/23/bust-open-secrets-south-africas-wonderful-access-to-information (accessed 2011-02-20): “... we demanded copies of every record relating to the decision and which would show how the mark adjustment decisions were made, which subjects were changed and how much”.
43. Umalusi (2011-02-23). Press statement by Umalusi Council chairmen, Prof Sizwe Mabizela: “We realised that non-disclosure is more damaging to the regulator ... we have nothing to hide. The process of standardisation s very rigorous and we have a great team of statisticians and academics. If the public wants to disclose every year, we are ready”.
44. The Times (2011-02-24) 3 “Umalusi denies cooking the books”; Sowetan (25-02-2011) 5 “Public trust in Umalusi wiped Out”; Pretoria News (2011-02-24) 3 “How matric marks were adjusted”.
45. The information officer has a legal duty to consider, provide and assist the requester with the application as determined by s 19(20) PAIA.
47. Umalusi was established by GENFETQA as an organ of state, responsible for moderation, monitoring, standardisation and certification of exit level exams.
48. See Umalusi’ s response to the media, that “... standardisation decisions are made behind closed doors because the work is highly complex, technical and qualitative, and because the welfare of many hundreds of thousands of candidates depends on that work. Disclosing the outcome of this rich discussion to the general public without the benefit of the plentiful debate that informed a particular outcome in respect of a particular subject is simply inappropriate.”
49. CCII systems (Pty) Ltd v Fakie 2000 4 SA 491 (T).
50 In the case of Umalusi this is equally appropriate: “At the press conference the grandiloquent prof Mabizela spent hours lecturing my colleagues on the finer points of statistics and ogive curves and the like, berating the press and
50. experts who dared to question the judgement of the esteemed experts of Umalusi.” Trench (http://www.trench.com.2011/03/23/bust-open-secrets-southafricas-wonderful-access-to-information-laws.htm (accessed 2011-08-05)).