- 1. Characteristics, structure and functions of the section on the Bulgarian language.
- 2.Content properties of the test in the Bulgarian language section.
- 3. Psychometric characteristics of the Bulgarian section of the test.
- 4. Measuring properties of the test in the Bulgarian language section.
- 5. Results and conclusions
- 6. Applications of the data
- Appendix: Bulgarian Language section - Specifics of the objectives
Nowadays in response to the newly emerging social needs and the respective new educational requirements, novel forms and procedures of assessment are being applied in Bulgarian secondary schools. The problem for the assessment of the level of knowledge of the Bulgarian language is interpreted in a framework outlined by: a) the existence of state requirements for educational content and state educational requirements for assessment in the teaching of the Bulgarian language and literature. The assessment requirements are still under refinement and establishment. All the requirements are in keeping with the European Union guidelines for language education (ML 1996).
The current paper focuses on the correlation between assessment and measurement. Assessment is a process of juxtaposing the aims of an activity (in this case the teaching of the Bulgarian language to 5 — 7 graders) and the results achieved through this activity to be compared to desired states of the assessed subject (in this case the students sitting for a nation-wide exam after finishing the 7 grade). Measurement is a procedure of correlating figures for persons under study (the examined 7th graders) with the aim to represent quntitatively determined properties and qualities characterizing the group of studied subjects. The deductive examination test is looked upon as a procedure aiming at eliciting a specific behavioral pattern in the examined subjects with view of drawing conclusions about some specific features. The measuring instrument is the nation-wide exam-test.
In 2000 and 2001 successively a nation-wide exam-test was conducted in the Bulgarian secondary schools for 7 graders. The 120 test items (60 items for the 2000 sessions and 60 for the 2001 exam sessions were chosen on the basis of initially set psychometric characteristics (see Section 3) from a bank database of priorly approved 1000 test items — 400 for the Bulgarian language, 400 for mathematics, 100 for the natural sciences and 100 for the social sciences respectively.
After approbation, expert evaluation and statistical processing of each of the banks for the four set sections, 4 equivalent variants of the test were deposited at the Ministry of Education. Each of the four variants contained:
А. 60 mutiple choise test-questions (with four possible answers presented) for the time allotment of 120 minutes. The test-questions were organized in four sections:
- 20 test items on the Bulgarian language;
- 20 test items in mathematics;
- 10 test items in the natural sciences;
- 10 test items in the social sciences
B. Answer key for each variant
C. Answersheet format for the test
D. A conversion table for transforming the test points into a six member evaluation scale. Each correct answer earns the testee one point, each wrong, ambigous or non-specified answer receives a 0.
a) discusses the preliminary results from the two exam sessions in which the nation-wide exam-test was applied in the 2000 and 2001 academic years;
b) encompasses the results from the entrance exams after the 7th grade (just before high school education);
c) concerns the Bulgarian language as a first language L1, i.e. the official language or as is the practice in Bulgaria to term this native tongue.
Although the exam was designed so as to identify the most successful students for the purposes of an entrance examination, the results of the exams can be used to draw significant conclusions about the general educational level of the students and more specifically the level of language literacy of the students.
Language education is viewed upon as a function of the teacher-guided process of language teaching (L1, L2, etc.). Moreover it is a means for enhancing the innate and acquired communicative competence in the sense in which psycho- and sociolinguists understand it. The main result from language education is the literacy of the studenst in a broader sense than is traditionally accepted.
As a term literacy, for the purposes of the current paper, covers not only the ability to comprehend the contetn of what has been read and to use the standard literary language, but also the ability to appropriately use language in accordance with the requirements of the specific, immediate sociocultural context. For the purposes of communication one of the levels is sufficient — that of literacy, which is the result of the necessary and sufficient language education. The other level — that of competence — provides the prerequisite language education that is required forr acceptable and free oral communication. Acceptable here means language behaviour compliant with the pragmatic postulates, while free implies using language without any hindrances or difficulties.
The tendency for broadening the contents of the notion of literacy is examplified by the introduction of the term cultural literacy and even intercultural literacy. While cultural literacy names the knowledge of the world expressed in and through language, a knowledge at a level higher than the everyday mundane practical knowledge and at the same time at a level lower than specialized knowledge (Hirsch 1987), intercultural literacy is interpreted as the cluster of abilities activated in communication that permit one to understand and adequately value, without any prejudices, cultures different from their native culture. Intercultural literacy facsilitates the avoidance of cultural assimilation while significantly contributing to the enhancement of tolerance and opennes towards others. School education and particularly the teaching of native language and literature play a major role for the acquisistion of competence understood in the broader sense outlined above. The level of literacy of the students is judged and assessed on the basis of the background knowledge (frames) which make it possible for the students to cope with the test items and questions.
1. Characteristics, structure and functions of the section on the Bulgarian language.
The concepts described in 0. above laid the foundations for the construction of the component Bulgarian language in the test. The Bulgarian language section aimed at assessing two separate dimensions — cognitive and content.
The cognitive dimension included:
a) evaluating the knowledge of the students — the use of metalanguage;
b) evaluating the abilities to appropriately apply the linguistics skills and language knowledge and the acquired communicative competence in specific sociocultural context — language use.
The content dimension represented the teaching material on whose basis the assessemnt was organized. The examination program imitated closely the obligatory educational program for teaching the Bulgarian language in secondary schools. The examination program was designed in a way that allowed the assessment of language preparation via testing.
The constructed test had as its main objective to assess the abilities of 7 graders to orient themselves in the communicative situation that generated a particular text, the abilities to comprehend the linguistically coded meanings, the skills for identifying and analyzing the structural peculiarities of texts, the skills for applying the norms of the standard language (for a detailed account of the aims of the test in keepng with the cognitive levels set for the test as a whole, see the specification). The enumerated characteristics made it necessary to create test tasks of higher cognitive level whose answering requires more than one cognitive operations. For instance, a check-up of expected successful acquisition of punctuation rules of the Bulgarian language could not be accomplished without an examination of the knowledge in Bulgarian syntax, necessary for adequately manipulating the punctuation rules.
The specificity of education in a first (native) language was taken into consideration in constructing the test tasks. As native speakers of the language, the seven graders have a well-developed sense of language and rich communicative experience accumulated not only at school. That is why the test examined and assessed their language preparation and the resultant communicative competence (see Bulgarian Language Section - Specifics of the Objectives).
The Bulgarian language section contained the following types of items: a) four reading comprehension items; b) three analogies items; c) three items on synonyms, phraseological units and antonyms; d) four spelling items; e) four punctuation items; f) two linguistic knowledge items.
2.Content properties of the test in the Bulgarian language section.
What was checked was the level to which seven grade students have acquired the obligatory minimum of knowledge and abilities in the Bulgarian langugae specified in an Examination Educational Program approved and decreeted by the Minitry of Science and Education.
The types and forms of the tasks covered the following cognitive levels and dimensions:
а) linguistic knowledge - 10% of the overall number of items on the Bulgarian language;
b) application of the acquired linguistic information and the accrued communicative experience - 90% of the Bulgarian language items.
The content validity of the test tasks and items was established via expert evaluations performed by supervisors from the Ministry of Science and Education and practicing Bulgarian language and literature teachers.
After discussions with the rest of the team memebers, the person responsible for the content section "Bulgarian language" described the aims, objectives and the contents of the test and outlined the respective content framework. The content framework specified the number of test items in each thematic sphere and cognitive level and dimensions. The content framework was submitted for expert evaluation. The overall specification of the test was designed on the basis of this content framework.
After discussing the specifications for each subject area, those responsible for the whole enterprise took the decision to adopt the most popular and functional taxonomy of educational objectives introduced as early as 1956 (Bloom 1956). For the purposes of the examinatrion test it was decided to simplify the taxonomy to two cognitive levels/dimensions: a) that of knowledge and b) that of their application (the abilities dimension) Thus the risk of uncertainties and contradictions in the expert evaluations was brought to a minimum.
To optimize the construction process, a five stage algorhythm, popular in the field of testing was used:
- choice of specific educational content;
- formulation of the main idea (postulates and rules);
- fixing of the cognitive level;
- choice of a new situation that fits the main idea;
- formulation of the task.
3. Psychometric characteristics of the Bulgarian section of the test.
The psychometriccharacteristics include the following parameters: standardiazation, validity, reliability.
Standardiazation here means the choice of a group whose test results are set up as a norm and fumction as the basis for an interpretive matrix to read the other results. The tasks and items were grouped in three standard sets — of low, medium and high achievements.
By validity is understood the degree to which the test is actually able to measure the achievements it has been designed to measure.
In testing, the term reliability is interpreted as presicion of the measurement. It is a well-established fact that a test cannot possibly be valid if it is not reliable. The reverse does not hold true.
4. Measuring properties of the test in the Bulgarian language section.
These properties are represented by the quantitative characteristics (numeric qualities) of the test items. The numeric qualities are indices of the distribution of the test results over their general totality. The main numeric qualities of each test item depend on two major factors: validity and difficulty. The most popular measurements of validity are discrimination and the point biserial correlation.
5. Results and conclusions
The constructed and performed test was designed in complience with the order from the Ministry of Science and Education. For this reason the Ministry preserves the right to decide when to publish the data that would interest linguists, methodologists and test-designers. At present the statistics and all other relevant information about the 8146 students who sat for the test in the city of Sofia are available. They make up around ¼ of all students in the country who apply after the seventh grade. All the data presented and commented upon here concerns these students specifically.
It is quite obvious that 20 items are insufficient to check everything that has been learnt for a whole academic year. The optimum number of items in an examination and evaluation test depends on its provisiory functions. The 400 set items covered all the material in the Bulgarian language taught to seven graders. The twenty test items were chosen by a computer lottery from all the high qulity items. As quility items qualified those that met the standards and requirements of the examination program, the educational program for the teaching of the Bulgarian language as well as those that corresponded to quantitative criteria fixed for each of the four sections of the test. These were the widely known in testing practices criteria for validity, level of difficulty and discriminatory capacity(see 4.).
The level of difficulty was measured between 0,60 and 0,70, the discriminating capacity - over 0,25, the point biserial correlation of the distractors had negative values.
0,25 was set up as good discriminatory capacity. If we compare the mean discriminatory capacity of the tasks in the Bulgarian language — 0,499 — to the accepted standard it will become perfectly clear that the test serves its main function to discriminate between students with good language competence and education from those with lesser achievements and showing some gaps in their education.
The quality of the test items is also manifest by the negative values of the point biserial correlation of the distractors. In other words, misleading answers have been chosen by students who scored low on the test.
Finally, we turn to the most important criterion — reliability or consistency of the test. This criterion shows whether the test has been a good instrument for measurment in examination conditions. This property of the test is represented via Cronbach's Alfa . For the Bulgarian language section its value amounts to 0,675. The closer the value to 1, the greater the reliability. The value for the test as a whole was 0,897.
70% of the test items had received correct answers. Among high achievers this figure was 80%, while in the group of underachievers it dropped to 60%.
The test data was processed with the help of the licensed system MicroCat — Testing System.
We could safely draw the conclusion that the exam-test functions as a reliable instrument for establishing whether the students from the seventh grade have successfully appropriated the required obligatory educational minimum in the Bulgarian language.
Seven graders have received language preparation that adequately reflects their competence in relation to: the ability to correctly comprehend a read text; to work freely with logical relations between concepts, to analyze the use of lexical and grammatical linguistic means of expression, to apply the norms of the standard language (spelling, punctuation); to recognize linguistic notions and terms that have been obligatorily taught and that make part of their linguistic competence.
6. Applications of the data
The pivotal application of the data is related to the creation of a data base concerning the language education of Bulgarian pupils in secondary shools and competence (linguistic and cultural) as its major component. On the basis of these data all achievements and weakness in the preparation and knowledge of students can be studied. Besides, the date can be used for the compilation of methodological systems for successful teaching of first (L1) and second (L2) language.
- Fundamental Consideration in Language Testing, Oxford: Oxford University Press., 1990.
- Taxonomy of Educational Objectives. Handbook 1: Cognitive Domain, New York: Mckay., 1956.
- Cultural Literacy. What Every American Needs to Know, Boston: Houghton Mifflin., 1987.
- Modern Languages: Learning, Teaching, Assesment. A Common European Framework of Reference, Strasbourg: Council of Europe., 1996.
- Modern Educational Measurement, New Jersy: Prentice Hall., 1981.
Appendix: Bulgarian Language section - Specifics of the objectives
The level of language preparation of the students from the seventh grade is checked and assessed on the basis of two cognitive dimensions: of knowledge (dimension A) and of the abilities (dimension B)
- First cognitive dimension — A is oriented towards fullfilling such objectives as recognition and reflction upon facts, concepts and regularities from the field of linguistics: the linguistic essence of language phenomena, the regularities of language related to the usage of the standard language, main linguistics terms and concepts such as text, sentence and so on included in the required provisionary obligatory minimum of comprehension, the norms of the standard language (grammatical, lexical, spelling, punctuation). This dimension comprises the following activities:
- type of text defined along the criterion ?field of communication? in which the speech event occurs ? the functions carried out by a particular text;
- type of syntactic means (on the level of the phrase and on higher levels);
- paratactic and hypotactic combining/compounding;
- means for intraphrasal connectivity in the text;
- lexical means of expression ? homonyms, paronyms, antonyms, polysemious words, metaphorically used words; phraseological units and deciphering of their lexical meaning.
- Recognition and comprehension of the properties of the morphemes;
- Recognition of spelling and punctuation errors and analysis of their nature
- synonymous relations between lexemes, synonymous relations between lexemes and composite expressions, between one composite expression and another (lexical and syntactic synonymy);
- relations of oppositeness between lexemes, relations of oppositeness between lexemes and composite expressions, between one composite expression and another.
- logical relations (genus - kind, part ? whole, etc.)
- Comprehension of the regualarity of the sound changes in cases of non-correspondence between spelling and pronunciation.
Recognition and comprehension of Second cognitive dimension - B- hypothetically termed "Appendix".
- It is designed to achieve aims of the type: applying linguistic information for solving a particular problem ? absorbing and analysing information from a text, formulation of judgmental and evaluative propositions related to the contents of a comprehended text. Identifying novel logic relations between concepts and notions and organizing them in new hierarchies. Applying the norms of the standard language and the rules for stylistic variation in speech production. Abilities to evaluate correspondences and discrapancies between communicative intent and speech realization. This level comprises the following activities:
- orientation in the communicative situiation ? evaluation of the correspondence between text and the situation in which it is used: topic and subject of the speech event, participants, aims and conditions of the communicative act.
- analysis of the structure of the text.
- distinguishsing between what is significant and insignificant in the information absorbed in the reading of a text and organizing the derived meanings of the text into a meaning-generation hierarchy.
- linguistic analysis (syntactic, lexico-semantic, morphological, morphemic, word formation, spelling, punctuation analysis) for uncovering divergences from the norms of the standard lanmguage and inconsistencies between cognitive intent and speech realization.
Ration between the tasks on the basis of cognitive levels:
10% - level A: linguistic knowledge tasks
90% - level B: reading comprehension tasks, logical relations uncovering tasks, word and expression relations tasks (synonymy, antonymy) aimed at analysing phraseological expressions, spelling and punctuation tasks.