Application Series 12
EDUCATED IN THE CASELESS CASE
Niels Veldhuijzen, Statistician at Cito, Arnhem, The Netherlands, uses SIR for the
National Assessment Project. He explains why he prefers the caseless, or relational, SIR
structure to the traditional hierarchical or case structure.
What is Cito?
The National Institute for Educational Measurement in the Netherlands (better known under
its Dutch acronym CITO) is a semi-governmental institute which provides the Dutch
educational system with many varieties of tests and other evaluation instruments. These
measure abilities of individual pupils and monitor school performance. More than 300 people
work at Cito; 6% of them at the Measurement and Research Department as statisticians and
psychometricians. Cito works for all kinds and types of schools except universities.
What is the National Assessment Project?
The National Assessment Project is a cyclic project performed on behalf of the Dutch
Department of Education. Every five years, a survey is held among eight-year-old and
twelve-year-old primary school children. Its main purpose is to measure the output of the
Dutch educational system with regard to subjects like language, arithmetic, geography and
physical training. A secondary purpose is to compare results over time, in order to see if
there are shifts in the general output levels for the two populations.
What does the National Assessment Project?
The project has two main stages: instrument construction and measuring. Instrument
construction is the process of producing test items (questions with direct answers and tasks
such as essays which are rated) which are meant to elicit responses from pupils, followed by
statistical and psychometric analyses in order to turn the items into measuring instruments
with known characteristics. The second stage consists of taking the measuring instruments
to the pupils, gathering the responses and analyzing and interpreting these responses.
About the data
In each survey, between four thousand and ten thousand pupils take part. For each subject
many items are available. As it is impossible to confront all pupils with all items, an
incomplete design is used. Items are divided into several subsets, and different subsets are
given to different subsets of pupils. The use of sophisticated psychometric models (so-called
item response models) makes it possible to link responses on one subset of items to those on
other subsets of items pertaining to the same subject matter. Due to the incompleteness of
the design and the large numbers of items and pupils, the structure of the data gathered is
very complex.
A Case for a Caseless Database
The National Assessment databases have evolved over many years of experience and
development, and one of the interesting features of today's databases is that they are
caseless, or relational. From the two stages of the National Assessment Project, instrument
construction and measuring, it can be deduced that there are two distinct kinds of objects in
the project, the test items and the pupils. These two objects are linked, of course, by a third
object: the responses of the pupils to the items. A caseless database allows all three objects
to be incorporated into a single database with very small keys, or links, between them. It
also allows for the incorporation of metadata, detailed information about the objects.
Responses
Of paramount interest for the project are the responses the pupils gave to the test items.
Two very simple but very large record types have been defined; one to hold responses to
specific questions and one to hold ratings of a task. Because items are divided into subsets,
each record contains at least two key fields: a pupil id and a subset id. The ratings record
type has the rater id as the third key field. By means of SIR's full report generator, it is very
easy to collect data pertaining to one pupil in an external file. The key fields provide the links
to the other two objects I mentioned earlier: pupils and items.
Pupils and Schools
One record type has been defined to hold background information about each pupil such as
sex, age, ethnicity. As some school characteristics are also studied, an important
background variable is the school id, which is incorporated in the pupil id. The school id
connects pupil records with a school record type, which holds school characteristics such as
the text books used by the school (CAT VARS does splendidly here!) and a school
category.
Subsets and items
A record type has been defined for item subsets with the subset id as its only key field. It
contains item ids for this subset. The order of the item ids corresponds with the order of the
items in the printed test booklets for this subset. Because of the incompleteness of the
design, some items are part of many subsets. These item ids point to item records, storing
data such as the kind of item (multiple choice, open ended question), the code of the correct
response for a multiple choice answer, and the possible response categories for other
items.
Data extraction
Thanks to the caseless, modular structure of the databases, data can be combined in many
ways. SIR's full report generator is the indispensable tool here. Collating records into lines
to be put in external files, aggregating data over variables, and many other data
manipulations are easy to do, resulting in data files with any format my colleagues and I like.
This is especially important for the psychometric analyses of test items, which at Cito is
mostly done by computer programs developed in house and require various and complex
files in a special format. A suite of PQL retrievals and programs has been written that makes
construction of these files very easy.
In fact, a two-stage procedure is followed. In the first stage, the data and metadata needed
are specified with the help of PQL-menus. The information supplied in the menus is used in
a procedure generator, which writes the PQL retrievals and programs to be used for data
extraction. In the second stage these generated procedures are run in batch mode, mostly at
night, to produce the required files. The next day, all the files which we have specified for
the execution of our psychometric analyses are waiting for us!
Updating the databases
Per subject matter and per age group, a survey is held every five years. The new data are
added to the databases, which will grow over the years. Adding data goes smoothly with the
help of the ADD REC utility. The data files to be added to the databases are prepared by
data entry specialists with the help of a dedicated SIR input database which provides input
screens with FIELD INPUTs and immediate data validation.
Final Remarks
The National Assessment Project databases are installed on a personal computer with 80386
and 80387 processors, a 33 MHz clock and a fast 500Mb SCSI-drive. At the start of the
project, my PC was called 'the BMW of PCs'; now it is called utterly out of date. But thanks
to the modular structure of the databases, the small key fields, the use of program generators
and, most of all, thanks to the many features of SIR and its beautiful PQL, managing the
National Assessment databases with SIR is a pleasant job.
For more information on The National Assessment Project contact:
Niels Veldhuijzen,
Cito,
Measurement & Research Dept
PO Box 1034
NL-6801 MG Arnhem
The Netherlands
Back Sir Home