Murdoch University Research Repository

Welcome to the Murdoch University Research Repository

The Murdoch University Research Repository is an open access digital collection of research
created by Murdoch University staff, researchers and postgraduate students.

Learn more

Psychometric profiling and aggregating of public examinations at the level of test scores

Tognolini, James Stephen (1989) Psychometric profiling and aggregating of public examinations at the level of test scores. PhD thesis, Murdoch University.

[img]
PDF - Whole Thesis
Available Upon Request

Abstract

This study reviews the current procedures used to form aggregates which are used to order persons competing for limited places at tertiary institutions in Australia. The review shows that most Australian states use a single index (called a TE Score), based on an examinable subset of Year 12 subjects, to select persons into tertiary institutions. It also examines the weaknesses and concerns of using a single index, and considers the debate between using profiles in preference to the total score in the tertiary entrance selection process. The thesis then develops a new method for scaling the scores from different persons who take different combinations of subjects onto the same measurement variable. This scaling is necessary in order to take into account the differences among the difficulties of the subjects and the differences in the way the examinations in the subjects spread the scores. For example, the subject Physics is considered more difficult than the subject Economics and subject Physics is considered to generally spread the scores less than the subject Economics. The main advantage of the proposed method over current procedures is that it has a strong measurement theory to support, the analysis of profiles, the construction of the variable and where appropriate, the ordering of candidates according to an aggregate.

The measurement model used to supervise the construction of the variable, called the Extended Logistic Model, is a generalisation of Rasch's Simple Logistic Model for dichotomously scored items. Four important interrelated characteristics arise from the application and use of this particular model. First, the process of variable construction and person measurement are separated from each other. This means that there is the possibility of estimating the parameters that characterise the difficulties and the spreads of the subjects independently of the distribution of abilities and that the measurement of persons can be independent of the subjects they take for examination. Second, for the stage of variable construction the data are compared to the model, and where appropriate, edited at the person-subject level of response in order to improve the overall accord between the data and the model. This editing also has the effect of improving the homogeneity of the correlations among the Third, the model not only provides an estimate of ability which is obtained from a person's total score on the subjects, but it also gives an indication of the standard error of this measurement. Fourth, the simple unweighted sum of the scores of different subjects that constitute a person's profile is used to obtain the summary subjects. score. This sum is sufficient, in the statistical sense, for the location of the person on the variable, and sufficiency implies that the sum captures all of the information in the profile. Therefore, the accord between the data and the model can be used to identify those persons whose summary scores do, and those who do not, capture all of the information in their profiles. For those that the total score does capture the full information, the total score can be used to estimate their abilities and to compare them for selection purposes. For those that the total score does not capture the full information, the profile may be examined in detail before decisions are made regarding selection into a tertiary institution.

An application of the model is demonstrated with a random sample of 580 persons from a population of 12,314, and nine subjects from a possible set of 28 subjects used for selection of persons into tertiary institutions in Western Australia in 1988. The Australian Scholastic Aptitude Test is used in the study as a tenth subject because every person has to take this subject which is currently used as an anchor variable in scaling subjects relative to each other. A sample is required because the computer programs used have been written for the microcomputer and as a consequence only relatively small data sets can be processed. The estimates of four parameters for each subject, interpreted specifically as the location, spread, skewness and kurtosis, are obtained, and the effects of editing of scores when they do not accord with the model, are demonstrated. It is shown that the homogeneity of correlations, and the magnitude of the correlations, improve after editing.

The estimates of the ability of persons on the variable and the calculation of the statistic which shows the degree to which the total score captures the information in the profile is demonstrated. It is shown that before the editing that eliminates unexpected responses is made at the 5% level of significance, approximately 3.5% of persons require their full profile to be examined because their profiles do not accord with the model, of variable construction and using the original unedited profiles in the stage of measuring persons, only approximately 12.5% of persons require their profiles to be considered in detail because the total score does not capture all of the information in the profile. After editing for the stage It is therefore demonstrated that it is not necessary to examine all the profiles of all the persons. The main conclusion is that the debate between those advocating the analysis of profiles and those advocating the use of aggregates can be reconciled by seeing the approaches as complementary. Specifically, where the total score captures all of the information in the profile, it is not necessary to study the whole profile; where it does not capture all of the information, the profile must be studied. It appears that the profiles that need to be studied are in the minority.

It is in the aspect of the measurement of persons that two main further research issues are identified. First, it is necessary to determine the effect of the editing process on the final ordering of persons on the variable. Second, it is necessary to conduct further research on the theoretical distribution of the test statistic used to identify those profiles that do not accord with the model.

Item Type: Thesis (PhD)
Notes: Note to the author: If you would like to make your thesis openly available on Murdoch University Library's Research Repository, please contact: repository@murdoch.edu.au. Thank you.
Supervisor(s): Andrich, David
URI: http://researchrepository.murdoch.edu.au/id/eprint/51451
Item Control Page Item Control Page