Commercial, Norm-Referenced, Standardized Exams

Definition

Used primarily on students in individual programs, courses or for a particular student cohort.

Advantages:

Can be adopted and implement quickly
Reduce/eliminate faculty time demands in instrument development and grading (i.e., relatively low frontloading and backloading effort)
Objective scoring
Provide for externally of measurement (i.e., externality validity is the degree to which the conclusions in your study would hold for other persons in other places and at other times -- ability to generalize the results beyond the original test group)
Provide norm group(s) comparison often required by mandates.
May be beneficial or required in instances where state or national standards exist for the discipline or profession.
Very valuable for benchmarking and cross-institutional comparison studies.

Disadvantages

May limit what can be measured
Eliminates the process of learning and clarification of goals and objectives typically
Unlikely to completely measure or assess the specific goals and objectives of a program, department, or institution
Relative standing results tend to be less meaningful than criterion-referenced (define and insert) results for program/student evaluation purposes.
Norm-referenced data is dependent on the institutions in comparison group(s) and methods of selecting students to be tested. (Caution: unlike many norm-referenced tests in higher education do not utilize, for the most part, randomly selected or well stratified national samples)
Group administered multiple-choice tests always include a potentially high degree or error, largely uncorrectable by guessing correction formulae which lowers validity.
Results unlikely to have direct implications for program improvement or individual student progress.
Someone must pay for obtaining these examinations; either the student or program.
If used repeatedly, there is a concern that faculty may teach to the exam as is done with certain AP high school courses.

Ways to Reduce Disadvantages

Choose the test carefully, and only after faculty have reviewed available instruments and determined a satisfactory degree of match between the test and the learning outcomes of the curriculum.
Request and review technical data, especially reliability and validity data and information on normative sample from test publishers.
Utilize on-campus measurement experts to review reports of test results and create more customized summary reports for the institution, faculty, etc.
Whenever possible, choose tests that also provide criterion-referenced results.
Assure that such tests are only one aspect of a multi-method approach in which no firm conclusions based on norm-referenced data are reached without validation from other sources.

Bottom Lines

Relatively quick and easy, but useful mostly where group-level performance and external comparisons or results are required. Not as useful for individual student or program evaluation. May not only be ideal, but many times the only alternative for benchmarking studies.

Bibliographic References:

Journal of Professional Issues in Engineering Education,

Journal of Engineering Education,