As the current coordinating officer for Div. 5, I have had the opportunity to meet and interact many of the division’s members. These interactions often include conversations about why the person chose to join Div. 5 and how they became interested in both psychology and methods. What is always fascinating to me are the multitudes of different paths that lead someone to become interested in methodology. In an effort to highlight some of the communalities between the division’s diverse membership (and hear some interesting stories), we are introducing a new (hopefully) regular column in The Score called “Meet a Div. 5 member.” And what better place to start than with Div. 5's President-elect, Kurt Geisinger, PhD.
Geisinger is the director of the Buros Center for Testing and Meierhenry Distinguished University Professor at the University of Nebraska - Lincoln. He serves as president of the International Test Commission and president-elect of Div. 5 of the American Psychological Association. He has edited or co-edited more than 12 volumes including the “APA Handbook of Testing and Assessment in Psychology,” “Psychological Testing of Hispanics” (two editions) and numerous “Mental Measurements Yearbooks,” and has edited “Applied Measurement in Education.”
Matt Fritz
University of Nebraska–Lincoln
Matt Fritz: In your own words, what is a methodologist?
Kurt Geisinger: First, I have a broad perspective on this question. Someone whose job it is to employ research methods to design studies, interpret the results of results, and be aware of alternate interpretations. There are many kinds of methodologists. Probably most are quantitative, but an increasing number are also qualitative. I note that I am a measurement specialist, which many will consider different than a methodologist. I have spent a considerable portion of my career building measures for various uses.
Can you describe one aspect of your career where you built measures?
KG: For almost 10 years, I was a consultant to New York City helping them to determine how to build civil service tests for police officers, fire fighters and other professions. When I built police officer examinations in New York City, my research began with reviewing previous job analyses, then progressed to observations and structured interviews, where I was mostly interested in how the job had changed since the previous job analysis. From this work, I developed a thorough task list of activities that police officers performed on the job. This task list was transformed into a survey which for each of the 700 tasks. The police officers reported how often they performed the task, how much time they spent completing the task, how important it was, what the consequence of error was if it were performed improperly, and finally, where they had learned how to perform this tasks. If they learned to perform the activity at the police academy, it was dropped from the list because it was ineligible for testing. Response rate was over 99 percent for two reasons. Police officers were given work time to complete it and they had to hand it in (in a sealed envelope) to their commanding officer. If only survey research were always so easy. Tasks that were perceived as unimportant, infrequent and so on were dropped, and the remaining tasks were subjected to a factor analysis. We typically came up with between 15-20 task clusters. We then used these data to build our examinations and justified them successfully in court when they were challenged, as they often were.
How did you become a methodologist?
KG:I did very well on the GRE-Q. Several graduate school programs reached out to me to consider their programs. I began my studies at the University of Georgia in a Psychology doctoral program entitled, Measurement and Human Differences. That program had students and faculty who ranged from statistically focused individuals through applied researchers, primarily in industrial-organizational psychology and educational research. I think that the primary focus, however, was on psychometric issues and testing. Many of the graduates went on to superb careers in testing and assessment and some to other areas of methodology. It has become an industrial-organizational program.
Who or what has had the largest impact on your career as a methodologist?
KG: Working with Anne Anastasi. I was Anne Anastasi’s largely self-selected replacement and worked with her for 15 years at the beginning of my post-PhD career. I learned from her the need to balance methodological (in her case and mine, testing and psychometrics) expertise with knowledge of issues in psychology. I also learned how important writing skills were, something I had also learned at Georgia from W.A. Owens, my master’s advisor, and William Rabinowitz, my doctoral advisor. Switching graduate schools midstream also helped me. I came to know testing and psychometrics at Georgia from a largely dustbowl empiricist perspective, and at Penn State from more of a construct validation/content validation/comprehensive-rationale perspective. I heard differing opinions on many methodological issues and had to reconcile them myself. Looking back, I was especially appreciative of courses by Paul Baltes and John Nesselroade on longitudinal and other research designs and factor analysis, respectively.
What is the most memorable thing that has happened to you as a methodologist?
KG: I actually must describe two events. First, while in graduate school at Penn State I took a job with a small personnel consulting firm. I was hired as director of research and project director of a court-ordered study to see if female officers performed as well as male officers in a large eastern city. I hired three men and three women -- all with appropriate graduate degrees and experience -- to ride with 100 men and 100 women who had been hired and had passed the police academy. They rated the probationary police officers using 17 behaviorally anchored rating scales, which I had helped develop based on the activities that the city had claimed women could not do as well as men. Ultimately, the study showed that women could do the job as well as men and since that time that city and virtually all of the United States has male and female officers.
Second, I have been active in analyzing and interpreting data on the use of standardized tests for people with disabilities. I have served as an expert witness in a number of cases in this regard, impacting the high school graduation tests in California, college admissions tests and professional school admissions. Not all psychometricians know that after the passage of the Rehabilitation Act of 1973 (specifically Section 504) testing agencies were supposed to provide testing to those with disabilities with accommodations, a principle that was enhanced with the passage of the Americans with Disabilities Act. However, most testing agencies did not wish to provide some adaptations without annotating the score, so compromise agreements permitted adding an asterisk to the score, indicating that the score was earned with an accommodation. Research has since shown, for the most part, that such scores can be used as valid scores and are indeed more valid than a score that would be earned without such an accommodation.
I really enjoy helping to impact social justice from an empirical perspective.
What advice would you give to a student who was thinking about applying to a methodology program?
KG: Do it. It can be a great career. But do not wish to learn to run statistical and psychometric programs as a primary orientation. I hear that from some students. Rather, learn to interpret findings; that is more professional and, frankly, is much harder. I would encourage individuals to decide some content-oriented psychological questions in which to have interest. Dealing with the difficult context of matching psychology-based research questions to appropriate methods is one of the most fun things to do in the field. There are many avenues in which one can be a methodologist. Jobs have generally been very good in the testing profession for decades, but when I was a dean and a vice president, I always told students to seek to do what they enjoy, not just a job. Life is too short to do something for years that you do not enjoy!