Professor Anna Brown is a psychometrician with an established reputation and extensive industry experience, currently teaching psychological methods and statistics at the School of Psychology of University of Kent. Previously, she taught short courses in applied psychometrics at the University of Cambridge. Her experiences outside of academia included research and test development at the Research Division of the UK largest occupational test publisher SHL Group.
Anna completed a 5-year master’s program in mathematics with distinction and a PhD in psychology with distinction. Her PhD research led to the development of the Thurstonian Item Response Theory (TIRT) model described as a breakthrough in scoring of ‘ipsative’ (relative-to-self) questionnaires, and received the prestigious “Best Dissertation” award from the Psychometric Society. Further advances in this methodology made by Brown enabled the development of novel personality assessment tools that are resistant to response biases and ‘faking good’ by candidates. Multiple world-leading test publishers and organizations, including SHL, Korn Ferry, and many others incorporated this methodology into millions of individual psychological assessments that have taken place in at least 37 different languages, across 40 different countries since 2010.
The following is a transcript of an interview of Brown conducted on July 27, 2023 by Jayci R. Pickering, PhD student at Oklahoma State University. This interview is reproduced with permission of Women in Research (WRM), and the interview was originally published on the WRM websiteopens in new window. Readers are encouraged to explore the website for many other wonderful interviews and resources. Additionally, WRM can also be found on LinkedIn opens in new window and their Facebook groupopens in new window.
What initiated your interest in research methods and thorough methodology?
“That's a really good question; my path in research methods is quite unusual because unlike many psychologists and social scientists, my undergraduate and master’s degree were in mathematics. I was studying calculus, algebra, mathematical statistics, numerical computational methods, etc. It was a given that this is what I wanted to do from an early age because, well, I guess I was quite good at it. So, for me, the question was not what interested me in methodology; the question was what application area to choose. Where to apply all of that knowledge. Many of my friends went into telecommunications and banking, very technical areas—and I guess I would be fine with that as well, but compared to what I do now? That’s just not of interest to me. I love what I do now. So, I met a PhD student in my department, mathematics, who was working with psychologists at the time. He was writing some algorithms for them, and I was helping with the analysis; the applications were purely psychometric—you know, measurement of individual differences—and I just found that so interesting, so I asked that friend to introduce me to those people. He did, and I started working with them very early in my career; I never looked back.”
How would you say your industry experience translates to your work in academia?
“I started with industry right after graduating; despite being very, very academic, I didn’t enter a PhD program immediately, as life was complicated. There were economic difficulties in my country, which was, at the time, the Soviet Union. But it was 1992—the collapse of the Soviet Union. Things were very hectic, and funding for research institutions or related work was basically collapsing. So, I didn’t go straight for the PhD. Instead, I chose to start working, first as a part-time research assistant in the psychology department. I mainly helped students with computations in the computing lab, and then I also had a baby, meaning I was a young mom who was super, super busy and needed money. So, I also worked on various cases with consultancy firms doing psychometric assessment in organizations; eventually, I got into SHL (Saville and Holdsworth, Ltd.), one of the largest world providers in the areas of management consulting and people management, talent assessment, selection, etc. It’s actually one of the largest employers of organizational psychologists here in the UK, and the company has many branches and offices around the world. Anyway, I started working for them at their Moscow branch in Russia. Eventually, personal circumstances led me to the UK in 1999, and I was lucky enough to start working with SHL’s head office. My overall employment history at SHL was probably about 12 years; I was a Support Technician, Research Technician, and ended up being a Principal Psychometrician or statistician in the company. It was an amazing experience. I ran very exciting projects, such as the Occupational Personality Questionnaire redevelopment using item response theory, and all sorts of test, translation, and adaptation projects with many international offices. It was an exciting and vibrant organization, and an interesting job because I did so many different things. My practical and industry experience actually drove many of my research questions—that's how I then decided to take a mid-career break, go into academia, and pursue a PhD. I was in my 40s at the time, which may be unusual; but it’s never late, really. One of my inspirations was the late Barbara Byrne, a famous professor who wrote exciting books about structural equation modeling. And I wanted to become a teacher—so I did. Now, I’m at the University of Kent. I’ve been here 10 years, and just got my professorship. So, to sum things up, the real-world questions always have, and continue to, inform my research; I can’t help it. I always think of things which are practical, needed, and useful, rather than focusing on some abstract questions. I try to ask questions I know my contacts and colleagues in the industry are looking to have answered. They are looking for solutions. And these very practical, applied problems are the things I’ve been researching using quite complex psychometric models.”
Sometimes it’s difficult to explain to others what goes on in academia. How important is methodology when translating our findings to those outside of the field?
“What straight comes to mind is Ben Goldacre, his books on bad science. He describes how people misunderstand science. And you know, our methodology is very technical, which means explaining the intricacies of what we do is not straightforward. I think students in particular need quite a lot of convincing when it comes to the usefulness and importance of our statistics and methodologies; they often seem overwhelmed by the idea, thinking, ‘I came here because I want to help people, and they make me do statistics.’ That’s the most common answer I hear, and it’s terrible. But I try to reason with students, asking them to imagine themselves as a practitioner who needs to make a diagnostic decision using some measurement tool. So, you’re using a tool that screens for depression... how do you know that what you are measuring is actually valid? How do you know whether you can rely on the scores and compare them over time? I try to get specific rather than treating things like a black box, and I think this helps persuade students that statistics are really quite important. When it comes to translating findings outside of the field and the classroom, I find that when my findings are motivated by a real need, a desire to solve some real-world problem, then I have no trouble translating my methodology and findings to people. And if it addresses someone’s problem, they will really want to listen; they will make an effort to understand exactly what you’re talking about.”
Can you speak to the necessity of rigorous methods within the field and throughout academia as a whole?
“I start the very first lecture with my master’s students—statistics and methodology for all psychology programs—by challenging their perceptions of the basic analyses they studied during their undergraduate degree. I tell them, ‘Sometimes you were taught to focus on such detail as to argue about the second and third decimal point of your p-values. And you picked up on these very specific and intricate things. But what I bet you haven’t considered, is how this thing that you’re analyzing—your dependent variable, for example—is measured. Whether the items you are using to measure this variable make sense, whether the measure is validated, how it was put together.”’ We start the course with this challenge before moving into the depths of psychometrics, where we talk about how things are often measured poorly, and we often make blunders. And this is part of my argument as to why rigorous methods are so important. I finish the course by asking the students to read an article by professor Denny Borsboom, from the University of Amsterdam; it’s quite a controversial one, called The Attack of the Psychometricians. The article doesn’t have formulas, so it’s less of a complex read. In it, Denny discusses why psychologists, who are substantive scientists, are so slow to adopt methodological innovations that already exist. And in sum, this is kind of my approach to discussing the importance of rigor in our methodology.”
How would you encourage early scholars who are interested in methodological issues? What specific opportunities, and with whom, might you encourage them to seek out?
“Well, again, I'm probably biased because of my own path in methodology, but I see no harm in getting out of academia and working in any industry that relies on methodology as their bread and butter—clinical trials analysis, educational interventions, and psychometric testing industry, etc. If a young person is really interested in methodology and they feel capable and willing to pursue that, I see no harm in them working outside of academia for a period to try something practical. They will learn so much, and this knowledge will make them valuable in both industry and academia. When you work in practice, you usually develop a much broader perspective on methodology; you can avoid, for instance, getting stuck in some narrow area of scientific interest without understanding the context it applies to. I see so many talented young researchers, specifically when I peer review articles, who start with some type of simulation. These are very popular in our discipline, and they’re usually vigorously performed, excellently programed and designed, everything. However, when I review these papers, I start questioning things, wondering why the authors chose to explore X & Y conditions when Z is typically used in practice. So, it’s useful to actually understand the context you are working in. And maybe those who don’t leave academia can build relationships with industry partners, allowing them to engage in practical work and develop a broader understanding of important context.”
How would you encourage scholars who find methodology to be intimidating and/or overwhelming?
“This is a difficult one, and it’s quite a common attitude, especially in the social sciences and psychology in particular. It’s a very female dominated subject in terms of student enrollment, meaning the student population is very female dominated. My own educational background was not in this country, as I said, but in the Soviet Union. So, I find it quite surprising that in the UK, and probably in the US and other countries in Europe as well, that girls shy away from math for some reason, thinking they aren’t as good as boys. But that’s just simply not true—there’s no evidence to suggest that is the truth. So, I think it’s partly a matter of lower self-esteem that develops into this fear of math, and stats; and it doesn’t need to be like that. I started in the mathematics department at Moscow State University in 1987, and the population of boys and girls in my department was 50/50. That’s unheard of, especially in applied mathematics and computing; computing was and is a male dominated subject. But it wasn’t like that for me, it was half and half. Clearly, females felt very capable of doing all kinds of stuff, which is the way it should be. So, I do think there is kind of a connection between finding methods intimidating and gender; that said, of course boys can find it intimidating, too. And that’s quite understandable. Another component probably involves insufficient, or maybe even dull teaching—because why else would people come into university already afraid? I would think part of that has to come from previous school experience, doesn’t it? As educators at a university, I think the best thing is to make things practical and provide our students with real world problems to solve. This may be controversial, but I always disliked jokey examples when it comes to teaching statistics. Some textbooks, for instance, might use jokey examples to demonstrate how we measure a T-test; they might say, measure whether chips which were fried and chips that were baked are different. I always disliked this because, I would argue, it makes students feel our subject is not particularly serious. If we’re just doing silly things like that, it all may seem like mathematical circles. So, I avoid using examples like that. Instead, I use examples from my own industry experience, so straight from organizational assessments. For instance, I used to work with large databases from child and adolescent mental health services in the UK; we would look at things like the prevalence of psychiatric disorders and mental health disorders among children, and how parents and clinicians report them. I always use these examples and others like them because I find they convey the importance of what we do. So, why is it important to engage in rigorous analysis? Because it leads to really important results and substantive conclusions that will perhaps influence policy or enhance our understanding of certain conditions. I think it’s especially important to give those who feel overwhelmed by methodology real world examples, showing them how we can analyze, interpret, and make decisions based about results. I don’t know if this is the best way or not, but it’s what I have intuitively done over the years. Each time I explain a topic, I provide an empirical example and walk students through how I would approach the data; what would I look at first? I would tell them, for instance, that I wouldn’t even start running a factor analysis before looking at distributions and correlations—because why would you need to run a factor analysis if nothing is correlated with anything else? I just try to walk them through my process; what I would do, when and why I would do it, how I would interpret the results, etc. That relaxed approach seems to help people recognize that it’s not all clear cut, so they don’t need to be afraid when they don’t find it very clear cut; I can almost guarantee we don’t find it clear cut either. At the end of the day, research methods are a tool helping us understand our interests—they are not something to be scared of. I’m trying at the moment to launch my book. It’s very practical work; looking and examining a data set, showcasing particular techniques on different data sets, explaining how to do things step-by-step... it’s really emerged from my teaching.”
What are some of the developments you foresee being relevant in regard to the future of research methodology?
“It’s always tricky to talk about the future. I think anyone who talks about the future finds themselves wrong after a short period of time; it’s impossible to predict. That said, an obvious one here would be to mention increase computational power. We can now do things we could never do before, like analyzing huge volumes of data. Artificial intelligence, machine learning… all of these things will have to shape the future. So, I don’t know. I’m sometimes scared to think how technology will shape the future, and what this will bring in terms of methodology. I also always want to mention the ethics of it. I’ve always been quite concerned when we get really excited about new technology and start applying it without thinking very much—just because we can. I think we need to be quite careful about implications and whether what we do is ethical. Even now, the things people say online—on various social media platforms, like LinkedIn, Facebook, or whatever—can be scraped and analyzed. Conclusions can be made using this data, and I’m scared of these things; because as a psychometrician who has specialized over the years in test validity and fairness, I’m not sure about the validity and the fairness of these practices. In all of these new technical developments, we need to catch up in terms of our rigorous standards for validity to make sure what we’re doing makes sense. That what we do is sound, and we are measuring what we want to measure. That we are not invading privacy. All of these things. There’s an old saying that goes, ‘Garbage in, garbage out.’ If we don’t ask the right questions when we collect information, it doesn’t matter how many AI engines you apply, we will never come up with the right answers. Because the meaning will not be there. So, I don’t think our job as methodologists and as researchers will ever be redundant because what we bring to the table with all of this technology is that judgment of what’s valid, meaningful, and useful.
In 2022, Lisa Lambert and Tina Köhler took over as co-editors of ORM, representing the first time the journal has been led by female scholars. Can you briefly speak on the importance of increased representation throughout the area of research methods? How might we continue to encourage this representation? How might we continue to champion and support one another?
“This is a really excellent development. I’ve published in this journal, so it’s very close to heart; it’s one of the journals I hold in high regard, in which I would like to publish more. This question brings to mind a poster I have in my officer; it’s the past presidents of the Psychometric Society, all their photographs. The title changes every year, and the society was established in 1935. So, the society has been in existence for almost 90 years now. And I checked yesterday– out of these almost 90 presidents, only six have been women. That’s amazing, isn’t it? Of course, the situation has gotten better; in more recent years we’ve been witness to progress. But would you believe that between 1935, when the society was first established, and 1996, there was only one female president. Quite early, actually, Dorothy Atkins was president in 1949. And again, that’s just amazing to me. You look at this photograph, and among all of these men, you see that one woman. And you think, ‘Wow, how did she make it?’ So, obviously things are better now, and we see representation in many places—at conferences, as Presidents, Professors, and now editors of ORM. This is great to see. And the lack of representation in previous years is despite great contributions women have made in our area, oftentimes behind the scenes. You know, there are more females than males in the “lower” levels of academia—more research assistants and PhD students—but as you trickle up, there becomes fewer and fewer representation. And I don’t think the situation will change unless and until it changes everywhere.