Assessment is one of psychology's defining contributions to science as well as health care. Yet rather than being well-known and celebrated, assessment seems to be on a retreat, with fewer undergraduate and graduate courses taught. The public is not aware of assessment options, and many methods that could help identify problems and improve outcomes are under-used because people do not even know about them, let alone where to find them or how to score them.
The WikiMedia Foundation has created websites that are open-access with crowdsourced content, of which Wikipedia is the largest and best known. Wikipedia has more articles than the Encyclopedia Britannica, rivals or exceeds Britannica in terms of accuracyopens in new window, and gets updated extremely rapidly (see articles about ebola outbreaks or Hurricane Florenceopens in new window for recent and high-stakes examples). Wikipedia is currently the fifth most visited site on the Internet, and it has a privileged relationship with Google (the No.1 most visited site), whereby Google will put relevant Wikipedia articles on the first screen full of hits generated in response to searches.
Div. 5 is partnering with three other APA divisions (Div. 2, Society for the Teaching of Psychology; Div. 12, Society of Clinical Psychology; and Div. 53, Society of Clinical Child and Adolescent Psychology) and a nonprofit service organization, Helping Give Away Psychological Scienceopens in new window (HGAPS), to explore ways of using open online platforms to improve the teaching, research, and application of psychological assessment. The current project, supported by a Committee on Division/APA Relations (CODAPAR) Interdivisional Collaboration Grant from APA, brings together students and content experts to build and update pages about psychological assessment. The grant is the latest in a series of eight small grants that have experimented with different ways of linking wiki editing and teaching.
The Challenge
In theory, anyone can edit Wikipedia. In practice, it can be challenging to make edits. Wikimedia is developing a visual editor that makes it easier for people with no computer coding experience to make edits, but some of the platforms and more advanced features still require coding. This creates a barrier to entry, and content experts may not have the interest or time to grow the requisite technical skills.
A second barrier is that many pages are locked to prevent vandalism or inept editing. Pages about public figures and pages about health care are categories that usually are locked, requiring an editor to accrue a track record of good faith editing before they have credit to edit pages.
A third barrier is that the Wikipedia editor community has its own social mores and etiquette. Most editors have no psychological science background, and often assume that academics will be more focused on self-promotion than adding content with a neutral point of view.
Making successful edits about psychological assessment requires two fairly rarified skill sets: content expertise about psychometrics and assessment, and coding skills with savvy in navigating wiki.
A Solution: Blended Teams
We have experimented with several different methods to try to grow the necessary skills or bring together the needed expertise. "Edit-a-thons" using food and prizes created incentives for people to come and learn about editing. These were fun and effective at creating exposure, but few projects were small enough to finish in a single session. Something different was needed to create sustained engagement.
One solution is to incorporate wiki editing into a class. The Wiki Education Foundationopens in new window has developed a set of tools and consultants to help do this, and there are a variety of examples online. Limitations of this method include that undergraduates, even in upper level courses, do not usually have a lot of sophistication about psychological assessment. The pushback from the editing community is a wild card that can make it difficult for the instructor to control the outcome of assignments. Even when all goes well, embedding the edits in a class also only engages students in sustained editing for 12-15 weeks.
A more successful model is to have students volunteering in a research lab learn to make edits and then continue to focus on the project over a longer term. HGAPSopens in new window spun off from the lab team, as the number of students interested in getting service experience and learning skills kept growing. The students created a service club, chartered with UNC Student Lifeopens in new window in 2016; they then created a non-profit incorporated in North Carolina in 2017, and obtained IRS recognition as a 501(C)(3) charitable organization in February 2018.
HGAPS has weekly meetings and uses grants and dues to pay for food at every meeting. The title of this article alludes to the success of the model. With students and early career contributors, "If you feed them, they will come!" Attendance has been one- to three dozen students, providing a solution to the problem of making sustained edits outside of a lab or course setting. We also were able to pilot a small set of travel awards and summer funding as ways of increasing the amount of focused activity. Additionally, now that HGAPS is an IRS-recognized charity, we will be able to experiment with GoFundMe style campaigns as a way of supporting specific goals.
The result: An open online ecology supporting psychological assessment
After several years of trial, error and learning, we are figuring out how to build an open-sourced online ecology supporting psychological assessment. We focus on creating or upgrading articles about assessments on Wikipedia (audience: general public), and building sister pages on Wikiversity (audience: Psychology students, researchers, and clinicians). As non-psychology editors challenged some of our content as becoming too technical, we moved it over to Wikiversity. This has been an effective way of trading greater control and freedom to develop more technical resources, while also keeping the information "within a click" for the general public.
We are using the Open Science Foundation (OSF) web site, OSF.IOopens in new window, to host copies of the measures, which we then link back to Wikiversity and Wikipedia. OSF.IO offers several advantages: it is endorsed by the Association for Psychological Science, so it has recognition and cachet among academics (whereas Wikipedia still gets mixed perceptions); it offers much more titrated options for accessing files (whereas Wikimedia requires "giving it away" in perpetuity); and projects and objects can be registered with a digital object identifier (DOI) — addressing attribution and credit issues essential for academic career advancement.
Products
We have a well-developed, still growing set of Wikiversity pages focused on evidence-based assessment. The home pageopens in new window and outline follow the theoretical framework and the arc of a typical clinical encounter, moving from things that can be done before the appointment is scheduled, to assessment in the waiting room, tactics for the first interview, and tips for assessing over the rest of treatment and into long-term monitoring and relapse prevention.
A second way of framing the content approaches the material in terms of clinical questions and disordersopens in new window, with the prototypical user scenario being a graduate student casting around to hook a package of materials to address a specific referral question (e.g., "How would I effectively assess whether or not a youth's academic and social struggles might be due to Attention-Deficit/Hyperactivity Disorder, a mood problem, or some other issue?").
A third way of organizing material uses case-based learningopens in new window, showing how different methods and measures are used to guide decisions with help-seeking individuals.
There have been several unexpected innovations as well. We have used the Wiki/OSF hybrid to support research engagement. Because the projects have been surprisingly social and fun, we refer to them as parties. Receiver operating characteristic (ROC) analysis is a much more pragmatic way of analyzing data for decision making than the Student's t-test, but it is not commonly taught in graduate programs. We have used Wiki/OSF.IO for team building and syntax-sharing, and a flipped classroom format to try to accelerate uptake of these methods, running a set of ROC partiesopens in new window, with more than a dozen peer reviewed papers connecting to the initiative. These have been effective at getting content experts and their teams to re-think how to package results to maximize clinical utility.
The poster parties have pushed the envelope of student engagement by taking psychological assessments related to course content (e.g., the Behavioral Inhibition/Behavioral Activation Scales, or the Big Five Inventory) or to clinical content (e.g., the Hypomania Checklist), having students take an anonymous survey, receive their deidentified scores and then discuss. Getting the score immediately raises a set of questions – "What does this number mean? How does it compare to others?" This leads quickly to a deeper understanding of norms. We have gone further, organizing motivated students into small teams that develop a question into a formal hypothesis and then learn about data visualization (both for checking assumptions and for presentation) as well as statistical analysis.
We have had a series of three waves of poster parties. These were designed as extracurricular activities that do not count as a course assignment or extra credit; instead, they have been focused on intrinsic motivation and career development. We have presented more than two dozen posters at regional, national, and international meetings. Stacking the posters together as slides in a PowerPoint deck, with essential syntax embedded in the speaker notes, creates a surprisingly powerful learning tool where people can see example paired with the code to generate the figures and results. This makes people more enthusiastic about working with the syntax as well as increasing the number of options in their functional toolkit.
Another surprise has been that positioning most of the materials online has made it more feasible for distributed teams to work together on projects. For all of the ROC and poster parties, half of the participants have been working long distance, using email and social media to supplement screen-sharing and videoconferencing during flipped classroom style sessions. Participants from Europe, Canada, Taiwan and Korea have all succeeded in generating co-authored work using the format. This also has revealed power of having one shared resource, rather than continually re-inventing materials for each group or class. Using Wikiversity or Wikipedia means that there is one platform for the English language, making it easier to find and share materials.
Key Results
Cumulatively, we have built or edited over 200 pages on Wikipedia and Wikiversity, including more than 80 in the evidence based assessmentopens in new window site alone. These have accrued more than 52 million page views since we started working on them. The screening center versions of the public domain questionnaires have had more than 3,500 users in half a year's time. Participants presented six posters and talks at the APA Annual Convention in San Francisco in 2018 — including student authors who came from Taiwan — and we aim to replicate and extend the model at the Chicago convention in 2019.
Growth Areas
More than just providing content, involving D5 experts has been crucial for identifying boundaries, ethical issues and places to proceed thoughtfully. Issues of how to represent measures that are copyrighted and commercially distributed, how to protect the security of test materials, and reactivity or social desirability are all crucial. These are larger issues that the field is sorting through with technological advances and open science in general. It is fascinating and exciting to have these conversations develop while engaging students in learning activities, rather than having them remain abstractions.
Ways to Get Involved
Intrigued? There are many ways to get involved! You can use the resources in your teaching. Pull down the measures for classroom exercises, use the recommended codebook and the scoring syntax will be available, too. If you want to have students learn about writing for the general public, as well as about dissemination in general, consider working with WikiEducation to set up some exercises in conjunction with a class.
Please provide feedback about pages. Conceptual suggestions and advice are as welcome as details about specific measures. You can make suggestions on the "Talk" or "Discuss" tabs on wiki pages, email themopens in new window or edit the wiki page directly. The CODAPAR grant officially ends Dec. 31, 2018, and we will update and share our key results then. But the goal is to build a set of resources and processes that will continue to get used and refined for years to come.