Why citational justice is integral to creating research with impact

As part of our blog series on creating policy impact, we thought it was important to reflect on the process of creating good evidence. In this piece we explore the responsibility that the people involved in the production of research have to ensure that the knowledge generated is as high quality and representative as possible.

We spoke with Sheila Craft-Morgan to learn what it takes to produce the best research, and why current research methodologies are falling short. She explains how to bring citational justice into your scholarship and how this can help create better policy.

Sheila is the Research Impact Librarian at Ohio State University and is an expert on citation bias and diversity, equity and inclusion in research. 

Overton: You recently did a study looking into the content of library guides on research impact topics at ARL institutions. Can you tell us about this investigation? 

Sheila Craft-Morgan: In August 2022, I began working as the Research Impact Librarian at Ohio State after over twenty years in institutional research. One of my first priorities was to review and update the resource guides that I 'owned'. I found that, while the guide was extensive, there were no references to citation bias, ‘Cite Black Women’ or any resources that addressed diversity, equity, inclusion, and justice in relation to research impact. So I was curious about what other institutions were including in their resource guides on this topic.

At this point, I looked at the guides of similar institutions, in this case Big Ten Institutions, to determine what type of information is most often included in their research impact resource guides. What I found was that the information fell into three big buckets - author identifiers, like ORCID ID and Scopus ID and researcher profiles; resources about citation metrics and alternative metrics; and then many of the institutions also included references to responsible research assessment. However, when I did this benchmarking, I didn't find information or resources that discussed bias in the research process, and in particular, citation bias, on most of the guides. I will note that the University of Michigan’s guide did include links to resources such as Unintended Cognitive and Systems Biases from the Declaration on Research Assessment (DORA).

So I decided to turn this into a research project. I submitted a poster proposal to ACRL and expanded my population to ARL libraries, specifically the top 50 ARL libraries, measured by the ARL investment index. For this study, I conducted a content analysis on the information contained in research impact resource guides and found that diversity, equity, inclusion and justice, or bias in relation to research impact was only referenced in about 14% of the resource guides. I’m now working on a paper that expands this project.

Can you tell us more about citation bias? What is it exactly? 

Citational bias has been heavily studied in other fields and has multiple definitions. In this context, I’m referring to the bias that results in publications being cited not because of the quality of the publication but because of the perceived qualifications of the authors. For example, various studies have found that publications are more likely to be cited if the authors are already highly cited or are published in a highly ranked journals, that men tend to self-cite more than women, women are less likely to be cited overall, and scholars from more prestigious institutions and certain regions are more likely to be cited. It’s not ‘intentional’, necessarily.  But it is important to note that this doesn't happen in a vacuum.  There is bias at all stages of the research cycle, from research to peer review to publishing. It’s a vicious cycle. It is also systemic, which was discussed in a recent study about how research is cited in a prominent medical journal.

Some researchers have referred to the effect that some research is not seen, not considered, or not included in the larger scholarly conversation because it is not cited as  “screening”, epistemological racism and epistemic exclusion among other terms.  

This issue is also personal for me as a woman, as a Black woman, and as someone who believes in the value of diversity, equity, inclusion and justice. It is disheartening to find that there is very little research within the library and information science field on this topic. To learn more about it you really have to go outside the field. Although I have found some information about this topic in the background information or additional resources, this issue is not directly addressed in responsible assessment of research frameworks. And I think it’s a glaring oversight.

This seems like a huge problem for knowledge production, and for society in general. We perhaps wouldn’t go so far as to say that the research produced in these systems is bad, but it certainly seems incomplete. This seems like a big issue that researchers and research facilitators should be trying hard to grapple with. Do you have any ideas why it’s not getting more attention?

That is a great question and one I am interested in exploring further. We live in a society that believes that it is a meritocracy while refusing to acknowledge the systemic barriers and institutional structures that are in place that prevent it from truly operating in accordance with that ideal. Within libraries, we are slowly reckoning with this cognitive dissonance with the work that is being done by authors writing in the area of critical librarianship. I especially like the concept of vocational awe, which is that although the library community holds this ideal that libraries are an equitable space, it was not always the case. It is important to be aware of the context of the history and the lived experience of both patrons and library employees. One of the sources that I reference frequently is the ACRL report, Open and Equitable Scholarly Communications. This research agenda reflects on the need for equity at all levels of scholarly communications - in people, content, and systems. And while it does mention citation bias, there is not an extensive discussion. Instead, the authors call for more research in the area. For me, this was my call to action.

I also think there is a lot of emphasis on ‘the data’ within the field of research impact and bibliometrics. While researchers have focused their attention on the accuracy of the calculations, what's missing is the context. How are people, and individual choices they make, affecting the data? It’s not just whether the data is factually correct, but rather whether the data is complete. Just focusing on the calculations and the science misses the full picture. 

Can you talk about how you’re trying to address this problem? What steps are you taking to affect change and get people to pay attention to this widespread methodological issue?

My starting point is just to talk about it. So whenever I do a workshop, I make sure to stress that metrics are just an indicator of quality and that many of the responsible research initiatives and resulting frameworks are encouraging us to expand the way that quality is defined and assessed. I also hope that the research that I'm doing is helping to start a conversation. As an EDI Officer on the LIS bibliometrics committee, I am working with my colleague to create a bibliography about citational bias and other forms of bias within the research lifecycle and developing webinars where we are discussing this issue.

On a personal level, I also take this learning into my own work. For the research I do, I think hard about who I cite. When I’m writing about a topic, I try to find sources outside of the US and the global north and explore other voices outside of the library and information science field to learn about the issue from all contexts.That is why I am very interested in the scholarship and teaching about inclusive citation, in the area of information literacy, and citational justice.  In essence, it’s about acknowledging all of the work that has already been done.

Thanks for raising that - I think this is an important point to pick up on. What does citational justice mean in practice? How can people apply this to their own work?

I think the main point is around awareness. It’s about acknowledging that while someone has been published in a highly cited journal, it doesn't necessarily mean that they're the only one who's talking about an issue. So, it’s about expanding your search. As a starting point, it may be as simple as going further down within your results and exploring sources that aren’t at the top of your list. It might mean using other databases. There is a language bias in many bibliometric databases and it’s difficult to find sources that aren’t in English. This is a real stumbling block if you’re trying to be more inclusive in this way. While this might be difficult to address on an individual level, if you’re at least aware of the existence of research in other languages on a topic then maybe you can use translation services. 

Finally, at Overton we’re obviously really concerned with the life cycle of research beyond publication. What happens to it after publication? What decisions is it used to inform in the real world? Can you explore why this citational bias - or these flawed methodologies - would pose a problem for evidence-based policymaking?

I’m not an expert in this specific aspect of it, but I think it all goes back to the issue of exclusion. For example, if someone in the legal field is searching for evidence to support a legal argument there is a strong chance that they’ll opt to use research written by a researcher from a particular institution or published in a specific journal because it is perceived to have more weight... But does that mean that it's the right source to use? Does that mean the researcher or research organisation is the only one producing knowledge on this topic? Absolutely not.

Similarly, if we don’t account for a diversity of opinions and perspectives within policymaking then people’s lives can be seriously impacted. It’s especially important to include those different perspectives for issues that directly affect specific populations because they may have experiences that don’t necessarily fit within the ‘accepted wisdom’ on a topic. One example that comes to mind is health equity and the concept of health equity tourism. This example really underscores how systemic this issue is, and how it affects the research being conducted and published as well as how it influences policy.

Dedicated to my dad, Johnny Craft Jr., who always encouraged me to follow my dreams and modeled treating everyone, especially myself, kindly and with respect. Love you always, rest in peace.

We're grateful to Sheila for making the time to speak with us and share her expertise.