Telling your research story with numbers: impact metrics for the humanities

Question

In each annual review, I’m feeling increasing pressure to quantify my “impact,” even though that is a ridiculous concept in my field. Do I have to play this game? Are there any ways to subvert it? I don’t use social media, so “alt-metrics” aren’t for me.

– Anonymous (medieval studies)

Dr. Editor’s response:

The idea that the value of scholarly work can be quantified in citations — let alone by indices that don’t track monographs — is indeed disturbing. And the rising tide of numbers isn’t limited to bibliometrics such as the h-index or systems like the U.K.’s REF scheme: in both Ontario and Alberta, we’ve seen provincial governments tie funding for universities to performance metrics such as graduation rates and student income after graduation (see CAUT 2020).

As Kathryn Maude persuasively argues, these metrics are problematic not just because they inaccurately measure the true number of citations for humanities monographs in particular: they also “can make our work less radical,” as the “pressure to publish limits intellectual curiosity and forces research down more conventional paths” (2014, p. 247).

In short, there’s good reason to be concerned about the pressure to demonstrate a quantifiable impact. It’s appropriate to use whatever power you have in your institution to resist metricization.

For many, however, the annual review and the tenure and promotion dossier won’t be the most effective place to mount such arguments. To determine which measures of influence are appropriate for faculty in the humanities, I spoke with Duke University librarian for literature and theater studies Arianne Hartsell-Gundy, who, like me, takes a pragmatist’s approach to impact metrics when working with individual scholars.

Arianne’s first piece of advice is, always, go to your favourite university librarian and work with them. Your institution’s librarians will be most familiar with your institutional context, and so are better positioned to support you than the following general guidelines. With that said, here are some starting points that Arianne recommends for making numerical claims about the quality of your humanities research:

1. “I publish in journals with X% acceptance rates”

The influence of a journal is too often assumed to be reflected in its Journal Impact Factor (JIF), an indicator of the average number of citations of articles published in a journal over a short term. As Marc Couture has argued, JIF is “an inaccurate and unreliable measurement,” and the connection between JIF and the quality of any single article it publishes is “sketchy at best” (2017).

If you publish in journals that don’t have a JIF, or in journals whose JIFs appear low when contrasted with those of other disciplines, one alternative metric is to cite a journal’s acceptance rate. The MLA Directory of Periodicals is fairly comprehensive for literary studies, and includes journals from other humanities disciplines; if you’re not a member of the MLA, you can find the directory through your university library. In the “submission details” section, look for the number of articles published per year, and divide that by the number of manuscripts submitted per year to calculate an acceptance rate.

2. “X number of libraries hold my book”

WorldCat is more than just a way to look for books to request through interlibrary loan. Through WorldCat’s Identities function, you can search for your own name and see a listing of each of your books. While the “editions” feature may be a source for confusion—WorldCat counts as separate editions soft and hard covers as well as physical and ebooks—you can reliably cite the number of WorldCat member libraries worldwide who hold a copy of your book. Search for your colleagues and you’ll quickly get a sense of whether or not finding your book held by 2,333 WorldCat member libraries is impressive.

3. “People from X number of countries are teaching my work”

When your work begins to be taught in university classrooms, you know you’re having an influence on the profession. The Open Syllabus Project contains over six million syllabi, and you can search for your own name and publications to see how many times your work has been taught. If your work appears in the Open Syllabus corpus, you’ll be able to click on each cited title and see a map of the countries in which your work is taught, as well as the names of specific institutions in which you’ve appeared on a syllabus.

Of course, not everyone who teaches a university course chooses to upload their syllabus to this database — but, then, all database searches are limited by the size of the dataset. The more researchers become invested in the principles of openness in research specifically and academic work generally, the more robust these datasets will become.

Read also: Performance-based funding comes to the Canadian postsecondary sector

I personally find the Open Syllabus Project easier to search than a platform like the Humanities Commons, which uses a more limited set of metadata for its syllabi, but the latter can show you how many people have downloaded your materials, which may be an useful metric to attest to your influence as an educator.

I’m thankful to Arianne for sharing these suggestions and resources with me. If you’d like support in developing a set of numbers that make sense for quantifying your work — or the work of your department — please consider starting with your university library’s subject librarians.