Big ships turn slowly, but they go faster with more tugboats.
When we applied for the HuMetricsHSS grant, our goal was to devise a set of parameters that our research-intensive (R1) university’s administration could use to accurately identify faculty in arts, humanities, and allied design disciplines who merited nomination for internal and external prestigious honorifics. Our university’s current stated criteria for these awards is typical: evidence of merit includes “peer-reviewed articles in leading scholarly journals,” “scholarly monographs or similar works,” “a consistent track record of external funding,” and “development of intellectual property”– all of which are deemed “high-impact scholarship” as measured by the h-index and similar bibliometrics. At the conceptual level, the idea of seeking discipline-appropriate, objective measures of research performance isn’t inherently exclusive. But in practice the terms “objective,” ”measures,” and even “research performance” privilege some disciplinary vocabularies at the expense of others. It would be difficult to ascribe these words to the interpretive, creative, multimedia, community-based, and public-facing forms of work common among scholars in the humanities, arts, and allied disciplines of design. This mismatch is unsurprising given that the STEM-focused institutional cultures that produced the dominant evaluative vocabulary are so distant from our own.
We can—and do—quibble with the reductionist existence of metrics as a concept. But we also live in the world they govern, work at an institution that values them, and want to be part of the conversation about transformative change for the better. As we reckoned with this position our question became: How can we bend the available metrics frame so that it makes our research as visible as possible? The pragmatic desire to create a shared space for cross-articulating the work of the humanities/arts/design and STEM fields led us into a deeper engagement with academic metrics and the tools used to generate them.
Answering this question involved knocking on many doors: the Office of Research, the Office of Institutional Planning and Research, the Libraries, and beyond. We learned deeply about Academic Analytics (its history, data logics, and sales packages), Google Scholar, Dimensions/Altmetric, and other commercial products. And we came to see the difficulty in aligning and connecting systems used by many different areas of the institution (public communication, tenure and promotion, and institutional assessment, just to name a few). At a comprehensive R1 institution, this is a massive barrier to change: no single entity has the power to align multiple enterprise systems even if they wanted to (which they might) AND has sufficient resources (which they don’t).
Along the way, we picked up allies in surprising places: the medical school, engineering, anthropology, and beyond. It turns out the systems our institution uses to tabulate and publicize the value of work in the arts, humanities, and design obscure value in many other places too—including on fields intimately allied with quantitative processes and focused in real-world impacts. Moreover, we learned that one of the fundamental limits to tracking research in the arts, humanities, and design is the ability to support a “forever home” for that research in the sense of a persistent digital identifier. Rather than cause distraction for faculty, bringing attention to the need to support/archive/embrace saving creative “expressions of scholarship” (to quote Emily Janke) for the future is an important outcome too.
Where are we at the end of this? In part because of the conversations we inaugurated as HumetricsHSS fellows (though only in part!) we have new software to play with and a mandate to explore how to share and aggregate impacts that, to a certain extent, we can define ourselves. We have more and better questions, and we have some concrete next steps to connect faculty to forever homes for their work.
What have we understood about our work through this? Seek to see the whole elephant, but don’t let that stop you from working on the part you can. Find the places with momentum to work from. Enter the conversations. Think about uses and misuses of the systems, and just when you think you’ve got it, look at it again upside down. Use your voice, and others will be glad for it; it takes a village to see the unknown unknowns.