The Syllabus as HuMetrics Case Study

Current metrics of humanities scholarship have been shown to be too blunt to capture the multiple dimensions of scholarly output and impact (see Haustein and Larivière). In addition, the inappropriate nature of current indicators can incentivize perverse scholarly practices (see The Metric Tide, Wilsdon et al.)…. In this context, our challenge and our responsibility is to articulate, incentivize, and reward practices that enrich our shared scholarly lives and expand our understanding of scholarship itself.

— Christopher Long, “Nurturing Fulfilling Scholarly Lives"

We chose the syllabus as guinea pig: what questions could we ask ourselves about our syllabi to ensure that they, as objects, embody the values (Equity, Openness, Collegiality, Quality, and Community) and practices we want to encourage and reward in the humanities community? 

—  Nicky Agate, "The Syllabus as Scholarship"

For nearly a week, the HuMetrics team circled in on a set of values and modes of scholarship that often go unrecognized and unrewarded when talking about scholarly productivity and academic excellence. Chris summarized prior posts and wrote about the intellectual framework that guided our deliberations, and Nicky outlined the value framework (very much a work in progress, and in need of feedback and testing) that can help shape the kinds of indicators we might want to gather. Values were important for us as a starting point because, as we reminded ourselves over the week, any kind of metric can be traced back to values, but those values are often tacitly present and uncritically examined, leading to the presumed status of metrics revealing value rather than reflecting it. How, then, might we reframe the indicators that help us tell the story of humanities and social science scholarship, starting with a set of values and practices (as Nicky remarks in her post) that we want to "encourage and reward in the humanities community."

The syllabus became an object of keen interest to our group as an interesting test case, as an object that reflects a set of values, of scholarly decisions, and importantly, of time investment. We wondered how the syllabus might enhance citation indicators in the humanities and how such indicators might help us rethink and influence notions of impact that currently favor an article-based intellectual economy. Does the syllabus reveal the circulation of ideas in humanities and social science subjects in more effective ways than traditional citation networks using only articles? Is the syllabus a site of more rapid or even potentially up-to-date scholarship, rather than that reflected in the slow burn of a book three years in the making?

Additionally, using the syllabus as a potential site for indicators reflects some of the values expressed in our previous postings, and stands to offer us a more comprehensive view of scholarship, citation, and influence. The syllabus serves as connective tissue between our research and teaching practices, where the latter is often under-rewarded and under-recognized labor for the purposes of promotion. The syllabus expands our conception of audience for considerations of impact, bringing students into the conversation and offering a more comprehensive view of scholarly networks. As an object that reveals choices and policies from the instructor, but also speaks to the impact of the authors featured, the syllabus offers multidirectional indicators. For example, what do we learn about the variety of scholarly modes of expression, from the blog post to the article, the archive to the monograph? What might we learn about the impact based on the authors cited? What about the diversity of representation for authors, regions, or disciplines? We might even stand to learn if citations used in the classroom align with citations in the articles and books in frequently-circulated research literature, or more about the role the syllabus plays in amplifying and influencing other (often more rewarded and recognized) forms of scholarship like the article or book.

Because of its unique format, where citations are contextualized by some form of time stamp (the class period, e.g.), we hope the syllabus can also help us capture engagement indicators, based on a ratio of the number of texts in a time spot and the time allotted. A syllabus might have one class meeting, for example, where there are two required readings, and the following class meeting where there are 13. Where certain kinds of citation metrics can lead to a game-the-system approach of citation flooding, an engagement metric based on a texts:time ratio inverts this approach, where a fewer number of texts during a time period would suggest a higher level of engagement, and a higher number of texts might suggest a lower level of engagement. While speculative at this point, such an indicator might inform the development of other metrics that serve as speed bumps, rather than accelerants, to practices that serve metrics alone rather than advancing scholarship.

This third post, following Nurturing Fulfilling Scholarly Lives and The Syllabus as Scholarship, serves as a summary of the HuMetrics team presentation on the final day of #TriangleSCI. In the coming weeks, our team will be co-authoring an article discussing our preliminary research, our deliberations, our attempts to reverse-engineer metrics so that we can begin with a question of values, and our next steps toward operationalizing this work in a series of case studies. Importantly, we welcome your feedback and input, so please comment, respond, challenge, and contribute.