Rethinking humane indicators of excellence in the humanities and social sciences

Tag: TriangleSCI

The Syllabus as HuMetrics Case Study

The Syllabus as HuMetrics Case Study

Current metrics of humanities scholarship have been shown to be too blunt to capture the multiple dimensions of scholarly output and impact (see Haustein and Larivière). In addition, the inappropriate nature of current indicators can incentivize perverse scholarly practices (see The Metric Tide, Wilsdon et al.)…. In […]

The Syllabus as Scholarship

The Syllabus as Scholarship

“A critical component of our emerging #Humetrics conversation at Triangle SCI involves finding ways to expose, highlight, and recognize the important scholarship that goes into the all-too-hidden work of peer review, syllabus development, conference organizing, mentoring, etc. Our current metrics fail to capture what is […]

Nurturing Fulfilling Scholarly Lives

Nurturing Fulfilling Scholarly Lives

In Aristotle’s Nicomachean Ethics, there is a famous passage in which he reminds us that “to be happy takes a complete lifetime; for one swallow does not make a spring, nor does one fine day; and similarly one day or a brief period of happiness does not make a person blessed and happy” (Nic. Eth., 1098a16–20).

This passage came to frame our conversations around #HuMetrics at this week’s Triangle Scholarly Communications Institute, because it reminds us that a fulfilling life  —  what Aristotle calls, eudaimonia, happiness, that is, a life well lived  —  requires cultivated habits rooted in core values that, when intentionally practiced, shape the character of a good life.

In the end, what we value should be embodied in what we do, not once or twice, but regularly over the course of a lifetime.

In framing our conversation about #HuMetrics with this ancient conception of ethics, excellence, and character, we seek also to advance and reinforce the idea that a scholarly life can only be well lived in communities of practice with others.

For the #HuMetrics team, this year’s Triangle SCI experience was a swallow that signifies but does not yet fully manifest the coming spring. It opened for us a space for the flowering of a community of practice oriented toward the question of how we might more broadly cultivate communities of practice that embody the values of fulfilling scholarly lives.

Five Core Excellences of Enriching Scholarship

Working out loud together, we identified five core excellences of enriching scholarship:

Rebecca Kennison wrote about Equity.

Simone Sacchi wrote about Openness. His point was amplified further by Rebecca in her post about The Value of Openness.

I wrote about Collegiality.

Jason Rhody wrote about Quality.
Nicky Agate wrote about Community.
And Stacy Konkiel sought to tie things together by distinguishing between enriching and corrosive values.

In writing together in this way, we seek to embody the excellences for which we advocate.

The question that animates our work is this:

The Winter of Our Discontent

For too long, we humanists have been allergic to metrics. This allergy has prevented us from engaging in a serious and sustained conversation about what practices of scholarship we might want to cultivate and incentivize both through the activities we measure and those we celebrate.

As a result, a large and growing battery of metrics have been developed based on the practices of more scientifically oriented scholarship or simply on what it was possible to use our technologies to measure.

Current metrics of humanities scholarship have been shown to be too blunt to capture the multiple dimensions of scholarly output and impact (see Haustein and Larivière). In addition, the inappropriate nature of current indicators can incentivize perverse scholarly practices (see The Metric Tide, Wilsdon et al.).

A critical component of our emerging #HuMetrics conversation at Triangle SCI involves finding ways to expose, highlight, and recognize the important scholarship that goes into the all-too-hidden work of peer review, syllabus development, conference organizing, mentoring, etc. Our current metrics fail to capture what is most substantive about the rich life of scholarship we practice together in living academic communities.

In this context, our challenge and our responsibility is to articulate, incentivize, and reward practices that enrich our shared scholarly lives and expand our understanding of scholarship itself.

Without being naïve about how difficult it is to change culture, we hope to begin to reshape the conversation about metrics around the values of enriching scholarly practices and the communities in which they thrive.

Although our time together at the Triangle SCI was only one swallow that does not yet make a spring, the seeds planted there may begin to take root over the weeks and months to come, and the communities of scholarship that blossomed there just might be “made glorious by this sun” that shines when a broader public is invited to join the conversation.

Changing Behavior by Changing Incentives

Changing Behavior by Changing Incentives

It’s our last day at Triangle SCI, and I’ve been contemplating overnight the feedback the #HuMetrics team got yesterday afternoon from our colleagues. In our presentation of what we’ve been doing this week, we attempted to sketch out, using the activity of mentoring, how actions […]

The Excellences of Scholarship: Collegiality

The Excellences of Scholarship: Collegiality

We began the day with a walk. The morning was cool and fresh, and the grounds around the DuBoise House at the Rizzo Center offered the six of us on the #HuMetrics team at the TriangleSCI the peace and space we needed to think out […]

On Quality

On Quality

Today is the second full day of the Triangle Scholarly Communication Institute, where I am part of a team that’s focusing on HuMetrics: Building Humane Metrics for the Humanities. We have each agreed to quickly blog some thoughts as part of our process; warning: what follows is quickly-crafted prose, full of rough edges.

The #HuMetrics team is making progress in attempts to reverse engineer that which we want to measure, by starting with the values than often govern the (in some ways broader, more comprehensive) range of work that we engage in (see Chris Long’s previous post for further discussion). We were able to distill a wide-ranging brainstorm of values into five categories, and have aims to think about how those value categories relate to the processes and products of our work. The important core of our conversation today centered around the fact that while metrics often seen (or are taken) as the end-goal, in point of fact the indicators always align to a value, and so part of the work here, in this free and open thinking space, is to be aspirational about the values we’d like to see elevated, incentivized, and rewarded. If openness as a value is prioritized, for example, one could imagine more weight given to scoring articles and/or journals that were OA rather than not. If that seems like an extreme example, it’s perhaps a worthy future exercise to consider the ways that certain ways of showing impact actually might already tweak the scale toward different values.

For our quick 20-minute afternoon exercise, each of us is taking a crack at writing about one of those five values: Equity, Openness, Collegiality, Quality, and Community. In our framework, Quality can take on one or more of the following characteristics:

  • Replication
  • Soundness
  • Pushing boundaries
  • Creativity
  • Originality
  • Advancing knowledge
  • Intentionality
  • Reproducibility

Replication and reproducibility have a certain emphasis in some of the social sciences, but the terms might in other contexts also be thought of in the sense of extensibility. It’s important to note that these are preliminary notions, and we welcome your feedback.

Keep in mind that we’re considering metrics than can apply to multiple kinds of academic processes and outputs, so not just whether or not your article or book has high quality (measured currently, say, based on whether or not the article is in a journal with a high impact score, or if your book receives a certain number of citations), but whether or not you play a role in helping measure the degree of quality for an object (say, serving as a peer reviewer for a grant application, a reviewer for a book, a referee for an article, etc.). Both of these kinds of activities are part of the transaction related to “quality,” but currently we overwhelmingly incentivize and reward the former rather than the latter. What such a focus on the transactional value in the sense of quality does is unpack the transactional relationship and scholarly networks that undergird much of our work, disturbing the notion of individual acts of scholarship by revealing the deep relationships behind scholarly works.

Another challenge to current methods for measuring impact related to quality is the well-noted challenge of addressing context (such as whether or not a citation is a positive one or a negative one), and the degree to which such measurements lend themselves to a certain kind of gaming the system (through overuse of citations to drive up citation scores). How do we successfully implement speed bumps (rather than roadblocks) that require some small additional effort that will likely not prevent gaming the system but may, to carry the metaphor, slow it to a reasonable speed? The use of “active citation” (a precise and annotated citation and link to the source), as argued by Andy Moravcsik (PDF) in the context of political science research, is one potential method, especially for more qualitative work.

Follow team #HuMetrics as we wrestle with humanities metrics. We are Christopher Long, Rebecca Kennison, Stacy Konkiel, Simone Sacchi, Jason Rhody, and Nicky Agate, and we’ll be writing here all week.

Community as a Humanistic Value

Community as a Humanistic Value

Today is the Day Two of the Triangle Scholarly Communication Institute, where I’m heading up a team that’s focusing on HuMetrics: Building Humane Metrics for the Humanities. Our team has focused a lot on the importance of working out loud, of process over product, and […]

On Openness

On Openness

Second day at #TriangleSCI with the #HuMetrics team. Today we focused on clustering our values into major value-categories (Equity, Openness, Collegiality, Quality, and Community) with the idea that excellence in scholarship is an expression and combination of these value-categories as they are embodied in scholarly […]

Equity as a Core Value

Equity as a Core Value

We’ve just completed Day 2 at #TriangleSCI, and more hard (but good) work is now behind the #HuMetrics team. Today we took our huge brainstorming list from yesterday and distilled the values we believe should underpin the development of “humane metrics.” We came up with five such values: Equity, Openness, Collegiality, Quality, and Community. I am looking here at equity. My fellow teammates Simone Sacchi, Chris Long, Jason Rhody, and Nicky Agate are tackling each of the others, and Stacy Konkiel is writing on the overarching question we looked to tackle: influence vs. impact.

What do we mean by “equity”? For us, equity is very much the concept described by Falk et al. (1993:2):

“Equity derives from a concept of social justice. It represents a belief that there are some things which people should have, that there are basic needs that should be fulfilled, that burdens and rewards should not be spread too divergently across the community, and that policy should be directed with impartiality, fairness and justice towards these ends.”

In the academy, valuing the principle of equity can (and we would argue should) inflect a number of everyday activities: creating courses, advising and mentoring students (and colleagues), organizing conferences, facilitating workshops, appointing or serving on search committees and editorial boards  —  and so much more. The results of embracing work done in the spirit of achieving an equitable academy? We want to imagine a world in which all who partake in teaching, learning, reading, researching, and writing  —  in short, all of us engaged in the scholarly enterprise  —  commit to actively listen and to openly question our own assumptions, to share, to amplify, and ultimately to empower.

Equality, Equity, No Barriers
Source: http://sydney.edu.au/science/physics/about/equity.shtml (Adapted from http://indianfunnypicture.com)

Follow team #HuMetrics as we wrestle with humanities metrics. We are Christopher Long, Rebecca Kennison, Stacy Konkiel, Simone Sacchi, Jason Rhody, and Nicky Agate, and we’ll be writing here all week.

Influence vs. Impact: Which Are Humanists Really Trying to Achieve?

Influence vs. Impact: Which Are Humanists Really Trying to Achieve?

Apologies for the false dichotomy I’ve set up by my framing of this post in its title as “impact versus influence.” It’s a result of the quickblogging process, one that Christopher Long, Rebecca Kennison, Nicky Agate, Simone Sacchi, Jason Rhody, and I agreed upon as […]


css.php