Rethinking humane indicators of excellence in the humanities and social sciences

Tag: TriangleSCI

The Syllabus as HuMetrics Case Study

The Syllabus as HuMetrics Case Study

Current metrics of humanities scholarship have been shown to be too blunt to capture the multiple dimensions of scholarly output and impact (see Haustein and Larivière). In addition, the inappropriate nature of current indicators can incentivize perverse scholarly practices (see The Metric Tide, Wilsdon et al.)…. In […]

The Syllabus as Scholarship

The Syllabus as Scholarship

“A critical component of our emerging #Humetrics conversation at Triangle SCI involves finding ways to expose, highlight, and recognize the important scholarship that goes into the all-too-hidden work of peer review, syllabus development, conference organizing, mentoring, etc. Our current metrics fail to capture what is […]

Nurturing Fulfilling Scholarly Lives

Nurturing Fulfilling Scholarly Lives

In Aristotle’s Nicomachean Ethics, there is a famous passage in which he reminds us that “to be happy takes a complete lifetime; for one swallow does not make a spring, nor does one fine day; and similarly one day or a brief period of happiness does not make a person blessed and happy” (Nic. Eth., 1098a16–20).

This passage came to frame our conversations around #HuMetrics at this week’s Triangle Scholarly Communications Institute, because it reminds us that a fulfilling life  —  what Aristotle calls, eudaimonia, happiness, that is, a life well lived  —  requires cultivated habits rooted in core values that, when intentionally practiced, shape the character of a good life.

In the end, what we value should be embodied in what we do, not once or twice, but regularly over the course of a lifetime.

In framing our conversation about #HuMetrics with this ancient conception of ethics, excellence, and character, we seek also to advance and reinforce the idea that a scholarly life can only be well lived in communities of practice with others.

For the #HuMetrics team, this year’s Triangle SCI experience was a swallow that signifies but does not yet fully manifest the coming spring. It opened for us a space for the flowering of a community of practice oriented toward the question of how we might more broadly cultivate communities of practice that embody the values of fulfilling scholarly lives.

Five Core Excellences of Enriching Scholarship

Working out loud together, we identified five core excellences of enriching scholarship:

Rebecca Kennison wrote about Equity.

Simone Sacchi wrote about Openness. His point was amplified further by Rebecca in her post about The Value of Openness.

I wrote about Collegiality.

Jason Rhody wrote about Quality.
Nicky Agate wrote about Community.
And Stacy Konkiel sought to tie things together by distinguishing between enriching and corrosive values.

In writing together in this way, we seek to embody the excellences for which we advocate.

The question that animates our work is this:

The Winter of Our Discontent

For too long, we humanists have been allergic to metrics. This allergy has prevented us from engaging in a serious and sustained conversation about what practices of scholarship we might want to cultivate and incentivize both through the activities we measure and those we celebrate.

As a result, a large and growing battery of metrics have been developed based on the practices of more scientifically oriented scholarship or simply on what it was possible to use our technologies to measure.

Current metrics of humanities scholarship have been shown to be too blunt to capture the multiple dimensions of scholarly output and impact (see Haustein and Larivière). In addition, the inappropriate nature of current indicators can incentivize perverse scholarly practices (see The Metric Tide, Wilsdon et al.).

A critical component of our emerging #HuMetrics conversation at Triangle SCI involves finding ways to expose, highlight, and recognize the important scholarship that goes into the all-too-hidden work of peer review, syllabus development, conference organizing, mentoring, etc. Our current metrics fail to capture what is most substantive about the rich life of scholarship we practice together in living academic communities.

In this context, our challenge and our responsibility is to articulate, incentivize, and reward practices that enrich our shared scholarly lives and expand our understanding of scholarship itself.

Without being naïve about how difficult it is to change culture, we hope to begin to reshape the conversation about metrics around the values of enriching scholarly practices and the communities in which they thrive.

Although our time together at the Triangle SCI was only one swallow that does not yet make a spring, the seeds planted there may begin to take root over the weeks and months to come, and the communities of scholarship that blossomed there just might be “made glorious by this sun” that shines when a broader public is invited to join the conversation.

Changing Behavior by Changing Incentives

Changing Behavior by Changing Incentives

It’s our last day at Triangle SCI, and I’ve been contemplating overnight the feedback the #HuMetrics team got yesterday afternoon from our colleagues. In our presentation of what we’ve been doing this week, we attempted to sketch out, using the activity of mentoring, how actions […]

The Excellences of Scholarship: Collegiality

The Excellences of Scholarship: Collegiality

We began the day with a walk. The morning was cool and fresh, and the grounds around the DuBoise House at the Rizzo Center offered the six of us on the #HuMetrics team at the TriangleSCI the peace and space we needed to think out […]

On Quality

On Quality

Today is the second full day of the Triangle Scholarly Communication Institute, where I am part of a team that’s focusing on HuMetrics: Building Humane Metrics for the Humanities. We have each agreed to quickly blog some thoughts as part of our process; warning: what follows is quickly-crafted prose, full of rough edges.

The #HuMetrics team is making progress in attempts to reverse engineer that which we want to measure, by starting with the values than often govern the (in some ways broader, more comprehensive) range of work that we engage in (see Chris Long’s previous post for further discussion). We were able to distill a wide-ranging brainstorm of values into five categories, and have aims to think about how those value categories relate to the processes and products of our work. The important core of our conversation today centered around the fact that while metrics often seen (or are taken) as the end-goal, in point of fact the indicators always align to a value, and so part of the work here, in this free and open thinking space, is to be aspirational about the values we’d like to see elevated, incentivized, and rewarded. If openness as a value is prioritized, for example, one could imagine more weight given to scoring articles and/or journals that were OA rather than not. If that seems like an extreme example, it’s perhaps a worthy future exercise to consider the ways that certain ways of showing impact actually might already tweak the scale toward different values.

For our quick 20-minute afternoon exercise, each of us is taking a crack at writing about one of those five values: Equity, Openness, Collegiality, Quality, and Community. In our framework, Quality can take on one or more of the following characteristics:

  • Replication
  • Soundness
  • Pushing boundaries
  • Creativity
  • Originality
  • Advancing knowledge
  • Intentionality
  • Reproducibility

Replication and reproducibility have a certain emphasis in some of the social sciences, but the terms might in other contexts also be thought of in the sense of extensibility. It’s important to note that these are preliminary notions, and we welcome your feedback.

Keep in mind that we’re considering metrics than can apply to multiple kinds of academic processes and outputs, so not just whether or not your article or book has high quality (measured currently, say, based on whether or not the article is in a journal with a high impact score, or if your book receives a certain number of citations), but whether or not you play a role in helping measure the degree of quality for an object (say, serving as a peer reviewer for a grant application, a reviewer for a book, a referee for an article, etc.). Both of these kinds of activities are part of the transaction related to “quality,” but currently we overwhelmingly incentivize and reward the former rather than the latter. What such a focus on the transactional value in the sense of quality does is unpack the transactional relationship and scholarly networks that undergird much of our work, disturbing the notion of individual acts of scholarship by revealing the deep relationships behind scholarly works.

Another challenge to current methods for measuring impact related to quality is the well-noted challenge of addressing context (such as whether or not a citation is a positive one or a negative one), and the degree to which such measurements lend themselves to a certain kind of gaming the system (through overuse of citations to drive up citation scores). How do we successfully implement speed bumps (rather than roadblocks) that require some small additional effort that will likely not prevent gaming the system but may, to carry the metaphor, slow it to a reasonable speed? The use of “active citation” (a precise and annotated citation and link to the source), as argued by Andy Moravcsik (PDF) in the context of political science research, is one potential method, especially for more qualitative work.

Follow team #HuMetrics as we wrestle with humanities metrics. We are Christopher Long, Rebecca Kennison, Stacy Konkiel, Simone Sacchi, Jason Rhody, and Nicky Agate, and we’ll be writing here all week.

Community as a Humanistic Value

Community as a Humanistic Value

Today is the Day Two of the Triangle Scholarly Communication Institute, where I’m heading up a team that’s focusing on HuMetrics: Building Humane Metrics for the Humanities. Our team has focused a lot on the importance of working out loud, of process over product, and […]

On Openness

On Openness

Second day at #TriangleSCI with the #HuMetrics team. Today we focused on clustering our values into major value-categories (Equity, Openness, Collegiality, Quality, and Community) with the idea that excellence in scholarship is an expression and combination of these value-categories as they are embodied in scholarly […]

Equity as a Core Value

Equity as a Core Value

We’ve just completed Day 2 at #TriangleSCI, and more hard (but good) work is now behind the #HuMetrics team. Today we took our huge brainstorming list from yesterday and distilled the values we believe should underpin the development of “humane metrics.” We came up with five such values: Equity, Openness, Collegiality, Quality, and Community. I am looking here at equity. My fellow teammates Simone Sacchi, Chris Long, Jason Rhody, and Nicky Agate are tackling each of the others, and Stacy Konkiel is writing on the overarching question we looked to tackle: influence vs. impact.

What do we mean by “equity”? For us, equity is very much the concept described by Falk et al. (1993:2):

“Equity derives from a concept of social justice. It represents a belief that there are some things which people should have, that there are basic needs that should be fulfilled, that burdens and rewards should not be spread too divergently across the community, and that policy should be directed with impartiality, fairness and justice towards these ends.”

In the academy, valuing the principle of equity can (and we would argue should) inflect a number of everyday activities: creating courses, advising and mentoring students (and colleagues), organizing conferences, facilitating workshops, appointing or serving on search committees and editorial boards  —  and so much more. The results of embracing work done in the spirit of achieving an equitable academy? We want to imagine a world in which all who partake in teaching, learning, reading, researching, and writing  —  in short, all of us engaged in the scholarly enterprise  —  commit to actively listen and to openly question our own assumptions, to share, to amplify, and ultimately to empower.

Equality, Equity, No Barriers
Source: http://sydney.edu.au/science/physics/about/equity.shtml (Adapted from http://indianfunnypicture.com)

Follow team #HuMetrics as we wrestle with humanities metrics. We are Christopher Long, Rebecca Kennison, Stacy Konkiel, Simone Sacchi, Jason Rhody, and Nicky Agate, and we’ll be writing here all week.

Influence vs. Impact: Which Are Humanists Really Trying to Achieve?

Influence vs. Impact: Which Are Humanists Really Trying to Achieve?

Apologies for the false dichotomy I’ve set up by my framing of this post in its title as “impact versus influence.” It’s a result of the quickblogging process, one that Christopher Long, Rebecca Kennison, Nicky Agate, Simone Sacchi, Jason Rhody, and I agreed upon as […]

Scales of Measurement and the Public Good

Scales of Measurement and the Public Good

Today is the first day of the Triangle Scholarly Communication Institute, where I am part of a team that’s focusing on HuMetrics: Building Humane Metrics for the Humanities. We have each agreed to quickly blog some thoughts as part of our process; warning: what follows […]

The Value of Openness

The Value of Openness

Openness to the skyDay 1 of the Triangle Scholarly Communication Institute is underway, and our HuMetrics: Building Humane Metrics for the Humanities team (Nicky Agate, Simone Sacchi, Christopher Long, Stacy Konkiel, Jason Rhody, and me) is already hard at work. We began by putting aside (for the moment) the “metrics” and focusing instead on the “humane,” which became a discussion of values that we want to encourage and reward.

As one might expect, many of those values were ones that embraced “openness”  —  whether “equitable access,” “open process,” “engagement,” “transparency,” “open source,” “candor,” “accessibility,” or “sharing.” What struck me, however, as someone who spends all day every day discussing “open access,” was what was missing from our list of explicitly “open” products of scholarship. What made that list? Not articles, not books, not even data; instead, we identified explicitly as “open” only “preprint OA” and “OERs” (open education resources).

Why only those two products? Was it because we assumed that “openness” was already inherent in other products and outputs? Was it because we were thinking intentionally about values that are currently not rewarded and so considered only tagging as “open” products that also are in that category? I don’t know. But it does point to the different kind of thinking that the Triangle SCI inspires. I am looking forward to tomorrow!

Follow team #HuMetrics as we wrestle with humanities metrics. We are Christopher Long, Rebecca Kennison, Stacy Konkiel, Simone Sacchi, Jason Rhody, and Nicky Agate, and we’ll be writing here all week.

First Day at #TriangleSCI Working with the #HuMetrics Team

First Day at #TriangleSCI Working with the #HuMetrics Team

First day at #TriangleSCI working with the #HuMetrics team. It is quite amazing what can happen when you put together in a room people from faculty administration, granting agencies, and societies with scholarly communication, information science, and metrics experts  —  without the constraints of their […]

HuMetrics Values

HuMetrics Values

Today is the first day of the Triangle Scholarly Communication Institute, where I’m heading up a team that’s focusing on HuMetrics: Building Humane Metrics for the Humanities. Our team has focused a lot on the importance of working out loud, of process over product, and […]

Enriching vs. Corrosive Values in Academia

Enriching vs. Corrosive Values in Academia

As part of the TriangleSCI HuMetrics working group, I spent the better part of this afternoon brainstorming and debating academic values, products, processes, and metrics in an attempt to lay a foundation for this week’s attempt at articulating “humane metrics” for the humanities.

As our discussion wound down, it occurred to the team that we were working with an important assumption re: values: we had spent the day identifying only those “enriching” values that we wanted to encourage (collaboration, generosity, inclusivity, quality, etc.), rather than examining the current set of values we wished to discourage, those that in many ways are “corroding” academia (competition, bureaucracy, exclusivity, etc.).

That’s in large part due to what we’re here in Chapel Hill to do: define and promote a means of appropriately measuring and rewarding quality scholarship (if not excellence) in the humanities  —  a group of disciplines currently underserved by citation-based metrics.

Yet citation-based metrics are increasingly being requested in research evaluation scenarios, by tenure and review committees, granting agencies, university administrators who wish to benchmark their departments against those of other universities. And those metrics often unintentionally reinforce so-called “corrosive” values: as Haustein and Larivière point out, the over-reliance upon simplistic, citation-based metrics for evaluation has led to undesired practices in academia like “salami-slicing,” self-citation, and ghost authorship; in the humanities, Nederhof explains that a focus upon both citation-based metrics and a push toward publishing in “preferred” venues has led to changing publication practices that are out of step with the realities of the field.

As our work begins tomorrow, we’ll attempt to explore those values that “enrich” academia, hopefully allowing us to “reverse engineer” or map metrics that measure scholars’ ability to achieve values we wish to encourage. No small feat, for sure.

Follow team #HuMetrics as we wrestle with humanities metrics. We are Christopher Long, Rebecca Kennison, Stacy Konkiel, Simone Sacchi, Jason Rhody, and Nicky Agate, and we’ll be writing here all week.

From Metrics to Values

From Metrics to Values

Recognizing that metrics drive practices in the academy (and elsewhere), we on the HuMetrics: Building a Humane Metrics for the Humanities team at this year’s Triangle Scholarly Communications Institute decided to approach our work by thinking first about the values that inform enriching scholarship. This […]