Last month, in collaboration with Wilfrid Laurier University Press, I completed the first round of open peer review on my prototype scholarly podcast, Secret Feminist Agenda. This open peer review process is part of a larger research project on podcasts as a form of non-traditional scholarly communication with a great deal of potential for knowledge mobilization and public scholarship. And like many forms of non-traditional scholarship — including digital prototypes, blog posts, videos, exhibitions, and performances — they face a significant barrier: recognition and legitimization within the formal apparatuses of peer review and publishing, which are in turn required for hiring, promotion, and tenure.
The peer review questions my collaborator Siobhan McMenemy and I formulated asked reviewers to reflect on the process of peer review itself, and how it can be adapted to a medium like podcasting. We also opted for an open peer review process, in keeping with the openness of podcasting itself. That first round of peer review, and my responses, are now available. And they are, I think, illuminating on the topic of how peer review may need to transform as our scholarship does.
Digital scholarship has relied on peer review as a way of legitimizing the unfamiliar to scholars’ peers and helping those working in emergent fields, like the digital humanities and media studies, to build sustainable careers. We might look, for example, at NINES: Nineteenth-century Scholarship Online, which was in part formed to address just this problem:
Digital humanities projects have long lacked a framework for peer review and thus have often had difficulty establishing their credibility as true scholarship. NINES exists in part to address this situation by instituting a robust system of review by some of the most respected scholars in the field of nineteenth-century studies, British and American.
But the forms of peer review appropriate to digital and other non-traditional scholarship are not necessarily identical to those practiced in the more traditional venues of scholarly journals and university presses. As the American Association of University Presses has recognized, “Scholarly digital initiatives are producing new modes and forms of publishing, and the dynamism of these developments requires ongoing assessment of conventional peer review processes.” Indeed, as I have learned, the peer review being practiced at university presses is nowhere near as traditional as I thought it was (see ‘single-blind peer review,’ below).
For this reason I thought it might be helpful to run through some of the existing forms of peer review, and take a closer look at how some different platforms are experimenting with the practice.
Varieties of Peer Review
Double-Blind Peer Review
Often referred to as the “gold standard” of peer review, double-blind means the author doesn’t know who the reviewers are, and the reviewers don’t know who the authors are. Most journals practice double-blind review, and anecdotally some academics have told me that their publications don’t “count” unless they’re double-blind reviewed. The logic behind double-blind review is that of removing common forms of bias, including age, race, and gender. But some work simply cannot be effectively made “blind” (and gosh, don’t even get me started on the ableism in the language of “blindness” in reviewing). Many of us who work in small fields have had the experience of reviewing an article, or receiving a peer review, and knowing exactly who the writer is. And, as Cheryl E. Ball, editor of Kairos: A Journal of Rhetoric, Technology, and Pedagogy, wrote in her review of Secret Feminist Agenda, “Digital media scholarship can’t be anonymously reviewed, period. […] It has to be reviewed collaboratively, which is a form of open review.” (More on how Kairos performs peer review below!)
Single-Blind Peer Review
In single-blind review, the authors are known but the reviewers are anonymous. It was a revelation to me upon beginning my collaboration with Siobhan at WLUP that many university presses use single-blind peer review because of the extraordinary difficulty of fully anonymizing a book manuscript. In fact, single-blind review may be the most common form of reviewing, and accounts for the impossibility of anonymizing certain kinds of scholarship, including scholarship in which the scholar’s identity is significant to the work or digital scholarship like podcasts in which the scholar’s voice clearly identifies them.
Open Peer Review
In open review, the authors and peer reviewers are known to each other, and often participate in an open exchange or dialogue around the editorial process. Kairos, for example, uses a multi-stage open peer review process, in which “The entire editorial board discusses the submission for two-to-three weeks, coming to a collaborative assessment of its quality and potential to be published in Kairos” and then “The editors assign a staff member to work with authors, as needed, to guide/facilitate revisions based on the editorial board’s comments and evaluation.” Similarly, Hybrid Pedagogy uses Google Docs to facilitate an open peer review process, in which authors are “paired with two peer editors, whom you’ll interact with directly throughout the process until your article is ready for one final production review by a third editor.” The peer reviewers are then clearly identified on the published article. Models of open peer review, while they differ in their specifics, are frequently adopted by platforms publishing non-traditional scholarship, including multimedia work. While some worry that the openness will reduce the rigour of the peer review process, with reviewers withholding critique out of a sense of politeness, others point to the greater space made in this process for non-traditional scholarship as well as the culture of collegiality and mentorship that open peer review can build. Open peer review is also strongly affiliated with open access publishing, which many institutions and funding bodies are actively encouraging.
Transparent Review
The next step beyond open peer review is a fully transparent review process, like the one we have adopted with Secret Feminist Agenda, in which both the reviews and my response to them are posted to the press website. Transparent peer review is particularly beneficial when working to develop the infrastructure for new forms of scholarship, as it opens up the conversation about how and why we are evaluating scholarship that pushes generic boundaries. Transparent peer review is also beneficial to public scholarship, as it allows members of the public to engage with the process rather than limiting the conversation to the traditional definition of peers.
*
Indeed, as digital scholarship expands our sense of what is possible in the world of scholarly communication, our definition of peer will surely need to transform. Public-facing scholarship engages a public of people who may not be affiliated with universities but who, by virtue of choosing to engage with the work, become part of the scholar’s intellectual community. In my case, because Secret Feminist Agenda speaks to a wide range of listeners, I’ve opted for multiple forms of peer review, encompassing the formalized process driven by WLUP as well as blog comments, listener emails, social media feedback, and even in-person conversations. Conceiving of those engaging with your work as peers whose responses should be considered seriously and carefully can only improve work, especially work that is attempted to find non-traditional audiences.
In a future posts, members of the CIT will tackle some practical strategies for navigating different forms of peer review and how to advocate for work that is peer reviewed in less traditional ways. In the meantime, start familiarizing yourself with these different forms of peer review (if you aren’t already) and spend some time looking at platforms that are practicing open and transparent review. If my suspicions are right, that’s where you’re going to find the most exciting scholarship.
