Improving the understanding and use of scholarly metrics in academia
My research proposal focuses on what I call metrics literacies: an integrated set of competencies, dispositions and knowledge that empowers individuals to recognize, interpret, critically assess and effectively and ethically use scholarly metrics.
I prepared the proposal earlier this year with the help of my research assistant Alyssa (who is also a student in this class), and Alice and Isabelle, two doctoral students affiliated with the ScholCommLab. We submitted the proposal to the SSHRC Insight Development Grant competition in February and----fingers crossed----are hoping to receive funding this summer.
Metrics literacies are an integrated set of competencies, dispositions and knowledge that empowers individuals to recognize, interpret, critically assess and effectively and ethically use scholarly metrics.
Scholarly metrics, such as the h-index and journal impact factor, are widely applied in academic tenure and funding decisions, but often inappropriately so. I think that the misuse of metrics and the quantification of research impact in general harm all scholarly disciplines because they create severe adverse effects. For example, in order to increase their publication counts researchers self-plagiarize or practice so-called salami publishing----that is, they publish multiple instead of one paper----or they cite their own articles a bit too frequently in order to increase their citation rates.
Scholarly metrics, such as the h-index and journal impact factor, are widely applied in academic tenure and funding decisions, but often inappropriately so.
That's why our research project aims to improve the understanding and use of scholarly metrics across academia to reduce the misuse of scholarly metrics. More specifically, we are trying to support researchers and research administrators in developing metrics literacies using multimedia open educational resources (OER). The h-index, one of the most popular scholarly metrics, will be used as a case study.
Research objectives and questions
Our project addresses the following research objectives:
Develop educational resources to support metrics literacies using audiovisual material such as videos, podcasts, or interactive online resources;
Experimentally test and identify the most effective types of educational resources for helping academics (e.g., professors, postdoctoral researchers, doctoral students) and research administrators (e.g., research managers, librarians) to appropriately apply and understand research metrics;
Disseminate the most effective resources under Creative Commons Attribution 4.0 International License (CC-BY) via a knowledge mobilization plan that includes the use of social media and community partnerships in an effort to improve metrics literacies in academia.
We want to answer the overarching research question:
How can the understanding and use of scholarly metrics in academia be improved?
More specifically we aim to answer the following research questions:
What are the characteristics of educational resources, in terms of audiovisual format, duration and level of human embodiment (i.e., audio/visual presence of a teacher or guide), that affect learners’ metrics literacies, as evidenced by their retention and knowledge about the h-index?
What are the characteristics of the educational resources that influence learners’ willingness to engage with them online, as evidenced by the extent to which they are viewed, shared and liked?
How do learners’ individual characteristics—such as discipline, career status (e.g., emerging or established), professional role (e.g., academic or administrative), or pre-existing beliefs and familiarity with research metrics—affect the answers to RQ1 and RQ2?
How can the results from the questions above be leveraged to ensure the widest possible adoption of effective metrics literacies resources by academics and research administrators?
Theoretical framework
Our research builds on Mayer’s cognitive theory of multimedia learning which states that combining auditory and visual elements reduces the cognitive load for a single channel. That means that audio-visual media are much more efficient and effective means of education.
We are also using the Knowledge to Action (K2A) framework to organize our research project into three phases and three teams.
Methodology
In Phase 1, graduate students will create three resources in different formats (e.g., video, podcast) with input from the bibliometrics expertise and support from the media production team. A short text document on the h-index will serve as the foundation for the content of the OERs as well as the control in the experiment in Phase 2. In order to ensure variation of formats, students will select the most appropriate from the following types:
video (e.g., lecture style, talking head; Khan Academy style, digital drawing board style; slide presentation),
podcast (e.g., interview, conversational, narrative nonfiction);
infographic (e.g., informational comic), or
interactive resource (e.g., choose-your-own-adventure books or games).
Phase 2 will focus on testing the effectiveness of the multimedia material as learning resources, using an experimental research design. In a randomized control trial, participants will be exposed to one of the three multimedia OERs or the text document, the traditional modality of scholarly communication. Participants will include emerging and established researchers, librarians and research managers, who will be surveyed three times: once before and after being exposed to the material, and then again one month later. The quantitative approach will be complemented by semistructured interviews asking participants to provide qualitative feedback.
In Phase 3, the community outreach team will publish and disseminate all educational resources online using a CC-BY license. In addition to supporting knowledge mobilization, this step will allow us to identify the most engaging resource(s) by measuring their online popularity (i.e., number of views, downloads and social media engagement). Our project will make empirical and practical contributions to bibliometrics and multimedia education. By developing, testing and distributing effective, engaging resources on the h-index, it will help improve metrics literacies among academics and research managers.
The timeline
We're hoping to receive the SSHRC funding this summer. If we are successful, we should be able to start our 2-year project in July 2020. Here's our timeline listing all the major work packages:
Our team
Our research project is quite unique as it brings together an interdisciplinary team of bibliometricians, science communicators, media producers, and education technology scholars. In our team of collaborators we have metrics subject experts as well as a film producer, Youtuber and podcast host, who will assist us producing high quality OERs which will explain scholarly metrics in an efficient, effective and engaging manner.
The team brings together a group of 13 talented and fun, mostly early-career, mostly female (12 kick-ass women + Peter) researchers and creatives and I can't wait to work with all of them!
Open Innovation Leaders
I am currently participating in the Open Innovation Leaders program by eLife to further improve the metrics literacies project. You can read more about my experience participating in eLife's program here. We are using design thinking and an open science approach, so I am documenting my progress in this Google Doc. Feel free to have a look and leave a comment!
We are currently working on creating a document on the h-index which includes all information about the indicator. Based on peer-reviewed literature, this so-called brief will contain all the facts and limitations of the h-index. The brief will serve as the basis for creating the different OERs, but also act as the control in our experimental setup (see below). Our roadmap for the next three months also includes other milestones such as creating instructions for the production of the OERs, drafting the survey and interview guides and applying the uOttawa Research Ethics Board (REB). Creating our first OER, a stop-motion video using LEGO, will be our last milestone before the summer.
Getting your feedback
Since community involvement is a major component and crucial to the success of the project, we would like to hear from you. Let us know in the comments below what you think about metrics education in general and our project in particular!