Evaluating data management plans

We have been involved in the reviewing and evaluation of data management plans for many years. You will find out below more information about these projects, plus how we review DMPs and common issues, which we have identified with the DMPs we have assessed.

ESRC DMPs

The ESRC introduced a requirement for a data management plan to be submitted with every grant application in April 2011. We evaluated an anonymous sample of 25 of the first submitted data management plans. This involved evaluating the quality of the information provided for each of those against the following eight topics:

1. An assessment of existing data that could be used for the research.
2. Information on new data that will be created.
3. Quality assurance of data.
4. Back-up and security of data.
5. Expected difficulties in data sharing, e.g. ethical or legal issues.
6. Copyright and Intellectual Property Right of data.
7. Data management responsibilities.
8. Preparation of data for sharing and archiving.

We then scored the answers to each question, using these numbers: ([1] insufficient, [2] sufficient, [3] excellent). Each plan was evaluated twice independently by various staff members of the UK Data Archive’s Research Data Management section.

The average quality score for a DMP was 17, with a minimum score of nine and a maximum score of 23. Nine DMPs scored below 16 (the score for sufficient information being provided for each topic). Six DMPs scored below 12.

DMPs on average provide good to excellent information on assessing existing data and describing new data to be created (the average scores being 2.4 and 2.3 respectively).

DMPs perform poorest on information about copyright and IPR of research data, with an average score of 1.8. Scores of 1 (insufficient information provided) were most common for copyright (7 plans), for data management responsibilities (5 plans) and for data preparation (5 plans).

It is also worth grant applicants viewing the ESRC guidance for peer reviewers of data management plans.

Rural Economy and Land Use (RELU) DMPs

This cross-disciplinary research programme (2005-2012) had its own data policy, with all projects preparing a data management plan at the start of a project (post funding). At the end of the plan, research data were deposited with the UK Data Archive and the Environmental Information Data Centre.

We reviewed the data management plans prepared by 29 large projects, plus four pilot projects that created and archived data and three fellowships (totalling 36 plans). During the review we considered whether:

• Information provided on the data planned to be produced is adequate and realistic according to the proposed research and methodology.
• All relevant data management aspects have been considered, with meaningful information provided in the plan.
• Where difficulties are anticipated to make data available for archiving,  possible solutions have been suggested.
• All possible obstacles to sharing data have been considered, such as ethical limitations and copyright ownership.
• A team member with data management responsibility is in place at each participating institution.

For each question we evaluated whether the information provided was: [1] insufficient (lacking clarity or detail), [2] adequate or [3] excellent. Each initial plan submitted could score a maximum of 18 and a minimum of 6.

Most plans contained sufficiently detailed lists of the various datasets planned to be produced. In a few cases, information was vague and award holders were asked to provide better or more detailed information.