University of Sheffield Research Software Survey (2020)

Robert Turner, Paul Richmond, University of Sheffield RSE Team September, 2021

Thank you!

Thanks for coming, thanks to the conference organisers, thanks to the RSE team at Sheffield, Paul Richmond in particular.

About me

Mix of software engineering and research experience.

RSE at Sheffield

13 RSEs, 35 projects / year worth ~£11m total

Why survey?

Improve our practise and procedures.
Prioritise our resources.
Data-driven advocacy.

I’m scared of surveys

from “The Art of Statistics: Learning from Data, David Spiegelhalter, 2020 paperback”

Reasoning

from “The Art of Statistics: Learning from Data, David Spiegelhalter, 2020 paperback”

Sources

Data and analysis software https://github.com/RSE-Sheffield/sssurvey.
Report in .pdf format https://github.com/RSE-Sheffield/sssurvey/releases/download/latest/report.pdf.

The Survey

Based on surveys carried out nationally in 2014 and at the University of Southampton in 2019, led by Simon Hettrick.
Asked about “demographics”, use and development of software, training and funding.

Analysis

Generally, “non responses” to optional (O) questions were ignored. Mandatory questions are marked (M).

Data anonymised manually.
Cleaned and annotated using Python scripts (with some manual decisions).
Plotted charts using Jupyter notebook.
Presentation and report written in markdown.
Outputs built using pandoc / GitHub actions.

Sample characteristics

This survey went to all PhD students and research staff with a prize incentive.
382 respondents.
A 2019 Southampton survey went to “all staff employed on an ERE contract (Education, Research and Enterprise) and all PhD students” with a prize incentive. As surveys were sent out on a faculty by faculty basis, it was possible to report a response rate of between 8% and 11% for all faculties.
2014 national survey - it is not clear what sampling strategy was used.

Faculty

Funders

Which organisations usually fund your research? (O)

Job

Importance of research software

How important is research software to your work? (M)

1 “Not at all”, to 5 “Vital”

Software development practise

Have you developed your own research software? (M)

Software Importance

91% of respondents use research software (92% nationally).
65% report that software is vital to their research (nationally, 69% report that “It would not be practical to conduct my work without software”).
27% develop their own code (56% nationally, 33% Southampton).

Sufficiency of training

Do you feel that you have received sufficient training to develop reliable software? (O)

Awareness around key skills

Current level of support

1: “poor” to 5: “excellent”

Training

69% (of the 27% subset who responded to this question) feel they have not had sufficient training to develop reliable software.
Current level of support for software development is questionable.

Funding for Software Development

Funding

Of the 54% of respondents who are involved with writing funding proposals, 45% expected to write software as part of the proposal.
Of those who expected to write software, 40% did not request funding for this (compared to 20%, nationally).

Staffing of software development

Have you or someone in your group ever hired someone specifically to develop software? (M)

Hiring practise

Recruit a developer (or fractional FTE equivalent of a developer) from a central University of Sheffield pool as needed (O)

Staffing

Pool RSE model seems preferable to hiring on a per-project basis
Some respondents see pool RSE model as unfavourable

High Performance Computing

Licensing and commercialisation

Would you be interested in the university helping you commercialise some of your research software? (O)

Do you feel that your research software is ready to be shared with a commercial partner? (M)

Actions (1)

Provide training / support to increase researcher confidence with version control, continuous integration and unit testing.
Investigate why 69% of respondents feel they have not had sufficient training. What training is needed? How much?
Investigate why participants responded as they did to a question about level of support for software development.

Actions (2)

Provide additional support to researchers to make research software outputs not intended for commercialisation freely available.
Discover if any action can be taken to help researchers who would like to use University of Sheffield HPC but don’t currently use to do so.
Advocate for researchers to include costs for software development in their funding applications.

Future work (Final Slide)

Can we say if responses are different for different funders, research subjects using subgroup analysis?
Does the national / international picture need to be revisited?