ACM Inroads: Archive

While there has been increasing focus in the U.K. in ensuring students are trained professionals at the end of their doctoral studies, much of the training has been devolved to central units, such as graduate schools. While this training has benefits, foregrounding training in the disciplinary context students are working in provides a more authentic learning environment, whilst sustaining a strong peer-to-peer community. Based on the last 10 years of our training program, we reflect on some of the benefits it has had on our computing students, as well as offering access to the resources we have developed [16].

Introduction

The model of doctoral studies in the U.K. has remained broadly the same over the past 100 years—the student–supervisor relationship, the written thesis, and an original knowledge contribution [1]. A full-time student typically studies for three to four years and is examined through an oral viva by two experts in the field. While other models exist—such as the EdD or the professional doctorate—these remain relatively niche. The U.K. standards body, the Quality Assurance Agency (QAA), characterizes doctoral studies as supporting candidates to "demonstrate an original contribution to knowledge in their subject, field or profession, through original research or the original application of existing knowledge or understanding." [18]

The current approach to doctoral training in the U.K. is the result of the Roberts report [21], commissioned by the U.K. government in 2001, which highlighted the need for transferable skills training and career development of researchers. One of the identified problems in the report was the poor ability of STEM graduates to translate their knowledge to the commercial sector, with a recommendation that government-funded studentships should be conditional on students completing two weeks of dedicated transferable skills training per academic year.

As a result, many institutes and funders signed the Concordat to Support the Career Development of Researchers, which is "an agreement between universities, research institutes and funders to support the career development of researchers in the UK." [19]

This high-level focus on training during doctoral studies was a positive move, but the latest data from the Postgraduate Research Experience Survey (PRES) indicates that the training provided may not meet student expectations [16]. The data from 2023 shows that less than half of students received training to develop transferable skills (47% full-time students, 33% for part-time students); only 35% of FT students received advice on career options (22% PT); and in terms of community, just over half (59% FT, 48% PT) felt part of a community of postgraduate research (PGR) students [16]. This may be due to training being delivered centrally (or not at all), rather than contextualized within a disciplinary area.

The main U.K. funder of computing research, the Engineering and Physical Sciences Research Council (EPSRC), has highlighted there is both a lack of studies considering doctoral training within the U.K., and even fewer that explore the outcomes and potential impact of doctoral education in specific disciplines [4]. Drawing on our experience of delivering a discipline-based training program for computing students that complements training delivered by our graduate school, we highlight how our training program, PGForum, compares and contrasts with the training other U.K. institutes provide for their computing doctoral students. Drawing on attendance and completion data and thesis acknowledgements, we consider some of the benefits of discipline-focused training.

Background

Across Europe, there is a great deal of diversity in how doctoral training is organized. There are three main approaches: supervisor-led, discipline-led, or central graduate schools. There has been a general trend toward central graduate schools, although many countries combine all three approaches [5].

In reviewing the literature regarding doctoral training, the EPSRC found there is a lack of U.K.-based studies regarding doctoral training, with the "vast number of studies [being] US-based, a system for doctoral education and career pathways that differs significantly from that in the UK." [4] The review highlights a strong need for tailored programs to support student development during their doctoral education, although the single U.K. study [8] focuses on placing that training into a broader interdisciplinary context.

Drawing on data from the U.S., George analyses a sample of 1,335 doctoral student reports to the Graduate Student Data Buddies Survey [7]. The results show that more than a third of computing doctoral students had considered leaving prior to completion, with the main factors including feeling isolated and unsupported by faculty.

The EPSRC review concludes that there is a need to go beyond a focus on individual supervisors (for example, Malekian, [14]), and that "greater support for doctoral students should be available to improve their doctoral experience" [4]. One example is through group supervision, which can provide sustained support and reduce feelings of isolation [10]. In the case outlined by Hutchings, group supervision sits outside the student/supervisor relationship, with training being regularly delivered by facilitators outside this relationship to an entire cohort of inter-disciplinary students.

Internationally, students studying for a doctoral degree often have high rates of depression and anxiety [11,23]. In the U.K., studies have indicated that Ph.D. students were much more likely to suffer from depression, compared to other working professionals [9]. While the survivor-biased view of doctoral training as a mechanism to sift out incapable or uncommitted students [15] is no longer commonly accepted, the structure of the training delivered to doctoral students is a significant factor in the student experience [12]. Some doctoral programs fail to support students robustly in becoming embedded within their departments, research communities, or professions [6,24].

Current Approaches to Doctoral Training within the U.K.

Before outlining our approach to doctoral training, we wanted to establish what other approaches currently exist in U.K. institutions.

We selected all institutes that responded to the Research Excellence Framework (REF) panel on 'Computer Science and Informatics' [19]. Across the September 23 and 24, 2024 we explored the institutional websites to establish:

Does the institute have a graduate school?
If so, does the graduate school provide training?
Does the computing school/department offer any training to doctoral students?
If so, what training is offered?

Ninety institutions submitted to the REF (including our own); eight institutions did not have sufficient information on their website to report. Of the remaining 81 institutions, 88% offer centrally delivered training and 22% offer some form of discipline-focused training, with 17% offering both. The full breakdown of our analysis can be seen in Table 1.

Two institutions offered formal qualifications as part of their training, both centrally delivered: Birmingham City University offers a Postgraduate Certificate in Research Practice, while the University of Strathclyde offers a PG Certificate in Researcher Professional Development.

The format of discipline-based training ranged from weekly workshops and courses to an annual colloquium or MSc courses taught. There was no notable difference in the training delivered at a discipline level between institutions that also have central training and those that do not. What is noticeable is the rarity of training delivered to Ph.D. students within their disciplinary areas.

We argue that this is a concern. While centrally delivered training is essential, discipline-based training can place these skills more firmly in the context of Ph.D. students' day-to-day work, as well as exploring disciplinary norms and community practices. Furthermore, while community-building is essential at an institutional level, providing a forum for Ph.D. students to socialize within their discipline strengthens networks, and provides opportunities for support embedded within their specialist area.

What Is PGForum?

PGForum (that is, Postgraduate Forum) is the training program for doctoral students in both the School of Computing and Communications and the Knowledge Media Institute at The Open University in the U.K. At any given time, PGForum caters to around 50 doctoral students, both full-time and part-time.

For more than 20 years, PGForum has been the focus of doctoral training for computing students at The Open University: part training program, part community-building, part social engagement. We meet weekly for a mix of presentations, discussion, and activities all centered around a theme of the week. The facilitator typically runs the sessions, with external speakers brought in to provide expertise and variety. Topics cover the entire doctoral journey, including reading, writing, study skills, career planning, publication, and examinations (that is, the 'craft skills' of research [22])—contextualized by community norms. Students are encouraged to ask for specific sessions, and to run sessions of their own.

A typical session is 90 minutes long, with the community meeting for around 35 weeks in any given year. Once a month we meet in the evening rather than the morning to support the engagement of part-time students in the whole community. Attendance is monitored and promoted to students and supervisors alike. Regular attendance is recommended; however, make it clear to students that we understand that other activities—such as field studies or conferences—may supersede their attendance.

Each weekly session starts with attendees sharing "Happy News," both professional (such as paper submissions) and personal (such as a holiday trip). While this may appear to be a small aspect, it has a significant impact on the training environment, creating a closer relationship between the students (as shown in Figure 1).

For a similar reason, we have a very basic house rule—what happens in PGForum, stays in PGForum. This leads to there being no recording of sessions (without prior agreement for specific sessions). It is made clear to all students that there is no judgement on contributions, creating an environment for rigorous debate and occasional 'non-official' statements, but there must be a space for anyone to be able to say anything. This again binds the community together.

The PGForum program culminates in a two-day conference. The program combines short talks from all of the students (15 minutes) with evening social activities, and additional sessions we can't provide during the PGForum program. We always close the conference with "Up against the Wall" (see Figure 2), where students line up from closest to submission to furthest from submission. Having posed a question (such as "What did you learn this year that you wish you'd known earlier?"), the later-stage students start, with good advice percolating down the line.

One of the challenges of running a training scheme across diverse research groups is that "each of the contributing disciplines has its own language, values, norms, and traditions—its own culture" [1]. This is both a challenge and a strength; while it raises difficulties in being precise (with different research areas having different expectations and standards in terms of, for example, analysis approaches), it leads to a broader discussion among peers about the expectations within their research communities, and how that differs and why, broadening their understanding and perspective.

The broad focus differs markedly from attempts to teach specific skills during intensive training courses (for example, research methods [1]) or mentorship within disciplines (for example, Levecque et al. [11]). It also provides a disciplinary perspective which is different from generic training delivered by central units, such as graduate schools [1]. While all of these training approaches are a valuable contribution to the development of doctoral students as researchers, our approach builds a sense of community, strengthens peer support, and provides disciplinary-relevant training.

Another challenge students have raised with regard to engaging in training is time, particularly "where they had supervisors who were not overtly supportive of participation in additional training." [4] One of the strengths of PGForum is the long-term support from supervisors and management within the two associated departments, ensuring supervisors actively recommend that students attend PGForum.

A thematic analysis of these acknowledgements showed that comments clustered around three key elements: advice and training, community building, and pastoral support. Some students acknowledged all three elements.

Our approach has strong overlaps with the program outlined by Yacoby et al. [24], with a focus on building a sense of community among students and development of key skills. It differs in not being exclusive to first-year students—with our later-stage students frequently supporting early-stage students, reciprocating the support they have previously received, and reflecting on the maturation of their perspective when topics recur. PGForum also covers a broader range of skill development and has a less explicit focus on mental health.

PGForum Experience Report

Having outlined the ethos of PGForum and highlighted its somewhat unusual features within the U.K. regarding training computing doctoral students, we need to consider the data we have indicating the program's success.

Given our close relationship with the students, we did not want to survey them directly as it was unlikely we would receive unbiased data. As such, we focus on what secondary data sources can tell us in two areas:

Comparing PGForum attendance data against the length of time to completion to examine whether attendance is associated with completing faster.
Trying to understand the student experience of attendance by examining the acknowledgement sections of completed theses.

The results align with work conducted around 10 years ago based on attendance data and exit interviews, demonstrating the long-running impact of the training program.

• Relationship between Attendance and Completion Time

We have been keeping a detailed record of attendance at PGForum from 2015, and we can extract a student's completion data from The Open University's registration system. This allows us to calculate the months to completion for a given student. We acknowledge that this timing data doesn't account for study breaks and other periods of approved leave as these records are not readily available. From this data, we have a pool of 65 students, with their attendance record (with regular attendance regarded as at least five sessions within a given year) and their completion time. Table 2 provides a broad outline of the completion profile of our students.

First, examining the data for full-time students, the t-test shows a significant difference between attendees (mean = 51.7) and non-attendees (mean = 60.7), (n = 42, t = -2.99, p<0.005). Perhaps unsurprisingly given the sample size, the t-test for part-time students shows no significant difference between attendees (mean = 75.7) and non-attendees (mean = 85.9), (n = 23, t = -0.66, p=0.25).

The data and associated analysis broadly indicates that students who engage regularly with PGForum complete more quickly than students who do not. Interpreting this data is challenging, and certainly no causality can be assumed. There is likely at least one confounding factor that stronger students are both more likely to complete more quickly and attend PGForum as they see the value in the training provided. However, the data does indicate that PGForum is successful in supporting students in their doctoral studies.

• Analyzing Acknowledgement Sections

In the U.K., acknowledgement sections within theses are commonly used to thank people who have supported the student through their doctoral studies and acknowledge those who have contributed along the way. As a secondary data source, they are thus ideal in trying to understand the student experience of attendance at PGForum, by first examining whether PGForum or the organizers are mentioned and then analyzing what elements the students have identified as important.

We read through the acknowledgement sections of the 65 students for whom we had attendance and completion time data. Nine of the 37 students who are marked as not attending still acknowledged the program; as did 21 of the 28 regular attendees.

A thematic analysis of these acknowledgements showed that comments clustered around three key elements: advice and training, community building, and pastoral support. Some students acknowledged all three elements.

"I also owe huge thanks to Emeritus Professor Marian Petre, erstwhile champion of the Post Graduate Forum group and incredible pillar of support throughout this process… This was such a tremendous help in improving the work and finally seeing me to the end. To everyone in the PGForum group, and its current facilitator Dr. Daniel Gooch, thank you. The regular sessions and opportunities to learn, laugh and sometimes cry together were a central part of my journey and undoubtedly helped me cross the finish line".

• Advice and Training

Twenty-four of the students noted, at least in part, how important the skills development and training PGForum provided were invaluable to their development as researchers:

"Marian Petre and Daniel Gooch, who untiringly organised and hosted the PGForum (Post-graduate Research Students Forum), at the School of Computing and Communications and the Knowledge Media Institute, which allowed me to improve my research skills. I would also like to extend my gratitude to all the guest speakers and colleagues that made the forum possible by sharing their knowledge and experience."

"I would like to thank Prof. Marian Petre and Dr. Daniel Gooch for managing the postgraduate forum sessions, which provided support for Ph.D. students in their journey. I am also grateful to all the people who provided feedback on my work during my presentations at the CRC student conferences, in particular Prof. Arosha Bandara, Prof. Marian Petre, and Dr. Daniel Gooch."

"I'd like to thank Prof. Marian Petre for running the PGForum, a forum aimed at postgraduate students in the computing department that served to not only bring together students from different disciplines, but also for providing guidance on how exactly one should go about their Ph.D."

• Community Support

Part of the design of PGForum is also to use the training sessions to support and develop a sense of community amongst the students. This is invaluable not only for professional development and knowledge transfer, but also to develop a network of pastoral support, with 10 of the students commenting on this aspect:

"The author is also grateful to Prof. Marian Petre for her work in sustaining the community of computing research students, and in particular ensuring the inclusion of external students such as the author. More generally, the author has benefited enormously from conversations with people in the CRC and KMi research community at the Open University. The author is grateful for having been part of that community."

"The next thank you must go to the supportive community that I have been fortunate to have around me during the process at the OU. The PGForum sessions run by Marian and now Dr. Daniel Gooch have continually been insightful and engaging. Thank you to my peers at the forum… thank you for always being supportive of my ideas and work."

• Pastoral

PGForum also provides students with access to support outside of their supervisory teams, and the formal complaint or support structures at the Open University. Four of the students explicitly commented on how much they appreciate that support, and the benefits this pastoral support gave them through their doctoral studies:

"Besides my supervisors, I would like to thank Prof. Marian Petre for hosting the post-graduate forum. Thanks to her counsels, I embraced a mentally and physically wholesome Ph.D. lifestyle that helped me to experience a fulfilling period of study."

"There simply are no words to thank Marian for her support during the past two years. She never, ever let me go; others surely would have. I also humbly and deeply thank the faculty and staff members from the Open University who ensured I could complete this research."

Final Thoughts

Our data shows that our discipline-focused training program makes a difference. Students complete their studies faster, and the acknowledgements show that they appreciate the training, pastoral, and community aspects of PGForum. So why doesn't every institute set up a similar program?

As with all initiatives, a cost/benefit analysis is needed. The discipline area needs to invest in such a scheme; it takes around 45 days in a given year, from an academic who is willing to engage with the students and champion their views. Furthermore, the program will only succeed if there is engagement and buy-in from supervisors. This is needed to encourage student attendance, find hosts for guest sessions, and ensure a variety of perspectives at the annual conference. As a side-benefit, supervisors gain a broader insight into the research occurring within their school or department, particularly outside of their research groups.

The other investment is in the time necessary to develop the training resources. This is a contribution we are offering to the community, by making our training materials available. Visit https://pragmaticphd.com/resources for more information; at that site, we have released a public set of resources under a Creative Commons license. They will need customizing for your own local context but will provide a baseline for developing a training program for computing Ph.D. students. We welcome any feedback, critique, or reports of use of these resources.

Ph.D. supervision is among the "most complex and subtle form[s] of teaching in which we engage." [1] The academy has started to acknowledge that it takes a village to train and support Ph.D. students. While there have been dramatic changes in how we approach this over the last 20 years, we still believe that supporting discipline-focused training offers opportunities and benefits that generic graduate school-level training does not. In future work, it will be important to consider whether such training is effective in improving student performance in other disciplines.

Acknowledgments

We would like to thank Trevor Collins, Wendy Stainton-Rogers, Verina Waights, Darrel Ince, and Anne de Roeck, who all supported the creation of PGForum. Thanks also to Robin Laney, Allistair Willis, and Paul Piwek, who have helped sustain and run the program. We would also like to thank all of the students and staff who have made PGForum such a positive and constructive program over the last 20 years.

References

1. Atkins M. and Brown, G. Effective teaching in higher education. Taylor & Francis, 2002.

2. Beloev, H. et al. Innovations in doctoral students education and training. In Proceedings of the 22nd Intern. Conf. on Computer Systems and Technologies (CompSysTech '21). Association for Computing Machinery, New York, NY, USA, 224–230. https://doi.org/10.1145/3472410.3472443

3. Berglund, A., Kinnunen, P., and Malmi, L. A doctoral course in research methods in computing education research: How should we teach it? In Proceedings of the 7th Baltic Sea Conf. on Computing Education Research 88 (Koli Calling '07). Australian Computer Society, Inc., AUS, 175–178.

4. EPSRC review of doctoral education: A review of the literature. https://www.ukri.org/publications/review-of-epsrc-funded-doctoral-education/, accessed 2024 Jan 3.

5. European University Association, Doctoral programmes in Europe's universities: Achievements and challenges; Report prepared for European universities and ministers of higher education, 2007.

6. Foot, R.E., It's not always what it seems: Exploring the hidden curriculum within a doctoral program. Ph.D. Thesis, Kent State University, 2017.

7. George, K. Thinking about leaving: Identifying threats to computing doctoral student persistence. In Proceedings of the 52nd ACM Technical Symp. on Computer Science Education (SIGCSE '21). Association for Computing Machinery, New York, NY, USA, p. 1259. https://doi.org/10.1145/3408877.3439679

8. Hancock, S. and Walsh, E. Beyond knowledge and skills: Rethinking the development of professional identity during the STEM doctorate. Studies in Higher Education 41, 1 (2014), 37–50. doi:10.1080/03075079.2014.915301.

9. Hazell, C.M., Niven, J.E., Chapman, L. et al. Nationwide assessment of the mental health of U.K. Doctoral Researchers. Humanit Soc. Sci. Commun 8, 305 (2021). https://doi.org/10.1057/s41599-021-00983-8

10. Hutchings, M. Improving doctoral support through group supervision: Analysing face-to-face and technology-mediated strategies for nurturing and sustaining scholarship. Studies in Higher Education 42, 3 (2015), 533–550. https://doi.org/10.1080/03075079.2015.1058352

11. Levecque, K. et al. Work organization and mental health problems in Ph.D. students. Research Policy 46, 4 (2017), 868–879, https://doi.org/10.1016/j.respol.2017.02.008.

12. Lovitts, B.E. Leaving the vory Tower: The Causes and Consequences of Departure from Doctoral Study, Rowman & Littlefield Publishers, U.K., 2001.

13. Lunn, S.J., Samary, M.M., and Peterfreund, A. Research experiences for graduate students (REGS): The evolution of computing education projects and creation of a virtual community. In Proceedings of the 54th ACM Technical Symp. on Computer Science Education 1 (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 701–707. https://doi.org/10.1145/3545945.3569858

14. Malekian, R. Effective supervision for enhancing quality of doctoral research in computer science and engineering. SN COMPUT. SCI. 4, 678 (2023); https://doi.org/10.1007/s42979-023-02167-4.

15. Nyquist, J.D. et al. On the road to becoming a professor: The graduate student experience. Change: The Magazine of Higher Learning, 31, 3, 18–27. https://doi.org/10.1080/00091389909602686.

16. Pragmatic PhD blog, https://pragmaticphd.com/resources.

17. Postgraduate Research Experience Survey, Advance HE; https://bit.ly/3QgovIV

18. QAA. Characteristics statement - doctoral degrees; https://bit.ly/40ScC0A

19. REF panel on Computer Science and Informatics; https://bit.ly/40SmVBU, 2021.

20. Researcher Development concordat; https://bit.ly/3QcXJkx

21. Roberts, G. SET for success: The supply of people with science, technology, engineering and mathematics skills: The report of Sir Gareth Roberts' review; https://bit.ly/4hSrbYK

22. Rugg, G. and Petre, M. The Unwritten Rules of PhD Research, third edition. Open University Press/McGraw Hill Education, 2020.

23. Satinsky, E.N., Kimura, T., Kiang, M.V. et al. Systematic review and metaanalysis of depression, anxiety, and suicidal ideation among Ph.D. students. Sci Rep 11, 14370 (2021). https://doi.org/10.1038/s41598-021-93687-7.

24. Yacoby, Y., Girash, J., and Parkes, D.C. Empowering first-year computer science Ph.D. students to create a culture that values community and mental health. In Proceedings of the 54th ACM Technical Symp. on Computer Science Education 1 (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 694–700. https://bit.ly/3EuIUam

Authors

Dr. Daniel Gooch
School of Computing and Communications
The Open University
Walton Hall
Milton Keynes, England

Prof. Marian Petre
School of Computing and Communications
The Open University
Walton Hall
Milton Keynes, England

Figures

Figure 1. Celebrating the submission of a thesis during the Covid-19 pandemic.