From d6c4bf8717715cb9ae667ecccc413d69227f426b Mon Sep 17 00:00:00 2001 From: Jim Madge Date: Tue, 12 Mar 2024 14:31:55 +0000 Subject: [PATCH 01/14] Add roadmap file --- ROADMAP.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 ROADMAP.md diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000000..5f167d50d8 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,17 @@ +# Roadmap + +This document sets out the plan for the evolution of the Data Safe Haven codebase. +It covers both short term goals, which are organised in sprint milestones, and long term goals, which are expected without a particular time frame. +It also collects desirable features which are unplanned or which we have decided not to pursue yet. + +Last updated: 2024-03-11 + +## Governance + +## Desired Features + +## Short Term + +### Milestones + +## Long Term From a4448ba1ed0129a27ba9150dff41af18b02106e5 Mon Sep 17 00:00:00 2001 From: Jim Madge Date: Tue, 12 Mar 2024 14:38:02 +0000 Subject: [PATCH 02/14] Add roadmap governance discussion --- ROADMAP.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index 5f167d50d8..ae5e7aab92 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -8,6 +8,30 @@ Last updated: 2024-03-11 ## Governance +This document is written by the code's [development lead](https://github.com/JimMadge) with input from the development team. + +When deciding on what features to include in the roadmap, and where to include them we consider, in no particular order, + +- The maintainability of the code (looking after the developers) +- Supporting users at The Alan Turing Institute (a significant user who we have a close relationship with) +- Supporting all users we are in contact with +- Integrating with, and building compatibility for, common infrastructure in the UK TRE landscape + +These factors are balanced when making decisions. +There is no formal process for assigning scores for the above factors, or the weight each should be given. +In general, we currently consider the needs of our colleagues at The Alan Turing institute as our single most important user. +However, we also have opinions and expertise in trusted research and aim to inform the Turing rather than be directed by them. +Furthermore, we wouldn't necessarily let the priorities or opinions of the Turing overrule those of other users. + +For example, + +- User A wants new feature X next, the Turing wants new feature Y next. + - We would be inclined to prioritise feature Y over X. +- The majority of the community feels the TRE should behave in manner X, a minority of the community, including the Turing, feels the TRE should behave in manner Y. + - We would be inclined to design the TRE to behave in manner X. + +At a time when the project has a large and active user base, we would like to better formalise how users feedback and ideas influence the roadmap. + ## Desired Features ## Short Term From 807690053511805b44007ccc5fffc6a6d8333bb8 Mon Sep 17 00:00:00 2001 From: Jim Madge Date: Tue, 12 Mar 2024 15:15:32 +0000 Subject: [PATCH 03/14] Add desired features --- ROADMAP.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index ae5e7aab92..dfb6d438ae 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -34,6 +34,52 @@ At a time when the project has a large and active user base, we would like to be ## Desired Features +These are features we would like to highlight but which we have actively decided not to incorporate into the long or short term roadmap. +This may be because, for example, we have decided we have insufficient resource, are lacking the right expertise or it is low priority. + +### Develop Outside, Run Inside + +A workflow enabling researchers to develop this research and analysis code outside of the TRE, using whatever tools they prefer and a familiar with, and bring this work into the TRE to run against the sensitive data. + +This would improve, + +- Reproducibility (as users would be encouraged and enabled to use public VCS repositories) +- User experience +- User efficiency +- Accuracy of results (as testing code is easier) + +This would involve, + +- A trusted mechanism to bring code into the TRE +- Synthetic or dummy data for testing outside of the TRE + +### User Defined Container Support + +Enable users to bring containers they have developed outside of the TRE into the TRE to be used in research/analysis. +The HPC focused container project Apptainer could be a good fit, providing performance and protecting against privilege escalation. +This work would compliment [](#develop-outside-run-inside) + +### Improve Resource Competition + +Researchers currently tend to share resources in SRDs which can cause problems if they are not careful to coordinate their work. +This could be improved by, + +- HPC style job submission queue +- Isolated resources per user (VMs, cgroups) + +# User-Facing Web App for Project Management + +There is currently little tooling for research teams working through processes or managing their environments. +A web app could be an accessible way to present features like, + +- Switching on and off resources (improves cost management) +- Scaling resources +- Opening support tickets +- Opening data ingress or egress request tickets +- Working through IG processes + +This work would be particularly powerful if the interfaces/processes can be abstracted so that the web app can be used by other TRE operators and integrated into their TRE implementations. + ## Short Term ### Milestones From a8c9775c28a42473ae9b832753e0293068161a82 Mon Sep 17 00:00:00 2001 From: Jim Madge Date: Tue, 12 Mar 2024 15:25:38 +0000 Subject: [PATCH 04/14] Add discussion of short-term goals and milestones --- ROADMAP.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index dfb6d438ae..5dbd787a5f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -82,6 +82,17 @@ This work would be particularly powerful if the interfaces/processes can be abst ## Short Term -### Milestones +Short-term goals are those which we have committed to and have planned an expected completion date. +These are organised into milestones. +Each milestone has, + +- A description of it's aims +- A target date +- A collection of issues which constitute the work to be done + +Milestones will usually correspond to new releases of Data Safe Haven. +Currently we aim to align milestones with Data Study Groups at the Turing, so that new releases are made in time to be used at these events. + +The milestones can be seen [on GitHub](https://github.com/alan-turing-institute/data-safe-haven/milestones) ## Long Term From 688d4601fc8b13060808d715d6a044598ee09ac2 Mon Sep 17 00:00:00 2001 From: Jim Madge Date: Tue, 12 Mar 2024 15:35:48 +0000 Subject: [PATCH 05/14] Add discussion of long-term goals --- ROADMAP.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index 5dbd787a5f..908edc8b72 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -96,3 +96,6 @@ Currently we aim to align milestones with Data Study Groups at the Turing, so th The milestones can be seen [on GitHub](https://github.com/alan-turing-institute/data-safe-haven/milestones) ## Long Term + +Long-term goals are developments which we have committed to but have not expected completion date. +This may be because they are lower priority than short-term goals or because the time required needs to be scoped. From 35d32fec4c396f42290f06e8f3b18a9df17bbda4 Mon Sep 17 00:00:00 2001 From: Jim Madge Date: Tue, 12 Mar 2024 15:48:20 +0000 Subject: [PATCH 06/14] Rearrange and add extra context --- ROADMAP.md | 56 ++++++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 23 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 908edc8b72..1b7b3b3983 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,10 +1,20 @@ # Roadmap +Last updated: 2024-03-12 + +## Background + This document sets out the plan for the evolution of the Data Safe Haven codebase. -It covers both short term goals, which are organised in sprint milestones, and long term goals, which are expected without a particular time frame. +It covers both short term-goals, which are organised in sprint milestones, and long-term goals, which are expected without a particular time frame. It also collects desirable features which are unplanned or which we have decided not to pursue yet. -Last updated: 2024-03-11 +The [short-term plans](#short-term) aim to give a clear indication of current work and when it may be finished. +We hope that a focus on defining releases will encourage development to balance new features with other improvements and address the needs of users. + +[Long-term goals](#long-term) give reassurance that we have committed to certain changes when we are unable to give a estimation of when they will be ready. + +Finally, [desirable features](#desireable-features) is a way for us to express developments we would like to pursue but have decided not to do yet. +This is a good indication of where we would like the project to go and areas we would be interested in collaborating. ## Governance @@ -32,9 +42,29 @@ For example, At a time when the project has a large and active user base, we would like to better formalise how users feedback and ideas influence the roadmap. +## Short Term + +Short-term goals are those which we have committed to and have planned an expected completion date. +These are organised into milestones. +Each milestone has, + +- A description of it's aims +- A target date +- A collection of issues which constitute the work to be done + +Milestones will usually correspond to new releases of Data Safe Haven. +Currently we aim to align milestones with Data Study Groups at the Turing, so that new releases are made in time to be used at these events. + +The milestones can be seen [on GitHub](https://github.com/alan-turing-institute/data-safe-haven/milestones) + +## Long Term + +Long-term goals are developments which we have committed to but have not expected completion date. +This may be because they are lower priority than short-term goals or because the time required needs to be scoped. + ## Desired Features -These are features we would like to highlight but which we have actively decided not to incorporate into the long or short term roadmap. +These are features we would like to highlight but which we have actively decided not to incorporate into the long- or short-term roadmap. This may be because, for example, we have decided we have insufficient resource, are lacking the right expertise or it is low priority. ### Develop Outside, Run Inside @@ -79,23 +109,3 @@ A web app could be an accessible way to present features like, - Working through IG processes This work would be particularly powerful if the interfaces/processes can be abstracted so that the web app can be used by other TRE operators and integrated into their TRE implementations. - -## Short Term - -Short-term goals are those which we have committed to and have planned an expected completion date. -These are organised into milestones. -Each milestone has, - -- A description of it's aims -- A target date -- A collection of issues which constitute the work to be done - -Milestones will usually correspond to new releases of Data Safe Haven. -Currently we aim to align milestones with Data Study Groups at the Turing, so that new releases are made in time to be used at these events. - -The milestones can be seen [on GitHub](https://github.com/alan-turing-institute/data-safe-haven/milestones) - -## Long Term - -Long-term goals are developments which we have committed to but have not expected completion date. -This may be because they are lower priority than short-term goals or because the time required needs to be scoped. From 67a4273a91ca73576cc6917c1202939b099a06a5 Mon Sep 17 00:00:00 2001 From: Jim Madge Date: Tue, 12 Mar 2024 15:49:08 +0000 Subject: [PATCH 07/14] Correct heading level --- ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index 1b7b3b3983..c6bc797e6b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -97,7 +97,7 @@ This could be improved by, - HPC style job submission queue - Isolated resources per user (VMs, cgroups) -# User-Facing Web App for Project Management +### User-Facing Web App for Project Management There is currently little tooling for research teams working through processes or managing their environments. A web app could be an accessible way to present features like, From 7eed3a82ad8f97ec650ddfe984642638916573ce Mon Sep 17 00:00:00 2001 From: Jim Madge Date: Tue, 12 Mar 2024 15:54:42 +0000 Subject: [PATCH 08/14] Add link text --- ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index c6bc797e6b..8f4c119d0d 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -87,7 +87,7 @@ This would involve, Enable users to bring containers they have developed outside of the TRE into the TRE to be used in research/analysis. The HPC focused container project Apptainer could be a good fit, providing performance and protecting against privilege escalation. -This work would compliment [](#develop-outside-run-inside) +This work would compliment [Develop Outside, Run Inside](#develop-outside-run-inside). ### Improve Resource Competition From dbdd6044482d15f2e9fc22e111354b63b93b8558 Mon Sep 17 00:00:00 2001 From: Jim Madge Date: Wed, 13 Mar 2024 16:08:27 +0000 Subject: [PATCH 09/14] Apply suggestions from code review Co-authored-by: Hari Sood <67151373+harisood@users.noreply.github.com> --- ROADMAP.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 8f4c119d0d..4953152ae9 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -11,7 +11,7 @@ It also collects desirable features which are unplanned or which we have decided The [short-term plans](#short-term) aim to give a clear indication of current work and when it may be finished. We hope that a focus on defining releases will encourage development to balance new features with other improvements and address the needs of users. -[Long-term goals](#long-term) give reassurance that we have committed to certain changes when we are unable to give a estimation of when they will be ready. +[Long-term goals](#long-term) give reassurance that we have committed to certain changes when we are unable to give an estimation of when they will be ready. Finally, [desirable features](#desireable-features) is a way for us to express developments we would like to pursue but have decided not to do yet. This is a good indication of where we would like the project to go and areas we would be interested in collaborating. @@ -69,7 +69,7 @@ This may be because, for example, we have decided we have insufficient resource, ### Develop Outside, Run Inside -A workflow enabling researchers to develop this research and analysis code outside of the TRE, using whatever tools they prefer and a familiar with, and bring this work into the TRE to run against the sensitive data. +A workflow enabling researchers to develop this research and analysis code outside of the TRE, using whatever tools they prefer and are familiar with, and bring this work into the TRE to run against the sensitive data. This would improve, From 4141fc719298738174db25de0b31d47cf7fa89a6 Mon Sep 17 00:00:00 2001 From: Jim Madge Date: Thu, 14 Mar 2024 14:31:42 +0000 Subject: [PATCH 10/14] Add addon desired feature --- ROADMAP.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index 4953152ae9..87668620d8 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -109,3 +109,20 @@ A web app could be an accessible way to present features like, - Working through IG processes This work would be particularly powerful if the interfaces/processes can be abstracted so that the web app can be used by other TRE operators and integrated into their TRE implementations. + +### Addon or Plugin System + +For a TRE to be effective, it needs to enable research. +DSH has always tried to strike a good balance between security and features/convenience for researchers. +A downside of this is it can lead to a complicated environment with many components. +This is especially true when working across domains, as each domain has specific tools that they rely on. +This create a lot of maintenance burden and spreads developer time thin. + +Another approach to this could be to break more TRE components out into option addon or plugin modules. +It would require building a system to integrate these into the TRE. +This would mean, + +- It is easier for communities to add functionality they want +- Broken features can be easily disabled without affecting the core TRE +- If we have no interest in continuing to support a module, the community can adopt and maintain it +- There is an easier entry point to contributing From 6a775d54947143844df1904b2b800d636425f6a4 Mon Sep 17 00:00:00 2001 From: JimMadge Date: Fri, 15 Mar 2024 00:17:21 +0000 Subject: [PATCH 11/14] Update SRD package versions --- .../packages/dbeaver-driver-versions.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deployment/secure_research_desktop/packages/dbeaver-driver-versions.json b/deployment/secure_research_desktop/packages/dbeaver-driver-versions.json index bac1536afd..d78e3652d8 100644 --- a/deployment/secure_research_desktop/packages/dbeaver-driver-versions.json +++ b/deployment/secure_research_desktop/packages/dbeaver-driver-versions.json @@ -3,6 +3,6 @@ "pgjdbc": "1.1.6", "postgis_geometry": "2023.1.0", "postgis_jdbc": "2023.1.0", - "postgresql": "42.7.2", + "postgresql": "42.7.3", "waffle_jna": "3.3.0" } \ No newline at end of file From ce1213d4110428e3e05c8d261404be02628ecd2c Mon Sep 17 00:00:00 2001 From: Jim Madge Date: Fri, 15 Mar 2024 14:14:14 +0000 Subject: [PATCH 12/14] Apply suggestions from code review Co-authored-by: Matt Craddock --- ROADMAP.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 87668620d8..970243817a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -13,7 +13,7 @@ We hope that a focus on defining releases will encourage development to balance [Long-term goals](#long-term) give reassurance that we have committed to certain changes when we are unable to give an estimation of when they will be ready. -Finally, [desirable features](#desireable-features) is a way for us to express developments we would like to pursue but have decided not to do yet. +Finally, [desired features](#desired-features) is a way for us to express developments we would like to pursue but have decided not to do yet. This is a good indication of where we would like the project to go and areas we would be interested in collaborating. ## Governance @@ -48,7 +48,7 @@ Short-term goals are those which we have committed to and have planned an expect These are organised into milestones. Each milestone has, -- A description of it's aims +- A description of its aims - A target date - A collection of issues which constitute the work to be done @@ -59,7 +59,7 @@ The milestones can be seen [on GitHub](https://github.com/alan-turing-institute/ ## Long Term -Long-term goals are developments which we have committed to but have not expected completion date. +Long-term goals are developments which we have committed to but have not planned an expected completion date. This may be because they are lower priority than short-term goals or because the time required needs to be scoped. ## Desired Features From fba59a2be2af1711585ee83ebf8ef40274a4e982 Mon Sep 17 00:00:00 2001 From: JimMadge Date: Tue, 19 Mar 2024 00:22:02 +0000 Subject: [PATCH 13/14] Update PyPI and CRAN allow lists --- .../package_lists/allowlist-full-python-pypi-tier3.list | 1 + 1 file changed, 1 insertion(+) diff --git a/environment_configs/package_lists/allowlist-full-python-pypi-tier3.list b/environment_configs/package_lists/allowlist-full-python-pypi-tier3.list index f8436f3c5c..8196a5ce61 100644 --- a/environment_configs/package_lists/allowlist-full-python-pypi-tier3.list +++ b/environment_configs/package_lists/allowlist-full-python-pypi-tier3.list @@ -411,6 +411,7 @@ odo olefile opencv-python opt-einsum +optree ordereddict orderedmultidict oscrypto From 17e6021a15b2a723732f731fc7f8b950739049c6 Mon Sep 17 00:00:00 2001 From: JimMadge Date: Fri, 22 Mar 2024 12:06:56 +0000 Subject: [PATCH 14/14] Update PyPI and CRAN allow lists --- .../package_lists/allowlist-full-python-pypi-tier3.list | 1 + 1 file changed, 1 insertion(+) diff --git a/environment_configs/package_lists/allowlist-full-python-pypi-tier3.list b/environment_configs/package_lists/allowlist-full-python-pypi-tier3.list index 8196a5ce61..ff94b97627 100644 --- a/environment_configs/package_lists/allowlist-full-python-pypi-tier3.list +++ b/environment_configs/package_lists/allowlist-full-python-pypi-tier3.list @@ -477,6 +477,7 @@ prettytable prometheus-client prompt-toolkit prophet +proto-plus protobuf psutil psycopg2