Codebase maintenance - Hotfixes and patches

[Davsarper]

Goal Title

What will this work achieve?
Ensure that codebase is kept up-to-date with bug fixes, security updates, external API changes etc.

Description

• Ensure that DSH code is always deployable
• Ensure that known security issues are remediated/minimised as soon as possible
• Ensure that documentation is up-to-date with code base

Definition of Done

When will this be considered as succesfully completed?
Ongoing

Details

• RACI
  • Accountable team or person: Data Safe Haven team
  • Responsible team or person: Data Safe Haven team
  • Informed and consulted people: Turing IT, @martintoreilly
• Estimated effort in total FTE: 0.25

Resourcing

August

• REG: 0.5

Checklist

• This Story have been agreed with project members, it tackles prioritised work
  • If not: Label as ForPrioritisation so it discussed in the next monthly meeting, do not set a status
• I have filled in the Team Accountabe field
• I have included this Story in the agreed upon Milestone, set status as Planned
  • If not: set status as Backlog, to be Planned via weekly meetings or async discussions
• The work to be done is likely to span over a month
• I have broken down the work in monthly issues and added them above
• I have labeled this issue according to its main project: SATRE, TRESA, CORE DSH (ELA) or other (please agree and create a new label if necessary)
• If known: sum the total effort (points) of tracked issues and add it on the body
• Select a Story level of effort in the project fields

Reporting

5 February to 8 April 24

Have worked on updating software used within SREs to ensure the security and functionality of the environment:

• Guacamole server updated PR
• Nexus server updated PR
• CodiMD server updated PR

Added and tested a script to handle SAS access tokens renewal, currently expiring yearly. These are required manage access to data storage (and therefore ingress and egress). The relevant PR is here alan-turing-institute/data-safe-haven#1739. In the process we realised SAS tokens are bound to Store Access Policies which could be modified to have no end date, we are currently considering the covenience of this approach versus potential security issues in alan-turing-institute/data-safe-haven#1751

Improved use of hardcoded domain names and IPs. The hardcoded lists are difficult to maintain and are prone to going out of date, despite not fully fixing the use of these improvements have been made for the 4.2.0 release by relaxing rules where security allows. For this the team checked individuals cases and applied where possible, no security issues where found and we added this as a specific thing to pent test. Related PR is alan-turing-institute/data-safe-haven#1745 and explanatory issue is alan-turing-institute/data-safe-haven#1549

An issue with Jupyter notebooks not being able to use Python when launched from the menu was found, despite extensive work a fix was not found and decided to let it be by documenting the right workaround: launching Jupyter Notebooks from the terminal. The issue is alan-turing-institute/data-safe-haven#1584

Worked on updating documentation to reflect Azure Active Directory name change to Microsoft Entra

8 January to 5 February

• v4.2.0 milestone issue review
  • Checking current issues
  • Some are likely to be "will not fix"
  • Ensuring time frame is reasonable
• v4.2.0 development progress
  • Multiple data provide guidance
• Database permission issue debugging
  • Identified unexpected behaviour, closer to identifying the problem
  • now fixed in PR
  • PostgreSQL user privileges were not correctly applied; now keeps them up to date with the security groups allocated on the DC
• ClamAV On-access was not running
  • On-access virus scanning is a DSPT requirement, and this process was not running correctly.
  • Fixed by PR
• Working on debugging PostgreSQL permissions issues
• DBeaver driver fix DBeaver database drivers missing from SRD data-safe-haven#1666
• Improve handling of file paths PR
• Checking out Azure feature retirement warnings Issue

5 December to 8 January 2024

• Investigating issues with Julia on AMD processors: During the building of VM images for deployment in SREs, Julia created and stored compiled versions of packages that were suitable only for Intel systems, causing crashes when users wanted to use AMD systems
• Investigating issues with DBeaver on Tier 2+ SREs: DBeaver drivers were not installing correctly during VM building, so it tries to download them from the internet. No problem on T1, but fails on T2.
• Testing solutions for cross-platform deployment: When deploying on Windows, some scripts fail if there are any spaces in the file path

1 November to 4 December

There has been work to address and improve issues and bugs related to last release while preparing for release 4.2.0.

Factoring storage creation and account deployments out of main deployment script now allows for a more resilient process (not having to re-run everything when one fails)

Also MS changing Azure Directory to Microsoft Entra ID has made necessary to spend time updating documentation and code, with the increased challenge that MS themsleves have not yet ocnsitently made the change.

• Name changing: PR for changes to documentation - Issue for changes to code
• Factor SHM storage creation out of main deployment script PR
• 4.2.0 milestone related issues https://github.com/alan-turing-institute/data-safe-haven/milestone/21

10 October to 30 October

• Add all contributors table to project README and docs
  • PR
  • Helps us better recognise contributors, especially those who do not appear in the commit history
  • Will require maintainers to upkeep the list
• Fixed an issue where the user could provide an invalid name for storage accounts

14 August to 18 September

• Removal of MSRDS which reduces support burden and codebase complexity, instead Guacamole implementaiton is more robust and secure
• Removal of CoCalc Reduces support burden for future releases by removing a largely unused feature
• Investigating adding arrow R package support

10 July to 14 August

Development/features

• Drop Microsoft Remote Desktop primarly for increased security as it shows more issues than Guacamole, in doing this several other open issues are resolved

Fixes/maintenance

• fixes
• Documentation improvements Add explanation of how to change allowed inbound IP addresses data-safe-haven#1484 and updates Remove MS Remote Desktop support data-safe-haven#1535

[Davsarper]

October: less prioritised, as focus is on the new code and funding applications

[Davsarper]

Dec planning

Reactive, but wiht DSGs going things will arise (and are arising)

[Davsarper]

@craddm @JimMadge today during Monthly we were asked this in relation to factoring out parts of the configuration:

What is the definition of done for the piece of work of 'factoring out configuration elements' (part of Codebase Maintenance)? What's the value we're aiming for and how do we know when we've achieved it?

Could you add what you consider the answers to be either here or in a (linked) item (issue, milestone...) in the corresponding repo?

[harisood]

Jan planning

We have a milestone with a number of issues that need work. This is well scoped and resource allocated

Rough allocation

• REG
  • 0.8 (@craddm)

[Davsarper]

February focus: release final version before next DSG
https://github.com/alan-turing-institute/data-safe-haven/milestone/21

[JimMadge]

March (and end of Februrary) Focus:
Close remaining v4.2.0 issues
Ready for release and pen test

[Davsarper]

After March release it should not be a priority in april

[JimMadge]

Reopening, to use for hot fixes and patch releases, which are not milestones.

[Davsarper]

Feel it will be a priority as use in production will reveal things to fix, not yet specific things planned. Plan time for this, be ready to allocate someone to this

[Davsarper]

DSG coming up, need to be on top but not known work.