Auditing / Reporting / Deletion Certification

[Danny-Cooke-CK]

Some institutions have regulatory and contractual conditions around their research environments which include the ability to Audit and Report and also produce Deletion Certificates. Auditing and reporting is probably more of a standard requirement but the need to produce "deletion certificates" to prove that a machine, which was home to sensitive data, has been destroyed is possibly more of a research institution specific requirement.

Some types of generic queries:
Delete Certificate - Proof that X person deleted a machine and associated Disks containing Data
Access - Who accessed a machine holding data
Data Transfer - When was the data transferred onto or off a machine and by who
Approval - Who approved the data transfer

Considerations:
Lets assume it would be researchers performing these actions and not IT personnel.

From a quick and easy point of view, it seems logical to offload the Data Analytics data and Cosmo DB records into a central data source somewhere to be queried via some sort of querying tool but that doesn't feel right as we can't be the only institution looking into this? So it might be a good idea to add some functionality to the TRE itself

Idea's on functionality:

1. Select the actions menu on a VM or Workspace and select audit ? Pass in some dates to the query and bring back all of the analytics logs between those dates?
2. When selecting Delete on a VM, should / could a deletion certificate be produced at the same time? Posted into a teams chat or emailed out to people? maybe have this switchable and so can turn it off if organisations don't require it ?
3. Airlock request's audit. What was transferred, when, who via an interface. this may require a history of airlock requests being accessible from a TRE view

We would like thoughts, suggestions and comments please from the community and Microsoft. We've left this open and high level at this point just to gauge interest but we feel a solution which makes it easily accessible for the researchers would be the best approach

[marrobi]

What actions do you ned to audit? There is currently an operations log for each deployed resource. It might be they could be linked up to event grid, in the same way that airlock events are to facilitate notifications.

Would this be a core feature for people with an audit role, as auditors might not have access into each workspace? Not sur ehaving audit on each item makes sense...

Also delete has at times been discussed, often organisations don't want the data within a workspace deleting, just the compute, in case they need to come back for it at a later point. Not for this discussion, but something to bear in mind.

[Danny-Cooke-CK]

For Audit we have these as a guide but as with most audits its not a definitive list.
Who built a machine
Who accessed a machine
Who accessed a machine when the data was present
Who transferred the data (airlock)
Who approved the airlock request
When was the data transferred.
Who destroyed a machine.

Happy for it to be a specific role such as audit. The want really is to keep the audit requirement away from an IT person. The operation log is useful for some aspects but more information is needed and also to make it searchable. For instance the Guacamole access logs. These seem to be the most useful for who accessed what what. The reason for making it searchable is also due to the length of time machines could be up i.e checking access to a machine during February 12 months ago etc.

As for the Delete and destruction certificate, would it be a compromise to have it optional ? action/delete (certify Y/N) possibly?

[marrobi]

For each of those you need to define where the action occurs/is logged - i.e. the data source. Some will be in the TRE operation logs, some will be as you say in OSS service logs - such as guacamole, some will be in Azure activity logs, or even say Databricks logs.

It might make sense for the to all sit in log analytics with custom attributes, rather than cosmos.