feat: Implement user-defined entity selection strategies in Presidio Structured

[miltonsim]

Change Description

I implemented an additional selection_strategy option in the generate_analysis() function, allowing users to define their preferred strategy for entity selection. Previously, the function only enabled selection based on the most common entity, which could result in scenarios where a small number of high-confidence entities were overshadowed by a larger number of low-confidence entities. Now, users have the flexibility to choose between three strategies: most common, highest_confidence or mixed.

I would love to hear to hear feedback and suggestions!

Issue reference

This PR fixes issue #1316

Checklist

• I have reviewed the contribution guidelines
• I have signed the CLA (if required)
• My code includes unit tests
• All unit tests and lint checks pass locally
• My PR contains documentation updates / additions if required

[omri374]

Thanks @mitonsim! Left a few minor comments

[omri374]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[omri374]

Please also add a note to the docs about the strategies: https://github.com/microsoft/presidio/blob/main/docs/structured/index.md

[miltonsim]

@omri374, thanks for your feedback! I've incorporated the requested changes.

However, I encountered an issue where my newly added test cases failed in the Azure Pipeline. The assertion assert key_recognizer_result_map["street"].entity_type == "NON_PII" is not passing because the 'street' column is incorrectly identified as a location, whereas, in my local environment, it's correctly identified as NON_PII.

Could you shed some light on this discrepancy and how I can resolve it?

[omri374]

Thanks for the updates @miltonsim! I'm not sure where the discrepancy is coming from. It could be the seed used for sampling, or a different version of the spaCy model. To reduce noise in unit tests, perhaps it would be easier to choose an entity detected using a simpler logic? like a phone number, email, credit card?

[omri374]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[omri374]

Hi @miltonsim would you be interested in continuing to work on this? I would suggest to replace the non-deterministic entities with others to make sure tests pass on all environments.

[miltonsim]

@omri374 Apologies for the delay. I'll provide an update by 14 March (Tues)

[omri374]

Thanks!

[miltonsim]

@omri374 Thank you for waiting. I figured that removing the street column might be the best solution. Please help me run the pipeline to check if it passes!

[omri374]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[omri374]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[elbud]

Could this update be pushed to PyPi? There is only the original release available at the moment. https://pypi.org/project/presidio-structured/#history

[omri374]

Hi @elbud, we plan to have a new release in the next couple of weeks.