Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(data-masking): add docs for data masking utility #3186

Merged
merged 178 commits into from
Feb 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
178 commits
Select commit Hold shift + click to select a range
236e223
Skeleton draft of data_masking docs
seshubaws Oct 10, 2023
025fc77
Added example getting started code to data masking docs
seshubaws Oct 11, 2023
0c0c4ac
Added SAM template example and fixed KMS permission info
seshubaws Oct 11, 2023
0112f69
Added clearer file names
seshubaws Oct 11, 2023
a45a4e3
Add testing your code example
seshubaws Oct 12, 2023
7b1645d
Added diagram and fixed highlighting in code examples
seshubaws Oct 12, 2023
3c83aa0
Added SAM template section to md file
seshubaws Oct 12, 2023
36faafb
Merge branch 'develop' into develop
leandrodamascena Oct 18, 2023
a2fbff8
Merge branch 'develop' into develop
leandrodamascena Oct 26, 2023
d26f78d
Merge branch 'develop' into develop
leandrodamascena Nov 6, 2023
3988f10
Separated examples into more tabs, fixed comments
seshubaws Nov 8, 2023
1609105
Fix mypy errors
seshubaws Nov 8, 2023
65a9e9f
Fix mypy errors
seshubaws Nov 14, 2023
9fc33b4
Fix mypy
seshubaws Nov 14, 2023
e631581
Remove itsdangerous
seshubaws Nov 14, 2023
2a959fb
fix mypy
seshubaws Nov 14, 2023
b206873
delete superflous init file
seshubaws Nov 14, 2023
bb1c2a9
Fix mypy
seshubaws Nov 14, 2023
ff8d490
Merge branch 'develop' into develop
seshubaws Nov 14, 2023
e87b05a
Merge branch 'develop' into develop
seshubaws Nov 21, 2023
9f0acb2
Reorganized data masking docs
seshubaws Nov 22, 2023
ae7deb6
nit fixes
seshubaws Nov 22, 2023
b8bd70f
Added itsdangerous as dev dep
seshubaws Nov 22, 2023
2f7c6b7
grammar fixes
seshubaws Dec 5, 2023
3199576
merging from develop
leandrodamascena Dec 8, 2023
b9c2c48
docs: refactor diag to make operations explicit
heitorlessa Dec 8, 2023
b35effd
docs: line editing for intro line
heitorlessa Dec 8, 2023
202d906
docs: line editing for one key feature
heitorlessa Dec 8, 2023
6687ebc
docs: Masking to Possible Operations to remove ambiguity
heitorlessa Dec 8, 2023
5942a70
Removing custom provider
leandrodamascena Dec 8, 2023
619292d
Adding banner
leandrodamascena Dec 8, 2023
c33f18e
docs: remove BYO from key features, highlight best practices
heitorlessa Dec 8, 2023
9ab5a40
Modifying SAM
leandrodamascena Dec 8, 2023
8965413
docs: line editing terminology
heitorlessa Dec 8, 2023
0762ba2
Revise docs
seshubaws Dec 9, 2023
3b49271
Modifying SAM
leandrodamascena Dec 11, 2023
bc97ca8
Merging from develop
leandrodamascena Dec 11, 2023
847c326
Removing itsdangerous dependency - we are not using
leandrodamascena Dec 11, 2023
ed768ca
Fixing mypy errors
leandrodamascena Dec 11, 2023
243e8cb
Adding more information
leandrodamascena Dec 11, 2023
b3f0884
Merge branch 'develop' into develop
leandrodamascena Dec 11, 2023
b6d0470
Added more info about fields param
seshubaws Dec 11, 2023
ef232de
Merge branch 'develop' into develop
leandrodamascena Dec 12, 2023
7111fba
Making error message actionable
leandrodamascena Dec 12, 2023
08fcadc
Making error message actionable
leandrodamascena Dec 12, 2023
3dc9d57
Making error message actionable
leandrodamascena Dec 12, 2023
410ed3b
docs: add first sequence diagram for operations
heitorlessa Dec 12, 2023
42a682b
docs: add encrypt operations sequence diagram
heitorlessa Dec 12, 2023
abe27f1
docs: remove dot notation from mask operation
heitorlessa Dec 12, 2023
a265b64
Adding typing
leandrodamascena Dec 12, 2023
e16833f
Fixes for SAM template comments
seshubaws Dec 13, 2023
ecf505e
Merge branch 'develop' into develop
seshubaws Dec 13, 2023
0cb967a
Add return types for mask()
seshubaws Dec 13, 2023
af86cd0
Merge branch 'develop' into develop
seshubaws Dec 13, 2023
e3c62fd
Addressing Seshu's feedback
leandrodamascena Dec 14, 2023
ebcc343
Merging from develop
leandrodamascena Dec 14, 2023
f41026b
Improving examples
leandrodamascena Dec 14, 2023
9413a26
Improving examples
leandrodamascena Dec 14, 2023
cfae267
docs: improve encrypt ops sequence diagram
heitorlessa Dec 14, 2023
74bbd60
docs: early caching msg before diagram
heitorlessa Dec 14, 2023
db318cd
docs: add caching in encryption sdk ops
heitorlessa Dec 14, 2023
fe184c4
Added max_bytes_encrypted to CMM
seshubaws Dec 14, 2023
65bf540
Fix mypy errors
seshubaws Dec 14, 2023
aacf0db
Adding logging + data type + variable names
leandrodamascena Dec 14, 2023
6970cbf
Added docstrings to baseprovider
seshubaws Dec 15, 2023
20f1315
Explain fields syntax more
seshubaws Dec 15, 2023
1879abc
Merge branch 'develop' into develop
heitorlessa Dec 15, 2023
5832e49
Merge branch 'develop' into develop
seshubaws Dec 15, 2023
007231a
Clarify fields param
seshubaws Dec 15, 2023
b395f12
docs: add decrypt operation diag
heitorlessa Dec 18, 2023
13a1f5d
docs: add encryption ctx, envelope encryption terminologies
heitorlessa Dec 18, 2023
0d8c530
docs: line editing terminology
heitorlessa Dec 18, 2023
352ed1e
docs: correct getting started, install sections
heitorlessa Dec 18, 2023
1dc130c
docs: add note on min memory and separation of concerns upfront
heitorlessa Dec 18, 2023
fd7fd71
docs: use newer Powertools log level env var
heitorlessa Dec 18, 2023
9b1bf22
docs: add missing logger ctx, remove metrics
heitorlessa Dec 18, 2023
7540dcc
docs: line editing comments
heitorlessa Dec 18, 2023
be60985
docs: add code annotation, further cleanup
heitorlessa Dec 18, 2023
963f8cc
docs: remove unused section
heitorlessa Dec 18, 2023
40644f1
docs: move navigation order, fix encryption typo
heitorlessa Dec 18, 2023
1b1f19b
Merge branch 'develop' into develop
heitorlessa Dec 18, 2023
9a33393
docs: line editing masking data
heitorlessa Dec 18, 2023
7733633
docs: fix file names in data masking
heitorlessa Dec 18, 2023
3d7b237
docs: add symmetric word in required resources
heitorlessa Dec 18, 2023
b816197
docs: line editing encrypting data
heitorlessa Dec 18, 2023
2bae940
docs: line editing decrypting data; code snippet fixes
heitorlessa Dec 18, 2023
8104cb3
docs: move load tests and traces upfront
heitorlessa Dec 18, 2023
a714623
docs: use pascal case for encryption sdk provider
heitorlessa Dec 18, 2023
fa24044
docs: remove redundant info from encryption sdk
heitorlessa Dec 18, 2023
0afe8fa
docs: add encryption message, link provider section
heitorlessa Dec 18, 2023
9b14252
docs: add initial encryption context section
heitorlessa Dec 18, 2023
b5a0aef
docs: add initial encryption context section
heitorlessa Dec 18, 2023
07cf100
docs: remove incorrect info on enc ctx being required, thus allowing …
heitorlessa Dec 18, 2023
24bcef7
docs: rename to AAD and add examples
heitorlessa Dec 18, 2023
ca7897f
docs: rename to AAD and add examples
heitorlessa Dec 18, 2023
fec33a6
Adding support to list/set/tuple fields + renaming the class of the p…
leandrodamascena Dec 18, 2023
74fccd7
Small fix
leandrodamascena Dec 18, 2023
a741e7e
Merge remote-tracking branch 'upstream/develop' into seshubaws/develop
leandrodamascena Dec 18, 2023
189bcba
Fixing typing
leandrodamascena Dec 18, 2023
483c1b4
Fixing typing
leandrodamascena Dec 18, 2023
79db91e
Small fixes
seshubaws Dec 18, 2023
e9d93bf
docs: rename to enc ctx and correct wording for upcoming API change t…
heitorlessa Dec 19, 2023
a0921a3
docs: use Dict as return as that's the expected return type
heitorlessa Dec 19, 2023
d9d9e31
docs: fix highlighting
heitorlessa Dec 19, 2023
93a20f8
docs: add choosing parts of your data section
heitorlessa Dec 19, 2023
e7ae1f1
docs: improve JSON section
heitorlessa Dec 19, 2023
ce37852
docs: break down field syntax into two sections
heitorlessa Dec 19, 2023
06fbb09
Merge branch 'develop' into develop
heitorlessa Dec 19, 2023
ea77812
refactor: allow customers to bring custom serializer/deserializer
heitorlessa Dec 19, 2023
f97ffb5
refactor: DataMasking depend on provider serializers
heitorlessa Dec 19, 2023
8b3c6d2
docs: add custom serializer example
heitorlessa Dec 19, 2023
f1c8dde
refactor: allow arbitrary encryption context
heitorlessa Dec 19, 2023
34f8e1b
docs: rename encryption context example
heitorlessa Dec 19, 2023
8519896
fix: encryption context propagation
heitorlessa Dec 19, 2023
7be9566
refactor: validate encryption context values
heitorlessa Dec 19, 2023
1ab12a2
docs: remove todo
heitorlessa Dec 19, 2023
e6889ba
pull out comparing enc_ctx into new method
seshubaws Dec 20, 2023
840b85b
Adding support to jsonpath-ng + refactoring tests
leandrodamascena Dec 20, 2023
2cad772
refactor: delegate encoding/decoding to shared fn as much as possible…
heitorlessa Dec 20, 2023
a770fa1
refactor: correct typing for encrypt
heitorlessa Dec 20, 2023
a918ced
refactor: correct typing for mask
heitorlessa Dec 20, 2023
f937d80
docs: correct code snippets typing
heitorlessa Dec 20, 2023
e9735d5
Adding flag to support raise on non existing field
leandrodamascena Dec 20, 2023
f763a11
docs: make it explicit the behaviour diff in encrypt & mask
heitorlessa Dec 20, 2023
634cfc2
Adding examples on how to access data
leandrodamascena Dec 20, 2023
4eef61e
Modifying perf test
leandrodamascena Dec 20, 2023
c284d4a
refactor: split lambda in partial + lambda to ease maintenance
heitorlessa Dec 20, 2023
b1da92e
refactor: decryption context for exact match
heitorlessa Dec 20, 2023
65b3d6a
chore: test encryption ctx validation
heitorlessa Dec 20, 2023
e0f4a3a
chore: fix linting on loop variable
heitorlessa Dec 20, 2023
fbed1a1
Adding complex examples + using ext ngjson
leandrodamascena Dec 20, 2023
d89cbcc
Adding text + increasing perf threshold
leandrodamascena Dec 20, 2023
5d1687e
Making utility public
leandrodamascena Dec 20, 2023
49b180a
Merge branch 'develop' into develop
leandrodamascena Dec 20, 2023
b0213a1
Adding chaging algorithm section
leandrodamascena Dec 20, 2023
64d6352
Merge branch 'develop' into develop
seshubaws Dec 20, 2023
95bc7a1
docs: address initial feedback
heitorlessa Dec 20, 2023
ace1c28
docs: re-incorporate initial feedback
heitorlessa Dec 20, 2023
429eb8a
Adding test
leandrodamascena Dec 20, 2023
85766bf
Fixed docstrings, added a test
seshubaws Jan 23, 2024
58c66f9
Merge develop from develop
seshubaws Jan 23, 2024
2328c50
Merge branch 'develop' into develop
leandrodamascena Jan 26, 2024
95098e5
Removed fields param from enc+dec methods
seshubaws Jan 29, 2024
af6932a
trying to fix pytest
seshubaws Jan 29, 2024
fbdcfd1
Merge branch 'develop' into develop
seshubaws Jan 29, 2024
9116ba6
trying to fix pytest
seshubaws Jan 29, 2024
5251e0c
Fix linting
seshubaws Jan 29, 2024
284d340
debug linting
seshubaws Jan 29, 2024
d01b657
debug linting
seshubaws Jan 29, 2024
00f6165
Changing mask to erase in docs, comments, etc
seshubaws Jan 29, 2024
5949cea
Fix for jsonpath upgrade
seshubaws Jan 29, 2024
cfa20c8
Fixing mypy typing
seshubaws Jan 29, 2024
7864bed
Merge branch 'develop' into develop
seshubaws Jan 29, 2024
eb66f19
test e2e
seshubaws Jan 30, 2024
0f01d50
Renaming files
leandrodamascena Jan 30, 2024
0d96349
Fixing examples
leandrodamascena Jan 30, 2024
990e8ad
Fixing examples
leandrodamascena Jan 30, 2024
cbc2b14
Merge branch 'develop' into develop
leandrodamascena Jan 30, 2024
a6bb1e3
Used AWS guidelines safe names in examples
seshubaws Jan 30, 2024
b7a3345
Merge branch 'develop' into develop
seshubaws Jan 30, 2024
62e1aba
Docs cleanup
seshubaws Jan 30, 2024
5ca7f99
Revise docs
seshubaws Jan 30, 2024
160b85f
Merge branch 'develop' into develop
leandrodamascena Jan 31, 2024
f116d3c
Update docs/utilities/data_masking.md
seshubaws Jan 31, 2024
839755a
Update docs/utilities/data_masking.md
seshubaws Jan 31, 2024
2a38326
Update docs/utilities/data_masking.md
seshubaws Jan 31, 2024
a6a975c
Added data masking to features lists, removed unnecessary tabs
seshubaws Jan 31, 2024
4156b3d
Made passing sdk args section more general
seshubaws Jan 31, 2024
f0cc727
Added using multiple keys section
seshubaws Jan 31, 2024
9d1d1e9
Fix lint
seshubaws Jan 31, 2024
581145d
Polishing few things
leandrodamascena Feb 1, 2024
b374d9f
Merge branch 'develop' into develop
leandrodamascena Feb 1, 2024
8f40a15
Addressing Heitor's feedback
leandrodamascena Feb 1, 2024
09f89ea
Adding workflow
leandrodamascena Feb 1, 2024
f19651d
Addressing GH hidden conversations
leandrodamascena Feb 1, 2024
f660683
Documentation
Feb 1, 2024
cfeb833
Final changes
leandrodamascena Feb 1, 2024
007fba7
Lock file
leandrodamascena Feb 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ dev:
pip install --upgrade pip pre-commit poetry
poetry config --local virtualenvs.in-project true
@$(MAKE) dev-version-plugin
poetry install --extras "all datamasking-aws-sdk redis"
poetry install --extras "all redis"
pre-commit install

dev-gitpod:
pip install --upgrade pip poetry
@$(MAKE) dev-version-plugin
poetry install --extras "all datamasking-aws-sdk redis"
poetry install --extras "all redis"
pre-commit install

format:
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ Powertools for AWS Lambda (Python) is a developer toolkit to implement Serverles
* **[Event source data classes](https://docs.powertools.aws.dev/lambda/python/latest/utilities/data_classes/)** - Data classes describing the schema of common Lambda event triggers
* **[Parser](https://docs.powertools.aws.dev/lambda/python/latest/utilities/parser/)** - Data parsing and deep validation using Pydantic
* **[Idempotency](https://docs.powertools.aws.dev/lambda/python/latest/utilities/idempotency/)** - Convert your Lambda functions into idempotent operations which are safe to retry
* **[Data Masking](https://docs.powertools.aws.dev/lambda/python/latest/utilities/data_masking/)** - Protect confidential data with easy removal or encryption
heitorlessa marked this conversation as resolved.
Show resolved Hide resolved
* **[Feature Flags](https://docs.powertools.aws.dev/lambda/python/latest/utilities/feature_flags/)** - A simple rule engine to evaluate when one or multiple features should be enabled depending on the input
* **[Streaming](https://docs.powertools.aws.dev/lambda/python/latest/utilities/streaming/)** - Streams datasets larger than the available memory as streaming data.

Expand Down
12 changes: 10 additions & 2 deletions aws_lambda_powertools/shared/functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -96,10 +96,18 @@ def resolve_env_var_choice(

def base64_decode(value: str) -> bytes:
try:
logger.debug("Decoding base64 record item before parsing")
logger.debug("Decoding base64 item to bytes")
return base64.b64decode(value)
except (BinAsciiError, TypeError):
raise ValueError("base64 decode failed")
raise ValueError("base64 decode failed - is this base64 encoded string?")


def bytes_to_base64_string(value: bytes) -> str:
try:
logger.debug("Encoding bytes to base64 string")
return base64.b64encode(value).decode()
except TypeError:
raise ValueError(f"base64 encoding failed - is this bytes data? type: {type(value)}")


def bytes_to_string(value: bytes) -> str:
Expand Down
174 changes: 0 additions & 174 deletions aws_lambda_powertools/utilities/_data_masking/base.py

This file was deleted.

5 changes: 0 additions & 5 deletions aws_lambda_powertools/utilities/_data_masking/constants.py

This file was deleted.

This file was deleted.

34 changes: 0 additions & 34 deletions aws_lambda_powertools/utilities/_data_masking/provider/base.py

This file was deleted.

This file was deleted.

Loading
Loading