Support compressed configuration [RHELDST-25461] #586
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Config has formerly always been stored as a String attribute on DynamoDB items. Let's support also loading it as a gzip-compressed Binary attribute. The motivation is to allow more room for the config to grow. The max DynamoDB item size is 400KB, and a realistic string-based config has already reached 70KB (about 18% of the limit). As the config is only ever expected to grow over time, we may eventually hit that limit without warning; this discourages us from adding more fields into the config. The same config compressed uses about ~5KB (about 1% of the limit), which seems small enough that we'll never have to worry about hitting the limit and we can freely add new fields into the config. This change should be fully deployed to all exodus-cdn environments before deploying the corresponding change to exodus-gw.
rohanpm
added a commit
to rohanpm/exodus-gw
that referenced
this pull request
Jul 29, 2024
Start writing the configuration in compressed binary form. For more info on the motivation, see the corresponding change in exodus-lambda: release-engineering/exodus-lambda#586
negillett
approved these changes
Jul 29, 2024
crungehottman
approved these changes
Jul 29, 2024
rohanpm
added a commit
to rohanpm/exodus-gw
that referenced
this pull request
Jul 29, 2024
Start writing the configuration in compressed binary form. For more info on the motivation, see the corresponding change in exodus-lambda: release-engineering/exodus-lambda#586 Regarding the test updates in this commit, the existing fixture used to mock the DynamoDB query for config was parametrized so that existing tests would cover both binary and non-binary config cases. However, since that was previously a global autouse fixture, that would have caused every test to run twice even if the test did not deal with the config at all. To avoid this, fixtures were split into a global autouse boto session fixture (ensuring all tests still avoid any real AWS calls) and a non-autouse fixture for the DynamoDB client. The new fixture had to be explicitly added to certain tests.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Config has formerly always been stored as a String attribute on DynamoDB items. Let's support also loading it as a gzip-compressed Binary attribute.
The motivation is to allow more room for the config to grow. The max DynamoDB item size is 400KB, and a realistic string-based config has already reached 70KB (about 18% of the limit). As the config is only ever expected to grow over time, we may eventually hit that limit without warning; this discourages us from adding more fields into the config.
The same config compressed uses about ~5KB (about 1% of the limit), which seems small enough that we'll never have to worry about hitting the limit and we can freely add new fields into the config.
This change should be fully deployed to all exodus-cdn environments before deploying the corresponding change to exodus-gw.