COSI-19, COSI-21: Enhance Scality COSI Driver with S3 and IAM Metrics, Logging, and IAM Client Improvements #85
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files
@@ Coverage Diff @@
## feature/COSI-65-instrument-cosi-drover-with-gprc-metrics #85 +/- ##
============================================================================================
- Coverage 93.67% 93.42% -0.26%
============================================================================================
Files 10 11 +1
Lines 680 745 +65
============================================================================================
+ Hits 637 696 +59
- Misses 37 41 +4
- Partials 6 8 +2 |
- use middleware for aws sdk for adding metrics for s3 and iam
anurag4DSB
force-pushed
the
feature/COSI-19-add-s3-iam-metrics
branch
from
7e94e8d to
d9b904f
Compare
anurag4DSB
force-pushed
the
feature/COSI-19-add-s3-iam-metrics
branch
from
d9b904f to
8880371
Compare
anurag4DSB
changed the title
anurag4DSB
changed the title
anurag4DSB
changed the title
| Name: "s3_requests_total", | ||
| Help: "Total number of S3 requests, categorized by method and status.", | ||
| }, | ||
| []string{"method", "status"}, |
There was a problem hiding this comment.
I think the S3 operation type is commonly called action throughout the product metrics.
There was a problem hiding this comment.
Okay, I will change that.
Thanks
|
|
||
| | Metric Name | Description | Labels | Example Values | | ||
| |---------------------------------------------------|------------------------------------------------------------|-------------------|----------------------------------| | ||
| | `scality_cosi_driver_iam_request_duration_seconds`| Histogram of IAM request durations in seconds. | `method`, `status`| `CreateUser`, `success`, `error`| |
There was a problem hiding this comment.
You may want to mention only one of success or error in the example values
There was a problem hiding this comment.
And in the table below for IAM Operations note that it is the method label
And add another table for the status label with success and error
|
|
||
| | Metric Name | Description | Labels | Example Values | | ||
| |---------------------------------------------------|------------------------------------------------------------|-------------------|----------------------------------| | ||
| | `scality_cosi_driver_s3_request_duration_seconds` | Histogram of S3 request durations in seconds. | `method`, `status`| `CreateBucket`, `success`, `error`| |
There was a problem hiding this comment.
ditto about mentioning only success or error
Comment on lines
+126
to
+129
| if [[ "$CREATE_BUCKET_COUNT" -ne "$EXPECTED_CREATE_BUCKET" ]]; then | ||
| echo "Error: CreateBucket count mismatch. Found: $CREATE_BUCKET_COUNT, Expected: $EXPECTED_CREATE_BUCKET" | tee -a "$LOG_FILE" | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
This is duplicated 4 times, could be a function
validate 'CreateBucket' $CREATE_BUCKET_COUNT $EXPECTED_CREATE_BUCKET $CREATE_BUCKET_DURATION
|
|
||
| 1. Verify the bucket exists. | ||
| 2. Use the `DeleteBucket` operation to delete the bucket. Only empty bucket deletion is supported. | ||
|
|
||
| ## Additional Resource | ||
|
|
||
| - [gRPC-Go Prometheus Metrics](https://github.com/grpc-ecosystem/go-grpc-prometheus) |
There was a problem hiding this comment.
same here update link to deprecated lib
BourgoisMickael
approved these changes
Jan 3, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces enhancements to the Scality COSI driver, including metrics for IAM and S3 operations, improved logging, and updates to the IAM client initialization process. Key highlights include:
Metrics and Observability:
IAM Client Enhancements:
Testing and Validation:
These changes improve observability, robustness, and clarity in the Scality COSI driver’s operation and metrics tracking.