Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bugfix][plugin][s3writer]S3writer: add pathStyleAccessEnabled config for fixed error 400 when upload to minio(set pathStyleAccessEnabled true when upload to minio) #1186

Merged
merged 2 commits into from
Nov 25, 2024
Merged

[bugfix][plugin][s3writer]S3writer: add pathStyleAccessEnabled config for fixed error 400 when upload to minio(set pathStyleAccessEnabled true when upload to minio) #1186

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
50e5a92
[bugfix][plugin][s3writer]S3writer: add pathStyleAccessEnabled config…
awol2005ex Nov 25, 2024
345deb6
[bugfix][plugin][s3writer]S3writer:don't submit upload when no record…
awol2005ex Nov 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions plugin/writer/s3writer/src/main/java/com/wgzhao/addax/plugin/writer/s3writer/S3Key.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,6 @@ public class S3Key extends Key
public static final String MAX_FILE_SIZE = "maxFileSize";

public static final String DEFAULT_SUFFIX = "defaultSuffix";

public static final String PATH_STYLE_ACCESS_ENABLED = "pathStyleAccessEnabled";
}
11 changes: 8 additions & 3 deletions plugin/writer/s3writer/src/main/java/com/wgzhao/addax/plugin/writer/s3writer/S3Util.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,23 @@ public static S3Client initS3Client(Configuration conf) {
Region region = Region.of(regionStr);
String accessId = conf.getString(S3Key.ACCESS_ID);
String accessKey = conf.getString(S3Key.ACCESS_KEY);
String pathStyleAccessEnabled =conf.getString(S3Key.PATH_STYLE_ACCESS_ENABLED,"");

return initS3Client(conf.getString(S3Key.ENDPOINT), region, accessId, accessKey);
return initS3Client(conf.getString(S3Key.ENDPOINT), region, accessId, accessKey ,pathStyleAccessEnabled);

}

public static S3Client initS3Client(String endpoint, Region region, String accessId, String accessKey) {
public static S3Client initS3Client(String endpoint, Region region, String accessId, String accessKey ,String pathStyleAccessEnabled) {
if (null == region) {
region = Region.of("ap-northeast-1");
}
try {
AwsBasicCredentials awsCreds = AwsBasicCredentials.create(accessId, accessKey);
return S3Client.builder()
return S3Client.builder().serviceConfiguration(e -> {
if("true".equals(pathStyleAccessEnabled)) {
e.pathStyleAccessEnabled(true);
}
})
.credentialsProvider(StaticCredentialsProvider.create(awsCreds))
.region(region)
.endpointOverride(URI.create(endpoint))
Expand Down
32 changes: 18 additions & 14 deletions plugin/writer/s3writer/src/main/java/com/wgzhao/addax/plugin/writer/s3writer/S3Writer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,22 +307,26 @@ public void startWrite(RecordReceiver lineReceiver)
completedParts.add(completedPart);
outputStream.reset();
}
// Finally, call completeMultipartUpload operation to tell S3 to merge all uploaded
// parts and finish the multipart operation.
CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder()
.parts(completedParts)
.build();
if(!completedParts.isEmpty()) {
// Finally, call completeMultipartUpload operation to tell S3 to merge all uploaded
// parts and finish the multipart operation.
CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder()
.parts(completedParts)
.build();

CompleteMultipartUploadRequest completeMultipartUploadRequest =
CompleteMultipartUploadRequest.builder()
.bucket(bucket)
.key(object)
.uploadId(uploadId)
.multipartUpload(completedMultipartUpload)
.build();
CompleteMultipartUploadRequest completeMultipartUploadRequest =
CompleteMultipartUploadRequest.builder()
.bucket(bucket)
.key(object)
.uploadId(uploadId)
.multipartUpload(completedMultipartUpload)
.build();

s3Client.completeMultipartUpload(completeMultipartUploadRequest);
LOG.info("end do write");
s3Client.completeMultipartUpload(completeMultipartUploadRequest);
LOG.info("end do write");
} else {
LOG.info("no content do write");
}
}

private String record2String(Record record)
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
<properties>
<!-- basic components -->
<aircompressor.version>0.25</aircompressor.version>
<awssdk.version>2.20.5</awssdk.version>
<awssdk.version>2.29.20</awssdk.version>
<avro.version>1.11.3</avro.version>
<commons.beanutils.version>1.9.4</commons.beanutils.version>
<commons.configuration.version>1.10</commons.configuration.version>
Expand Down