Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Question about Using AWS S3 in region "cn-northwest-1"(China) #294

Open
zhangsimingshannonai opened this issue Oct 15, 2019 · 4 comments
Open

Question about Using AWS S3 in region "cn-northwest-1"(China) #294

zhangsimingshannonai opened this issue Oct 15, 2019 · 4 comments

Comments

@zhangsimingshannonai
Copy link

zhangsimingshannonai commented Oct 15, 2019 •
edited
Loading

My S3 bucket is in AWS-CHINA, region is "cn-northwest-1", but I didn't find anything about this in the src, Is ice support china-region?

I used ice in kubernetes way.
Here is my configmap-ice.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ice
data:
  ice.properties: |
    ice.processor=true
    ice.reader=true
    ice.reservationCapacityPoller=false
    ice.reservationPeriod=oneyear
    ice.reservationUtilization=HEAVY
    ice.highstockUrl=https://code.highcharts.com/stock/4.2.1/highstock.js
    ice.urlPrefix=aws.shannonai
    ice.fromEmail=xxxxxxxxxxxxxxxxxxxxxxxx
    ice.ondemandCostAlertThreshold=250
    ice.s3SecretKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    ice.billing_s3bucketname=xxxxxxxxxxxx
    ice.billing_s3bucketregion=cn-northwest-1
    ice.billing_accessRoleName=xxxxxxxxxxxxxxxxx
    ice.startmillis=1364774400000
    ice.companyName=shannonai
    ice.work_s3bucketname=xxxxxxxxxxxxxxxxxx
    ice.work_s3bucketprefix=ice/
    ice.work_s3bucketregion=cn-northwest-1
    ice.processor.localDir=/mnt/ice_processor
    ice.reader.localDir=/mnt/ice_reader
    ice.monthlycachesize=12

Here is my deployment.yaml's env part

  - name: ice
          image: harbor.shannonai.com/test/ice:latest
          ports:
            - containerPort: 8080
          imagePullPolicy: IfNotPresent
          env: 
            - name: GRAILS_OPTS 
              value: "-server -Xmx2G -Xms1G -Dfile.encoding=UTF-8"
            - name: EC2_REGION
              value: cn-northwest-1
            - name: AWS_DEFAULT_REGION
              value: cn-northwest-1
          command:
            - /opt/ice/grailsw
            - -Djava.net.preferIPv4Stack=true
            - -Djava.net.preferIPv4Addresses
            - -Duser.timezone=Asia/Shanghai
            - -Dice.s3AccessKeyId=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
            - -Dice.s3SecretKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
            - run-app

The error I met is

2019-10-15 18:17:32,210 [localhost-startStop-1] INFO  BootStrap  - Starting ice...
| Error com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; the authorization component "Credential=xxxxxxxxxxxxxx/20191015/us-east-1/s3/aws4_request" is malformed. (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: 194A470802DB493B), S3 Extended Request ID: U0DPCCZy6ZswY91v4lav0jjMX7SEtAn6S+OHsCTMIU3Z7BGo9Q1mRNJF/eUk1NyPlm91vKRH0Ak=

It seems that ice is still connect to the default s3 region "us-east-1", where did I misconfig? Please help me to figure it out, thanks very much!
@nfonrose
Copy link
Member

nfonrose commented Oct 15, 2019 via email

@zhangsimingshannonai
Copy link
Author

zhangsimingshannonai commented Oct 15, 2019 •
edited
Loading

@nfonrose

Thanks for your reply, but I still not know where should I change, what is you mean "code"?
I need to rebuild a dockerimage for china? Please tell me what file should I modify? Thanks

@zhangsimingshannonai
Copy link
Author

zhangsimingshannonai commented Oct 15, 2019 •
edited
Loading

You said "Changing the region for the bucket is not enough for AWS China", but I even not change region successfully.
my error is

2019-10-15 18:17:32,210 [localhost-startStop-1] INFO  BootStrap  - Starting ice...
| Error com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; the authorization component "Credential=xxxxxxxxxxxxxx/20191015/us-east-1/s3/aws4_request" is malformed. (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: 194A470802DB493B), S3 Extended Request ID: U0DPCCZy6ZswY91v4lav0jjMX7SEtAn6S+OHsCTMIU3Z7BGo9Q1mRNJF/eUk1NyPlm91vKRH0Ak=

It is still connect to "us-east-1", but I set "ice.billing_s3bucketregion=cn-north-1" in ice.properties.
I don't know why, please help me

@zhangsimingshannonai
Copy link
Author

zhangsimingshannonai commented Oct 15, 2019 •
edited
Loading

Finally change AwsUtils.java file's endpoint, nothing happend.

$ ls
AwsUtils.java  Dockerfile

$ ack AwsUtils.java
Dockerfile
6:COPY AwsUtils.java src/java/com/netflix/ice/common/AwsUtils.java

$ ack Endpoint
AwsUtils.java
100:                s3Client.setEndpoint("s3.amazonaws.com");
103:                s3Client.setEndpoint("s3-" + System.getProperty("EC2_REGION") + ".amazonaws.com.cn");
123:                    simpleDBClient.setEndpoint("sdb.amazonaws.com");
126:                    simpleDBClient.setEndpoint("sdb." + System.getProperty("EC2_REGION") + ".amazonaws.com.cn");
276:                s3Client.setEndpoint("s3-" + bucketFileRegion + ".amazonaws.com.cn");

Could anyone answer my question? I would be greatful appreciate.

@debora-ito debora-ito mentioned this issue Oct 18, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nfonrose @zhangsimingshannonai