AWS S3 With Python-github.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Amazon Simple Storage Service (Amazon S3)** is an object storage service that offers scalability, data availability, security, and performance.\n",
    "\n",
    "\n",
    "Amazon S3 is designed for 99.999999999% (11 9's) of durability, and stores data for millions of applications for companies all around the world.\n",
    "\n",
    "\n",
    "An **Amazon S3 bucket** is a storage location to hold files. S3 files are referred to as **objects**.\n",
    "\n",
    "\n",
    "\n",
    "**Boto 3** Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python, which allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Create an Amazon S3 bucket**\n",
    "\n",
    "The name of an Amazon S3 bucket must be unique across all regions of the AWS platform. The bucket can be located in a specific region to minimize latency or to address regulatory requirements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:12:07.806893Z",
     "iopub.status.busy": "2024-08-09T19:12:07.803269Z",
     "iopub.status.idle": "2024-08-09T19:12:08.297764Z",
     "shell.execute_reply": "2024-08-09T19:12:08.297764Z",
     "shell.execute_reply.started": "2024-08-09T19:12:07.806893Z"
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import boto3\n",
    "from botocore.exceptions import ClientError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:25:53.259946Z",
     "iopub.status.busy": "2024-08-09T19:25:53.259946Z",
     "iopub.status.idle": "2024-08-09T19:25:53.291730Z",
     "shell.execute_reply": "2024-08-09T19:25:53.289507Z",
     "shell.execute_reply.started": "2024-08-09T19:25:53.259946Z"
    }
   },
   "outputs": [],
   "source": [
    "AWS_ACCESS_KEY_ID = \"xxx\"\n",
    "AWS_SECRET_ACCESS_KEY = \"xxx\"\n",
    "region_name = \"us-east-1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:22:13.715353Z",
     "iopub.status.busy": "2024-08-09T19:22:13.715353Z",
     "iopub.status.idle": "2024-08-09T19:22:13.747221Z",
     "shell.execute_reply": "2024-08-09T19:22:13.745664Z",
     "shell.execute_reply.started": "2024-08-09T19:22:13.715353Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create an S3 client\n",
    "s3_client = boto3.client(\n",
    "    's3',\n",
    "    region_name=region_name,\n",
    "    aws_access_key_id=AWS_ACCESS_KEY_ID,\n",
    "    aws_secret_access_key=AWS_SECRET_ACCESS_KEY\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:28:52.371398Z",
     "iopub.status.busy": "2024-08-09T19:28:52.371398Z",
     "iopub.status.idle": "2024-08-09T19:28:52.846822Z",
     "shell.execute_reply": "2024-08-09T19:28:52.846822Z",
     "shell.execute_reply.started": "2024-08-09T19:28:52.371398Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "my-s3-python\n",
      "my-s3-python-v2\n"
     ]
    }
   ],
   "source": [
    "# Retrieve the list of buckets\n",
    "response = s3_client.list_buckets()\n",
    "\n",
    "# Extract bucket names\n",
    "buckets = response['Buckets']\n",
    "\n",
    "# Print bucket names\n",
    "for bucket in buckets:\n",
    "    print(bucket['Name'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:29:30.662410Z",
     "iopub.status.busy": "2024-08-09T19:29:30.662410Z",
     "iopub.status.idle": "2024-08-09T19:29:31.153332Z",
     "shell.execute_reply": "2024-08-09T19:29:31.152248Z",
     "shell.execute_reply.started": "2024-08-09T19:29:30.662410Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bucket my-s3-python-v2 created successfully.\n"
     ]
    }
   ],
   "source": [
    "# Define the name of the bucket\n",
    "bucket_name = \"my-s3-python-v2\"\n",
    "\n",
    "from botocore.exceptions import NoCredentialsError, PartialCredentialsError, ClientError\n",
    "\n",
    "try:\n",
    "    if region_name == 'us-east-1':\n",
    "        # No LocationConstraint required for us-east-1\n",
    "        response = s3_client.create_bucket(\n",
    "            Bucket=bucket_name\n",
    "        )\n",
    "    else:\n",
    "        # Specify LocationConstraint for all other regions\n",
    "        response = s3_client.create_bucket(\n",
    "            Bucket=bucket_name,\n",
    "            CreateBucketConfiguration={\n",
    "                'LocationConstraint': region_name\n",
    "            }\n",
    "        )\n",
    "    print(f'Bucket {bucket_name} created successfully.')\n",
    "except NoCredentialsError:\n",
    "    print('Credentials not available.')\n",
    "except PartialCredentialsError:\n",
    "    print('Incomplete credentials provided.')\n",
    "except ClientError as e:\n",
    "    print(f'Client error: {e}')\n",
    "# ERROR: lientError: An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location-constraint is not valid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:30:01.528912Z",
     "iopub.status.busy": "2024-08-09T19:30:01.525901Z",
     "iopub.status.idle": "2024-08-09T19:30:01.548868Z",
     "shell.execute_reply": "2024-08-09T19:30:01.548356Z",
     "shell.execute_reply.started": "2024-08-09T19:30:01.528912Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ResponseMetadata': {'RequestId': 'BKW1TCD0H67NSZHE',\n",
       "  'HostId': 'ZPBwjnaNADT6zI/B9AUibqtbmxpO0JOI241355fwtVYsw9/PgSLSrkYufLuMfQgVnHdjGb8kB/w=',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amz-id-2': 'ZPBwjnaNADT6zI/B9AUibqtbmxpO0JOI241355fwtVYsw9/PgSLSrkYufLuMfQgVnHdjGb8kB/w=',\n",
       "   'x-amz-request-id': 'BKW1TCD0H67NSZHE',\n",
       "   'date': 'Fri, 09 Aug 2024 19:29:32 GMT',\n",
       "   'location': '/my-s3-python-v2',\n",
       "   'server': 'AmazonS3',\n",
       "   'content-length': '0'},\n",
       "  'RetryAttempts': 0},\n",
       " 'Location': '/my-s3-python-v2'}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Listing Buckets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:31:05.108369Z",
     "iopub.status.busy": "2024-08-09T19:31:05.108369Z",
     "iopub.status.idle": "2024-08-09T19:31:05.127039Z",
     "shell.execute_reply": "2024-08-09T19:31:05.127039Z",
     "shell.execute_reply.started": "2024-08-09T19:31:05.108369Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_buckets(s3_client):\n",
    "    # Retrieve the list of buckets\n",
    "    response = s3_client.list_buckets()\n",
    "    \n",
    "    # Extract bucket names\n",
    "    buckets = response['Buckets']\n",
    "    \n",
    "    # Print bucket names\n",
    "    for bucket in buckets:\n",
    "        print(bucket['Name'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:31:22.742242Z",
     "iopub.status.busy": "2024-08-09T19:31:22.740650Z",
     "iopub.status.idle": "2024-08-09T19:31:23.238677Z",
     "shell.execute_reply": "2024-08-09T19:31:23.238129Z",
     "shell.execute_reply.started": "2024-08-09T19:31:22.742242Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "my-s3-python\n",
      "my-s3-python-v2\n"
     ]
    }
   ],
   "source": [
    "get_buckets(s3_client=s3_client)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Uploading files\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:38:40.514156Z",
     "iopub.status.busy": "2024-08-09T19:38:40.514156Z",
     "iopub.status.idle": "2024-08-09T19:38:40.538640Z",
     "shell.execute_reply": "2024-08-09T19:38:40.538131Z",
     "shell.execute_reply.started": "2024-08-09T19:38:40.514156Z"
    }
   },
   "outputs": [],
   "source": [
    "bucket_name = \"my-s3-python-v2\"\n",
    "\n",
    "# List of files to upload\n",
    "file_list = [\n",
    "    'corpus/dog/d1.jpg',\n",
    "    'corpus/dog/d2.jpg',\n",
    "    'corpus/dog/d3.jpg'    \n",
    "]\n",
    "\n",
    "def upload_files_to_s3(s3_client, bucket_name, file_list):\n",
    "    for file_path in file_list:\n",
    "        try:\n",
    "            # Extract the file name from the path --> object name = file_name in Bucket\n",
    "            file_name = file_path.split('/')[-1]\n",
    "            \n",
    "            # Upload the file\n",
    "            s3_client.upload_file(file_path, bucket_name, file_name)\n",
    "            print(f'Successfully uploaded {file_name} to bucket {bucket_name}.')\n",
    "        except FileNotFoundError:\n",
    "            print(f'File not found: {file_path}')\n",
    "        except NoCredentialsError:\n",
    "            print('Credentials not available.')\n",
    "        except PartialCredentialsError:\n",
    "            print('Incomplete credentials provided.')\n",
    "        except ClientError as e:\n",
    "            print(f'Client error: {e}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:38:41.835591Z",
     "iopub.status.busy": "2024-08-09T19:38:41.835591Z",
     "iopub.status.idle": "2024-08-09T19:38:43.134567Z",
     "shell.execute_reply": "2024-08-09T19:38:43.134567Z",
     "shell.execute_reply.started": "2024-08-09T19:38:41.835591Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully uploaded d1.jpg to bucket my-s3-python-v2.\n",
      "Successfully uploaded d2.jpg to bucket my-s3-python-v2.\n",
      "Successfully uploaded d3.jpg to bucket my-s3-python-v2.\n"
     ]
    }
   ],
   "source": [
    "# Upload the files\n",
    "upload_files_to_s3(s3_client, bucket_name, file_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload as File Object\n",
    "\n",
    "To upload a file to an S3 bucket as a file object using boto3, you can use the upload_fileobj method. \n",
    "\n",
    "This method allows you to upload a file-like object directly to S3. \n",
    "\n",
    "This is particularly useful when you have files coming from sources like in-memory objects or files opened with Python's file handling methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:44:51.134077Z",
     "iopub.status.busy": "2024-08-09T19:44:51.134077Z",
     "iopub.status.idle": "2024-08-09T19:44:51.157013Z",
     "shell.execute_reply": "2024-08-09T19:44:51.155847Z",
     "shell.execute_reply.started": "2024-08-09T19:44:51.134077Z"
    }
   },
   "outputs": [],
   "source": [
    "bucket_name = \"my-s3-python-v2\"\n",
    "\n",
    "# List of files to upload\n",
    "file_list = [\n",
    "    'corpus/dog/d1.jpg',\n",
    "    'corpus/dog/d2.jpg',\n",
    "    'corpus/dog/d3.jpg'    \n",
    "]\n",
    "\n",
    "def upload_fileobj_to_s3(s3_client, bucket_name, file_path):\n",
    "    try:\n",
    "        # Open the file in binary read mode\n",
    "        with open(file_path, 'rb') as file_obj:\n",
    "            # Extract the file name from the path\n",
    "            file_name = \"file_object_\" + file_path.split('/')[-1]\n",
    "            \n",
    "            # Upload the file object\n",
    "            s3_client.upload_fileobj(file_obj, bucket_name, file_name)\n",
    "            print(f'Successfully uploaded {file_name} to bucket {bucket_name}.')\n",
    "    except FileNotFoundError:\n",
    "        print(f'File not found: {file_path}')\n",
    "    except NoCredentialsError:\n",
    "        print('Credentials not available.')\n",
    "    except PartialCredentialsError:\n",
    "        print('Incomplete credentials provided.')\n",
    "    except ClientError as e:\n",
    "        print(f'Client error: {e}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:44:56.325151Z",
     "iopub.status.busy": "2024-08-09T19:44:56.322731Z",
     "iopub.status.idle": "2024-08-09T19:44:57.543078Z",
     "shell.execute_reply": "2024-08-09T19:44:57.537163Z",
     "shell.execute_reply.started": "2024-08-09T19:44:56.322731Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully uploaded file_object_d1.jpg to bucket my-s3-python-v2.\n",
      "Successfully uploaded file_object_d2.jpg to bucket my-s3-python-v2.\n",
      "Successfully uploaded file_object_d3.jpg to bucket my-s3-python-v2.\n"
     ]
    }
   ],
   "source": [
    "# Upload each file in the list\n",
    "for file_path in file_list:\n",
    "    upload_fileobj_to_s3(s3_client, bucket_name, file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:45:29.869053Z",
     "iopub.status.busy": "2024-08-09T19:45:29.869053Z",
     "iopub.status.idle": "2024-08-09T19:45:30.343324Z",
     "shell.execute_reply": "2024-08-09T19:45:30.343324Z",
     "shell.execute_reply.started": "2024-08-09T19:45:29.869053Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "my-s3-python\n",
      "my-s3-python-v2\n"
     ]
    }
   ],
   "source": [
    "get_buckets(s3_client=s3_client)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get list all files (or objects) in given S3 bucket\n",
    "\n",
    "This method returns a list of objects in the bucket, which you can then iterate over to get the file names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:48:04.669652Z",
     "iopub.status.busy": "2024-08-09T19:48:04.669652Z",
     "iopub.status.idle": "2024-08-09T19:48:05.128384Z",
     "shell.execute_reply": "2024-08-09T19:48:05.127873Z",
     "shell.execute_reply.started": "2024-08-09T19:48:04.669652Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "d1.jpg\n",
      "d2.jpg\n",
      "d3.jpg\n",
      "file_object_d1.jpg\n",
      "file_object_d2.jpg\n",
      "file_object_d3.jpg\n"
     ]
    }
   ],
   "source": [
    "def list_files_in_bucket(s3_client, bucket_name):\n",
    "    try:\n",
    "        # List objects in the specified bucket\n",
    "        response = s3_client.list_objects_v2(Bucket=bucket_name)\n",
    "\n",
    "        # Check if the bucket contains objects\n",
    "        if 'Contents' in response:\n",
    "            # Extract file names\n",
    "            files = [obj['Key'] for obj in response['Contents']]\n",
    "            return files\n",
    "        else:\n",
    "            print(f'No objects found in bucket {bucket_name}.')\n",
    "            return []\n",
    "    except NoCredentialsError:\n",
    "        print('Credentials not available.')\n",
    "        return []\n",
    "    except PartialCredentialsError:\n",
    "        print('Incomplete credentials provided.')\n",
    "        return []\n",
    "    except ClientError as e:\n",
    "        print(f'Client error: {e}')\n",
    "        return []\n",
    "\n",
    "# Get the list of files\n",
    "files = list_files_in_bucket(s3_client, bucket_name)\n",
    "\n",
    "# Print the file names\n",
    "for file in files:\n",
    "    print(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pagination: If your bucket contains a large number of files, you may need to handle pagination. The list_objects_v2 method returns a maximum of 1000 objects by default. To handle more objects, use the ContinuationToken to paginate through results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:49:15.557098Z",
     "iopub.status.busy": "2024-08-09T19:49:15.553864Z",
     "iopub.status.idle": "2024-08-09T19:49:15.578258Z",
     "shell.execute_reply": "2024-08-09T19:49:15.577747Z",
     "shell.execute_reply.started": "2024-08-09T19:49:15.557098Z"
    }
   },
   "outputs": [],
   "source": [
    "def list_files_in_bucket_pagination(s3_client, bucket_name):\n",
    "    try:\n",
    "        paginator = s3_client.get_paginator('list_objects_v2')\n",
    "        file_list = []\n",
    "        for page in paginator.paginate(Bucket=bucket_name):\n",
    "            if 'Contents' in page:\n",
    "                file_list.extend([obj['Key'] for obj in page['Contents']])\n",
    "        return file_list\n",
    "    except NoCredentialsError:\n",
    "        print('Credentials not available.')\n",
    "        return []\n",
    "    except PartialCredentialsError:\n",
    "        print('Incomplete credentials provided.')\n",
    "        return []\n",
    "    except ClientError as e:\n",
    "        print(f'Client error: {e}')\n",
    "        return []\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T19:49:32.071689Z",
     "iopub.status.busy": "2024-08-09T19:49:32.071689Z",
     "iopub.status.idle": "2024-08-09T19:49:32.761961Z",
     "shell.execute_reply": "2024-08-09T19:49:32.759180Z",
     "shell.execute_reply.started": "2024-08-09T19:49:32.071689Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "d1.jpg\n",
      "d2.jpg\n",
      "d3.jpg\n",
      "file_object_d1.jpg\n",
      "file_object_d2.jpg\n",
      "file_object_d3.jpg\n"
     ]
    }
   ],
   "source": [
    "# Get the list of files\n",
    "files = list_files_in_bucket_pagination(s3_client, bucket_name)\n",
    "\n",
    "# Print the file names\n",
    "for file in files:\n",
    "    print(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra Args\n",
    "Both upload_file and upload_fileobj accept an optional ExtraArgs parameter that can be used for various purposes.\n",
    "\n",
    "Some Important ExtraArgs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:04:14.058892Z",
     "iopub.status.busy": "2024-08-09T20:04:14.057881Z",
     "iopub.status.idle": "2024-08-09T20:04:14.649112Z",
     "shell.execute_reply": "2024-08-09T20:04:14.647146Z",
     "shell.execute_reply.started": "2024-08-09T20:04:14.058892Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully uploaded corpus/dog/d1.jpg to bucket my-s3-python-v2 with public-read access.\n"
     ]
    }
   ],
   "source": [
    "# permission = public\n",
    "file_path = 'corpus/dog/d1.jpg'\n",
    "object_name = \"public_\" + file_path.split('/')[-1]\n",
    "\n",
    "#response = s3_client.upload_file(file_path, bucket_name, object_name, ExtraArgs={\"ACL\": \"public-read\"})\n",
    "# S3UploadFailedError: Failed to upload corpus/dog/d1.jpg to my-s3-python-v2/publicd1.jpg: \n",
    "# An error occurred (AccessControlListNotSupported) when calling the PutObject operation: The bucket does not allow ACLs\n",
    "\n",
    "try:\n",
    "    # Upload the file with public-read permission\n",
    "    s3_client.upload_file(\n",
    "        Filename=file_path,\n",
    "        Bucket=bucket_name,\n",
    "        Key=\"public_\" + file_path.split('/')[-1],\n",
    "        ExtraArgs={'ACL': 'public-read'}\n",
    "    )\n",
    "    print(f'Successfully uploaded {file_path} to bucket {bucket_name} with public-read access.')\n",
    "except FileNotFoundError:\n",
    "    print(f'File not found: {file_path}')\n",
    "except NoCredentialsError:\n",
    "    print('Credentials not available.')\n",
    "except PartialCredentialsError:\n",
    "    print('Incomplete credentials provided.')\n",
    "except ClientError as e:\n",
    "    print(f'Client error: {e}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading files\n",
    "\n",
    "The methods provided by the AWS SDK for Python to download files are similar to those provided to upload files.\n",
    "\n",
    "\n",
    "The download_file method accepts the names of the bucket and object to download and the filename to save the file to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:08:21.078515Z",
     "iopub.status.busy": "2024-08-09T20:08:21.078515Z",
     "iopub.status.idle": "2024-08-09T20:08:21.699293Z",
     "shell.execute_reply": "2024-08-09T20:08:21.698029Z",
     "shell.execute_reply.started": "2024-08-09T20:08:21.078515Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully downloaded d1.jpg from bucket my-s3-python-v2 to static/images/d1.jpg.\n"
     ]
    }
   ],
   "source": [
    "object_key = 'd1.jpg'  # The key (path) of the object in S3\n",
    "local_file_path = 'static/images/' + object_key   # Path to save the downloaded file locally\n",
    "\n",
    "try:\n",
    "    # Download the file from S3\n",
    "    s3_client.download_file(bucket_name, object_key, local_file_path)\n",
    "    print(f'Successfully downloaded {object_key} from bucket {bucket_name} to {local_file_path}.')\n",
    "except NoCredentialsError:\n",
    "    print('Credentials not available.')\n",
    "except PartialCredentialsError:\n",
    "    print('Incomplete credentials provided.')\n",
    "except ClientError as e:\n",
    "    print(f'Client error: {e}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:10:22.755540Z",
     "iopub.status.busy": "2024-08-09T20:10:22.755540Z",
     "iopub.status.idle": "2024-08-09T20:10:22.789360Z",
     "shell.execute_reply": "2024-08-09T20:10:22.787427Z",
     "shell.execute_reply.started": "2024-08-09T20:10:22.755540Z"
    }
   },
   "outputs": [],
   "source": [
    "def download_fileobj_from_s3(s3_client, bucket_name, object_key, file_path):\n",
    "    try:\n",
    "        # Open a file-like object to write the downloaded data\n",
    "        with open(file_path, 'wb') as file_obj:\n",
    "            # Download the file from S3 to the file-like object\n",
    "            s3_client.download_fileobj(bucket_name, object_key, file_obj)\n",
    "            print(f'Successfully downloaded {object_key} from bucket {bucket_name} to {file_path}.')\n",
    "    except NoCredentialsError:\n",
    "        print('Credentials not available.')\n",
    "    except PartialCredentialsError:\n",
    "        print('Incomplete credentials provided.')\n",
    "    except ClientError as e:\n",
    "        print(f'Client error: {e}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:12:48.151192Z",
     "iopub.status.busy": "2024-08-09T20:12:48.151192Z",
     "iopub.status.idle": "2024-08-09T20:12:48.832193Z",
     "shell.execute_reply": "2024-08-09T20:12:48.831681Z",
     "shell.execute_reply.started": "2024-08-09T20:12:48.151192Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully downloaded file_object_d2.jpg from bucket my-s3-python-v2 to static/images/file_object_d2.jpg.\n"
     ]
    }
   ],
   "source": [
    "object_key = \"file_object_d2.jpg\"\n",
    "local_file_path = 'static/images/' + object_key\n",
    "\n",
    "# Download the file object\n",
    "download_fileobj_from_s3(s3_client, bucket_name, object_key, local_file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## File transfer configuration\n",
    "\n",
    "\n",
    "When uploading, downloading, or copying a file or S3 object, the AWS SDK for Python automatically manages retries and multipart and non-multipart transfers.\n",
    "\n",
    "The management operations are performed by using reasonable default settings that are well-suited for most scenarios. To handle a special case, the default settings can be configured to meet requirements.\n",
    "\n",
    "## Multipart transfers\n",
    "\n",
    "Multipart transfers occur when the file size exceeds the value of the multipart_threshold attribute.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:22:04.015889Z",
     "iopub.status.busy": "2024-08-09T20:22:04.015889Z",
     "iopub.status.idle": "2024-08-09T20:22:04.047677Z",
     "shell.execute_reply": "2024-08-09T20:22:04.047677Z",
     "shell.execute_reply.started": "2024-08-09T20:22:04.015889Z"
    }
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "from boto3.s3.transfer import TransferConfig\n",
    "from botocore.exceptions import NoCredentialsError, PartialCredentialsError, ClientError\n",
    "\n",
    "\n",
    "def upload_large_file_to_s3(s3_client, file_path, bucket_name, \n",
    "                            multipart_threshold=1024 * 1024 * 5,  # 5 MB\n",
    "                            multipart_chunksize=1024 * 1024 * 5,  # 5 MB\n",
    "                            max_concurrency=4):\n",
    "    \"\"\"\n",
    "    Uploads a large file to an S3 bucket with multipart upload.\n",
    "\n",
    "    :param s3_client: S3 client.\n",
    "    :param file_path: Local path to the file to upload.\n",
    "    :param bucket_name: Name of the S3 bucket.    \n",
    "    :param multipart_threshold: The size threshold for multipart uploads in bytes.\n",
    "    :param multipart_chunksize: The size of each part in the multipart upload in bytes.\n",
    "    :param max_concurrency: Number of threads to use for uploading parts in parallel.\n",
    "    \"\"\"    \n",
    "    # Configure multipart upload threshold and chunk size\n",
    "    config = TransferConfig(\n",
    "        multipart_threshold=multipart_threshold,\n",
    "        multipart_chunksize=multipart_chunksize,\n",
    "        max_concurrency=max_concurrency,\n",
    "        use_threads=True\n",
    "    )\n",
    "\n",
    "    try:\n",
    "        # Upload the file using multipart uploads if needed\n",
    "        s3_client.upload_file(\n",
    "            Filename=file_path,\n",
    "            Bucket=bucket_name,\n",
    "            Key=\"big_\" + file_path.split('/')[-1],\n",
    "            Config=config  # Apply the custom TransferConfig\n",
    "        )\n",
    "        print(f'Successfully uploaded {file_path} to bucket {bucket_name}.')\n",
    "    except NoCredentialsError:\n",
    "        print('Credentials not available.')\n",
    "    except PartialCredentialsError:\n",
    "        print('Incomplete credentials provided.')\n",
    "    except ClientError as e:\n",
    "        print(f'Client error: {e}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:22:04.997042Z",
     "iopub.status.busy": "2024-08-09T20:22:04.997042Z",
     "iopub.status.idle": "2024-08-09T20:22:05.663377Z",
     "shell.execute_reply": "2024-08-09T20:22:05.662139Z",
     "shell.execute_reply.started": "2024-08-09T20:22:04.997042Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully uploaded corpus/dog/d3.jpg to bucket my-s3-python-v2.\n"
     ]
    }
   ],
   "source": [
    "file_path = 'corpus/dog/d3.jpg'\n",
    "# object_name = \"big_\" + file_path.split('/')[-1]\n",
    "upload_large_file_to_s3(\n",
    "    s3_client=s3_client,\n",
    "    file_path=file_path,\n",
    "    bucket_name=bucket_name        \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Presigned URLs\n",
    "\n",
    "A user who does not have AWS credentials or permission to access an S3 object can be granted temporary access by using a presigned URL.\n",
    "\n",
    "A presigned URL is generated by an AWS user who has access to the object. The generated URL is then given to the unauthorized user. The presigned URL can be entered in a browser or used by a program or HTML webpage. The credentials used by the presigned URL are those of the AWS user who generated the URL.\n",
    "\n",
    "A presigned URL remains valid for a limited period of time which is specified when the URL is generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:27:45.368913Z",
     "iopub.status.busy": "2024-08-09T20:27:45.368913Z",
     "iopub.status.idle": "2024-08-09T20:27:45.392201Z",
     "shell.execute_reply": "2024-08-09T20:27:45.391121Z",
     "shell.execute_reply.started": "2024-08-09T20:27:45.368913Z"
    }
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "from botocore.exceptions import NoCredentialsError, PartialCredentialsError, ClientError\n",
    "from urllib.parse import quote_plus\n",
    "\n",
    "def generate_presigned_url_get(s3_client, bucket_name, object_key, expiration=3600):\n",
    "    \"\"\"\n",
    "    Generate a presigned URL to download an S3 object.\n",
    "\n",
    "    :param s3_client\n",
    "    :param bucket_name: Name of the S3 bucket.\n",
    "    :param object_key: The key (path) of the object in S3.    \n",
    "    :param expiration: Time in seconds for the presigned URL to remain valid (default is 1 hour).\n",
    "    :return: Presigned URL as a string.\n",
    "    \"\"\"    \n",
    "    try:\n",
    "        # Generate the presigned URL for the GET operation\n",
    "        response = s3_client.generate_presigned_url('get_object',\n",
    "                                                    Params={'Bucket': bucket_name, 'Key': object_key},\n",
    "                                                    ExpiresIn=expiration)\n",
    "        return response\n",
    "    except NoCredentialsError:\n",
    "        print('Credentials not available.')\n",
    "    except PartialCredentialsError:\n",
    "        print('Incomplete credentials provided.')\n",
    "    except ClientError as e:\n",
    "        print(f'Client error: {e}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:28:49.346601Z",
     "iopub.status.busy": "2024-08-09T20:28:49.346601Z",
     "iopub.status.idle": "2024-08-09T20:28:49.373237Z",
     "shell.execute_reply": "2024-08-09T20:28:49.373237Z",
     "shell.execute_reply.started": "2024-08-09T20:28:49.346601Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Presigned URL: https://my-s3-python-v2.s3.amazonaws.com/d1.jpg?AWSAccessKeyId=AKIAQJ2NLI3ANYMUWVFU&Signature=K8X3MaetLYJoyNhGZmfb0bk8%2Fe0%3D&Expires=1723238929\n"
     ]
    }
   ],
   "source": [
    "object_key = 'd1.jpg'  # The key (path) of the object in S3\n",
    "presigned_url = generate_presigned_url_get(\n",
    "    s3_client=s3_client,\n",
    "    bucket_name=bucket_name,\n",
    "    object_key=object_key    \n",
    ")\n",
    "\n",
    "print(f'Presigned URL: {presigned_url}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bucket policies\n",
    "\n",
    "An S3 bucket can have an optional policy that grants access permissions to other AWS accounts or AWS Identity and Access Management (IAM) users. Bucket policies are defined using the same JSON format as a resource-based IAM policy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieve a Bucket Policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:31:28.922042Z",
     "iopub.status.busy": "2024-08-09T20:31:28.922042Z",
     "iopub.status.idle": "2024-08-09T20:31:28.947153Z",
     "shell.execute_reply": "2024-08-09T20:31:28.947153Z",
     "shell.execute_reply.started": "2024-08-09T20:31:28.922042Z"
    }
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "from botocore.exceptions import NoCredentialsError, PartialCredentialsError, ClientError\n",
    "\n",
    "def get_bucket_policy(s3_client, bucket_name):\n",
    "    \"\"\"\n",
    "    Retrieve the policy of an S3 bucket.\n",
    "\n",
    "    :param s3_client\n",
    "    :param bucket_name: Name of the S3 bucket.\n",
    "    \n",
    "    :return: Bucket policy as a JSON string or an error message.\n",
    "    \"\"\"    \n",
    "    try:\n",
    "        # Retrieve the bucket policy\n",
    "        response = s3_client.get_bucket_policy(Bucket=bucket_name)\n",
    "        policy = response.get('Policy')\n",
    "        return policy\n",
    "    except NoCredentialsError:\n",
    "        return 'Credentials not available.'\n",
    "    except PartialCredentialsError:\n",
    "        return 'Incomplete credentials provided.'\n",
    "    except ClientError as e:\n",
    "        return f'Client error: {e}'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:32:06.391674Z",
     "iopub.status.busy": "2024-08-09T20:32:06.391674Z",
     "iopub.status.idle": "2024-08-09T20:32:06.912752Z",
     "shell.execute_reply": "2024-08-09T20:32:06.912752Z",
     "shell.execute_reply.started": "2024-08-09T20:32:06.391674Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bucket Policy: {\"Version\":\"2012-10-17\",\"Id\":\"Policy1723233518257\",\"Statement\":[{\"Sid\":\"Stmt1723233514556\",\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"s3:GetObject\",\"Resource\":\"arn:aws:s3:::my-s3-python-v2/*\"}]}\n"
     ]
    }
   ],
   "source": [
    "bucket_policy = get_bucket_policy(\n",
    "    s3_client=s3_client,\n",
    "    bucket_name=bucket_name\n",
    ")\n",
    "\n",
    "print(f'Bucket Policy: {bucket_policy}')\n",
    "# Bucket Policy: Client error: An error occurred (AllAccessDisabled) when calling the GetBucketPolicy operation: All access to this object has been disabled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set a bucket policy\n",
    "\n",
    "A bucket's policy can be set by calling the put_bucket_policy method.\n",
    "\n",
    "The policy is defined in the same JSON format as an IAM policy. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Policy Format\n",
    "\n",
    "The **Sid (statement ID)** is an optional identifier that you provide for the policy statement. You can assign a Sid value to each statement in a statement array.\n",
    "\n",
    "The **Effect** element is required and specifies whether the statement results in an allow or an explicit deny. Valid values for Effect are Allow and Deny.\n",
    "\n",
    "By default, access to resources is denied. \n",
    "\n",
    "Use the **Principal** element in a policy to specify the principal that is allowed or denied access to a resource.\n",
    "\n",
    "You can specify any of the following principals in a policy:\n",
    "\n",
    "- AWS account and root user\n",
    "- IAM users\n",
    "- Federated users (using web identity or SAML federation)\n",
    "- IAM roles\n",
    "- Assumed-role sessions\n",
    "- AWS services\n",
    "- Anonymous users\n",
    "\n",
    "\n",
    "The **Action** element describes the specific action or actions that will be allowed or denied. \n",
    "\n",
    "We specify a value using a service namespace as an action prefix (iam, ec2, sqs, sns, s3, etc.) followed by the name of the action to allow or deny.\n",
    "\n",
    "The **Resource** element specifies the object or objects that the statement covers. We specify a resource using an ARN. Amazon Resource Names (ARNs) uniquely identify AWS resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define a policy that enables any user to retrieve any object stored in the bucket identified by the bucket_name variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:41:20.404367Z",
     "iopub.status.busy": "2024-08-09T20:41:20.404367Z",
     "iopub.status.idle": "2024-08-09T20:41:20.423522Z",
     "shell.execute_reply": "2024-08-09T20:41:20.423270Z",
     "shell.execute_reply.started": "2024-08-09T20:41:20.404367Z"
    }
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "from botocore.exceptions import NoCredentialsError, PartialCredentialsError, ClientError\n",
    "import json\n",
    "\n",
    "# Define the bucket policy\n",
    "custom_bucket_policy = {\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Id\": \"Policy1723233518257\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Sid\": \"Stmt1723233514556\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Principal\": \"*\",\n",
    "            \"Action\": \"s3:GetObject\",\n",
    "            \"Resource\": f\"arn:aws:s3:::{bucket_name}/*\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "def set_bucket_policy(s3_client, bucket_name, bucket_policy):\n",
    "    \"\"\"\n",
    "    Set a bucket policy to allow public read access to all objects in the bucket.\n",
    "\n",
    "    :param s3_client: S3 client.\n",
    "    :param bucket_name: Name of the S3 bucket.    \n",
    "    :param bucket_policy\n",
    "    \"\"\"    \n",
    "    # Convert the policy to a JSON string\n",
    "    policy_string = json.dumps(bucket_policy)\n",
    "\n",
    "    try:\n",
    "        # Set the bucket policy\n",
    "        s3_client.put_bucket_policy(\n",
    "            Bucket=bucket_name,\n",
    "            Policy=policy_string\n",
    "        )\n",
    "        print(f'Successfully set the policy for bucket {bucket_name}.')\n",
    "    except NoCredentialsError:\n",
    "        print('Credentials not available.')\n",
    "    except PartialCredentialsError:\n",
    "        print('Incomplete credentials provided.')\n",
    "    except ClientError as e:\n",
    "        print(f'Client error: {e}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:44:29.184308Z",
     "iopub.status.busy": "2024-08-09T20:44:29.182788Z",
     "iopub.status.idle": "2024-08-09T20:44:29.743674Z",
     "shell.execute_reply": "2024-08-09T20:44:29.738424Z",
     "shell.execute_reply.started": "2024-08-09T20:44:29.184308Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully set the policy for bucket my-s3-python-v2.\n"
     ]
    }
   ],
   "source": [
    "# Example usage\n",
    "set_bucket_policy(\n",
    "    s3_client=s3_client,\n",
    "    bucket_name=bucket_name,\n",
    "    bucket_policy=custom_bucket_policy    \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete a bucket policy\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:43:47.181046Z",
     "iopub.status.busy": "2024-08-09T20:43:47.181046Z",
     "iopub.status.idle": "2024-08-09T20:43:47.203062Z",
     "shell.execute_reply": "2024-08-09T20:43:47.203062Z",
     "shell.execute_reply.started": "2024-08-09T20:43:47.181046Z"
    }
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "from botocore.exceptions import NoCredentialsError, PartialCredentialsError, ClientError\n",
    "\n",
    "def delete_bucket_policy(s3_client, bucket_name):\n",
    "    \"\"\"\n",
    "    Delete the policy of an S3 bucket.\n",
    "\n",
    "    :param s3_client\n",
    "    :param bucket_name: Name of the S3 bucket.\n",
    "    \n",
    "    \"\"\"\n",
    "    try:\n",
    "        # Delete the bucket policy\n",
    "        s3_client.delete_bucket_policy(Bucket=bucket_name)\n",
    "        print(f'Successfully deleted the policy for bucket {bucket_name}.')\n",
    "    except NoCredentialsError:\n",
    "        print('Credentials not available.')\n",
    "    except PartialCredentialsError:\n",
    "        print('Incomplete credentials provided.')\n",
    "    except ClientError as e:\n",
    "        print(f'Client error: {e}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:44:09.665682Z",
     "iopub.status.busy": "2024-08-09T20:44:09.665682Z",
     "iopub.status.idle": "2024-08-09T20:44:10.204696Z",
     "shell.execute_reply": "2024-08-09T20:44:10.203499Z",
     "shell.execute_reply.started": "2024-08-09T20:44:09.665682Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully deleted the policy for bucket my-s3-python-v2.\n"
     ]
    }
   ],
   "source": [
    "# Example usage\n",
    "delete_bucket_policy(\n",
    "    s3_client=s3_client,\n",
    "    bucket_name=bucket_name    \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CORS Configuration\n",
    "\n",
    "Cross Origin Resource Sharing (CORS) enables client web applications in one domain to access resources in another domain. An S3 bucket can be configured to enable cross-origin requests. The configuration defines rules that specify the allowed origins, HTTP methods (GET, PUT, etc.), and other elements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieve a bucket CORS configuration\n",
    "\n",
    "Retrieve a bucket's CORS configuration by calling the AWS SDK for Python get_bucket_cors method.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:48:04.898539Z",
     "iopub.status.busy": "2024-08-09T20:48:04.898539Z",
     "iopub.status.idle": "2024-08-09T20:48:04.929374Z",
     "shell.execute_reply": "2024-08-09T20:48:04.929374Z",
     "shell.execute_reply.started": "2024-08-09T20:48:04.898539Z"
    }
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "from botocore.exceptions import NoCredentialsError, PartialCredentialsError, ClientError\n",
    "\n",
    "def get_bucket_cors(s3_client, bucket_name):\n",
    "    \"\"\"\n",
    "    Retrieve the CORS configuration of an S3 bucket.\n",
    "\n",
    "    :param s3_client\n",
    "    :param bucket_name: Name of the S3 bucket.    \n",
    "    :return: CORS configuration as a JSON string or an error message.\n",
    "    \"\"\"    \n",
    "    try:\n",
    "        # Retrieve the CORS configuration\n",
    "        response = s3_client.get_bucket_cors(Bucket=bucket_name)\n",
    "        cors_configuration = response.get('CORSRules')\n",
    "        return cors_configuration\n",
    "    except NoCredentialsError:\n",
    "        return 'Credentials not available.'\n",
    "    except PartialCredentialsError:\n",
    "        return 'Incomplete credentials provided.'\n",
    "    except ClientError as e:\n",
    "        return f'Client error: {e}'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:48:22.111987Z",
     "iopub.status.busy": "2024-08-09T20:48:22.111987Z",
     "iopub.status.idle": "2024-08-09T20:48:22.598351Z",
     "shell.execute_reply": "2024-08-09T20:48:22.596330Z",
     "shell.execute_reply.started": "2024-08-09T20:48:22.111987Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CORS Configuration: Client error: An error occurred (NoSuchCORSConfiguration) when calling the GetBucketCors operation: The CORS configuration does not exist\n"
     ]
    }
   ],
   "source": [
    "cors_config = get_bucket_cors(\n",
    "    s3_client=s3_client,\n",
    "    bucket_name=bucket_name\n",
    ")\n",
    "\n",
    "print(f'CORS Configuration: {cors_config}')\n",
    "# CORS Configuration: Client error: An error occurred (NoSuchCORSConfiguration) \n",
    "# when calling the GetBucketCors operation: The CORS configuration does not exist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set Bucket CORS\n",
    "\n",
    "To set the CORS (Cross-Origin Resource Sharing) configuration for an Amazon S3 bucket using Python and boto3, you will define a CORS configuration in JSON format and apply it to the bucket using the put_bucket_cors method of the boto3 S3 client.\n",
    "\n",
    "Here's a step-by-step guide to set the CORS configuration for an S3 bucket:\n",
    "\n",
    "Example: Setting a Bucket CORS Configuration\n",
    "\n",
    "### CORS Configuration Format\n",
    "\n",
    "The CORS configuration is defined in JSON format and typically includes rules that specify:\n",
    "+ Allowed origins (domains that can access the resources).\n",
    "+ Allowed methods (HTTP methods like GET, POST).\n",
    "+ Allowed headers (HTTP headers that can be used in requests).\n",
    "+ Expose headers (headers that the client can access)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:52:21.950838Z",
     "iopub.status.busy": "2024-08-09T20:52:21.950838Z",
     "iopub.status.idle": "2024-08-09T20:52:21.974253Z",
     "shell.execute_reply": "2024-08-09T20:52:21.973244Z",
     "shell.execute_reply.started": "2024-08-09T20:52:21.950838Z"
    }
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "from botocore.exceptions import NoCredentialsError, PartialCredentialsError, ClientError\n",
    "import json\n",
    "\n",
    "def set_bucket_cors(s3_client, bucket_name):\n",
    "    \"\"\"\n",
    "    Set the CORS configuration for an S3 bucket.\n",
    "\n",
    "    :param s3_client\n",
    "    :param bucket_name: Name of the S3 bucket.    \n",
    "    \"\"\"\n",
    "    # Define the CORS configuration\n",
    "    cors_configuration = {\n",
    "        \"CORSRules\": [\n",
    "            {\n",
    "                \"AllowedHeaders\": [\"*\"],\n",
    "                \"AllowedMethods\": [\"GET\", \"PUT\", \"POST\", \"DELETE\"],\n",
    "                \"AllowedOrigins\": [\"*\"],\n",
    "                \"ExposeHeaders\": [\"ETag\"],\n",
    "                \"MaxAgeSeconds\": 3000\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "\n",
    "    # Convert the CORS configuration to a JSON string\n",
    "    cors_configuration_json = json.dumps(cors_configuration)\n",
    "    \n",
    "    try:\n",
    "        # Set the CORS configuration\n",
    "        s3_client.put_bucket_cors(\n",
    "            Bucket=bucket_name,\n",
    "            CORSConfiguration=cors_configuration\n",
    "        )\n",
    "        print(f'Successfully set the CORS configuration for bucket {bucket_name}.')\n",
    "    except NoCredentialsError:\n",
    "        print('Credentials not available.')\n",
    "    except PartialCredentialsError:\n",
    "        print('Incomplete credentials provided.')\n",
    "    except ClientError as e:\n",
    "        print(f'Client error: {e}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:52:42.306379Z",
     "iopub.status.busy": "2024-08-09T20:52:42.306379Z",
     "iopub.status.idle": "2024-08-09T20:52:42.871257Z",
     "shell.execute_reply": "2024-08-09T20:52:42.871257Z",
     "shell.execute_reply.started": "2024-08-09T20:52:42.306379Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully set the CORS configuration for bucket my-s3-python-v2.\n"
     ]
    }
   ],
   "source": [
    "# Example usage\n",
    "set_bucket_cors(\n",
    "    s3_client=s3_client,\n",
    "    bucket_name=bucket_name\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-08-09T20:53:27.669989Z",
     "iopub.status.busy": "2024-08-09T20:53:27.669989Z",
     "iopub.status.idle": "2024-08-09T20:53:28.159895Z",
     "shell.execute_reply": "2024-08-09T20:53:28.159895Z",
     "shell.execute_reply.started": "2024-08-09T20:53:27.669989Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CORS Configuration: [{'AllowedHeaders': ['*'], 'AllowedMethods': ['GET', 'PUT', 'POST', 'DELETE'], 'AllowedOrigins': ['*'], 'ExposeHeaders': ['ETag'], 'MaxAgeSeconds': 3000}]\n"
     ]
    }
   ],
   "source": [
    "# Review again CORS\n",
    "cors_config = get_bucket_cors(\n",
    "    s3_client=s3_client,\n",
    "    bucket_name=bucket_name\n",
    ")\n",
    "\n",
    "print(f'CORS Configuration: {cors_config}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}