#2: Implemented lifecycle of an EC2 instance #6

tomuben · 2022-06-16T12:48:01Z

fixes #2

2. moved method STACK_NAME into class CloudFormationStack

…key_file_manager.py

exasol_script_languages_developer_sandbox/lib/random_string_generator.py

exasol_script_languages_developer_sandbox/lib/run_setup_ec2.py

exasol_script_languages_developer_sandbox/templates/ec2_cloudformation.jinja.yaml

exasol_script_languages_developer_sandbox/lib/deployer.py

2. Changed implementation of random_string_generator.py 3. renamed function execute_setup_ec2() => run_lifecycle_for_ec2() 4. minor changes

exasol_script_languages_developer_sandbox/lib/cf_stack.py

exasol_script_languages_developer_sandbox/lib/key_file_manager.py

tkilias

Some questions and thoughts. Should we use aws localstack for testing the aws_access class. And can we refactor things, such that we have the context manager for keys and cloud formation, but on the other hand, use creation and closing idependently as CLI. Means the create CLI stores everything necessary on disk and close loads this and then closes everything. Reason is, I need the exact functionalities for handling ec2 instances also for GPU tests and development.
Another question, could we add the Key creation into the cloud formation template.

test/test_key_file_manager.py

tomuben · 2022-07-04T10:23:20Z

Another question, could we add the Key creation into the cloud formation template.

It is possible, see here, but I there are a few reasons to not do this:

For some cases (maybe debugging/tests) we might want to use an existing KeyPair. In order to implement that we would have to assemble the cloudformation yaml file dynamically, which makes the code more difficult.
We still need to get access to the private key of the Keypair. I assume the best way would be to write the private key to a temporary file (as we do now). Then the management of the lifecycle of this file would become independent of the lifecycle of the KeyPair (which is managed within the lifecycle of the cloudformation stack)
The code to get access to the private key is not much easier: First one has to find the Key-Id, and after that use AWS SSM to get the value of the private key.

tomuben · 2022-07-04T14:39:27Z

And can we refactor things, such that we have the context manager for keys and cloud formation, but on the other hand, use creation and closing idependently as CLI. Means the create CLI stores everything necessary on disk and close loads this and then closes everything. Reason is, I need the exact functionalities for handling ec2 instances also for GPU tests and development.

Of, course, I am happy if the code will be reused. But the question is if you want to use it from within Python code, or via CLI. Let's discuss offline.

tomuben · 2022-07-05T17:27:04Z

Should we use aws localstack for testing the aws_access class.

I added new tests which use localstack. However I realized that localstack simply mocks many things. It's not possible to test error scenarios (e.g. verify that a client error is thrown when a cloudformation stack is being accessed after it has been deleted).

…er()/CloudformationStack()

test/test_serialization.py

test/test_run_lifecycle_for_ec2.py

test/test_key_file_manager.py

exasol_script_languages_developer_sandbox/lib/deployer.py

2. For EC2 Key use full uuid string 3. For Cloudformation stack check if stack name exists

exasol_script_languages_developer_sandbox/lib/cf_stack.py

exasol_script_languages_developer_sandbox/lib/random_string_generator.py

exasol_script_languages_developer_sandbox/lib/cf_stack.py

exasol_script_languages_developer_sandbox/lib/random_string_generator.py

test/test_serialization.py

tomuben added 2 commits June 16, 2022 09:47

#2: Implemented launch of an EC2 instance

3d416ca

Added git to dev-dependencies

7549c33

tomuben had a problem deploying to AWS June 16, 2022 12:56 Failure

Fixed integration test

bab8e0e

tomuben temporarily deployed to AWS June 16, 2022 13:01 Inactive

1. Moved key-file parameters outside of aws_options.py

52e52d3

2. moved method STACK_NAME into class CloudFormationStack

tomuben temporarily deployed to AWS June 16, 2022 14:34 Inactive

Refactored run_setup_ec2.py and added tests for run_setup_ec2.py and …

aa686e1

…key_file_manager.py

tomuben temporarily deployed to AWS June 17, 2022 10:23 Inactive

tomuben changed the title ~~#2: Implemented launch of an EC2 instance~~ #2: Implemented lifecycle of an EC2 instance Jun 17, 2022

tkilias suggested changes Jun 17, 2022

View reviewed changes

1. Added tags to AWS resources

0432187

2. Changed implementation of random_string_generator.py 3. renamed function execute_setup_ec2() => run_lifecycle_for_ec2() 4. minor changes

tomuben temporarily deployed to AWS June 20, 2022 05:59 Inactive

tkilias suggested changes Jun 20, 2022

View reviewed changes

exasol_script_languages_developer_sandbox/lib/cf_stack.py Show resolved Hide resolved

exasol_script_languages_developer_sandbox/lib/key_file_manager.py Show resolved Hide resolved

tomuben added 2 commits June 30, 2022 09:26

converted CloudformationStack into a ContextManager

c940986

converted KeyFileManager into a ContextManager

ed3c83f

tomuben had a problem deploying to AWS June 30, 2022 12:53 Failure

tomuben temporarily deployed to AWS June 30, 2022 12:56 Inactive

tkilias suggested changes Jul 3, 2022

View reviewed changes

test/test_key_file_manager.py Outdated Show resolved Hide resolved

Fixed comment from review

c1dab85

tomuben temporarily deployed to AWS July 4, 2022 14:43 Inactive

Add a localstack test

0457d59

tomuben temporarily deployed to AWS July 5, 2022 15:01 Inactive

Added more tests to localstack test

b75552e

tomuben temporarily deployed to AWS July 5, 2022 17:17 Inactive

Removed ContextManager behavior from class KeyFileManager

e055462

tomuben had a problem deploying to AWS July 8, 2022 13:49 Failure

Use separate class KeyFileManagerContextManager

7d2efbd

tomuben temporarily deployed to AWS July 8, 2022 14:03 Inactive

Added tests for serialization/deserialization of classes KeyFileManag…

36cf245

…er()/CloudformationStack()

tomuben had a problem deploying to AWS July 8, 2022 16:18 Failure

fixed tests

4e7cf87

tomuben temporarily deployed to AWS July 8, 2022 16:35 Inactive

tkilias suggested changes Jul 11, 2022

View reviewed changes

test/test_serialization.py Outdated Show resolved Hide resolved

tkilias suggested changes Jul 11, 2022

View reviewed changes

test/test_run_lifecycle_for_ec2.py Show resolved Hide resolved

test/test_key_file_manager.py Show resolved Hide resolved

exasol_script_languages_developer_sandbox/lib/deployer.py Show resolved Hide resolved

1. Use base64 for getting random string based on UUID

4722b90

2. For EC2 Key use full uuid string 3. For Cloudformation stack check if stack name exists

tomuben had a problem deploying to AWS July 11, 2022 18:09 Failure

Fixed test_run_lifecycle_for_ec2.py

308941b

tomuben temporarily deployed to AWS July 11, 2022 18:27 Inactive

tkilias suggested changes Jul 12, 2022

View reviewed changes

exasol_script_languages_developer_sandbox/lib/cf_stack.py Outdated Show resolved Hide resolved

exasol_script_languages_developer_sandbox/lib/random_string_generator.py Outdated Show resolved Hide resolved

tomuben added 2 commits July 12, 2022 12:15

Fixed tests

dbd2db0

Use same version for localstack docker image as Python package

ceb82a5

tomuben temporarily deployed to AWS July 12, 2022 16:48 Inactive

Added documentation

87331f8

tomuben temporarily deployed to AWS July 12, 2022 17:02 Inactive

Fixed test_serialization.py

262ee35

tomuben temporarily deployed to AWS July 13, 2022 13:12 Inactive

tkilias suggested changes Jul 13, 2022

View reviewed changes

Minor fixes from review

1057d90

tomuben temporarily deployed to AWS July 13, 2022 13:46 Inactive

tkilias suggested changes Jul 13, 2022

View reviewed changes

test/test_serialization.py Outdated Show resolved Hide resolved

test/test_serialization.py Outdated Show resolved Hide resolved

Minor fixes from review

ad44555

tomuben temporarily deployed to AWS July 13, 2022 14:14 Inactive

tkilias approved these changes Jul 13, 2022

View reviewed changes

tomuben merged commit beea108 into main Jul 13, 2022

tomuben deleted the feature/2_implement_launch_of_an_ec2_instance branch July 13, 2022 14:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#2: Implemented lifecycle of an EC2 instance #6

#2: Implemented lifecycle of an EC2 instance #6

tomuben commented Jun 16, 2022 •

edited

Loading

tkilias left a comment •

edited

Loading

tomuben commented Jul 4, 2022 •

edited

Loading

tomuben commented Jul 4, 2022

tomuben commented Jul 5, 2022

#2: Implemented lifecycle of an EC2 instance #6

#2: Implemented lifecycle of an EC2 instance #6

Conversation

tomuben commented Jun 16, 2022 • edited Loading

tkilias left a comment • edited Loading

Choose a reason for hiding this comment

tomuben commented Jul 4, 2022 • edited Loading

tomuben commented Jul 4, 2022

tomuben commented Jul 5, 2022

tomuben commented Jun 16, 2022 •

edited

Loading

tkilias left a comment •

edited

Loading

tomuben commented Jul 4, 2022 •

edited

Loading