Fix standard-IA pattern matching to remove it from reco calculations (#…

…10)

* #9 S-IA fix

* Adding support for no savings in a region from a reco

* v1.1.0 release

reco/CHANGELOG.md

@@ -1,2 +1,5 @@
+### 2022-12-21 - Version 1.1.0 - switch180
+* BUG: Fix bug that includes S-IA capacity units in reservation recommendation. This created potentially incorrect recommendations in regions where customers have tables in the Standard Infrequent Access table class
+
 ### 2022-10-01 - Version 1.0.0 - switch180
 * Initial release on GitHub

reco/README.md

@@ -22,11 +22,9 @@ The tool uses your existing [AWS Cost and Usage Reports](https://docs.aws.amazon
 ## Documentation
 ```
 $ python3 src/ddbr.py reco -h
-usage: ddbr.py reco [-h] [--athena-sql] [--debug] [--disable-analytics]
-                    [--file-name FILE_NAME] [--term {1,3,all}]
-                    [--file-type {cur,dli,dli_rt}]
-                    [--output {plain,csv,dict,all}] [--start-time START_TIME]
-                    [--end-time END_TIME] [--package PACKAGE]
+usage: ddbr.py reco [-h] [--athena-sql] [--debug] [--file-name FILE_NAME] [--term {1,3,all}]
+    [--file-type {cur,dli,dli_rt}] [--output {plain,csv,dict,all}] [--start-time START_TIME]
+    [--end-time END_TIME] [--package PACKAGE] [--version]
 
 optional arguments:
   -h, --help            show this help message and exit
@@ -36,17 +34,14 @@ optional arguments:
                         File name or path to file where usage data resides.
   --term {1,3,all}      RC term length to consider.
   --file-type {cur,dli,dli_rt}
-                        CUR Athena query (cur) or DBR file (dli*). Detailed
-                        line items (dli). DLI with resources and tags (dli_rt)
+                        CUR Athena query (cur) or DBR file (dli*). Detailed line items (dli). DLI with resources and tags (dli_rt)
   --output {plain,csv,dict,all}
                         Format of text to be displayed
   --start-time START_TIME
-                        Start time with leading zeroes in format --start-time
-                        "%m/%d/%y %H:%M:%S"
-  --end-time END_TIME   End time with leading zeroes in format --end-time
-                        "%m/%d/%y %H:%M:%S"
-  --package PACKAGE     Should output be ZIP'd into a user-deliverable format.
-                        Provide the package ZIP suffix
+                        Start time with leading zeroes in format --start-time "%m/%d/%y %H:%M:%S"
+  --end-time END_TIME   End time with leading zeroes in format --end-time "%m/%d/%y %H:%M:%S"
+  --package PACKAGE     Should output be ZIP'd into a user-deliverable format. Provide the package ZIP suffix
+  --version             Print version and exit.
 ```
 
 
@@ -128,9 +123,9 @@ Loading CSV into memory. Please wait.
 Generating recommendations.
 ############################################################
 DynamoDB Reserved Capacity Report
-These recommendations are based on data from 05/01/19 00:00:00 to 05/31/19 23:00:00. RC term length for the report is 1 year(s)
-Generated on 03/25/20 19:14:34
-Please consult your account’s active reserved capacity reservations to determine the amount of capacity to own. The amounts below do not factor in what you already own. Instead, they reflect the amount you should have in your account. Please contact AWS for a final recommendation on the amount to buy if it’s not attached to this report.
+These recommendations are based on data from 05/01/19 00:00:00 to 05/31/19 23:00:00. The time span represents 30 days of data. RC term length for the report is 1 year(s)
+Please consult your account’s active reserved capacity reservations to determine the amount of capacity to own. The amounts below do not factor in what you already own. Instead, they reflect the amount you should have in your account. Please be aware reservations are not available for rWCUs, the table class S-IA, or for the on-demand capacity mode.
+Generated on 12/21/22 10:37:57 with reco version v1.1.0
 ############################################################
 ************************************************************
                Tokyo region - ap-northeast-1
@@ -150,13 +145,13 @@ Totals: Effective monthly rate: $9,810.57, monthly rate after first month: $5,40
                           Glossary
 ************************************************************
 Reserved capacity to own: How much RC this payer account should own. You must compare the amount to own against your current RC reservations to determine the amount to buy. See report header.
-Effective monthly rate: The cost per month, factoring in the upfront RC purchase cost. This is the 'monthly rate after first month' + (upfront cost / months in the RC term). If you already own RC, this amount will be incorrect.
-Monthly rate after first month: The price per month if you owned all the recommended RC, not including the upfront cost. This cost includes the usage above the recommended RC plus the RC hourly rate for the recommended number of units.
+Effective monthly rate: The cost per month, factoring in the upfront RC purchase cost. This is the 'monthly rate after first month' + (upfront cost / months in the RC term). If you already own RC, this amount will be incorrect. This amount is only valid if the time range is one month, otherwise this field is actually the effective cost over the time period selected.
+Monthly rate after first month: The price per month if you owned all the recommended RC, not including the upfront cost. This cost includes the usage above the recommended RC plus the RC hourly rate for the recommended number of units. This amount is only valid if the time range is one month, otherwise this field is actually the sum of the cost over the time period selected.
 '...savings over the public rate card': The impact of owning this much RC! This is the money you'll save if you own the suggested amount. These prices are calculated using the public pricing and do not include any credits or negotiated discounts.
 'XXX units of 100': DynamoDB reserved capacity is bought in batches of 100 units.
 ############################################################
 End of report.
-Totals: Effective monthly rate: $9,810.57, Total savings over the public rate card: $84,736.56 (41.85%), Total upfront purchase cost: $51,825.80
+Totals: Effective monthly rate: $9,810.57, Total savings over the public rate card for duration of term: $84,736.56 (41.85%), Total upfront purchase cost: $51,825.80
 ############################################################
 ```
 

reco/src/ddb_rc_reco/config.py

@@ -1,8 +1,9 @@
 test_file_loc = 'test/b8f1d493-9aa2-4e5b-a5e8-121b7cfa131e_shrunk.csv.gz'
 
-wcu_regex = r'.*(-|^)(?!<Repl)WriteCapacityUnit.*'
-rcu_regex = r'.*ReadCapacityUnit.*'
+wcu_regex = r'(?!.*IA-)(.*(^|-)WriteCapacityUnit.*)'
+rcu_regex = r'(?!.*IA-)(.*(^|-)ReadCapacityUnit.*)'
 
+version = '1.1.0'
 
 #1yr RC rates listed only
 pricing = {"NRT":{"ut_region_match_filter":r'^APN1-', "dto":0.14, "wcu":0.000742,"rcu":0.0001484,"rwcu":0.001113,"rc1":{"rcu":{"base":34.2,"hour":0.0029},"wcu":{"base":171.4,"hour":0.0147}}, "rc3":{"rcu":{"base":41.0,"hour":0.0018},"wcu":{"base":205.6,"hour":0.0093}}},
@@ -36,7 +37,7 @@
     "\n'...savings over the public rate card': The impact of owning this much RC! This is the money you'll save if you own the suggested amount. These prices are calculated using the public pricing and do not include any credits or negotiated discounts." +
     "\n'XXX units of 100': DynamoDB reserved capacity is bought in batches of 100 units.")
 
-plain_report_warning_top = "Please consult your account’s active reserved capacity reservations to determine the amount of capacity to own. The amounts below do not factor in what you already own. Instead, they reflect the amount you should have in your account. Please contact AWS for a final recommendation on the amount to buy if it’s not attached to this report."
+plain_report_warning_top = "Please consult your account’s active reserved capacity reservations to determine the amount of capacity to own. The amounts below do not factor in what you already own. Instead, they reflect the amount you should have in your account. Please be aware reservations are not available for rWCUs, the table class S-IA, or for the on-demand capacity mode."
 
 descriptions = {
 "NRT": {"en_us": {"region": {"code": "ap-northeast-1", "short_name": "Tokyo"}}},

reco/src/ddb_rc_reco/reco.py

@@ -12,7 +12,7 @@
 from datetime import datetime, timedelta
 from multiprocessing import Pool, JoinableQueue, Process, Queue
 from ddb_rc_reco.config import test_file_loc, pricing, descriptions, ideal_sql
-from ddb_rc_reco.config import plain_report_footer, rcu_regex, wcu_regex
+from ddb_rc_reco.config import plain_report_footer, rcu_regex, wcu_regex, version
 
 
 
@@ -135,6 +135,8 @@ def generate_reco_tables(region_hours, wish):
             reco_table[region]['_totals']['rc_upfront'] += reco_table[region][usage_type]['recommendation']['rc_upfront']
         if reco_table[region]['_totals']['od_only_rate'] > 0:
             reco_table[region]['_totals']['percent_savings_over_od']  = round(((reco_table[region]['_totals']['od_only_rate'] - reco_table[region]['_totals']['mixed_rate_total'])/ reco_table[region]['_totals']['od_only_rate']) * 100.0, 2)
+        else:
+            reco_table[region]['_totals']['percent_savings_over_od'] = 0.0
         totals['od_only_rate'] += reco_table[region]['_totals']['od_only_rate']
         totals['mixed_rate_total'] += reco_table[region]['_totals']['mixed_rate_total']
         totals['rc_upfront'] += reco_table[region]['_totals']['rc_upfront']
@@ -231,7 +233,7 @@ def print_section_header(some_str, seperator='#'):
     end_time = table_input['_meta']['end_time'][0]
     rc_term = table_input['_meta']['rc_term']
     days_in_report = (parse_dt(end_time)-parse_dt(start_time)).days
-    print_section_header("DynamoDB Reserved Capacity Report\nThese recommendations are based on data from {} to {}. The time span represents {} days of data. RC term length for the report is {} year(s)\n{}\nGenerated on {}".format(start_time, end_time, days_in_report, rc_term, descriptions['banner'][lang]['warning_top']['text'],  print_dt(datetime.now()) ))
+    print_section_header("DynamoDB Reserved Capacity Report\nThese recommendations are based on data from {} to {}. The time span represents {} days of data. RC term length for the report is {} year(s)\n{}\nGenerated on {} with reco version v{}".format(start_time, end_time, days_in_report, rc_term, descriptions['banner'][lang]['warning_top']['text'],  print_dt(datetime.now()), version ))
     if days_in_report < 28 or days_in_report > 31:
         dynamic_warning = "WARNING: Non-standard number of days detected in the report. The fields marked monthly in this report represent the total cost for the selected time period, not the true monthly cost.\nRe-run the report with data for only one month to receive accurate monthly cost estimates."
         print_section_header(dynamic_warning, "*")

reco/src/ddbr.py

@@ -11,7 +11,7 @@
 from ddb_rc_reco.reco import refresh_csv_index, process_csv, get_range_time, dt_format
 from ddb_rc_reco.reco import parse_dt, print_dt, open_file_read, region_list
 from ddb_rc_reco.reco import generate_reco_tables, output_table, output_csv
-from ddb_rc_reco.config import ideal_sql
+from ddb_rc_reco.config import ideal_sql, version
 '''
 
 DDBR CLI for DynamoDB RC recommendations
@@ -44,12 +44,15 @@ def main():
     reco_parser.add_argument('--start-time', help="Start time with leading zeroes in format --start-time \"{}\"".format(dt_format.replace('%', '%%')), type=parse_dt)
     reco_parser.add_argument('--end-time', help="End time with leading zeroes in format --end-time \"{}\"".format(dt_format.replace('%', '%%')), type=parse_dt)
     reco_parser.add_argument('--package', help="Should output be ZIP'd into a user-deliverable format. Provide the package ZIP suffix", type=str)
+    reco_parser.add_argument('--version', action='store_true', help='Print version and exit.')
     # TODO reco_parser.add_argument('--region', type=str, choices=pricing.keys(), help='Airport code for region to process')
 
     args = main_parser.parse_args()
     if args.debug is True:
         logging.getLogger().setLevel(10)
-    if args.athena_sql is True:
+    if args.version is True:
+        print("reco v{}".format(version))
+    elif args.athena_sql is True:
         print(ideal_sql)
     elif args.file_name:
         csv_loc = args.file_name

reco/test/test_unit.py

@@ -10,7 +10,7 @@
 from ddb_rc_reco.reco import simulate_purchase, output_table, output_csv, open_file_read, refresh_csv_index
 
 class test_reco(TestCase):
-    test_csv_apn1 = 'test/APN1.csv.gz'
+    test_csv_apn1 = 'test/APN1-IA.csv.gz'
     test_csv_apn1_reads_only = 'test/APN1_reads.csv.gz'
     test_csv_apn1_daily_granularity = 'test/APN1_daily_granularity.csv.gz'
     test_csv_usw2_rwcu_wcu = 'test/USW2_rWCU_WCU.gz'
@@ -209,11 +209,11 @@ def eval_regex(regex, passers, failers):
         always_fail = ['APN1-HeavyUsage:dynamodb.write', 'APN1-HeavyUsage:dynamodb.read', 'USW2-DataTransfer-Out-Bytes']
         regex = wcu_regex
         passers = ['USW2-WriteCapacityUnit-Hrs', 'WriteCapacityUnit-Hrs']
-        failers = ['USW2-ReplWriteCapacityUnit-Hrs', 'ReplWriteCapacityUnit-Hrs']
+        failers = ['USW2-ReplWriteCapacityUnit-Hrs', 'ReplWriteCapacityUnit-Hrs', 'EUC1-IA-WriteCapacityUnit-Hrs', 'EUC1-IA-ReadCapacityUnit-Hrs']
         eval_regex(regex, passers, failers + always_fail)
         regex = rcu_regex
         passers = ['USW2-ReadCapacityUnit-Hrs', 'ReadCapacityUnit-Hrs']
-        failers = ['USW2-ReplWriteCapacityUnit-Hrs', 'USW2-WriteCapacityUnit-Hrs']
+        failers = ['USW2-ReplWriteCapacityUnit-Hrs', 'USW2-WriteCapacityUnit-Hrs', 'EUC1-IA-ReadCapacityUnit-Hrs', 'EUC1-IA-WriteCapacityUnit-Hrs']
         eval_regex(regex, passers, failers + always_fail)
     def test_process_csv_gt(self):
         expected_start = parse_dt('11/01/19 00:00:00')