r.2024.07.31.0

Summary:

A new feature enhancing storage costs for OpenShift running on Azure as well as a few minor backend improvements.

What's New?

Koku/Cost Management is now able to better report and distribute OpenShift storage costs, especially when running OpenShift on Azure. Enhancements for AWS, GCP and on-premise will follow.

Customers utilizing the Azure Disk CSI Driver storage class to dynamically provision persistent volumes, in conjunction with the latest release of the cost-management-metrics-operator, will now see improved correlation of Azure disk costs to OpenShift projects within the Azure filtered by OpenShift view. The latest operator enables us to collect the unique identifier for disks created by the CSI driver during the creation of persistent volumes.

For customers not using the latest operator, the previous method of matching persistent volume names to Azure disk resource identifiers will still be in use. However, this method can result in uncorrelated costs. We highly recommend adopting the new approach for optimal cost visibility.

Additionally, we've made significant improvements in assigning Azure disk costs to OpenShift projects. Through our extended efforts, we now map the capacity of Azure disks to the product name. This allows us to assign a portion of the disk cost based on the requested persistent volume capacity. The cost allocation is calculated using the following equation:

(Persistent volume capacity / Disk Capacity) * Cost of the Disk
The remaining unallocated portion of the disk cost has now been added under a new project called Storage unattributed. This project will now also include the cost of persistent volumes that have no persistent volume claims.

What's Changed

• [COST-5319] Export EC2 compute response data to CSV by @djnakabaale in #5235
• Update ocp-smoke-tests keywords by @esebesto in #5240

Full Changelog: r.2024.07.29.0...r.2024.07.31.0

r.2024.07.29.0

Summary:

• Minor bug fixes and improvements

What's Changed

• COST-5288 Fix currency for network distribution by @myersCody in #5223
• fix log message for failed parquet conversion by @lcouzens in #5234
• COST-5141 vaccum expired trino partitions by @myersCody in #5171
• COST-5317 Support order by account_alias by @djnakabaale in #5233
• COST-5318 Support exclude instance_name and operating_system params by @djnakabaale in #5232
• COST-4741 - Separate AWS node network costs into a Network unattributed namespace by @samdoran in #5104
• COST-5315 Return network_unattributed_distributed costs in the OCP costs API by @samdoran in #5231

Full Changelog: r.2024.07.23.0...r.2024.07.29.0

r.2024.07.23.0

Summary:

• AWS: New API endpoint for EC2 instances, gated by unleash
• Bug-fix: Fix inadvertently skipping polling after finding no new reports on previous day

What's Changed

• COST-5299 clear cache when table is dropped by migration by @maskarb in #5221
• Update nise and AWS DataTransferGenerator resource IDs by @samdoran in #5222
• COST-5289 EC2 compute API - update openapi doc by @djnakabaale in #5224
• COST-4945 Create a new API to return list of EC2-compute instances by @bacciotti in #5117
• COST-5180 Fix inadvertently skipping polling after finding no new reports on previous day by @lcouzens in #5205
• Consolidate local volume mounts by @maskarb in #5215
• COST-5198 Skip customer save on get request by @chambridge in #5225
• COST-5280 rename cpu/memory request/limits parameters by @maskarb in #5226
• add additional info to INSERT operation by @maskarb in #5227
• COST-4993 gate ec2 endpoint with unleash by @maskarb in #5229

Full Changelog: r.2024.07.18.0...r.2024.07.23.0

r.2024.07.18.0

What's Changed

• [Tech-Debt] Remove dead code from Trino upgrade. by @myersCody in #5153
• COST-5252 cache schema/table exists by @maskarb in #5213
• COST-5093 Update mapped tags in EC2 summary table by @djnakabaale in #5147
• COST-4912 Calculate node based unattributed storage (Azure) by @myersCody in #5066
• COST-5198 Only store user object on write requests by @chambridge in #5214
• COST-5125 Trino migration command improvements by @samdoran in #5191
• COST-4741 Add UsageType and Operation to daily data by @samdoran in #5216
• COST-5114 Add type ahead support for the new EC2 fields by @djnakabaale in #5159
• COST-5287 Prevent azure cost deduplication on disk-id resources by @myersCody in #5219
• COST-4283 OCP: set start-date based on data in file being processed by @maskarb in #5212

Full Changelog: r.2024.07.12.0...r.2024.07.18.0

r.2024.07.12.0

Summary:

Handle exception when querying Azure for VM ID, add csi_volume_handle back in SQL, enable using readreplica db, dependency updates.

What's Changed

• COST-5198 - split read traffic to read replica db using nginx proxy by @chambridge in #5188
• remove unused method by @maskarb in #5208
• Bump certifi from 2024.6.2 to 2024.7.4 in the pip group across 1 directory by @dependabot in #5207
• chore(image): update base image by @github-actions in #5203
• Handle case when resource ID cannot be obtained by @chambridge in #5209
• COST-5148 update EC2 compute insert sql by @djnakabaale in #5206
• Unpause the csi volume handle sql by @myersCody in #5175
• [COST-5244, COST-5245, COST-5246, COST-5247] dependency update by @maskarb in #5211
• COST-4981 reset local unleash by @maskarb in #5148

Full Changelog: r.2024.07.08.0...r.2024.07.12.0

r.2024.07.08.0

Summary:

Upgrade to Python 3.11, adjustment of logs, and logging of S3 batch deletes. Corrections in VM usage reporting, and processing of OCPGCP network data.

What's Changed

• COST-5076 upgrade to python 3.11 by @maskarb in #4444
• COST-5228 log outside for loop by @djnakabaale in #5202
• log s3 batch deletes by @lcouzens in #5204
• COST-5219 Correctly report VM usage for metering when billing record is split by @cgoodfred in #5201
• COST-4745 OCPGCP Network data processing SQL by @cgoodfred in #5058

Full Changelog: r.2024.07.04.0...r.2024.07.08.0

r.2024.07.04.0

Summary:

Fix and improve S3 operations, Docker build improvements, and filtering optimization changes.

What's Changed

• COST-5213 - fix S3 prepare by @lcouzens in #5194
• COST-5214 Remove directive to make the build work with docker by @samdoran in #5196
• COST-5216 Delete filtering optimization by @myersCody in #5197
• Revert "[COST-5216] Delete filtering optimization" by @myersCody in #5200
• COST-5226 - Skip S3 delete (daily flow) if we have marked deletion complete. by @lcouzens in #5198
• COST-5213 - Improve collection for delete from S3. by @lcouzens in #5194

Full Changelog: r.2024.06.28.0...r.2024.07.04.0

r.2024.06.28.0

Summary:

Prevent downloads queuing for a provider when a current download is already in progress

What's Changed

• COST-4844 Tweak serializer for storageclass order by @myersCody in #5174
• COST-5078 Switch to RHEL 8 VMs for CI jobs by @samdoran in #5193
• COST-5180 Only attempt to fetch new reports if current processing is complete
  by @lcouzens in #5181
• COST-5214 Update containerfile to build with podman by @samdoran in #5195

Full Changelog: r.2024.06.27.0...r.2024.06.28.0

r.2024.06.27.0

Summary:

Made a few changes to data processing to ensure tasks are sent to the correct workers and that task caching works as expected.

What's Changed

• Improve processing times by using queues more efficiently by @cgoodfred in #5186
• COST-5196 Improve processing times by using queues more efficiently by @lcouzens in #5187
• COST-5176 Correctly pass context dictionary within log_json function call by @djnakabaale in #5182
• COST-5178 Improve processing S3 deletions by @lcouzens in #5180
• Bump urllib3 from 1.26.18 to 1.26.19 in the pip group across 1 directory by @dependabot in #5172
• Add flower as a development dependency by @samdoran in #5189

Full Changelog: r.2024.06.24.0...r.2024.06.27.0

r.2024.06.24.0

Summary:

Optimizations in SQL and cost model task, and other operational improvements.

What's Changed

• [COST-4916] add v3.3.0 operator commits by @maskarb in #5143
• [COST-5124] Improve Trino migration management command by @samdoran in #5163
• Filter accounts by matching criteria during subs processing by @cgoodfred in #5184
• Update tasks.py by @lcouzens in #5185
• clean up grafana dashboard by @lcouzens in #5183

Full Changelog: r.2024.06.21.0...r.2024.06.24.0