v4.4.0

4.4.0 (2024-02-22)

Features

• Add --url to Oracle connections add options (#1083) (2f078c2)
• Add PostgreSQL OID support (#1076) (58f8fcb)
• Add support to generate a JSON config file only for applications purposes (#1089) (d463038)
• set default oracle sql alchemy arraysize to 500 (#1088) (1672ac5)
• Support for Kubernetes (#1058) (fdbdbe0)

Bug Fixes

• Add support for cx_Oracle's DB_TYPE_LONG_RAW (#1095) (90547ef)
• Better casts to string for binary floats/doubles (#1078) (15bfc4c)
• case-insensitive comparison field support (#1103) (d28786f)
• Fix merge issue for Teradata empty dataframes (#1100) (cc91fa2)
• increase upper limit on recursion columns (#1090) (c599ebf)
• Remove DDL automatically issued by Ibis for Postgres connections (#1067) (c2b660b)
• Row validation primary key columns >64bit int/float are cast to string (#1080) (9e70e9e)
• Spanner generate-partition to use BQ dialect (#1066) (f3cc565)
• spanner hash function to return string instead of bytes (#1062) (722dff9)

Documentation

• Add Airflow Kubernetes pod operator samples (#1087) (7d5ea91)
• Updates on nested column limitations, contributing guide examples and incorrect example (#1082) (cc0f60a)

v4.3.0

4.3.0 (2023-11-28)

Features

• Adding Exclude columns flag for aggregations in column validations (#961) (faa32dc)
• support query parameter for MSSQL connection (#1026) (48b0355)

Bug Fixes

• --dry-run for SQLAlchemy clients with valid raw SQL (#1047) (c1e0e34)
• Add Spanner RawSQL operation to enable filtering (#1054) (3a01503)
• Adding credentials as parameter for Spanner (#1031) (367658e)
• Adjust find-tables to properly get Oracle and Postgres schemas (#1034) (45fb40a)
• Cast should treat nullable and non-nullables as the same (#1037) (5e5c5eb)
• Fix --grouped-columns issue for Oracle validation (#1050) (3473a27)
• Fix decimal separator to "." (dot) on Oracle (#1042) (14cc7ef)
• Teradata SSLMODE issue fix (#1014) (e7aab6b)

Documentation

• Add CLOB to Oracle BLOB validation document (#1029) (8c76c1b)
• Update connections.md to add supported version of DB2 (#1030) (44b4be7)

v4.2.0

4.2.0 (2023-09-28)

Features

• Add more mappings to the allowlist configuration files for Oracle schema validations (#953) (0fed588)
• Include date columns for min/max/sum validations (#984) (6de9921)
• Include date columns in scope of wildcard_include_timestamp option (#989) (a4cf773)
• Support BQ decimal precision and scale for schema validation (#960) (b1d4942)
• Support standard deviation for column agg (#964) (bb81701)

Bug Fixes

• Add exception handling for invalid value to cast a comparison field (#957) (703ca75)
• Add missing SnowflakeDialect mapping for BINARY data type (#959) (9ad529a)
• Add not-null string to accepted date types in append_pre_agg_calc_field() (#980) (76fcfc6)
• Adjust set up for randow row batch size default value, but it maintains as 10,000 (#986) (a20ccab)
• custom query row validation failing when SQL contains upper cased columns (#994) (a9fed41)
• Fix warning and precision detection when target precision higher than source (#965) (5f00ce1)
• generate-table-partitions- fixes Issue 945 and Issue 950 (#962) (c53f2fc)
• Prevent failure of column validation config generation if source column other than allow-list not present in target table. (#974) (40a073e)
• Prevent Oracle blob throwing exceptions during column validation (#1005) (8df1cfa)
• support for case insensitive PKs and Snowflake random row (#998) (1a157ae)
• support for null columns, support for access locks (#976) (f54bb4d)
• yaml validation files in gcs (#977) (bf0fa0a)

Documentation

• Add a new sample code for row hash validation of Oracle BLOB (#997) (0bd48a2)

v4.1.0

4.1.0 (2023-08-18)

Features

• support timestamp aggregation for Oracle and TD (#941) (911bae8)

Bug Fixes

• Issues with validate column for time zoned timestamps (#930) (ee7ae9a)
• Schema validations ignore not null on Teradata and BigQuery (#935) (936744b)
• Support casting TD PKs to VARCHAR (#946) (2171532)

v4.0.0

4.0.0 (2023-08-02)

⚠ BREAKING CHANGES

• Ibis Upgrade to 5.1.0 (#894)
• Partition based on non-numeric and multiple keys (#889)

Features

• Adding Random-Row support for Custom Query (#891) (fc42c61)
• Adding RawSQL function for Redshift (#903) (c25d690)
• Enhance validate schema to support time zoned timestamp columns (#919) (aed1505)
• generate-table-partitions: Works on all 7 platforms - BigQuery, Hive, MySQL, Oracle, Postgres, SQL Server and Teradata. (#922) (aa84d7a)
• Ibis Upgrade to 5.1.0 (#894) (b5db4c0)
• Partition based on non-numeric and multiple keys (#889) (7b6a530)
• Snowflake support (#921) (e1d590b)
• Support allow list decimals having a range for precision and scale. Also add --allow-list-file. (#888) (7783beb)

Bug Fixes

• Adding date and timestamp formatting for Hive (#876) (65a090a)
• Adding enhancements to allow-list in schema validation (#881) (c83df2b)
• Adding UTF encoding for Oracle hash generation (#878) (2e24eae)
• No column filtering for csv/json text output. Reverts part of change for issue 753 (#890) (ba641e0)
• redshift bug for custom query (#911) (f1018b5)
• teradata NUMBER with no precision/scale, small doc fix after Ibis upgrade (#914) (f9db68f)
• validate column sum/min/max issue for decimals with precision beyond int64/float64 (#918) (5a8d691)

Documentation

• Add sample shell script and documentation to execute validations at a BigQuery dataset level (#910) (a84da45)

v3.2.0

3.2.0 (2023-05-31)

Features

• Add --dry-run option to validate. (#778) (8989350)
• Add Impala flags for http_transport and http_path (#829) (d966b9e)
• Add support for SQL Server's IMAGE, BINARY, VARBINARY, NCHAR, NTEXT, NVARCHAR data types (#859) (6ebece3)
• Add support for SQL Server's MONEY data type (#837) (0749c9e)
• Move source credentials to secret manager (#824) (1dd5fea)
• Redshift integration for Normal row and Custom-Query Validation. (#817) (92ab215)

Bug Fixes

• Add missing operations for SQL Server - ExtractEpochSeconds, ExtractDayOfYear, ExtractWeekOfYear (#870) (709dd4c)
• Adding datetime and timestamp format logic (#840) (eb095c9)
• dry-run bug when running configs, added CODEOWNERS, and docs (#865) (1779772)
• handle numeric datatype mapping in teradata schema and fix int mapping as per teradata doc (#874) (333eadb)
• split connection names from second last period instead of first from front (#864) (1462deb)
• Support for sum/min/max included for oracle number greater than int64 (#809) (73bda66)

Documentation

• Fix typos on README (#801) (14ddcc5)
• update installation guide about Python 3.11 (#815) (88cd281)
• Update our documentation about find-tables command and the score-cutoff parameter (#846) (54403e3)

v3.1.0

3.1.0 (2023-04-21)

Features

• add db2 hash and concat support (#800) (c16e2f7)
• add Impala connection optional parameters (#743) (#790) (414d7f8)
• added source_type in output while listing connections list (#803) (056275b)
• Adding Custom-Query support for DB2. (#807) (a8085d3)
• Option for simpler report output grid (#802) (b92eb91)

Bug Fixes

• Mysql fix to support row hash validations, random row validation, and filter (#812) (ae07fa4)
• schema validation fixes for Oracle/SQL Server float64 and SQL Server datetimeoffset (#796) (ad0e64f)

Documentation

• add README for Airflow DAG sample, update code formatting in other docs (#722) (f4c3241)
• score-cutoff changed to 1 (#779) (d3aabca)

v3.0.0

3.0.0 (2023-03-28)

⚠ BREAKING CHANGES

• issue673 optimize CLI tools arg parser (#701)

Features

• ✨ Add support for source/target inline sql queries for validate custom-query command (#734) (c5e7a37)
• gcp secret manger support for DVT (#704) (d6c40f1)
• ibis_bigquery strftime support for DATETIME columns (#737) (b1141de)

Bug Fixes

• Add support for numeric and precision with length and precision in Postgres Custom Query (#723) (742b77e)
• Adding Decimal datatype support for MSSQL custom query validation (#771) (0d5c5eb)
• Better detection of Oracle client (#736) (efce0b8)
• Cater for query driven comparisons in date format override code (#733) (0a22643)
• issue 740 teradata strftime function (#747) (9fd102a)
• issue673 optimize CLI tools arg parser (#701) (26bb8e9)
• Protect column and row validation calculated column names from Oracle 30 character identifier limit (#749) (89413c1)
• remove secret manager warnings (#781) (7e72bfd)

Documentation

• formatting fixes and fix broken link (#739) (7306dfc)
• oracleuserpriv (#746) (a7889bf)

v2.9.0

2.9.0 (2023-02-16)

Features

• Added Partition support to generate multiple YAML config files (#653) (Issue #619,#662) (f79c308)
• added run_id to output (#708) (17720f2)
• Divert cast of PostgreSQL decimal with scale>0 to to_char (#721) (3542851)
• Use centralized date/time format in order to compare row data across engines (#720) (0de823b)

Bug Fixes

• Error handling for batch processing of config files (#663) (21a26af)
• Protect non-date columns from astype(str) date workaround (#726) (489ee27)
• schema validation fix for different base names of source and destination data types (#710) (d7b44b0)

Documentation

• updated Oracle parameter from user_name to user and changed underscores to hypens across the document (#689) (8777e00)

v2.8.0

2.8.0 (2023-01-19)

Features

• Logic to add allow-list to support datatype matching with a provided list in case of mismatched datatypes between source and target (#643) (269f8dc)

Bug Fixes

• making logmech as optional for TD connection (#665) (500caa3)