Export of internal ZetaSQL changes.

Update bazel version to 5.3.0; Upgrade protobuf and grpc versions -- Change by Matthew Brown <[email protected]>: Fix FUNCTION<> type argument parsing to deal with type parameters and collation -- Change by John Fremlin <[email protected]>: Move Bison parser helper macros into header -- Change by ZetaSQL Team <[email protected]>: Enables ResolvedAggregationThresholdAggregateScan node to support GROUP BY ROLLUP. -- Change by ZetaSQL Team <[email protected]>: Inline a helper function. This function was attempting to reduce code duplication between handling inline subqueries and inline lambda definitions. -- Change by ZetaSQL Team <[email protected]>: Fix for subquery expressions in differential_privacy/anonymization select list -- Change by ZetaSQL Team <[email protected]>: Add testing for collated sort key in analytic function in reference implementation. -- Change by ZetaSQL Team <[email protected]>: Use a btree_set to keep track of subquery parameters in sql_function_inliner instead of using an unordered set and and then separately sorting. -- Change by ZetaSQL Team <[email protected]>: use argument name in signature mismatch message. -- Change by ZetaSQL Team <[email protected]>: Fix for subquery expressions in differential_privacy/anonymization select list -- Change by ZetaSQL Team <[email protected]>: Remove table_scan_groups from analyzer tests -- Change by ZetaSQL Team <[email protected]>: Minor import fix -- Change by Matthew Brown <[email protected]>: Fix bug improperly invoking function with repeated and named arguments -- Change by ZetaSQL Team <[email protected]>: Fix for subquery expressions in differential_privacy/anonymization select list -- Change by John Fremlin <[email protected]>: Log stack, wall and CPU time usage for processing steps -- Change by Matthew Brown <[email protected]>: Fix bug when zero repetitions in functions like F(REPEATED, NAMED) -- Change by ZetaSQL Team <[email protected]>: internal cleanup to remove deprecated Type::TypeNameWithParameters function. -- Change by ZetaSQL Team <[email protected]>: Add test cases for WITH REPORT -- Change by ZetaSQL Team <[email protected]>: Add SampleScan for bounding max rows contributed. -- Change by Matthew Brown <[email protected]>: Update interface of sql_function; split out helper function from simple_catalog_util -- Change by ZetaSQL Team <[email protected]>: Output of (BYTES || BYTES) has a BYTES type. -- Change by Matthew Brown <[email protected]>: Refactor Analyer Tests to make catalogs more hermetic -- Change by Matthew Brown <[email protected]>: Update ZetaSQL Docs -- Change by ZetaSQL Team <[email protected]>: Make the reference implementation for `AggregateOp` aware of arrays known/unknown orders. This will stop the reference implementation from enforcing incidental array orders in RQG tests. -- Change by ZetaSQL Team <[email protected]>: Make reference implementation for `SortOp` aware of arrays known/unknown orders. This will stop the reference implementation from enforcing incidental orders on query results when those query's ORDER BY arrays with unknown element order. -- Change by ZetaSQL Team <[email protected]>: Make reference implementation for `$equals` and other comparison operators aware of array known/unknown orders. This will mean the stop the reference implementation from enforcing incidental orders in compliance tests (including RQG tests) when V_1_1_ARRAY_EQUALITY is enabled. -- Change by ZetaSQL Team <[email protected]>: Add some example compliance tests that exhibit broken array comparisons in the reference implementation. -- Change by ZetaSQL Team <[email protected]>: Disable Geography in all constructors of the ReferenceDriver. -- Change by ZetaSQL Team <[email protected]>: Do not add extra spaces around map constructors. -- Change by ZetaSQL Team <[email protected]>: Fix formatting of proto extension fields in braced map constructor. -- Change by ZetaSQL Team <[email protected]>: Refactor value formatting logic. -- Change by ZetaSQL Team <[email protected]>: Support partition key with collation in reference implementation. -- Change by Matthew Brown <[email protected]>: Fix bugs to enable F(Repeated ANY, <named-only argument>) -- Change by Matthew Brown <[email protected]>: Comprehensive tests for function resolution -- Change by ZetaSQL Team <[email protected]>: Fix non-deterministic mark for array find functions in reference implementation -- Change by ZetaSQL Team <[email protected]>: Remove in_development flag for FEATURE_RANGE_TYPE -- Change by ZetaSQL Team <[email protected]>: Change RANGE argument validation from InternalError to InvalidArgumentError -- Change by ZetaSQL Team <[email protected]>: Add space between } and AS -- Change by ZetaSQL Team <[email protected]>: Remove 'in_development' annotation from FEATURE_EXTENDED_TYPES -- Change by ZetaSQL Team <[email protected]>: Add support for inlining sql-defined aggregate functions that call other aggregate functions, and compliance tests for this case. -- Change by ZetaSQL Team <[email protected]>: Add initial compliance tests for queries invoking SQL-defined aggregate functions. -- Change by ZetaSQL Team <[email protected]>: Document NULLIFZERO and ZEROIFNULL -- Change by ZetaSQL Team <[email protected]>: Add support for inlining the most basic trivial SQL-defined aggregations. Those with no arguments and no invocations of aggregate functions in their bodies. GitOrigin-RevId: 23a39bcc92c42c56dbad29b7e47dc9c58e0bb533 Change-Id: I5bb7e20bcf496780bb65302f6191073cfb9b236d
google · Jul 14, 2023 · 5133c6e · 5133c6e
1 parent 0a028ce
commit 5133c6e
Show file tree

Hide file tree

Showing 195 changed files with 13,475 additions and 5,838 deletions.
diff --git a/.bazelversion b/.bazelversion
@@ -1 +1 @@
-4.0.0
+5.3.0
diff --git a/README.md b/README.md
@@ -20,26 +20,23 @@ analytic functions.
 This codebase is being open sourced in multiple phases:
 
 1. Parser and Analyzer **Complete**
-   - Initial release includes only a subset of tests
 2. Reference Implementation **In Progress**
    - Base capability **Complete**
    - Function library **In Progress**
 3. Compliance Tests **Complete**
    - includes framework for validating compliance of arbitrary engines
 4. Misc tooling
-   - Improved Formatter **In Progress**
+   - Improved Formatter **Complete**
 
 Multiplatform support is planned for the following platforms:
 
  - Linux (Ubuntu 20.04 is our reference platform, but others may work).
+   - gcc-9+ is required, recent versions of clang may work.
  - MacOS (Experimental)
  - Windows (version TDB)
 
-Until all this code is released, we cannot provide any guarantees of API
-stability and cannot accept contributions. We will also be releasing more
-documentation over time, particular related to developing engines with this
-framework. Documentation on the [language](docs/) itself is fairly
-complete.
+We do not provide any guarantees of API stability and *cannot accept
+contributions*.
 
 
 ## Flags
@@ -51,16 +48,16 @@ debugging purposes only and may change, stop working or be removed at any time.
 ## How to Build
 
 ZetaSQL uses [bazel](https://bazel.build) for building and dependency
-resolution. After installing bazel (we maintain support for 1.0,
-but other versions may work), simply run:
+resolution. After installing bazel (check .bazelversion for the specific version
+of bazel we test with, but other versions may work), simply run:
 
 ```bazel build ...```
 
 ## How to add as a Dependency in bazel
 See the (WORKSPACE) file, as it is a little unusual.
 
 ### With docker
- TODO: Add docker build script.
+ TODO: Add docker build instructions.
 
 ## Example Usage
 A very basic command line tool is available to run simple queries with the

diff --git a/WORKSPACE b/WORKSPACE
@@ -47,9 +47,9 @@ http_archive(
 # gRPC Java
 http_archive(
     name = "io_grpc_grpc_java",
-    url = "https://github.com/grpc/grpc-java/archive/v1.43.2.tar.gz",
-    strip_prefix = "grpc-java-1.43.2",
-    sha256 = "6c39c5feecda4f1ccafe88d8928d9a0f2a686d9a9a9c03888a2e5ac92f7ee34a",
+    url = "https://github.com/grpc/grpc-java/archive/v1.56.0.tar.gz",
+    strip_prefix = "grpc-java-1.56.0",
+    sha256 = "4af5ecbaed16455fcda9fdab36e131696f5092858dd130f026069fcf11817a21",
 )
 
 load("@rules_jvm_external//:repositories.bzl", "rules_jvm_external_deps")

diff --git a/bazel/maven_install.json b/bazel/maven_install.json
diff --git a/bazel/zetasql_deps_step_2.bzl b/bazel/zetasql_deps_step_2.bzl
@@ -235,33 +235,25 @@ py_library(
                 strip_prefix = "farmhash-816a4ae622e964763ca0862d9dbd19324a1eaf45",
             )
     if analyzer_deps:
+        # We need to override protobuf's native dep, because they patch the
+        # the repo in an unhelpful way.
+        if not native.existing_rule("upb"):
+            http_archive(
+                name = "upb",
+                urls = ["https://github.com/protocolbuffers/upb/archive/0ea9f73be35e35db242ccc65aa9c87487b792324.tar.gz"],
+                sha256 = "046b5f134523eaad9265a41a2ec0701cc45973841070af2772e3578a9f3bfed0",
+                strip_prefix = "upb-0ea9f73be35e35db242ccc65aa9c87487b792324",
+            )
+
         # Protobuf
         if not native.existing_rule("com_google_protobuf"):
             http_archive(
                 name = "com_google_protobuf",
-                urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.19.3.tar.gz"],
-                sha256 = "390191a0d7884b3e52bb812c440ad1497b9d484241f37bb8e2ccc8c2b72d6c36",
-                strip_prefix = "protobuf-3.19.3",
+                urls = ["https://github.com/protocolbuffers/protobuf/archive/refs/tags/v4.23.3.tar.gz"],
+                sha256 = "21fcb4b0df6a8e6279e5843af8c9f2245919cf0d3ec2021c76fccc4fc4bf9aca",
+                strip_prefix = "protobuf-4.23.3",
             )
 
-    # Required by gRPC
-    if not native.existing_rule("build_bazel_rules_apple"):
-        http_archive(
-            name = "build_bazel_rules_apple",
-            urls = ["https://github.com/bazelbuild/rules_apple/archive/0.18.0.tar.gz"],
-            sha256 = "53a8f9590b4026fbcfefd02c868e48683b44a314338d03debfb8e8f6c50d1239",
-            strip_prefix = "rules_apple-0.18.0",
-        )
-
-    # Required by gRPC
-    if not native.existing_rule("build_bazel_apple_support"):
-        http_archive(
-            name = "build_bazel_apple_support",
-            urls = ["https://github.com/bazelbuild/apple_support/archive/0.7.1.tar.gz"],
-            sha256 = "140fa73e1c712900097aabdb846172ffa0a5e9523b87d6c564c13116a6180a62",
-            strip_prefix = "apple_support-0.7.1",
-        )
-
     if testing_deps:
         if not native.existing_rule("com_google_file_based_test_driver"):
             http_archive(
@@ -276,9 +268,9 @@ py_library(
         if not native.existing_rule("com_github_grpc_grpc"):
             http_archive(
                 name = "com_github_grpc_grpc",
-                urls = ["https://github.com/grpc/grpc/archive/v1.43.2.tar.gz"],
-                sha256 = "b74ce7d26fe187970d1d8e2c06a5d3391122f7bc1fdce569aff5e435fb8fe780",
-                strip_prefix = "grpc-1.43.2",
+                urls = ["https://github.com/grpc/grpc/archive/v1.56.0.tar.gz"],
+                sha256 = "e034992a0b464042021f6d440f2090acc2422c103a322b0844e3921ccea981dc",
+                strip_prefix = "grpc-1.56.0",
             )
 
     if analyzer_deps:

diff --git a/bazel/zetasql_deps_step_3.bzl b/bazel/zetasql_deps_step_3.bzl
@@ -22,11 +22,11 @@ load("@com_github_grpc_grpc//bazel:grpc_deps.bzl", "grpc_deps")
 load("@com_google_googleapis//:repository_rules.bzl", "switched_rules_by_language")
 
 def zetasql_deps_step_3():
+    protobuf_deps()
+    grpc_deps()
     switched_rules_by_language(
         name = "com_google_googleapis_imports",
         java = True,
         cc = True,
     )
     google_common_workspace_rules()
-    protobuf_deps()
-    grpc_deps()
diff --git a/bazel/zetasql_deps_step_4.bzl b/bazel/zetasql_deps_step_4.bzl
@@ -18,12 +18,7 @@
 
 # This must be loaded _after_ com_github_grpc_grpc, so we move it to this file
 
-# Needed by (at least) com_github_grpc_grpc
-load("@build_bazel_rules_apple//apple:repositories.bzl", "apple_rules_dependencies")
-
-# Needed by (at least) com_github_grpc_grpc
-load("@build_bazel_apple_support//lib:repositories.bzl", "apple_support_dependencies")
+load("@com_github_grpc_grpc//bazel:grpc_extra_deps.bzl", "grpc_extra_deps")
 
 def zetasql_deps_step_4():
-    apple_rules_dependencies()
-    apple_support_dependencies()
+    grpc_extra_deps()
diff --git a/bazel/zetasql_java_deps.bzl b/bazel/zetasql_java_deps.bzl
@@ -22,26 +22,26 @@ load("@bazel_tools//tools/build_defs/repo:java.bzl", "java_import_external")
 load("@io_grpc_grpc_java//:repositories.bzl", "IO_GRPC_GRPC_JAVA_ARTIFACTS")
 
 ZETASQL_MAVEN_ARTIFACTS = [
-    "com.google.api.grpc:proto-google-common-protos:1.12.0",
+    "com.google.api.grpc:proto-google-common-protos:2.21.0",
     "com.google.code.findbugs:jsr305:3.0.2",
     "com.google.errorprone:error_prone_annotations:2.11.0",
-    "com.google.guava:guava:31.0.1-jre",
-    "com.google.guava:guava-testlib:31.0.1-jre",
-    "io.grpc:grpc-context:1.43.2",
-    "io.grpc:grpc-core:1.43.2",
-    "io.grpc:grpc-api:1.43.2",
-    "io.grpc:grpc-netty:1.43.2",
-    "io.grpc:grpc-protobuf-lite:1.43.2",
-    "io.grpc:grpc-protobuf:1.43.2",
-    "io.grpc:grpc-stub:1.43.2",
+    "com.google.guava:guava:32.1.1-jre",
+    "com.google.guava:guava-testlib:32.1.1-jre",
+    "io.grpc:grpc-context:1.56.0",
+    "io.grpc:grpc-core:1.56.0",
+    "io.grpc:grpc-api:1.56.0",
+    "io.grpc:grpc-netty:1.56.0",
+    "io.grpc:grpc-protobuf-lite:1.56.0",
+    "io.grpc:grpc-protobuf:1.56.0",
+    "io.grpc:grpc-stub:1.56.0",
     "io.netty:netty-common:4.1.34.Final",
     "io.netty:netty-transport:4.1.34.Final",
     "io.opencensus:opencensus-api:0.21.0",
     "io.opencensus:opencensus-contrib-grpc-metrics:0.21.0",
     "javax.annotation:javax.annotation-api:1.2",
     "joda-time:joda-time:2.10.13",
     "com.google.code.gson:gson:jar:2.8.9",
-    "com.google.protobuf:protobuf-java:3.19.3",
+    "com.google.protobuf:protobuf-java:3.23.1",
     "com.google.truth:truth:1.1.3",
     "com.google.truth.extensions:truth-proto-extension:1.1.3",
     "junit:junit:4.13.2",

diff --git a/docs/approximate_aggregate_functions.md b/docs/approximate_aggregate_functions.md
@@ -130,9 +130,10 @@ APPROX_QUANTILES(
 **Description**
 
 Returns the approximate boundaries for a group of `expression` values, where
-`number` represents the number of quantiles to create. This function returns
-an array of `number` + 1 elements, where the first element is the approximate
-minimum and the last element is the approximate maximum.
+`number` represents the number of quantiles to create. This function returns an
+array of `number` + 1 elements, sorted in ascending order, where the
+first element is the approximate minimum and the last element is the approximate
+maximum.
 
 Returns `NULL` if there are zero input rows or `expression` evaluates to
 `NULL` for all rows.

diff --git a/docs/conditional_expressions.md b/docs/conditional_expressions.md
@@ -310,6 +310,60 @@ SELECT NULLIF(10, 0) as result
  *--------*/
 ```
 
+### `NULLIFZERO`
+
+```sql
+NULLIFZERO(expr)
+```
+
+**Description**
+
+Returns `NULL` if the value of `expr` is `0`. Otherwise, returns `expr`. `expr`
+must be a zeroable type.
+
+**Return Data Type**
+
+Type of `expr`.
+
+**Example**
+
+```sql
+SELECT NULLIFZERO(0) AS result
+
+/*--------*
+ | result |
+ +--------+
+ | NULL   |
+ *--------*/
+```
+
+### `ZEROIFNULL`
+
+```sql
+NULLIFZERO(expr)
+```
+
+**Description**
+
+Returns `0` if the value of `expr` is `NULL`. Otherwise, returns `expr`. `expr`
+must be a zeroable type.
+
+**Return Data Type**
+
+Type of `expr`.
+
+**Example**
+
+```sql
+SELECT ZEROIFNULL(NULL) AS result
+
+/*--------*
+ | result |
+ +--------+
+ | 0      |
+ *--------*/
+```
+
 <!-- mdlint off(WHITESPACE_LINE_LENGTH) -->
 
 [cond-exp-supertype]: https://github.com/google/zetasql/blob/master/docs/conversion_rules.md#supertypes

diff --git a/docs/conversion_rules.md b/docs/conversion_rules.md
@@ -483,6 +483,7 @@ or more supertypes, including itself, which defines its set of supertypes.
       </td>
     </tr>
 
+
   </tbody>
 </table>