Release Note Preview for v1.5.0-incubating

[yaooqinn]

[KYUUBI #2070][FLINK] Support Flink job submission on yarn-session mode
[KYUUBI #2089] Add debugging instructions for Flink engine
[KYUUBI #1948][FOLLOWUP] Relocate fb303 classes
[KYUUBI #2078] logCaptureThread does not catch sparksubmit exception
[KYUUBI #1936] Send credentials when opening session and wait for completion
[KYUUBI #2079] Update kyuubi layer source file to add flink and trino…
[KYUUBI #2075] Using thread-safe FastDateFormat instead of SimpleDateFormat
[KYUUBI #2063] Fix engine idle timeout lose efficacy for Flink Engine
[KYUUBI #1215][DOC] Document incremental collection
[KYUUBI #2043] Upgrade log4j/2.x/ to 2.17.2
[KYUUBI #2060] Clear job group for init SQL
[KYUUBI #2054] [KYUUBI-1819] Support closing Flink SQL engine process
[KYUUBI #2055] correct the log service name
[KYUUBI #2047] Support more MySQL JDBC driver versions
[KYUUBI #2012] rename EventLoggingService and creates an correct directory
[KYUUBI #2042] [KYUUI #2036] Redirect Issues/PR Notifications to [email protected]
[KYUUBI #2039] Remove LegacyFlinkOperationSuite
[KYUUBI #1961] Upgrade hive-service-rpc to 3.1.2
[KYUUBI #2008] Support engine type and subdomain in kyuubi-ctl
[KYUUBI #2006] Improve the readability of ServiceControl list/get operations
[KYUUBI #1999] Update quick_start_with_helm.md
[KYUUBI #2000][DEPS] Bump Hadoop 3.3.2
[KYUUBI #2004] Sync contents for CONTRIBUTING & COMMUNITY between web and main repo
[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf
[KYUUBI #1986] Apache Kyuubi(Incubating) Maturity Self Assessment
[KYUUBI #1988] Tune initialization of vars depend on system level confs
[KYUUBI #1992] Remove SPARK_HOME predication.
[KYUUBI #1960] Skip creating sparksession and starting engine if the max initialization time exceeds
[KYUUBI #1958] Fix flaky test: statement redact for engine tab
[KYUUBI #1983] [[KYUUBI #1806][FOLLOWUP] Fix LaunchEngine may be closed multiple times
[KYUUBI #1981] Remove unused artifacts for common and parent module
[KYUUBI #1972] Fix flaky test: SchedulerPoolSuite: Scheduler pool
[KYUUBI #1969] Fix race on some service during start and stop phase
[KYUUBI #1979] Simplify Kyuubi Event
[KYUUBI #1975] [BUILD] Run TPC-DS test using the default Spark version
[KYUUBI #1974] Support merge small files in multi insert statement
[KYUUBI #1963] Make plan only mode skippable for configuable plans
[KYUUBI #1970] Replace mirror-cn profile with Apache officially suggested dlcdn
[KYUUBI #1948][FOLLOWUP] Remove TFramedTransport in extension-spark-3-1 and extension-spark-3-2
[KYUUBI #1965] [DOCS] Use the newly added official logo at asf svn in README
[KYUUBI #1948] Upgrade thrift version to 0.16.0
[KYUUBI #1964] [BUILD] Update scalatest.exclude.tags
[KYUUBI #1883] Support max result rows for Flink queries
[KYUUBI #1920] Skip the Plan parsing of UseStatement
[KYUUBI #1944] Upgrade Spark 3.1.3
[KYUUBI #1957] Skip html comments in merge commit test body from PR desc
[KYUUBI #1947] Revamp Github Action settings for CI
[KYUUBI #1950] Remove ambiguous SPARK_HADOOP_VERSION
[KYUUBI #1949] [INFA] Add homepage for every module in maven central
[KYUUBI #1946] Close the kyuubi connection on launch engine operation failure
[KYUUBI #1918] Build and test on Linux ARM64
[KYUUBI #1945] [BUILD] Remove spark-hadoop from matrix to simplify workflow
[KYUUBI #1940] [TEST] Fix EngineSecurityAccessorSuite
[KYUUBI #1876][FOLLOWUP] Update iceberg.name
[KYUUBI #1937] Build: Use Spark 3.2 for cross version verification
[KYUUBI #1939] Build: Bump Jacoco maven plugin 0.8.7
[KYUUBI #1796] Destroy the process after the engine starts
[KYUUBI #1931] AbstractSession._lastIdleTime should be reset in acquire method
[KYUUBI #1919] Add more enum values for OperationModes
[KYUUBI #1930] Export TRINO_ENGINE_HOME in load-kyuubi-env script
[KYUUBI #1926] Remove useless constants in Constants
[KYUUBI #1923] Add GetFunctions for trino engine
[KYUUBI #1906] Add GetColumns for trino engine
[KYUUBI #1914] Update metric doc
[KYUUBI #1867] Support PlanOnlyStatement operation like Spark SQL engine
[KYUUBI #1911] Add GetTypeInfo for trino engine
[KYUUBI #1876] Set Spark 3.2.1 as default
[KYUUBI #1536] Add session conf advisor
[KYUUBI #1899] Bump Iceberg 0.13.1
[KYUUBI #1887] Add GetTables for trino engine
[KYUUBI #1909] [DOCS] Remove unnecessary docs link in readme
[KYUUBI #1886] Add GetSchemas for trino engine
[KYUUBI #1905] log4j2-defaults.properties is not working since log4j 2 is always intialized by default
[KYUUBI #1908] Codecov for flink/trino/beeline/jdbc modules
[KYUUBI #1903] Add GetTableTypes for trino engine
[KYUUBI #1901] [GA] Fix checkout version
[KYUUBI #1898] Enhance release guide add more information about how to…
[KYUUBI #1896] Make Query Details and Failure Reason sortable on the Kyuubi Query Engine page
[KYUUBI #1811] TPC-DS benchmark expose cli args breakdown and results-dir
[KYUUBI #1893] Bump Delta 1.0.1
[KYUUBI #1880] Fixed: forcedMaxOutputRows extension for join
[KYUUBI #1889] Improve the user experience of the configuration parameters of the document
[KYUUBI #1885] Add GetCatalogs for trino engine
[KYUUBI #1821] Add trino ExecuteStatement
[KYUUBI #1884] Fix compile SQL measurement
[KYUUBI #1816] Implement KyuubiHistoryServerPlugin
[KYUUBI #1873] Fix incorrect applicationId
[KYUUBI #1878] Fix typo in log4j2-test.properties
[KYUUBI #1838] Clean up query results after query operations finish
[KYUUBI #1868] Upgrate netty from 4.1.68 to 4.1.73
[KYUUBI #1872] Make ProcBuilder decouple with YARN
[KYUUBI #1875] Upgrade codahale.metrics.version to 4.2.8 and its deps
[KYUUBI #1870] Upgrate k8s client from 5.5.0 to 5.12.1
[KYUUBI #1871] Upgrade minikube action from 2.4.2 to 2.4.3
[KYUUBI #1804][FOLLOWUP][GA] Avoid push docker image for forked repos
[KYUUBI #1844] Add step for generating release notes
[KYUUBI #1849] Bump Iceberg 0.13.0 and enable Iceberg test with Spark 3.2
[KYUUBI #1861] Migrate bind-api and activation-api from javax to jakarta
[KYUUBI #1848] Bump Spark 3.2.1
[KYUUBI #1769][FOLLOWUP] Exclude log4j 12 deps from spark-hive and fix detect deps change workflow
[KYUUBI #1769][FOLLOWUP] Revert the dependency change for kyuubi-hive-jdbc(shaded) modules
[KYUUBI #1846] Add improvement issue template
[KYUUBI #1858] Rename test log4j2 properties to log4j2-test.properties
[KYUUBI #1850] Bump Prometheus 0.14.1
[KYUUBI #1855] Suppress logging to console in scalatest
[KYUUBI #1847] Bump Flink 1.14.3
[KYUUBI #1829] Support executing statement async for Flink SQL engine
[KYUUBI #1852] Fix UT property of log4j2 conf
[KYUUBI #1769][FOLLOWUP] Some cleanups for log4j properties
[KYUUBI #1839] Minor refactor OperationUtil
[KYUUBI #1813] Enable the internal secure authentication between Kyuubi server and engine
[KYUUBI #1769] [BUILD] Migrate from log4j1 to log4j2
[KYUUBI #1832] Fixed: forcedMaxOutputRows extension for subquery
[KYUUBI #1836] [RELEASE] Enhance release guide
[KYUUBI #1840] Minor fix for general_incubator_vote script
[KYUUBI #1834] [BUILD] Add support of external mvn with default mvn jvm opts when making distribution
[KYUUBI #1835] Fix error to helm manifest
[KYUUBI #1786] Fix the Spark sql engine logger level changed to WARN issue
[KYUUBI #1685] Support jar and lib start option
[KYUUBI #1778] Support Flink Set/Reset Operations
[KYUUBI #1825] Generate appName for Flink applications
[KYUUBI #1831] [DOCS] Fix the jdbc connection url format doc
[KYUUBI #1610] Rest frontend service jetty server has started at null
[KYUUBI #1806] Fix LaunchEngine may be closed multiple times
[KYUUBI #1698] Add kyuubi server operation state rate
[KYUUBI #1808] Update the link in the PULL_REQUEST_TEMPLATE
[KYUUBI #1800] Remove oom hook
[KYUUBI #1780] Log error message on REST endpoints
[KYUUBI #1784][FOLLOWUP][TRINO] Fix float types lose precision
[KYUUBI #1784][FOLLOWUP][FLINK] Support float type
[KYUUBI #1797] Only exposing jdbc url for server side
[KYUUBI #1804]Avoid nightly build for forked repos
[KYUUBI #1764] Reuse the workingDir in ProcBuilder
[KYUUBI #1699] Improve codahale metric
[KYUUBI #1784] Fix float types lose precision
[KYUUBI #1787] Export FLINK_ENGINE_HOME in load-kyuubi-env script
[KYUUBI #1774] Export FLINK_HOME in load-kyuubi-env script
[KYUUBI #1775] Add kyuubi.server.name configuration
[KYUUBI #1785] Stop Discovery services properly
[KYUUBI #1643][FOLLOWUP] Fix Flink GetFunctions result schema
[KYUUBI #1788] Update greetings.yml
[KYUUBI #1776] Improve EnginePage
[KYUUBI #1648] Implement Flink GetTypeInfo operation
[KYUUBI #1771] Change rate unit in JsonReporterService
[KYUUBI #1696] Add fetch logs and results rows rate
[KYUUBI #1704] Complete Flink-Hive data type mapping
[KYUUBI #1645] Implement Flink engine GetSchemas operation
[KYUUBI #1714] Add executeScala api for KyuubiStatement
[KYUUBI #1756] Clean up code of FlinkOperation
[KYUUBI #1766] [TEST] Remove SessionManagerSuite to reduce duplicated tests
[KYUUBI #1763] [INFRA] Remove scala installation inbuild/mvn and using warn level for the ci
[KYUUBI #1643] Implement GetFunctions operation
[KYUUBI #1761] [SubTask][THTTP] Make abstract TFrontendService for thrift http protocol support
[KYUUBI #1757] Extract workingDir from engine specific process builde…
[KYUUBI #1510] Add TRINO engine type and TrinoProcessBuilder
[KYUUBI #1748] bin/kyuubi-zk-cli should has execute permission
[KYUUBI #1759] Remove duplicate close operation api
[KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
[KYUUBI #1738] Unify kubernetes integration test
[KYUUBI #1660] Add restart option for bin/kyuubi
[KYUUBI #1746] Move sparkEngineMajorMinorVersion into SparkQueryTests to make HiveJDBCTestHelper spark free
[KYUUBI #1743] Fix parallelism of DataGenerator and other enhancements
[KYUUBI #1740] Use testcontainers scalatest in trino module
[KYUUBI #1745] [KYUBBI #1744] Fix trino client flaky test
[KYUUBI #1737] [WIP][KYUUBI #1731][FEATURE] Kyuubi server should not fail to start when Hadoop FileSystem class not found
[KYUUBI #1716] Unify Hive deps for server and client
[KYUUBI #1729] Extract java lint workflow to a single yml and disable it in other jobs
[KYUUBI #1735] [CI] Add current ip address, long hostname and short hostname to /etc/hosts
[KYUUBI #1736] Bump up GA actions/checkout to latest (v2.3.5)
[KYUUBI #1732] Reduce spark driver memory configuration to protect fa…
[KYUUBI #1730] Use getHostName instead of getCanonicalHostName
[KYUUBI #1672] Add VersionInfo in REST API
[KYUUBI #1726] Extract dep workflow to a single yml and disable it in other jobs
[KYUUBI #1719] javax.activation:activation:jar duplicates
[KYUUBI #1720] Extract scalastyle workflow to a single yml and disable it in other jobs
[KYUUBI #1646] support flink get tables
[KYUUBI #1723] Create greetings.yml
[KYUUBI #1722] [NOTICE] Update NOTICE for 2022
[KYUUBI #1721] Recover maven deplop plugin in all module
[KYUUBI #1709] Introduce a IT module for Kyuubi and FlinkSQL Engine t…
[KYUUBI #1718] Extract rat workflow to a single yml and disable it in other jobs
[KYUUBI #1717] Extract rat configurations from pom to file
[KYUUBI #1715] For kyuubi-beeline, using kyuubi-hive-jdbc instead of kyuubi-hive-jdbc-shaded
[KYUUBI #1712] Bump up maven.version to 3.8.4
[KYUUBI #1710] Support to specify OPERATION_LANGUAGE with TExecuteStatementReq confOverlay
[KYUUBI #1647] Change GetCatalog returned schema key name to support JDBC specification
[KYUUBI #1690] Checking whether launchEngineOpHandle is null for KyuubiConnection:: getExecLog
[KYUUBI #1697] BackendService method avg call times and rate
[KYUUBI #1681] Extend Flink ExecuteStatement to support more operations
[KYUUBI #1705] Support decimal type for Flink SQL Engine
[KYUUBI #1691] Implement api: /${version}/operations/${operation_identifier}/rowset
[KYUUBI #1641] Add Trino client
[KYUUBI #1647] Implement GetTableTypes operation
[KYUUBI #1477] Use KyuubiHadoopUtils.newHadoopConf instead of new Con…
[KYUUBI #1678] Fixed: The value of hasResultSet is always set to true in Kyuubi Server
[KYUUBI #1679] Remove invalid column and field number check
[KYUUBI #1673] [BUILD] Improve log4j configurations
[KYUUBI #1568] [FOLLOWUP] Add getSessionEvent in session
[KYUUBI #1675][FEATURE] Redact secret information from SparkSQLEngine log
[KYUUBI #1674] Uncache cached tables when session closed
[KYUUBI #1654][Bug] Flaky OperationsResourceSuite
[KYUUBI #1665][FOLLOWUP] Rename FlinkEngineProcessBuilder to FlinkProcessBuilder to align with SparkProcessBuilder
[KYUUBI #1594] SparkProcessBuild submit add --conf spark.driver.host=
[KYUUBI #1665] Rename FlinkEngineProcessBuilder to FlinkProcessBuilder to align with SparkProcessBuilder
[KYUUBI #1656] Fix dist failed with --spark-provided
[KYUUBI #1658] Revamp swagger ui dependencies
[KYUUBI #1662] Turn up maven options in mvn script
[KYUUBI #1597] fix row value in Query Execution should be queryExecution
[KYUUBI #1649] Determine the yarn command in stop-application.sh.
[KYUUBI #1634][FEATURE] Redact secret information from SparkSQLEngine UI
[KYUUBI #1639] Make ApiRequestContext provide KyuubiRestFrontendService directly
[KYUUBI #1629] Flink backend implementation
[KYUUBI #1631] Migrating existing rest fe tests to real cases
[KYUUBI #1556] Upgrade swagger-ui version from 4.1.0 to 4.1.3
[KYUUBI #1614] Fix hardcoded version in OpenAPIDefinition
[KYUUBI #1623] KyuubiSessionEvent shall always have Id
[KYUUBI #1618] Correct hadoop version in RELEASE file
[KYUUBI #1581] Add SchemaHelper for trino engine
[KYUUBI #1620] Implement api: /${version}/operations/${operation_identifier}/log
[KYUUBI #1613] Record the engine's applicationId in KyuubiSessionEvent
[KYUUBI #1611] Fix swagger-ui redirect wrong to other service address
[KYUUBI #1601] Align SparkStatementEvent to KyuubiOperationEvent
[KYUUBI #1592] [TEST][ICEBERG][DELTA] Introduce row level operation test for data lake format
[KYUUBI #1582] Use ClientTypeSignature to further analysis of trino column type
[KYUUBI #1607] [DOC] Add operation and backend service metrics in metrics doc
[KYUUBI #1579] Implement basic ability of executing statement in Flink engine
[KYUUBI #1575] Implement api: /${version}/operations/${operation_identifier}/resultsetmetadata
[KYUUBI #1593] use user set host or ip instead of read hostname from user set
[KYUUBI #1591] Watchdog support for Spark-3.2
[KYUUBI #1587] [SUB-TASK][KPIP-2] Bump Flink 1.14.2
[KYUUBI #1577] Add DropIgnoreNonexistent Rule.
[KYUUBI #1586] Add time metric on each KyuubiBackendService method
[KYUUBI #1578] Introduce flink-sql-client dependency to reduce some duplicated code
[KYUUBI #1499] Introduce DataFrameHolder for cli result fetching
[KYUUBI #1550] Provide a specific user guide about connecting to kerberized kyuubi
[KYUUBI #1555][FLINK] Bump flink version to 1.14.0
[KYUUBI #1554] Add kyuubi.engine.pool.name configuration
[KYUUBI #1460] Bump Hudi 0.10.0
[KYUUBI #1559] Bump Jersey 2.34
[KYUUBI #1568] Replace sessionDetail with sessionEvent
[KYUUBI #1565] Move time functions to RowSetUtils and move schema to spark package
[KYUUBI #1567] [Bug] Expose HADOOP_CONF_DIR to Kyuubi beeline to fix "GSS initiate failed" error
[KYUUBI #1571] [BUILD] Correct build command in build/dist
[KYUUBI #1558] Bump Jackson 2.12.5
[KYUUBI #1556] Shorten kyuubi.session.engine.check.interval to one minute
[KYUUBI #1560] Align all Jackson libs version
[KYUUBI #1539] Replace operationDetail with KyuubiEvent
[KYUUBI #1557] [DOC] The TTL Of Kyuubi Engines
[KYUUBI #1174][FOLLOWUP] change configuration(ha.zookeeper.publish.configs) default value to false
[KYUUBI #1515] Add RowSet for trino engine
[KYUUBI #1525][FLINK] Implement GetCatalogs operation and do the further initialization
[KYUUBI #1548] Get docs version from project.version
[KYUUBI #1544] Update the mailing list to send announcement
[KYUUBI #1542] Move FetchIterator into kyuubi-common for multi-engine support
[KYUUBI #1541] Remove keys sync step for Kyuubi finalize release
[KYUUBI #1533] Add documentation about Kyuubi docker-image-tool using Spark image as BASE_IMAGE
[KYUUBI #1535] Remove duplicate hive client version property
[KYUUBI #1531] Add kyuubi version in engineSpace
[KYUUBI #1528] Record the kyuubi server ip address in event log
[KYUUBI #1527] [DOC] Improve High Availability Guide
[KYUUBI #1516] Implement api: /${version}/operations/${operation_identifier}
[KYUUBI #1480] Fix bug in docker-image-tool & Provider image way to config SPARK instead of local copy
[KYUUBI #1393][TASK-2] Add operation number metric
[KYUUBI #1523] Refine usage about option '--namespace' in kyuubi-ctl
[KYUUBI #1522] Pass the client ip to engine
[KYUUBI #1526] [DOC] Update kyuubi_layers.drawio to add driver layer
[KYUUBI #1521] [BUILD] Migrate to actions/cache@v2
[KYUUBI #1489] Support operation log for ExecuteScala
[KYUUBI #1520] [BUILD] Switch to setup-java@v2
[KYUUBI #1509] Make KYUUBI_WORK_DIR_ROOT as the default root path.
[KYUUBI #1511] Initialize kyuubi trino engine module
[KYUUBI #1513] Added support liveness and readiness for kyuubi pod
[KYUUBI #1501] Introduce operationsResource
[KYUUBI #1503] Introduce FlinkThriftBinaryFrontendService to do the further initialization
[KYUUBI #1487] Correct the KEYS link and add script to generate vote and announcement
[KYUUBI #1496] Support tpcds benchmark
[KYUUBI #1508] Add Kyuubi version in UI
[KYUUBI #1504] Support to capture console out/error into interpreter output for ExecuteScala
[KYUUBI #1505] Move ServiceNodeInfo to a separate file
[KYUUBI #1492] Initialize kyuubi flink sql engine module
[KYUUBI #1445] [Kyuubi #1444] Implement api: /${version}/sessions/${session_identifier}/operations/${operation_identifier}
[KYUUBI #1490] Introduce the basic framework for running scala
[KYUUBI #1493] Make a reasonable thread pool size for listing file statuses
[KYUUBI #1379] Make SESSION_ENGINE_LAUNCH_ASYNC true by default
[KYUUBI #1486] Enable delta test on Spark 3.2 with delta 1.1.0
[KYUUBI #1484] Add a dependency describe for kyuubi on azure delta-lake doc.
[KYUUBI #1483] Revamp KyuubiSessionImpl and KyuubiSyncThriftClient
[KYUUBI #1458] Delta lake table columns won't show up in DBeaver.
[KYUUBI #1382] Sync the flink-provided profile in docs and testing suite comments
[KYUUBI #1481] Add z-order docs
[KYUUBI #1449] Fix when KyuubiServer initialize fail but not exit
[KYUUBI #1478] Docker-image-tool.sh awareness of incorrectly configured SPARK_HOME
[KYUUBI #1472] allow the engine not to self-terminate.
[KYUUBI #1473] Exit gracefully when engine idle
[KYUUBI #1471] Improve log if user specify the spark principal and keytab but not same with proxy user
[KYUUBI #1446] Decouple zookeeper from abstract ServiceDiscovery
[KYUUBI #1470] Git ignore kyuubi-server metrics
[KYUUBI #1467] Add The Doc of Intergrate Azure Blob Storage Based Delta Lake with Kyuubi
[KYUUBI #1469] Add cancel vote template and add release notes into vote template
[KYUUBI #1462] [DOC] Use relative paths instead of absolute paths
[KYUUBI #1466] Unify the guava version for kyuubi server and kyuubi beeline
[KYUUBI #1463] Enabled to preserve symbolic links when copying dist dir
[KYUUBI #1454] Add Doc About Deploy Kyuubi On Kubernetes
[KYUUBI #1459] Kyuubi Server Event Test change
[KYUUBI #1461] [DOC] Remove deprecated dist repo directory for deprecated release candidates
[KYUUBI #1396][K8S] Add docker image tools and replace Dockerfile
[KYUUBI #1443] MaxPartitionStrategy support file source based table.
[KYUUBI #1050] Add KyuubiServerInfoEvent
[KYUUBI #1439] Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl.
[KYUUBI #1442] Fix encode MySQLHandshakePacket UT