Skip to content

Commit

Permalink
docs: Update references (#372)
Browse files Browse the repository at this point in the history
* Initial commit

* Update xrefs

* Update xref
  • Loading branch information
Techassi authored Sep 21, 2023
1 parent b50bac1 commit 58bbd18
Show file tree
Hide file tree
Showing 3 changed files with 60 additions and 34 deletions.
13 changes: 8 additions & 5 deletions docs/modules/hive/pages/getting_started/first_steps.adoc
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
= First steps

After going through the xref:getting_started/installation.adoc[] section and having installed all the operators, you will now deploy a Hive metastore cluster and it's dependencies. Afterwards you can <<_verify_that_it_works, verify that it works>>.
After going through the xref:getting_started/installation.adoc[] section and having installed all the operators, you
will now deploy a Hive metastore cluster and it's dependencies. Afterwards you can
<<_verify_that_it_works, verify that it works>>.

== Setup

Several requirements should have already been installed in the xref:getting_started/installation.adoc[Installation guide]:

* xref:commons-operator::index.adoc[Commons Operator]
* xref:secret-operator::index.adoc[Secret Operator]
* xref:hive::index.adoc[Hive Operator]
* xref:commons-operator:index.adoc[Commons Operator]
* xref:secret-operator:index.adoc[Secret Operator]
* xref:hive:index.adoc[Hive Operator]
* PostgreSQL
* MinIO for S3

Expand Down Expand Up @@ -68,7 +70,8 @@ NAME READY AGE
hive-postgres-s3-metastore-default 1/1 48s
----

For further testing we recommend to use e.g. the python https://github.com/quintoandar/hive-metastore-client[hive-metastore-client] in order to e.g. create tables or connect a product like Trino via the xref:trino::index.adoc[Stackable Trino Operator].
For further testing we recommend to use e.g. the python https://github.com/quintoandar/hive-metastore-client[hive-metastore-client]
in order to e.g. create tables or connect a product like Trino via the xref:trino:index.adoc[Stackable Trino Operator].

== What's next

Expand Down
32 changes: 20 additions & 12 deletions docs/modules/hive/pages/getting_started/installation.adoc
Original file line number Diff line number Diff line change
@@ -1,23 +1,29 @@
= Installation

On this page you will install the Stackable Operator for Apache Hive and all required dependencies. For the installation of the dependencies and operators you can use Helm or stackablectl.
On this page you will install the Stackable Operator for Apache Hive and all required dependencies. For the installation
of the dependencies and operators you can use Helm or `stackablectl`.

The stackablectl command line tool is the recommended way to interact with operators and dependencies. Follow the xref:stackablectl::installation.adoc[installation steps] for your platform if you choose to work with stackablectl.
The `stackablectl` command line tool is the recommended way to interact with operators and dependencies. Follow the
xref:management:stackablectl:installation.adoc[installation steps] for your platform if you choose to work with
`stackablectl`.

== Dependencies

First you need to install MinIO and PostgreSQL instances for the Hive metastore. PostgreSQL is required as a database for Hive's metadata, and MinIO will be used as a data store, which the Hive metastore also needs access to.
First you need to install MinIO and PostgreSQL instances for the Hive metastore. PostgreSQL is required as a database
for Hive's metadata, and MinIO will be used as a data store, which the Hive metastore also needs access to.

There are two ways to install the dependencies:

1. Using xref:stackablectl::index.adoc[stackablectl]
2. Using Helm
. Using xref:management:stackablectl:index.adoc[stackablectl]
. Using Helm

WARNING: The dependency installations in this guide are only intended for testing and not suitable for production!

=== stackablectl

stackablectl was designed to install Stackable components, but its xref:stackablectl::commands/stack.adoc[Stacks] feature can also be used to install arbitrary Helm Charts. You can install MinIO and PostgreSQL using the Stacks feature as follows, but a simpler method via Helm is shown <<Helm, below>>.
`stackablectl` was designed to install Stackable components, but its xref:management:stackablectl:commands/stack.adoc[Stacks]
feature can also be used to install arbitrary Helm Charts. You can install MinIO and PostgreSQL using the Stacks feature
as follows, but a simpler method via Helm is shown <<Helm, below>>.

[source,bash]
----
Expand Down Expand Up @@ -54,16 +60,17 @@ include::example$getting_started/release.yaml[]

The release definition already references the required operators for this Getting Started guide.

Now call stackablectl and reference those two files:
Now call `stackablectl` and reference those two files:

[source,bash]
----
include::example$getting_started/getting_started.sh[tag=stackablectl-install-minio-postgres-stack]
----

This will install MinIO and PostgreSQL as defined in the Stacks, as well as the Operators. You can now skip the <<Stackable Operators>> step that follows next.
This will install MinIO and PostgreSQL as defined in the Stacks, as well as the Operators. You can now skip the
<<Stackable Operators>> step that follows next.

TIP: Consult the xref:stackablectl::quickstart.adoc[Quickstart] to learn more about how to use stackablectl.
TIP: Consult the xref:management:stackablectl:quickstart.adoc[Quickstart] to learn more about how to use `stackablectl`.

=== Helm

Expand All @@ -89,8 +96,8 @@ After the dependencies are deployed, you can start to install the operators.

There are 2 ways to run Stackable Operators:

1. Using xref:stackablectl::index.adoc[stackablectl]
2. Using Helm
. Using xref:management:stackablectl:index.adoc[stackablectl]
. Using Helm

=== stackablectl

Expand Down Expand Up @@ -126,7 +133,8 @@ Then install the Stackable operators:
include::example$getting_started/getting_started.sh[tag=helm-install-operators]
----

Helm will deploy the operators in a Kubernetes Deployment and apply the CRDs for the Apache Hive service (as well as the CRDs for the required operators). You are now ready to deploy the Apache Hive metastore in Kubernetes.
Helm will deploy the operators in a Kubernetes Deployment and apply the CRDs for the Apache Hive service (as well as the
CRDs for the required operators). You are now ready to deploy the Apache Hive metastore in Kubernetes.

== What's next

Expand Down
49 changes: 32 additions & 17 deletions docs/modules/hive/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,49 +2,64 @@
:description: The Stackable Operator for Apache Hive is a Kubernetes operator that can manage Apache Hive metastores. Learn about its features, resources, dependencies and demos, and see the list of supported Hive versions.
:keywords: Stackable Operator, Hadoop, Apache Hive, Kubernetes, k8s, operator, engineer, big data, metadata, storage, query

This is an operator for Kubernetes that can manage https://hive.apache.org[Apache Hive] metastores.
The Apache Hive metastore (HMS) was originally developed as part of Apache Hive. It stores information on the location of tables and partitions in file and blob storages such as xref:hdfs:index.adoc[Apache HDFS] and S3 and is now used by other tools besides Hive as well to access tables in files.
This Operator does not support deploying Hive itself, but xref:trino:index.adoc[Trino] is recommended as an alternative query engine.
This is an operator for Kubernetes that can manage https://hive.apache.org[Apache Hive] metastores. The Apache Hive
metastore (HMS) was originally developed as part of Apache Hive. It stores information on the location of tables and
partitions in file and blob storages such as xref:hdfs:index.adoc[Apache HDFS] and S3 and is now used by other tools
besides Hive as well to access tables in files. This Operator does not support deploying Hive itself, but
xref:trino:index.adoc[Trino] is recommended as an alternative query engine.

== Getting started

Follow the xref:getting_started/index.adoc[Getting started guide] which will guide you through installing the Stackable Hive Operator and its dependencies. It walks you through setting up a Hive metastore and connecting it to a demo Postgres database and a Minio instance to store data in.
Follow the xref:getting_started/index.adoc[Getting started guide] which will guide you through installing the Stackable
Hive Operator and its dependencies. It walks you through setting up a Hive metastore and connecting it to a demo
Postgres database and a Minio instance to store data in.

Afterwards you can consult the xref:usage-guide/index.adoc[] to learn more about tailoring your Hive metastore configuration to your needs, or have a look at the <<demos, demos>> for some example setups with either xref:trino:index.adoc[Trino] or xref:spark-k8s:index.adoc[Spark].
Afterwards you can consult the xref:usage-guide/index.adoc[] to learn more about tailoring your Hive metastore
configuration to your needs, or have a look at the <<demos, demos>> for some example setups with either
xref:trino:index.adoc[Trino] or xref:spark-k8s:index.adoc[Spark].

== Operator model

The Operator manages the _HiveCluster_ custom resource. The cluster implements a single `metastore` xref:home:concepts:roles-and-role-groups.adoc[role].
The Operator manages the _HiveCluster_ custom resource. The cluster implements a single `metastore`
xref:home:concepts:roles-and-role-groups.adoc[role].

image::hive_overview.drawio.svg[A diagram depicting the Kubernetes resources created by the Stackable Operator for Apache Hive]

For every role group the Operator creates a ConfigMap and StatefulSet which can have multiple replicas (Pods). Every role group is accessible through its own Service, and there is a Service for the whole cluster.
For every role group the Operator creates a ConfigMap and StatefulSet which can have multiple replicas (Pods). Every
role group is accessible through its own Service, and there is a Service for the whole cluster.

The Operator creates a xref:concepts:service_discovery.adoc[service discovery ConfigMap] for the Hive metastore instance. The discovery ConfigMap contains information on how to connect to the HMS.
The Operator creates a xref:concepts:service_discovery.adoc[service discovery ConfigMap] for the Hive metastore
instance. The discovery ConfigMap contains information on how to connect to the HMS.

== Dependencies

The Stackable Operator for Apache Hive depends on the Stackable xref:commons-operator:index.adoc[commons] and xref:secret-operator:index.adoc[secret] operators.
The Stackable Operator for Apache Hive depends on the Stackable xref:commons-operator:index.adoc[commons] and
xref:secret-operator:index.adoc[secret] operators.

== Required external component: An SQL database

The Hive metastore requires a database to store metadata.
Consult the xref:required-external-components.adoc[required external components page] for an overview of the supported databases and minimum supported versions.
The Hive metastore requires a database to store metadata. Consult the xref:required-external-components.adoc[required
external components page] for an overview of the supported databases and minimum supported versions.

== [[demos]]Demos

Three demos make use of the Hive metastore.

The xref:stackablectl::demos/spark-k8s-anomaly-detection-taxi-data.adoc[] and xref:stackablectl::demos/trino-taxi-data.adoc[] use the HMS to store metadata information about taxi data. The first demo then analyzes the data using xref:spark-k8s:index.adoc[Apache Spark] and the second one using xref:trino:index.adoc[Trino].
The xref:demos:spark-k8s-anomaly-detection-taxi-data.adoc[] and xref:demos:trino-taxi-data.adoc[] use the HMS to store
metadata information about taxi data. The first demo then analyzes the data using xref:spark-k8s:index.adoc[Apache Spark]
and the second one using xref:trino:index.adoc[Trino].

The xref:stackablectl::demos/data-lakehouse-iceberg-trino-spark.adoc[] demo is the biggest demo available. It uses both Spark and Trino for analysis.
The xref:demos:data-lakehouse-iceberg-trino-spark.adoc[] demo is the biggest demo available. It uses both Spark and
Trino for analysis.

== Why is the Hive query engine not supported?

Only the metastore is supported, not Hive itself.
There are several reasons why running Hive on Kubernetes may not be an optimal solution.
The most obvious reason is that Hive requires YARN as an execution framework, and YARN assumes much of the same role as Kubernetes - i.e. assigning resources.
For this reason we provide xref:trino:index.adoc[Trino] as a query engine in the Stackable Data Platform instead of Hive. Trino still uses the Hive Metastore, hence the inclusion of this operator as well. Trino should offer all the capabilities Hive offers including a lot of additional functionality, such as connections to other data sources.
Only the metastore is supported, not Hive itself. There are several reasons why running Hive on Kubernetes may not be an
optimal solution. The most obvious reason is that Hive requires YARN as an execution framework, and YARN assumes much of
the same role as Kubernetes - i.e. assigning resources. For this reason we provide xref:trino:index.adoc[Trino] as a
query engine in the Stackable Data Platform instead of Hive. Trino still uses the Hive Metastore, hence the inclusion of
this operator as well. Trino should offer all the capabilities Hive offers including a lot of additional functionality,
such as connections to other data sources.

Additionally, Tables in the HMS can also be accessed from xref:spark-k8s:index.adoc[Apache Spark].

Expand Down

0 comments on commit 58bbd18

Please sign in to comment.