Updating docs with references to kind property (#4712)

* Updating docs with references to kind property

Also adding more details in reference sections along with a page around structuring your project

* Removing -- @kind: model from examples

* Updating kind to type

Plus other small fixes

docs/docs/build/connect/connect.md

@@ -50,6 +50,7 @@ When you add a source using the UI, a code definition will automatically be crea
 In your Rill project directory, create a `source_name.yaml` file in the `sources` directory with the following content:
 
 ```yaml
+type: source
 connector: local_file
 path: /path/to/local/data.csv
 ```
@@ -81,6 +82,7 @@ When you add a source using the UI or CLI, a code definition will automatically
 For example, to create a remote http(s) source, create a `source_name.yaml` file in the `sources` directory with the following contents:
 
 ```yaml
+type: source
 connector: https
 uri: https://data.example.org/path/to/file.parquet
 ```

docs/docs/build/connect/glob-patterns.md

@@ -88,6 +88,7 @@ Schema relaxation is enabled by default. Set `ingest.allow_schema_relaxation: fa
 
 For example:
 ```yaml
+type: source
 connector: "gcs"
 uri: "gs://my-bucket/y=2023/m=*/d=0[1-7]/H=01/*.parquet" 
 

docs/docs/build/connect/multiple-connectors.md

@@ -35,13 +35,15 @@ For example, let's say we had `sourceA` and `sourceB` defined that point to diff
 
 **sourceA.yaml**:
 ```yaml
+type: source
 connector: "snowflake-a"
 dsn: "user:password@account_identifier/DB_A/SCHEMA_A?warehouse=COMPUTE_WH&role=ACCOUNTADMIN"
 sql: "select * from table_A"
 ```
 
 **sourceB.yaml**:
 ```yaml
+type: source
 connector: "snowflake-b"
 dsn: "user:password@account_identifier/DB_B/SCHEMA_B?warehouse=COMPUTE_WH&role=ACCOUNTADMIN"
 sql: "select * from table_B"

docs/docs/build/dashboards/customize.md

@@ -69,10 +69,10 @@ A list of measures that should be visible by default. Operates the same as the `
 
 In your Rill project directory, create a `<theme_name>.yaml` file in any directory. Rill will automatically ingest the theme next time you run `rill start` or deploy to Rill Cloud and change the color scheme of your dashboard.
 
-First, create the yaml file as below. In this example, the charts and hover in Rill will change to Plum while spinners will change to Violet.
+First, create the YAML file as below. In this example, the charts and hover in Rill will change to Plum while spinners will change to Violet.
 
 ```yaml
-kind: theme
+type: theme
 colors:
   primary: plum
   secondary: violet 
@@ -97,8 +97,9 @@ Formatting for individual measures on a dashboard can be controlled through the
 Instead of displaying a numeric measure in USD currency, the euro currency can be used instead through `format_preset`. In the following example `dashboard.yaml`, the `Total Revenue` measure will be displayed using euros in the resulting dashboard.
 
 ```yaml
+type: metrics_view
 title: example_dashboard
-mode: example_model
+model: example_model
 measures:
   - label: "Total Revenue"
     expression: SUM(revenue)

docs/docs/build/dashboards/dashboards.md

@@ -51,6 +51,7 @@ When you add a metrics definition using the UI, a code definition will automatic
 In your Rill project directory, after the `dashboard_name.yaml` file is created in the `dashboards` directory, its configuration or definition can be updated as needed by updating the YAML file directly, using the following template as an example:
 
 ```yaml
+type: metrics_view
 model: model_name
 title: Dashboard name
 default_time_range: ""

docs/docs/build/models/environments.md

@@ -63,6 +63,7 @@ rill start --env <name_of_environment>
 Environment overrides can be applied to source properties in the [YAML configuration](/reference/project-files/sources.md) of a source. For example, let's say that you have a [S3](/reference/connectors/s3.md) source defined but you only wanted to read from a particular month partition during local development and make sure that [source refreshes](/build/connect/source-refresh.md) are only applied _in production_ (i.e. when a project is deployed on Rill Cloud). Then, in your `source.yaml` file, you can define it as:
 
 ```yaml
+type: source
 connector: s3
 path: s3://path/to/bucket/*.parquet
 env:
@@ -102,6 +103,7 @@ Similar to the example in the previous section, let's say we had a S3 source def
 Now, for your `source.yaml` file (and combined with templating), you could do something like:
 
 ```yaml
+type: source
 connector: "duckdb"
 sql: SELECT * FROM read_parquet('s3://path/to/bucket/*.parquet') {{ if dev }} where updated_at >= '2024-01-01' AND updated_at < '2024-02-01' {{ end }}
 ```

docs/docs/build/structure/_category_.yml

@@ -0,0 +1,4 @@
+position: 50
+label: Structure Project
+collapsible: true
+collapsed: true

docs/docs/build/structure/structure.md

@@ -0,0 +1,42 @@
+---
+title: Structure your project
+sidebar_label: Structure Project
+sidebar_position: 00
+---
+
+After creating your initial set of sources, models, and dashboards, you may have noticed the following _native_ folders that exist in your Rill project directory:
+- [Sources](/reference/project-files/sources)
+- [Models](/reference/project-files/models)
+- [Dashboards](/reference/project-files/dashboards)
+
+By default, any new sources, models, and dashboards will be created in their respective native folder. However, this does not necessarily have to be the case and Rill Developer allows a flexible project directory structure, including nested folders or even storing objects in non-native folders. This is a powerful feature that allows you as a developer to organize your project to meet your specific team needs.
+
+## Adding new resources or parent folders
+
+Within Rill Developer, from the left-hand side (file explorer), you should be able to click on the `Add` button to add a new base resource, such as a new source, model, or dashboard. Furthermore, you will also have the ability to add a new parent folder to store groups of resources (can be mixed). If you choose to add a new folder, you should see the folder structure reflected if you check the project directory via the CLI. 
+
+![Adding objects](/img/build/structure/adding-objects.png)
+
+:::warning Make sure to include the `type` property
+
+For backward-compatibility purposes, any resource that belongs in the `sources`, `models`, and `dashboards` native folders are assumed to be a source, model, or dashboard respectively (including nested folders that belong within a native folder). 
+
+However, if you'd like to create a resource outside one of these native folders, make sure to include the `type` property in the resource definition or Rill will not be able to properly resolve the resource type! For more details, see our [reference documentation](/reference/project-files/index.md).
+
+:::
+
+## Moving resources within your project
+
+From the UI, within the file explorer, you should be able to drag resources / objects around and move them from / to folders as necessary. 
+
+:::info Using the CLI
+
+For developers who prefer to use the CLI, the project structure can still be controlled or adjusted directly via the CLI and/or using your preferred IDE of choice (e.g. VSCode).
+
+:::
+
+## Adding a nested folder
+
+Rather than creating a parent folder and moving it into another folder manually, there is a shortcut to create a nested folder by directly hovering over an existing folder in the file explorer > click on the "triple dots" > `New folder`.
+
+![Adding nested folder](/img/build/structure/adding-nested-folder.png)

docs/docs/deploy/performance.md

@@ -50,6 +50,7 @@ There are a few ways to achieve this:
 
 There are different ways this can be achieved and the method also depends heavily on the datasource being used. For example, assuming we had a [S3 source](/reference/connectors/s3.md) that was well partitioned by year and month (and written into a partitioned bucket), the recommended pattern would be to leverage the `path` [source property](/reference/project-files/sources.md) and a [glob pattern](/build/connect/glob-patterns.md) to limit the size of the ingestion in your development environment. Something like (as your `source.yaml`):
 ```yaml
+type: source
 connector: s3
 path: s3://bucket/path/**/*.parquet
 env:
@@ -59,6 +60,7 @@ env:
 
 By leveraging the [environment YAML syntax](/build/models/environments.md), this ensures that only data from December 2023 will be read in from this S3 source when using Rill Developer locally while the full range of data will still be used in production (on Rill Cloud). However, if this data was **not** partitioned, then we could simply leverage DuckDB's ability to read from S3 files directly and _apply a filter post-download_ on the source. Taking this same example and using some [templating](templating.md), the `source.yaml` could be rewritten to something like the following:
 ```yaml
+type: source
 connector: "duckdb"
 sql: SELECT * FROM read_parquet('s3://bucket/path/*.parquet') {{ if dev }} where timestamp_column >= '2023-12-01' AND timestamp_column < '2024-01-01' {{ end }}
 ```

docs/docs/deploy/templating.md

@@ -64,7 +64,8 @@ Let's say that we have a [Snowflake](/reference/connectors/snowflake.md) source
 
 In this hypothetical scenario, our `source.yaml` might look something like the following:
 ```yaml
-type: "snowflake"
+type: source
+connector: "snowflake"
 sql: "select * from <table_name> {{if dev}} limit 1 {{end}}"
 dsn: "{{if dev}}SUPPORT_TEST{{else}}PROD_USER{{end}}@<account_identifier>/<database>/<schema>?warehouse=<warehouse>&role=<role>N&authenticator=SNOWFLAKE_JWT&privateKey=..."
 ```
@@ -76,7 +77,8 @@ dsn: "{{if dev}}SUPPORT_TEST{{else}}PROD_USER{{end}}@<account_identifier>/<datab
 Let's say that we have a [GCS](/reference/connectors/gcs.md) source created where Rill is reading in some CSV data (in this case we have some sample [Citi Bike trip data](https://citibikenyc.com/system-data) loaded onto both a "test" and "prod" GCS bucket). In this case, let's imagine that we want to connect to this "test" bucket for local development purposes but we want to make sure that our production data hosted on our "prod" bucket is what's being used to power this same source once the project has been deployed to Rill Cloud. In such a scenario, our `source.yaml` might look like:
 
 ```yaml
-type: "duckdb"
+type: source
+connector: "duckdb"
 sql: "select * from read_csv('gs://{{if dev}}<test_bucket>{{else}}<prod_bucket>{{end}}/201306-citibike-tripdata.csv', auto_detect=true, ignore_errors=1, header=true)"
 ```
 
@@ -90,6 +92,7 @@ In another example, let's say we had a [S3](/reference/connectors/s3.md) source
 Fortunately, we can leverage DuckDB's ability to read from S3 files directly and _apply a filter post-download_ using templating logic in the SQL. In this case, because there is an existing `updated_at` timestamp column, we can use it to filter and retrieve only one week's worth of data. For example, our `source.yaml` file may end up looking something like:
 
 ```yaml
+type: source
 connector: "duckdb"
 sql: SELECT * FROM read_parquet('s3://bucket/path/*.parquet') {{ if dev }} where updated_at >= '2024-03-01' AND updated_at < '2024-03-07' {{ end }}
 ```

docs/docs/integrate/custom-apis/index.md

@@ -16,7 +16,7 @@ we support two types of custom APIs:
     write queries against a [model](/build/models/models.md) that you have created. It should have the following structure:
 
     ```yaml
-    kind: api
+    type: api
     sql: SELECT abc FROM my_table
     ```
     where `my_table` is your model name. Read more details about [SQL apis](./sql-api.md).
@@ -25,7 +25,7 @@ we support two types of custom APIs:
 It should have the following structure:
 
     ```yaml
-    kind: api
+    type: api
     metrics_sql: SELECT dimension, AGGREGATE(measure) FROM my_metrics GROUP BY dimension
     ```
     where `my_metrics` is your metrics view name, `measure` is a custom metrics that you have defined. 

docs/docs/integrate/custom-apis/metrics-sql-api.md

@@ -9,7 +9,7 @@ Metrics SQL API allows you to query data from your predefined [metrics view](/bu
 Example:
 
 ```yaml
-kind: api
+type: api
 metrics_sql: SELECT dimension, AGGREGATE(measure) FROM my_metrics GROUP BY dimension
 ```
 

docs/docs/integrate/custom-apis/metrics-sql.md

@@ -17,6 +17,7 @@ Metrics SQL transforms queries that reference `dimensions` and `measures` within
 
 Consider a metrics view configured as follows:
 ```
+type: metrics_view
 title: Ad Bids
 model: ad_bids
 timeseries: timestamp

docs/docs/integrate/custom-apis/sql-api.md

@@ -9,7 +9,7 @@ against a [model](/build/models/models.md) or [source](../../reference/project-f
 It should have the following structure:
 
 ```yaml
-kind: api
+type: api
 sql: SELECT abc FROM my_table
 ```
 
@@ -29,7 +29,7 @@ See integration docs [here](/integrate/custom-api.md) to learn how to these are
 
 Assume a API endpoint defined as `my-api.yaml`:
 ```yaml
-kind: api
+type: api
 sql: |
   SELECT count("measure")
     {{ if ( .user.admin ) }} ,dim  {{ end }} 

docs/docs/reference/connectors/googlesheets.md

@@ -11,7 +11,8 @@ sidebar_position: 13
 Rill has the ability to read from any http(s) URL endpoint that produces a valid data file in a supported format. For example, to bring in data from [Google Sheets](https://www.google.com/sheets/about/) as a CSV file directly into Rill as a source ([leveraging the direct download link syntax](https://www.highviewapps.com/blog/how-to-create-a-csv-or-excel-direct-download-link-in-google-sheets/)), you can create a `source_name.yaml` file in the `sources` directory of your Rill project directory with the following content:
 
 ```yaml
-type: "duckdb"
+type: source
+connector: "duckdb"
 path: "select * from read_csv_auto('https://docs.google.com/spreadsheets/d/<SPREADSHEET_ID>/export?format=csv&gid=<SHEET_ID>', normalize_names=True)"
 ```
 

docs/docs/reference/olap-engines/clickhouse.md

@@ -67,9 +67,7 @@ If you would like to connect Rill to an existing ClickHouse instance, please don
 You'll also need to update the `olap_connector` property in your project's `rill.yaml` to change the default OLAP engine to ClickHouse:
 
 ```yaml
-
 olap_connector: clickhouse
-
 ```
 
 :::note

docs/docs/reference/olap-engines/multiple-olap.md

@@ -72,6 +72,7 @@ For more information about available configurations for `rill.yaml`, please see
 For each dashboard that is using a separate OLAP engine (than the default), you'll want to set the `connector` and `table` properties in the underlying [dashboard YAML](../project-files/dashboards.md) configuration to the OLAP engine and corresponding [external table](build/olap/olap.md#external-olap-tables) that exists in your OLAP store respectively.
 
 ```yaml
+type: metrics_view
 title: <dashboard_name>
 connector: <olap_engine>
 table: <external_table_in_olap>

docs/docs/reference/project-files/apis.md

@@ -9,10 +9,10 @@ See comprehensive documentation on how to define and use [custom APIs](/integrat
 
 ## Properties
 
-_**`kind`**_ — should always be `api` _(required)_
+_**`type`**_ — Refers to the resource type and must be `api` _(required)_.
 
 Either one of the following:
 
-- _**`sql`**_ — General SQL query referring a [model](/build/models/models.md) _(required)_
+- _**`sql`**_ — General SQL query referring a [model](/build/models/models.md) _(required)_.
 
-- _**`metrics_sql`**_ — SQL query referring metrics definition and dimensions defined in the [metrics view](/build/dashboards/dashboards.md) _(required)_
+- _**`metrics_sql`**_ — SQL query referring metrics definition and dimensions defined in the [metrics view](/build/dashboards/dashboards.md) _(required)_.