Add Clickhouse Schemas

[marcus-snx]

• generate schemas based on event ABIs
• save schemas to file system (right now in /parquet-data/indexers/schemas/{network_name}/{protocol_name}/{table_name}.sql)

[Tburm]

@marcus-snx how do we use these schemas to define the raw tables? Does this get included somehow when we define sources in dbt? I wouldn't think we need the CREATE TABLE statements, but let me know where these will eventually go.

[marcus-snx]

@Tburm

@marcus-snx how do we use these schemas to define the raw tables? Does this get included somehow when we define sources in dbt? I wouldn't think we need the CREATE TABLE statements, but let me know where these will eventually go.

These schemas get saved locally and are called when we start the indexer, and the (empty) tables get created in ClickHouse. Then, when the listener detects new data, it will insert it into these tables.

[marcus-snx]

These tables are then used as sources in dbt.

[Tburm]

Won't this result in these CREATE TABLE statements not being written and checked into dbt though? I would think we want to do something like define these types in a sql file that gets stored, or in the sources.yml file.

For example, what happens if an ABI suddenly changes and has one more field? It will create a new sql file, but it won't run because the table already exists.

[marcus-snx]

Won't this result in these CREATE TABLE statements not being written and checked into dbt though? I would think we want to do something like define these types in a sql file that gets stored, or in the sources.yml file.

For example, what happens if an ABI suddenly changes and has one more field? It will create a new sql file, but it won't run because the table already exists.

We would need to handle such migrations (for example by checking if a generated schema differs from an existing schema), but I think this would still be handled outside dbt?