Add support for FX pools #29

gmbronco · 2024-01-11T08:57:36Z

This pull request adds:

support for FX pools by adding a parser for the subgraph data. In the mapper I'm extracting the structure we had in the pool-creator-service and splitting it into two parts:
- mapping data from subgraph to DB schema
- building prisma create object with all the nested tables.
  This helps with testing and adding updates, or upserts will be easier, since we have the clean data object.
I changed foreign keys constrains related to pools to allow for deleting a pool with all associated records.
added fishery for setting up object factories in tests

franzns

Overall nice changes! I wonder whether we should extract the json thing to it's own PR and make it for all pool types, maybe even including pool specific dynamic data.

franzns · 2024-01-11T09:47:15Z

modules/pool/subgraph-mapper.ts

+    nestedPools: { id: string; address: string }[],
+) => {
+    const type = mapSubgraphPoolTypeToPoolType(pool.poolType!);
+    const version = mapPoolTypeVersion(type, pool.poolTypeVersion!);


need to pass pool.poolType other wise it wont ever have phantonStable as pool type

franzns · 2024-01-11T09:48:16Z

modules/pool/subgraph-mapper.ts

+import { Chain, PrismaPoolType } from '@prisma/client';
+import { BalancerPoolFragment } from '../subgraphs/balancer-subgraph/generated/balancer-subgraph-types';
+import { AddressZero } from '@ethersproject/constants';
+import * as dataMappers from './pool-data';


Can we explicitely import the dataMappers instead of doing it with wildcards and exports?

absolutely, we have the option to individually import all the dataMappers. before I change that, let me ask what you think. my rationale for using a namespace is to keep it clear where these functions are coming from. especially around handling types. to make it work with individual imports i'd probably need to add a suffix, like fxDataMapper and it would end up quite similar to the current dataMappers.fx.

updated to have all the imported functions in the header.

franzns · 2024-01-11T09:57:57Z

modules/pool/subgraph-mapper.ts

+    const prismaPoolRecordWithAssociations = {
+        data: {
+            ...dbData.base,
+            data: dbData.data, // DISCUSS: simplify DB schema by migrating from individual tables to a JSON column with types enforced on read. And same with dynamic data.


Few thoughts:

instead if calling it data we should call it poolTypeSpecificData

If I understand this correctly, fxPool type data is only under data but linear, stable etc. speicifc data is under their own tables AND under data? I wonder if we should remove the data from this PR and export that change into its own PR, also removing the pool type specific tables.

Adding to above suggested to extract that change to its own PR: It could then also get rid of the pool type specific dynamic data tables and have a poolTypeSpecificDynamicData column (or table? thinking about update patterns)

Are we sure that we dont use/aggregate the data in the pool type specific tables? (or dynamic data tables?)

data column in now renamed to poolTypeSpecificData. Ready to start the work on migrating other tables in the new PR.

franzns · 2024-01-11T10:11:28Z

modules/pool/lib/pool-creator.service.ts

@@ -192,18 +132,13 @@ export class PoolCreatorService {
        await prismaBulkExecuteOperations(operations);
    }

-    private async createPoolRecord(pool: BalancerPoolFragment, blockNumber: number) {
-        const poolType = this.mapSubgraphPoolTypeToPoolType(pool.poolType || '');
+    private async createPoolRecord(


also needs to pass the nestedPools when called in syncNewPoolsFromSubgraph?

good point, fixed

franzns · 2024-01-11T12:05:42Z

modules/pool/subgraph-mapper.test.ts

+        expect(result.data.type).toBe('WEIGHTED');
+    });
+
+    it('should return correct object for stable pool', () => {


add test for phantom stable, will probably fail

added one and it was failing indeed.

modules/pool/pool-data/fx.ts

franzns · 2024-01-11T12:17:22Z

modules/pool/pool.prisma

@@ -65,7 +66,7 @@ model PrismaPoolLinearData {

    id                  String
    poolId              String
-    pool                PrismaPool          @relation(fields:[poolId, chain], references: [id, chain])
+    pool                PrismaPool          @relation(fields:[poolId, chain], references: [id, chain], onDelete: Cascade)


Since there are delete cascade now, maybe we can also simplify poolService.deletePool()?

true, cleaned it up a bit

franzns

Overall nice changes! I wonder whether we should extract the json thing to it's own PR and make it for all pool types, maybe even including pool specific dynamic data.

covered by syncAllPoolsFromSubgraph

covered by syncAllPools

to have references in the header

franzns

Shall we leave out the dynamicData for now?
Also, we are now saving the poolSpecificData for all pool types, but are not reading it. Only for fx pools. Are you planning to extract the 'jsob' from this PR or are you creating a new PR with that uses the data on the 'jsonb' column? I would not add it here, as this PR is already big enough..

franzns · 2024-01-15T15:11:37Z

modules/pool/subgraph-mapper.ts

+export type StableDynamicData = ReturnType<typeof stableDynamic>;
+export type LinearDynamicData = ReturnType<typeof linearDynamic>;


can we remove this? it's unused afaik.

franzns · 2024-01-15T15:11:41Z

modules/pool/subgraph-mapper.ts

 const dynamicMapper = {
-    STABLE: dataMappers.stableDynamic,
-    COMPOSABLE_STABLE: dataMappers.stableDynamic,
-    META_STABLE: dataMappers.stableDynamic,
-    LINEAR: dataMappers.linearDynamic,
+    STABLE: stableDynamic,
+    COMPOSABLE_STABLE: stableDynamic,
+    META_STABLE: stableDynamic,
+    LINEAR: linearDynamic,


can we remove this? it's unused afaik.

mapper is subgraph schema related, so more data we parse to DB schema the better. think about it as a layer. it makes it easy to compose what is needed for any pool syncing we would want to do.

we still read these data from subgraph in pool create jobs, once removed where would it go? just keep it blank until onchain calls fill it in?

gmbronco · 2024-01-18T10:06:02Z

I'd like to merge it as it is. Next steps:

migrate other pool types data tables to jsonb
and dynamic data to jsonb

Perhaps in 2 PRs, to keep it easy on us.

franzns reviewed Jan 11, 2024

View reviewed changes

gmbronco force-pushed the fix-fx branch from 28bcb88 to 90ebc90 Compare January 12, 2024 18:04

gmbronco added 20 commits January 12, 2024 19:06

docker-compose setup

956a895

more tests

dcfd0d2

allow deleting pools with associations

ee6e0e7

pool defaults for 24h volume and fees

001d1df

pools data json column

9d5c936

add fx data to subgraph

b5af3e5

subgraph to DB mappers

827943c

FX pools parsing in SOR

7ab4104

FX pools in the graph schema

64d756b

subgraph pool to DB update

5d63418

obsolete poolSyncAllPoolTypesVersions

65d9451

include pool tokens in subgraph sync

6aa8aac

obsolete poolReloadPoolTokenIndexes

9c533b2

covered by syncAllPoolsFromSubgraph

obsolete poolReloadPoolNestedTokens

5497a1d

covered by syncAllPoolsFromSubgraph

obsolete poolSync mutations

d5fb8e8

covered by syncAllPools

fix pool type mapping for StablePhantom

d6e71f0

add nestedPool list when creating new pools

63fd0b2

renaming pool.data to pool.poolTypeSpecificData

2dee404

import all of the functions directly

64ca22b

to have references in the header

delete cleanup based on constraints

48c1f90

gmbronco force-pushed the fix-fx branch from 90ebc90 to 48c1f90 Compare January 12, 2024 18:07

franzns reviewed Jan 15, 2024

View reviewed changes

franzns added 3 commits January 18, 2024 13:54

renaming

00fd3e0

rename column

8fb588a

prisma schema

9eb0ea2

franzns merged commit bf17d89 into v3-canary Jan 18, 2024
1 check passed

gmbronco deleted the fix-fx branch March 11, 2024 17:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for FX pools #29

Add support for FX pools #29

gmbronco commented Jan 11, 2024

franzns left a comment

franzns Jan 11, 2024

gmbronco Jan 12, 2024

franzns Jan 11, 2024

gmbronco Jan 11, 2024

gmbronco Jan 12, 2024

franzns Jan 11, 2024

gmbronco Jan 12, 2024

franzns Jan 11, 2024

gmbronco Jan 12, 2024

franzns Jan 11, 2024

gmbronco Jan 12, 2024

franzns Jan 11, 2024

gmbronco Jan 12, 2024

franzns left a comment

franzns left a comment

franzns Jan 15, 2024

franzns Jan 15, 2024

gmbronco Jan 18, 2024

gmbronco commented Jan 18, 2024

		export type StableDynamicData = ReturnType<typeof stableDynamic>;
		export type LinearDynamicData = ReturnType<typeof linearDynamic>;

Add support for FX pools #29

Add support for FX pools #29

Conversation

gmbronco commented Jan 11, 2024

franzns left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

franzns left a comment

Choose a reason for hiding this comment

franzns left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gmbronco commented Jan 18, 2024