Skip to content

Dealing with Rate Differences from Multiple Sources

Jump to bottom
martiliones edited this page Jun 20, 2024 · 11 revisions

Algorithm

In cases of significant data differences, Currencyinfo selects accurate rates using the following algorithm:

Breaking Down Rates into Groups

The app begins with searching for significant price differences but doesn't immediately block a rate if such differences are found, allowing for alternative handling, such as accepting a rate if 90% of sources agree on it. Instead, the app defines boundaries by creating groups of prices, with the rateDifferencePercentThreshold setting the acceptable range of percentage difference between the lowest and highest prices within each group.

You can edit your config file as follows to set the threshold to 30%:

{ // config.jsonc
  "rateDifferencePercentThreshold": 30,
  // ...
}

For better understanding, take a look at the example where we get the made-up rate prices from 7 different sources with rateDifferencePercentThreshold set to 25% (default value):

flowchart TD
    R2[0.025] --> Group2
    R1[0.02] --> Group2
    R3[0.03] --> Group1
    R4[0.031] --> Group1
    R5[0.04] --> Group3
    R6[0.044] --> Group3
    R7[0.05] --> Group3

    R2 --> Group1

    subgraph Group2["0.02-0.025"]
    end

    subgraph Group1["0.025-0.031"]
    end

    subgraph Group3["0.04-0.05"]
    end

    classDef green fill:#9EE09E,stroke:#4CAF50,color:#000000
    classDef red fill:#FFCDD2,stroke:#F44336,color:#000000

    class Group2,Group1,Group3 green
Loading

The example illustrates how prices are divided based on the rateDifferencePercentThreshold:

  1. Prices 0.02 and 0.025 are in the same group because their 22% difference is within the acceptable deviation. However, 0.02 and 0.03, with a 40% difference, cannot be in the same group together.
  2. Since 0.025 and 0.03 have only an 18% difference, they can form a second group, which also includes 0.031 because its difference from 0.025 is within the threshold.
  3. The highest price in the second group (0.031) and the lowest in the third group (0.04) have a 25.35% difference, so the groups don't share a common price.

Note

The same price from the same source can be present in 2 groups simultaneously.

Choosing Authoritative Group

After identifying significant price changes, you need to determine if the differing sources are important. For example, if you have 100 sources and one gives false or outdated data, you would usually ignore that source, even if its price differs significantly. This example results in two groups: one with 99 reliable sources and one with the wrong source, making the first group represent 99% of the sources.

The groupPercentage parameter sets the percentage of sources needed in a group to consider it trustworthy. For instance, if you set groupPercentage to 90%, the group must include at least 90% of the available sources in order to save the rate.

Tip

If you want Currencyinfo to notify you about any significant changes, set groupPercentage to 100.

You can also use weights to prioritize certain sources over others. A weight is a relative value that shows how much influence a source has. For example, if you have four sources with weights 20, 10, 10, and 10, the first source counts as much as two sources. In another example, if 9 out of 10 sources have a weight of 100 and the last source has a weight of 1, the last source won't affect the trustworthy group selection with groupPercentage set to 90%, but it will still be included in the group and participate in calculating the final rate.

{ // config.jsonc
  "groupPercentage": 90, // The group must have a weight of at least 90% sources
  "currency_api": {
    // ...
    "weight": 20 // Set custom weight for Currency API
  },
  // ...
}

Let's revisit the earlier illustration. The second and third groups have the same number of sources, so with the same default weights for every source, it's unclear which group Currencyinfo should choose. To resolve this, let's assign some weights to each rate price:

flowchart TD
    W2["Weight 20"] --> R2
    W1["Weight 10"] --> R1
    W3["Weight 50"] --> R3
    W4["Weight 60"] --> R4
    W5["Weight 10"] --> R5
    W6["Weight 10"] --> R6
    W7["Weight 20"] --> R7

    R2[0.025] --> W8
    R1[0.02] --> W8
    R3[0.03] --> W9
    R4[0.031] --> W9
    R5[0.04] --> W10
    R6[0.044] --> W10
    R7[0.05] --> W10

    R2 --> W9

    W8["Group Weight 30 (16%)"]
    W9["Group Weight 130 (65%)"]
    W10["Group Weight 40 (22%)"]

    W11["Total weight: 200 (100%)"]

    W8 --> W11 
    W9 --> W11 
    W10 --> W11 

    classDef green fill:#9EE09E,stroke:#4CAF50,color:#000000
    classDef orange fill:#ffdccd,stroke:#f48f36,color:#000000
    classDef red fill:#ffcdea,stroke:#f43685,color:#000000
    classDef brightGreen fill:#cdffd4,stroke:#a5f436,color:#000000
    classDef white fill:#ffffff,stroke:#000000,color:#000000
    classDef transparent fill:#00000000,stroke: #00000000

    class W9 brightGreen
    class R2,R3,R4 green
    class W1,W2,W3,W4,W5,W6,W7 transparent
    class W8,W10 red
    class W11 white
Loading

Calculating the Final Rate

When multiple sources return different but close rates, we still have to decide which one to save. Currencyinfo applies the selected strategy exclusively to the rates within the selected group. You can set strategy to one of the options to determine the merge strategy:

Option Description
avg Saves the average price from all the available sources.
min Saves the minimal price from the available sources.
max Opposite to min. Saves the maximum price from the available sources.
priority Saves the price from a higher source in the priorities list.
weight Saves the price from the source with the highest weight value.

Example with custom weights and different strategies:

  flowchart TD
    C((ADM/USD))
    C -->|"Coinmarketcap (Weight 10)"| D[0.02] --> Group2
    C -->|"Coingecko (Weight 50)"| F[0.025] --> Group2
    C -->|"CryptoCompare (Weight 20)"| E[0.03] --> Group1

    F --> Group1

    subgraph Group2["0.02-0.025 (Weight 60)"]
    G[0.02]
    H[0.025]
    end

    subgraph Group1["0.025-0.03 (Weight 30)"]
    K[0.025]
    L[0.03]
    end

    Group2 --> R(avg: 0.0225)
    Group2 --> O(max: 0.025)
    Group2 --> O2(min: 0.02)
    Group2 --> O3(weight: 0.02)

    classDef green fill:#9EE09E,stroke:#4CAF50,color:#000000
    classDef red fill:#FFCDD2,stroke:#F44336,color:#000000

    class Group2 green
    class Group1 red
Loading

If you want to use weights only for choosing the authoritative group and have a list of prioritized sources independently through priorities, you can use the priority strategy:

{ // config.jsonc
  "strategy": "priority",
  "priorities": [
    "CurrencyApi",
    "ExchangeRateHost",
    "MOEX",
    "Coinmarketcap",
    "CryptoCompare",
    "Coingecko"
  ],
  // ...
}

If you use the configuration above for the previous example, then Currency would choose Coinmarketcap because even though it weighs less, it is higher in the priorities list.

Minimum number of sources

To avoid situations where there are too few sources to determine the accuracy of the rate, specify minSources in your settings. For example, if you don't want a pair to be saved unless it has been fetched from at least three different sources and does not significantly differ between them:

{ // config.jsonc
  "minSources": 3,
  // ...
}

Warning

If a rate is configured to be fetched from fewer sources than the minSources setting, it will only be saved if all those sources successfully return information about the coin. Currencyinfo will warn about such coins in the logs.

Example

Use case

A user wants to configure Currencyinfo to display the most competitive rate available. The rate should not be updated if there is more than a 30% difference between prices from different sources. However, if this 30% difference is detected, it can be accepted if it is from a source with low trust due to slower update times. Additionally, the rate should only be saved if fetched from all three sources.

Configuration

{
  "rateDifferencePercentThreshold": 30,
  "minSources": 3,
  "strategy": "min",
  "groupPercentage": 90,
  "SlowAPI": {
    // ...
    "weight": 5
  },
  "FastAPI1": {
    // ...
    "weight": 100
  },
  "FastAPI2": {
    // ...
    "weight": 100
  }
}
  • rateDifferencePercentThreshold - 30 - The maximum acceptable difference between trusted sources.
  • groupPercentage - 90 - The authoritative group must have a weight of 90% of the total weight of all available sources. This means the group can only be authoritative if it contains both FastAPI1 and FastAPI2.
  • strategy - min - Save the minimal rate from the authoritative group.
  • minSources - 3 - The rate should only be saved if it is fetched from all three sources.
Clone this wiki locally