Blog

CADLI Part 3: Understanding Latest Tick Conversions

This blog explores the inner workings of the CADLI Latest Tick Data API, highlighting our innovative methodologies and the sophisticated technology behind them. From direct index pairs to complex synthetic conversions, we explore how CADLI navigates the intricacies of digital asset conversions, offering precision and clarity in a market known for its volatility and complexity.

  • February 13, 2024
  • Vlad Cealicu

In the digital asset industry, the need for accurate and reliable conversions of cryptocurrencies to stable and globally recognised currencies like the US Dollar is paramount. Enter CADLI, our cutting-edge index, designed to revolutionise the way digital assets are understood and valued. CADLI not only facilitates price discovery for a wide range of assets, both liquid and illiquid, but also plays a crucial role in assessing their value in USD terms.

‍

This blog post delves into the inner workings of the CADLI Latest Tick Data API, highlighting our innovative methodologies and the sophisticated technology behind them. From direct index pairs to complex synthetic conversions, we explore how CADLI navigates the intricacies of digital asset conversions, offering precision and clarity in a market known for its volatility and complexity.

‍

With CADLI, we offer three distinct methods for providing data on requested instruments or pairs:

‍

  • Direct Index Pairs: For pairs such as BTC-USD, we directly provide the data as calculated by the CADLI index. This method is straightforward and efficient, offering users immediate access to the calculated USD values of these direct pairs.
  • Synthetic Inverted Index Pairs: In cases like USD-BTC, we employ a synthetic inversion method. This involves reversing the direct pair values and accordingly adjusting the base and quote volumes. This method allows users to view the value from the perspective of the inverted pair, providing a different angle of analysis.

Here is an example of the invert methodology in action, this is the response of the API Latest Tick endpoint for the BTC-USD and the USD-BTC instruments:

{
 "Data": {
   "BTC-USD": {
     "TYPE": "266",
     "MARKET": "cadli",
     "INSTRUMENT": "BTC-USD",
     "CCSEQ": 73519761,
     "VALUE": 37763.1644092693,
     "VALUE_FLAG": "DOWN",
     "VALUE_LAST_UPDATE_TS": 1700960977,
     "VALUE_LAST_UPDATE_TS_NS": 687000000,
     //......
     "CURRENT_WEEK_VOLUME": 1880114.02261408,
     "CURRENT_WEEK_QUOTE_VOLUME": 70033703406.5464,
     "CURRENT_WEEK_VOLUME_TOP_TIER": 1017268.86560116,
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER": 37885981495.9981,
     "CURRENT_WEEK_VOLUME_DIRECT": 211850.903249879,
     "CURRENT_WEEK_QUOTE_VOLUME_DIRECT": 7889741168.56429,
     "CURRENT_WEEK_VOLUME_TOP_TIER_DIRECT": 170674.75439911,
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER_DIRECT": 6356614799.099,
     "CURRENT_WEEK_OPEN": 37388.3505763348,
     "CURRENT_WEEK_HIGH": 38407.8791777158,
     "CURRENT_WEEK_LOW": 35702.7363290693,
     "CURRENT_WEEK_TOTAL_INDEX_UPDATES": 8222612,
     "CURRENT_WEEK_CHANGE": 374.8138329345,
     "CURRENT_WEEK_CHANGE_PERCENTAGE": 1.00248828085971
     //......
   },
   "USD-BTC": {
     "TYPE": "987", //987 is a synthetic instrument calculated though INVERSION
     "MARKET": "cadli",
     "INSTRUMENT": "USD-BTC",
     "CCSEQ": 73519761, // same CCSEQ as above
     "VALUE": 0.0000264808316687185, // 1/BTC-USD.VALUE
     "VALUE_FLAG": "UP", // opposite of BTC-USD.VALUE_FLAG
     "VALUE_LAST_UPDATE_TS": 1700960977,
     "VALUE_LAST_UPDATE_TS_NS": 687000000,
     //......
     "CURRENT_WEEK_VOLUME": 70033703406.5464, //BTC-USD.CURRENT_WEEK_QUOTE_VOLUME
     "CURRENT_WEEK_QUOTE_VOLUME": 1880114.02261408, //BTC-USD.CURRENT_WEEK_VOLUME
     "CURRENT_WEEK_VOLUME_TOP_TIER": 37885981495.9981, //BTC-USD.CURRENT_WEEK_QUOTE_VOLUME
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER": 1017268.86560116, //BTC-USD.CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER
     "CURRENT_WEEK_VOLUME_DIRECT": 7889741168.56429, //BTC-USD.CURRENT_WEEK_QUOTE_VOLUME_DIRECT
     "CURRENT_WEEK_QUOTE_VOLUME_DIRECT": 211850.903249879, //BTC-USD.CURRENT_WEEK_VOLUME_DIRECT
     "CURRENT_WEEK_VOLUME_TOP_TIER_DIRECT": 6356614799.099, //BTC-USD.CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER_DIRECT
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER_DIRECT": 170674.75439911, //BTC-USD.CURRENT_WEEK_VOLUME_TOP_TIER_DIRECT
     "CURRENT_WEEK_OPEN": 0.0000267462989028715, // 1/BTC-USD.CURRENT_WEEK_OPEN
     "CURRENT_WEEK_HIGH": 0.0000280090576470968, // 1/BTC-USD.CURRENT_WEEK_LOW
     "CURRENT_WEEK_LOW": 0.0000260363243534727, // 1/BTC-USD.CURRENT_WEEK_HIGH
     "CURRENT_WEEK_TOTAL_INDEX_UPDATES": 8222612,
     "CURRENT_WEEK_CHANGE": -2.65467234152998e-7,
     "CURRENT_WEEK_CHANGE_PERCENTAGE": -0.992538201704226
     //......
   }
 },
 "Err": {}
}

  • Synthetic Divided Index Pairs: For pairs that involve forex conversions, such as BTC-EUR, or two-index conversions like BTC-ETH, we use a synthetic division approach. This method involves dividing the values of one index pair by another, allowing for the creation of new synthetic pairs that might not directly trade in the market. It’s particularly useful for estimating values in different currencies or for creating pairs between two digital assets. This is the most complicated conversion, it needs to take into account not only the average prices over certain periods of BTC-USD and ETH-USD to correctly calculate OPEN, HIGH, LOW values for all the periods, but it also needs to take into account the ‘age’ of the market.

Our methodology for calculating historical metrics in synthetic pairs hinges on considering the “age” of the constituent instruments. This strategy not only ensures accuracy and consistency in our calculations but also reduces computational costs, a crucial factor in handling a vast array of instruments.

Given the sheer volume of assets and forex pairs we cover in CADLI— at the time of writing this blog post approximately 10,000 assets and 200 forex pairs — the total potential combinations of trading and synthetic instruments reach a staggering 104,029,800. The introduction of Real World Assets on-chain and the quadratic growth of possible combination instruments necessitates a solution for rapid estimations of historical metrics on derived synthetic instruments. This is especially critical given our current API usage of 25 billion data requests per month, from over 50 million unique users, all while maintaining low cache times (under 10 seconds for each instrument).

Our approach is designed to offer accurate estimations of historical metrics on tick data, tailored to the needs of most free and premium users. We reserve the most precise calculations for our enterprise users, which involve significant on-the-fly computation and extra data lookups, while providing reliable and quick estimations for our broader user base. This balance between precision and efficiency is key in our ongoing effort to deliver valuable data insights across all levels of service.

‍

Calculating Volumes for Synthetic Instruments

Calculating synthetic volumes is a task that demands a nuanced and sophisticated approach. Take the BTC-ETH or ETH-BTC instruments, even though they are derived from the same trading pairs (BTC-USD and ETH-USD) they differ in their order and present unique challenges in accurately and consistently representing their historical market dynamics.

When deciding how to tackle the issue of estimated historical volumes based on limited access to data we investigated two methodologies:

Methodology 1: Using the Latest Price — Applies the latest price to calculate synthetic historical base and quote volumes (CURRENT_DAY, WEEK, MONTH, YEAR, MOVING_24H, MOVING_7_DAY, 30_DAY, 90_DAY, 180_DAY, 356_DAY and LIFETIME). While this seems to be the easiest option, it does not accurately capture historical volumes during periods of market volatility (it’s synthetic figures anyway but we’d prefer to use better estimations).

Methodology 2: Using an Average Price for Various Periods — This approach calculates volumes using an average price over different time frames, including shorter spans like the CURRENT_HOUR, CURRENT_DAY, and MOVING_24_HOUR, as well as longer durations like MOVING_365_DAY or LIFETIME. If there is no volume in the historical period, we use the latest price instead. The advantage of using this is that it offers a more balanced view by averaging out price fluctuations across various periods.

The big problem with Methodology 2 is that depending on when either side started trading you’ll get an average price of the older trading pair over a longer period. Imagine ASSET5-USD started trading two days ago. To get the LIFETIME volumes for ASSET5-BTC we would use the 2 days of volumes from ASSET5-USD as base volumes and we’d use the average lifetime BTC-USD price to calculate the quote volume, but in reality, when it comes to LIFETIME values, only using the last two days of the BTC-USD average price would give us a better estimate.

In most cases this would not be that big of a problem, for example, with the BTC-ETH instrument, ETH-USD began trading after BTC-USD, we use the base volume of ETH (from ETH-USD) as the quote volume for the BTC-ETH pair for each of the periods we cover. This ETH volume is then converted to BTC using average period values for BTC-ETH and used as base volume. In the case of ETH-BTC, the ETH-USD base volume is used as the base for the ETH-BTC pair, we then do the conversion using average period values to calculate the quote volume (in BTC).

By using Methodology 2 we ensure a consistent and accurate depiction of trading activities in synthetic pairs, regardless of their configuration, be it BTC-ETH or ETH-BTC. It also aligns seamlessly with our objective to deliver a thorough analysis. This analysis effectively cushions the effects of short-term price fluctuations and provides adaptability in time frames. Crucially, it achieves this without necessitating excessive resource expenditure or data lookups, relying instead on precomputed values. By adopting this methodology, we can present a more stable and historically faithful perspective on the volume dynamics of synthetic pairs.

To implement Methodology 2, we have the quoteVolumeTransform function in our API:

quoteVolumeTransform: function (fieldName, baseData, quoteData, dividedInstrument, baseVolumeFieldName) {
   if (baseData[baseVolumeFieldName] === 0) {
       return 0;
   }
   if (quoteData[baseVolumeFieldName] === 0) {
       return utilTransformModule.multiplyWithPrecision(baseData[baseVolumeFieldName], dividedInstrument.VALUE);
   }
   const averageQuoteValue = quoteData[fieldName] / quoteData[baseVolumeFieldName];
   const averageBaseValue = baseData[fieldName] / baseData[baseVolumeFieldName];
   return utilTransformModule.multiplyWithPrecision(baseData[baseVolumeFieldName], averageBaseValue / averageQuoteValue);
},
//Example usage:
//ethBtcData.CURRENT_HOUR_QUOTE_VOLUME = quoteVolumeTransform('CURRENT_HOUR_QUOTE_VOLUME', ethUsdData, btcUsdData, ethBtcData, 'CURRENT_HOUR_VOLUME');

This function ensures that our users receive accurate and representative volume data for synthetic pairs across various time frames, capturing our volume conversion logic effectively.

It is worth noting that on the higher Enterprise plans we do offer a much more accurate volume figure even for synthetic instruments by using underlying exchange data and historical data instead of precomputed tick values, however, that comes at a much higher compute cost for us and we can’t offer it on the free or premium plans.

Here is an example of the methodology in action, this is the response of the API Latest Tick endpoint for the BTC-USD, ETH-USD and ETH-BTC instruments.

{
 "Data": {
   "BTC-USD": {
     "TYPE": "266",
     "MARKET": "cadli",
     "INSTRUMENT": "BTC-USD",
     "CCSEQ": 73403478,
     "VALUE": 37781.2242950387,
     "VALUE_FLAG": "UP",
     "VALUE_LAST_UPDATE_TS": 1700950612,
     "VALUE_LAST_UPDATE_TS_NS": 517000000,
     //......
     "CURRENT_WEEK_VOLUME": 1870373.14930039,
     "CURRENT_WEEK_QUOTE_VOLUME": 69665646938.1537,
     "CURRENT_WEEK_VOLUME_TOP_TIER": 1012977.4447722,
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER": 37723838932.5028,
     "CURRENT_WEEK_VOLUME_DIRECT": 210979.420329009,
     "CURRENT_WEEK_QUOTE_VOLUME_DIRECT": 7856824366.18103,
     "CURRENT_WEEK_VOLUME_TOP_TIER_DIRECT": 169993.88929574,
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER_DIRECT": 6330893012.14093,
     //......
   },
   "ETH-USD": {
     "TYPE": "266",
     "MARKET": "cadli",
     "INSTRUMENT": "ETH-USD",
     "CCSEQ": 56406252,
     "VALUE": 2082.69091011576,
     "VALUE_FLAG": "UP",
     "VALUE_LAST_UPDATE_TS": 1700950612,
     "VALUE_LAST_UPDATE_TS_NS": 872000000,
     //......
     "CURRENT_WEEK_VOLUME": 18956477.9222111,
     "CURRENT_WEEK_QUOTE_VOLUME": 38658997848.3347,
     "CURRENT_WEEK_VOLUME_TOP_TIER": 9818123.49899776,
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER": 20019060997.1869,
     "CURRENT_WEEK_VOLUME_DIRECT": 1517485.0510653,
     "CURRENT_WEEK_QUOTE_VOLUME_DIRECT": 3095647614.92368,
     "CURRENT_WEEK_VOLUME_TOP_TIER_DIRECT": 1503459.69216905,
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER_DIRECT": 3067071858.33926,
     //......
   },
   "ETH-BTC": {
     "TYPE": "985", //985 is a synthetic instrument calculated though DIVISON
     "MARKET": "cadli",
     "INSTRUMENT": "ETH-BTC",
     "CCSEQ": 129809730, // the sum of the constituent CCSEQ
     "VALUE": 0.0551250243732639, // ETH-USD.VALUE / BTC-USD.VALUE
     "VALUE_FLAG": "UP", // based on ETH-USD since it is the most recent value
     "VALUE_LAST_UPDATE_TS": 1700950612,
     "VALUE_LAST_UPDATE_TS_NS": 872000000,
     //......
     "CURRENT_WEEK_VOLUME": 18956477.9222111, // ETH-USD.CURRENT_WEEK_VOLUME
     "CURRENT_WEEK_QUOTE_VOLUME": 1037911.1474925, // ETH-USD.CURRENT_WEEK_VOLUME * ((ETH-USD.CURRENT_WEEK_QUOTE_VOLUME / ETH-USD.CURRENT_WEEK_VOLUME) / (BTC-USD.CURRENT_WEEK_QUOTE_VOLUME / BTC-USD.CURRENT_WEEK_VOLUME))
     "CURRENT_WEEK_VOLUME_TOP_TIER": 9818123.49899776, // same methodology as CURRENT_WEEK_VOLUME
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER": 537560.805832965, // same methodology as CURRENT_WEEK_QUOTE_VOLUME
     "CURRENT_WEEK_VOLUME_DIRECT": 0, // 0 since this is based on the index conversions, for enterprise plans we offer this value based on the direct ETH-BTC trading pair on exchnages
     "CURRENT_WEEK_QUOTE_VOLUME_DIRECT": 0, // 0 since this is based on the index conversions, for enterprise plans we offer this value based on the direct ETH-BTC trading pair on exchnages
     "CURRENT_WEEK_VOLUME_TOP_TIER_DIRECT": 0, // 0 since this is based on the index conversions, for enterprise plans we offer this value based on the direct ETH-BTC trading pair on exchnages
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER_DIRECT": 0, // 0 since this is based on the index conversions, for enterprise plans we offer this value based on the direct ETH-BTC trading pair on exchnages
     //......
   }
 },
 "Err": {}
}

‍

Calculating OPEN, HIGH, LOW for Synthetic Instruments

Calculating the OPEN, HIGH, and LOW values for synthetic divided index pairs, such as ETH-BTC, presents a very similar challenge to the volume one discussed above. The challenge intensifies when considering different historical periods, ranging from short-term intervals like hours and days to long-term frames such as months and the entire lifetime of the instruments.

When deciding how to tackle the issue of estimated historical OPEN, HIGH and LOW values based on limited access to data we investigated three methodologies.

‍

Methodology 1: Using the Latest Quote Asset Value

This method applies the latest quote asset to USD value to the OPEN, HIGH, and LOW values of the base asset to USD (e.g., ETH-BTC. CURRENT_MONTH_OPEN = ETH-USD.CURRENT_MONTH_OPEN / BTC-USD.VALUE).

  • Advantage: Offers immediate reflection of the latest market conditions, essential for short-term trading strategies.
  • Limitation: The crypto market’s volatility may cause significant skews in OPEN, HIGH, and LOW values due to short-term jumps or drops in the quote asset’s value.

‍

Methodology 2: Combination of Specific and Averaged Data

This approach combines the ETH-USD.{PERIOD}_OPEN value with the average price of BTC-USD over the same period (e.g., ETH-BTC. CURRENT_MONTH_OPEN = ETH-USD.CURRENT_MONTH_OPEN / (BTC-USD.CURRENT_MONTH_QUOTE_VOLUME / BTC-USD.CURRENT_MONTH_VOLUME)).

  • Advantage: Provides a comprehensive view, blending specific start points with long-term average trends.
  • Limitation: Risks smoothing out significant historical market movements, losing granular details. Also, it leads to always changing open values based on the trading activity of the quote asset to USD pair. Furthermore, it leads to unbalanced HIGHs and LOWs over different time-periods, for example the CURRENT_YEAR_HIGH could be lower than the CURRENT_MONTH_HIGH

‍

Methodology 3: Period-Specific Open Ratio Calculation

This approach calculates OPEN, HIGH, and LOW values using period-specific ratios of the base and quote assets (e.g., ETH-BTC. CURRENT_MONTH_OPEN = ETH-USD.CURRENT_MONTH_OPEN / BTC-USD.CURRENT_MONTH_OPEN).

  • Advantage: Offers consistent time-based comparisons, crucial for accurate period analysis. For example unlike Methodology 1 and 2 the open of each period will not change with the latest tick.
  • Limitation: Does not adequately capture intra-period trends and market evolutions. This method can significantly inflate HIGH values and reduce LOW values, as it’s unlikely for both instruments to reach their peaks or troughs simultaneously, leading to skewed representations in these metrics.

We decided to use two of the methodologies listed above instead of sticking to just one as we did with volume.

We decided to adopt Methodology 3 for calculating the OPEN value of synthetic instruments like ETH-BTC, which involves using period-specific ratios, such as ETH-USD.CURRENT_MONTH_OPEN / BTC-USD.CURRENT_MONTH_OPEN. This methodology was selected due to its ability to provide consistent time-based comparisons, crucial for an accurate and clear understanding of the market’s opening conditions at the beginning of each period. By effectively capturing the real market scenario at the start of a timeframe, it offers a reliable baseline for further analysis, ensuring that the OPEN value accurately reflects the initial market conditions.

We chose Methodology 2 for the HIGH and LOW calculations in synthetic instruments like ETH-BTC. This methodology entails blending specific period values, such as the HIGH of ETH-USD, with averaged data like the average price of BTC-USD over the same period. The strength of this approach lies in its comprehensive market view, capturing both the extremities and average trends of the market. By marrying specific period highs and lows with long-term averages, it encompasses the market’s peaks, troughs, and general behaviour over a period. This balanced perspective is particularly beneficial in the crypto market, known for its rapid and significant price fluctuations, as it helps smooth out volatility spikes or drops, providing a more stabilized and realistic view of market movements.

To calculate the OPEN values over multiple periods we use the following function:

openTransform: function (fieldName, baseData, quoteData) {
   return utilTransformModule.divideWithPrecision(baseData[fieldName], quoteData[fieldName]);
},
//Example usage:
//ethBtcData.CURRENT_HOUR_OPEN = openTransform('CURRENT_HOUR_OPEN', ethUsdData, btcUsdData);

  • This function applies Methodology 3 for calculating the OPEN value. It takes a field name (like ‘CURRENT_HOUR_OPEN’) and the respective data from both the base and quote instruments (ETH-USD and BTC-USD).
  • The function uses utilTransformModule.divideWithPrecision to divide the base data's OPEN value by the quote data's OPEN value, ensuring precision in the division.
  • Example: ethBtcData.CURRENT_HOUR_OPEN = openTransform('CURRENT_HOUR_OPEN', ethUsdData, btcUsdData);
  • This approach ensures time-aligned and period-specific OPEN values for the synthetic pair, reflecting accurate market opening conditions.

To calculate the HIGH and LOW values over multiple periods we use the following function:

highLowTransform: function (fieldName, baseData, quoteData, dividedInstrument, baseVolumeFieldName, quoteVolumeFieldName) {
   if (quoteData[baseVolumeFieldName] === 0) {
       return utilTransformModule.divideWithPrecision(baseData[fieldName], quoteData.VALUE);
   }
   const averageBaseQuotePrice = quoteData[quoteVolumeFieldName] / quoteData[baseVolumeFieldName];
   return utilTransformModule.divideWithPrecision(baseData[fieldName], averageBaseQuotePrice);
},
//Example usage:
//ethBtcData.CURRENT_HOUR_HIGH = highLowTransform('CURRENT_HOUR_HIGH', ethUsdData, btcUsdData, ethBtcData, 'CURRENT_HOUR_VOLUME', 'CURRENT_HOUR_QUOTE_VOLUME');

  • The highLowTransform function implements Methodology 2, applicable for calculating HIGH and LOW values.
  • It also takes a field name (e.g., ‘CURRENT_HOUR_HIGH’), base and quote data, and additional fields for volumes.
  • If the base volume of the quote data is zero, it divides the base data’s field value by the latest value of the quote asset. Otherwise, it calculates the average price from the quote data’s volume and uses this for division.
  • Example: ethBtcData.CURRENT_HOUR_HIGH = highLowTransform('CURRENT_HOUR_HIGH', ethUsdData, btcUsdData, ethBtcData, 'CURRENT_HOUR_VOLUME', 'CURRENT_HOUR_QUOTE_VOLUME');
  • This function effectively blends specific period values with averaged data, offering a more comprehensive view of market highs and lows.

For our Enterprise clients, we’ve developed a refined approach, Methodology 3+, to enhance the accuracy of OPEN, HIGH, and LOW value calculations for synthetic instruments. This methodology is particularly focused on instances where the opening times of the base and quote data do not align on the same dates. Instead of the standard calculation of dividing the base OPEN by the quote OPEN regardless of their timing, Methodology 3+ incorporates an alignment with the actual opening dates using data from our historical records. This ensures a more accurate reflection of market conditions, particularly for the OPEN value. Similarly, for the HIGH and LOW values, Methodology 3+ diverges from averaging over a period, opting instead to use the close of the specific date. This alignment with specific closing values provides a more precise capture of market highs and lows, offering a tailored and accurate representation for our Enterprise clients.

However, the implementation of Methodology 3+ comes with additional computational costs, hence it is only available for our Enterprise clients. This enhanced accuracy requires an extra lookup to our historical data storage, a step that is more resource-intensive and time-consuming compared to standard methodologies. Accessing specific historical data points, such as the exact open value from a previous date or the close of a particular date, involves querying and retrieving data from another database, increasing the processing time and demanding more from our data systems. While this approach offers significantly improved accuracy and detail, it does so at the expense of increased computational resources. For our Enterprise clients, who prioritize detailed and precise market analysis, this trade-off is often justified, as the value of the more aligned and accurate data provided by Methodology 3+ outweighs the additional resource requirements.

Here is an example methodologies 2 and 3 in action, this is the response of the API Latest Tick endpoint for the BTC-USD, ETH-USD and ETH-BTC pairs

{
 "Data": {
   "BTC-USD": {
     "TYPE": "266",
     "MARKET": "cadli",
     "INSTRUMENT": "BTC-USD",
     "CCSEQ": 73508998,
     "VALUE": 37781.2651134514,
     "VALUE_FLAG": "DOWN",
     "VALUE_LAST_UPDATE_TS": 1700960099,
     "VALUE_LAST_UPDATE_TS_NS": 597000000,
     //......
     "CURRENT_WEEK_VOLUME": 1879027.54239257,
     "CURRENT_WEEK_QUOTE_VOLUME": 69992660200.2589
     //......
     "CURRENT_WEEK_OPEN": 37388.3505763348,
     "CURRENT_WEEK_HIGH": 38407.8791777158,
     "CURRENT_WEEK_LOW": 35702.7363290693,
     "CURRENT_WEEK_TOTAL_INDEX_UPDATES": 8211850,
     "CURRENT_WEEK_CHANGE": 392.9145371166,
     "CURRENT_WEEK_CHANGE_PERCENTAGE": 1.0509009653004
      //......
   },
   "ETH-USD": {
     "TYPE": "266",
     "MARKET": "cadli",
     "INSTRUMENT": "ETH-USD",
     "CCSEQ": 56496535,
     "VALUE": 2082.5473810759,
     "VALUE_FLAG": "UP",
     "VALUE_LAST_UPDATE_TS": 1700960099,
     "VALUE_LAST_UPDATE_TS_NS": 249000000,
     //......
     "CURRENT_WEEK_VOLUME": 19063626.0641834,
     "CURRENT_WEEK_QUOTE_VOLUME": 38882198951.2168,
     //......
     "CURRENT_WEEK_OPEN": 2012.62574245691,
     "CURRENT_WEEK_HIGH": 2131.64857740821,
     "CURRENT_WEEK_LOW": 1932.53171758948,
     "CURRENT_WEEK_TOTAL_INDEX_UPDATES": 6963272,
     "CURRENT_WEEK_CHANGE": 69.92163861899,
     "CURRENT_WEEK_CHANGE_PERCENTAGE": 3.47415006893598
     //......
   },
   "ETH-BTC": {
     "TYPE": "985", //985 is a synthetic instrument calculated though DIVISON
     "MARKET": "cadli",
     "INSTRUMENT": "ETH-BTC",
     "CCSEQ": 130005533,
     "VALUE": 0.0551211658694415,
     "VALUE_FLAG": "UP",
     "VALUE_LAST_UPDATE_TS": 1700960099,
     "VALUE_LAST_UPDATE_TS_NS": 597000000,
     //......
     "CURRENT_WEEK_VOLUME": 19063626.0641834,
     "CURRENT_WEEK_QUOTE_VOLUME": 1043834.06101564,
     //......
     "CURRENT_WEEK_OPEN": 0.0538302896873663, // ETH-USD.CURRENT_WEEK_OPEN / BTC-USD.CURRENT_WEEK_VOLUME
     "CURRENT_WEEK_HIGH": 0.057226377397171, // ETH-USD.CURRENT_WEEK_HIGH / (BTC-USD.CURRENT_WEEK_QUOTE_VOLUME / BTC-USD.CURRENT_WEEK_OPEN)
     "CURRENT_WEEK_LOW": 0.0518808731302432, // ETH-USD.CURRENT_WEEK_LOW/ (BTC-USD.CURRENT_WEEK_QUOTE_VOLUME / BTC-USD.CURRENT_WEEK_OPEN)
     "CURRENT_WEEK_TOTAL_INDEX_UPDATES": 0, // 0 since this has not really had any index updates since it's calculated  synthetically
     "CURRENT_WEEK_CHANGE": 0.0012908761820752,
     "CURRENT_WEEK_CHANGE_PERCENTAGE": 2.3980479941168897
     //......
   }
 },
 "Err": {}
}

‍

Adapting the Forex Market Dynamics to Work With Synthetic Conversions

For forex (foreign exchange) conversions within the CADLI Data API endpoints, we employ a distinct methodology, tailored to the unique nature of forex markets. Unlike digital assets, forex conversions typically do not come with volume data. To address this, we have devised a system that ensures accurate and meaningful forex average rate calculations, despite the absence of traditional volume information.

To navigate the lack of volume data in forex conversions, we use a concept of synthetic volumes on the forex pairs. This is where fields like CURRENT_HOUR_VOLUME_SYNTHETIC_FOR_AVERAGE_VALUE_CALCULATION come into play. This field represents a synthetically generated base volume of the forex instrument for the current day. It's important to note that this volume is not derived from actual trading data but is instead created to facilitate the calculation of an average value (rate or price) over a specific period.

For each forex update we receive, since actual volume data is unavailable, we assume a base volume of 1. This synthetic base volume is then used alongsideCURRENT_HOUR_QUOTE_VOLUME_SYNTHETIC_FOR_AVERAGE_VALUE_CALCULATION. This latter field represents a synthetically generated quote volume of the forex instrument for the current hour, again not based on actual trading data but designed for calculating an average value over the hour.

The average value is then derived by dividing the CURRENT_DAY_QUOTE_VOLUME_SYNTHETIC_FOR_AVERAGE_VALUE_CALCULATION by the CURRENT_DAY_VOLUME_SYNTHETIC_FOR_AVERAGE_VALUE_CALCULATION.

This method allows us to maintain a consistent and rational approach to forex rate calculations, despite the lack of traditional volume data. By synthesizing volumes and employing them in our calculations, we ensure that the average values we present are not only logical but also representative of the forex market dynamics over the given period.

‍

Ensuring Accurate and Reliable Forex Rates

The CADLI API’s approach to forex conversions stands as a testament to our commitment to accuracy and reliability. Understanding the unique challenges posed by the forex market, we’ve tailored our system to ensure that the rates we provide are both meaningful and useful for our users. This method allows for a more nuanced and realistic representation of forex rates, crucial for those who rely on these conversions for their digital asset trading and analysis.

In this blog post, we’ve explored the intricate workings of the CADLI Data API, focusing on its sophisticated approach to converting and understanding digital assets in USD.

CADLI plays a crucial role in providing accurate USD conversion pathways, essential for both liquid and illiquid assets. The simplicity of direct index pairs like BTC-USD is contrasted with the complexity of synthetic pairs requiring additional conversion steps. We’ve detailed three distinct methods used in CADLI: Direct Index Pairs for straightforward conversions, Synthetic Inverted Index Pairs for inverted analysis, and Synthetic Divided Index Pairs for creating new synthetic pairs, useful in forex conversions and two-index combinations.

Particularly for synthetic instruments, we’ve delved into the methodologies for calculating historical metrics, focusing on OPEN, HIGH, and LOW values. We’ve discussed the merits and limitations of using the latest quote asset value, a combination of specific and averaged data, and period-specific ratios. Our approach caters to a wide user base, balancing precision and efficiency. We’ve highlighted the enhanced Methodology 3+ for our Enterprise clients, which, despite its higher computational cost, offers greater accuracy by aligning OPEN, HIGH, and LOW values with actual market dates. The implementation of this methodology is exemplified through functions like openTransform and highLowTransform in our API, showcasing our commitment to providing reliable and nuanced market insights.

Additionally, we’ve adapted our methodologies to the unique dynamics of the forex market within the CADLI Data API, employing synthetic volumes to calculate meaningful average rates despite the absence of traditional volume data. This innovative approach underscores our dedication to maintaining consistent and rational forex rate calculations, ensuring that our clients receive accurate and reliable market data. The CADLI API stands as a robust solution for digital asset conversions and analysis, reflecting our unwavering commitment to precision, reliability, and user-centric innovation in the digital asset sector.

‍

It Takes a Village to Build the Global Standard for Digital Asset Data and Indices

Before finishing and if you’re still around at the end of one of the longest blog posts we have ever published, I think it’s important to acknowledge the diverse and collaborative effort that underpins our ability to provide access to over 100 million potential asset combinations.

This undertaking truly exemplifies the notion that “it takes a village.” The journey begins with our exchange integrations teams, who play a crucial role in collecting comprehensive trade and swap data from over 300 markets, covering more than 1 million instruments. The seamless operation of this system is also heavily reliant on the great work of our internal tools team, who, among other things, provide indispensable support to our asset management and instrument mapping teams. These teams, in turn, diligently maintain over 10,000 assets and accurately map over 120,000 instruments ensuring we have a consistent view across all the industry.

Further enhancing our capabilities, our index team aggregates this vast data pool to create detailed indices, while our API team develops interfaces and additional functionalities, leveraging the rich base exchange data and the indices. The success of these technical endeavours is driven by our product and research teams, who ensure we are building the right products to meet market needs. Keeping us all on track, our data analysis and quality assurance teams maintain high standards of accuracy and reliability.

Our marketing team plays a pivotal role in helping people discover our products, maintaining a consistent and appealing brand image and making sure our sales team is well supported. In parallel, the sales team works tirelessly to onboard and guide both new and existing clients, while our support and account management team ensures that everyone receives the assistance they need.

Last but not least, the backbone of our smooth operation is our devops team, who ensure that everything runs seamlessly and securely. Together, these teams form the village that powers our service, enabling us to offer sophisticated, reliable, and accessible data analysis in the digital asset industry.

Related Products

CCData's Adaptive Diversified Liquidity Index (CADLI)

Real-time average digital asset market price across all available markets for 4,000 cryptocurrencies.
Read more

Related blogs

CADLI Part 3: Understanding Latest Tick Conversions

In the digital asset industry, the need for accurate and reliable conversions of cryptocurrencies to stable and globally recognised currencies like the US Dollar is paramount. Enter CADLI, our cutting-edge index, designed to revolutionise the way digital assets are understood and valued. CADLI not only facilitates price discovery for a wide range of assets, both liquid and illiquid, but also plays a crucial role in assessing their value in USD terms.

‍

This blog post delves into the inner workings of the CADLI Latest Tick Data API, highlighting our innovative methodologies and the sophisticated technology behind them. From direct index pairs to complex synthetic conversions, we explore how CADLI navigates the intricacies of digital asset conversions, offering precision and clarity in a market known for its volatility and complexity.

‍

With CADLI, we offer three distinct methods for providing data on requested instruments or pairs:

‍

  • Direct Index Pairs: For pairs such as BTC-USD, we directly provide the data as calculated by the CADLI index. This method is straightforward and efficient, offering users immediate access to the calculated USD values of these direct pairs.
  • Synthetic Inverted Index Pairs: In cases like USD-BTC, we employ a synthetic inversion method. This involves reversing the direct pair values and accordingly adjusting the base and quote volumes. This method allows users to view the value from the perspective of the inverted pair, providing a different angle of analysis.

Here is an example of the invert methodology in action, this is the response of the API Latest Tick endpoint for the BTC-USD and the USD-BTC instruments:

{
 "Data": {
   "BTC-USD": {
     "TYPE": "266",
     "MARKET": "cadli",
     "INSTRUMENT": "BTC-USD",
     "CCSEQ": 73519761,
     "VALUE": 37763.1644092693,
     "VALUE_FLAG": "DOWN",
     "VALUE_LAST_UPDATE_TS": 1700960977,
     "VALUE_LAST_UPDATE_TS_NS": 687000000,
     //......
     "CURRENT_WEEK_VOLUME": 1880114.02261408,
     "CURRENT_WEEK_QUOTE_VOLUME": 70033703406.5464,
     "CURRENT_WEEK_VOLUME_TOP_TIER": 1017268.86560116,
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER": 37885981495.9981,
     "CURRENT_WEEK_VOLUME_DIRECT": 211850.903249879,
     "CURRENT_WEEK_QUOTE_VOLUME_DIRECT": 7889741168.56429,
     "CURRENT_WEEK_VOLUME_TOP_TIER_DIRECT": 170674.75439911,
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER_DIRECT": 6356614799.099,
     "CURRENT_WEEK_OPEN": 37388.3505763348,
     "CURRENT_WEEK_HIGH": 38407.8791777158,
     "CURRENT_WEEK_LOW": 35702.7363290693,
     "CURRENT_WEEK_TOTAL_INDEX_UPDATES": 8222612,
     "CURRENT_WEEK_CHANGE": 374.8138329345,
     "CURRENT_WEEK_CHANGE_PERCENTAGE": 1.00248828085971
     //......
   },
   "USD-BTC": {
     "TYPE": "987", //987 is a synthetic instrument calculated though INVERSION
     "MARKET": "cadli",
     "INSTRUMENT": "USD-BTC",
     "CCSEQ": 73519761, // same CCSEQ as above
     "VALUE": 0.0000264808316687185, // 1/BTC-USD.VALUE
     "VALUE_FLAG": "UP", // opposite of BTC-USD.VALUE_FLAG
     "VALUE_LAST_UPDATE_TS": 1700960977,
     "VALUE_LAST_UPDATE_TS_NS": 687000000,
     //......
     "CURRENT_WEEK_VOLUME": 70033703406.5464, //BTC-USD.CURRENT_WEEK_QUOTE_VOLUME
     "CURRENT_WEEK_QUOTE_VOLUME": 1880114.02261408, //BTC-USD.CURRENT_WEEK_VOLUME
     "CURRENT_WEEK_VOLUME_TOP_TIER": 37885981495.9981, //BTC-USD.CURRENT_WEEK_QUOTE_VOLUME
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER": 1017268.86560116, //BTC-USD.CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER
     "CURRENT_WEEK_VOLUME_DIRECT": 7889741168.56429, //BTC-USD.CURRENT_WEEK_QUOTE_VOLUME_DIRECT
     "CURRENT_WEEK_QUOTE_VOLUME_DIRECT": 211850.903249879, //BTC-USD.CURRENT_WEEK_VOLUME_DIRECT
     "CURRENT_WEEK_VOLUME_TOP_TIER_DIRECT": 6356614799.099, //BTC-USD.CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER_DIRECT
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER_DIRECT": 170674.75439911, //BTC-USD.CURRENT_WEEK_VOLUME_TOP_TIER_DIRECT
     "CURRENT_WEEK_OPEN": 0.0000267462989028715, // 1/BTC-USD.CURRENT_WEEK_OPEN
     "CURRENT_WEEK_HIGH": 0.0000280090576470968, // 1/BTC-USD.CURRENT_WEEK_LOW
     "CURRENT_WEEK_LOW": 0.0000260363243534727, // 1/BTC-USD.CURRENT_WEEK_HIGH
     "CURRENT_WEEK_TOTAL_INDEX_UPDATES": 8222612,
     "CURRENT_WEEK_CHANGE": -2.65467234152998e-7,
     "CURRENT_WEEK_CHANGE_PERCENTAGE": -0.992538201704226
     //......
   }
 },
 "Err": {}
}

  • Synthetic Divided Index Pairs: For pairs that involve forex conversions, such as BTC-EUR, or two-index conversions like BTC-ETH, we use a synthetic division approach. This method involves dividing the values of one index pair by another, allowing for the creation of new synthetic pairs that might not directly trade in the market. It’s particularly useful for estimating values in different currencies or for creating pairs between two digital assets. This is the most complicated conversion, it needs to take into account not only the average prices over certain periods of BTC-USD and ETH-USD to correctly calculate OPEN, HIGH, LOW values for all the periods, but it also needs to take into account the ‘age’ of the market.

Our methodology for calculating historical metrics in synthetic pairs hinges on considering the “age” of the constituent instruments. This strategy not only ensures accuracy and consistency in our calculations but also reduces computational costs, a crucial factor in handling a vast array of instruments.

Given the sheer volume of assets and forex pairs we cover in CADLI— at the time of writing this blog post approximately 10,000 assets and 200 forex pairs — the total potential combinations of trading and synthetic instruments reach a staggering 104,029,800. The introduction of Real World Assets on-chain and the quadratic growth of possible combination instruments necessitates a solution for rapid estimations of historical metrics on derived synthetic instruments. This is especially critical given our current API usage of 25 billion data requests per month, from over 50 million unique users, all while maintaining low cache times (under 10 seconds for each instrument).

Our approach is designed to offer accurate estimations of historical metrics on tick data, tailored to the needs of most free and premium users. We reserve the most precise calculations for our enterprise users, which involve significant on-the-fly computation and extra data lookups, while providing reliable and quick estimations for our broader user base. This balance between precision and efficiency is key in our ongoing effort to deliver valuable data insights across all levels of service.

‍

Calculating Volumes for Synthetic Instruments

Calculating synthetic volumes is a task that demands a nuanced and sophisticated approach. Take the BTC-ETH or ETH-BTC instruments, even though they are derived from the same trading pairs (BTC-USD and ETH-USD) they differ in their order and present unique challenges in accurately and consistently representing their historical market dynamics.

When deciding how to tackle the issue of estimated historical volumes based on limited access to data we investigated two methodologies:

Methodology 1: Using the Latest Price — Applies the latest price to calculate synthetic historical base and quote volumes (CURRENT_DAY, WEEK, MONTH, YEAR, MOVING_24H, MOVING_7_DAY, 30_DAY, 90_DAY, 180_DAY, 356_DAY and LIFETIME). While this seems to be the easiest option, it does not accurately capture historical volumes during periods of market volatility (it’s synthetic figures anyway but we’d prefer to use better estimations).

Methodology 2: Using an Average Price for Various Periods — This approach calculates volumes using an average price over different time frames, including shorter spans like the CURRENT_HOUR, CURRENT_DAY, and MOVING_24_HOUR, as well as longer durations like MOVING_365_DAY or LIFETIME. If there is no volume in the historical period, we use the latest price instead. The advantage of using this is that it offers a more balanced view by averaging out price fluctuations across various periods.

The big problem with Methodology 2 is that depending on when either side started trading you’ll get an average price of the older trading pair over a longer period. Imagine ASSET5-USD started trading two days ago. To get the LIFETIME volumes for ASSET5-BTC we would use the 2 days of volumes from ASSET5-USD as base volumes and we’d use the average lifetime BTC-USD price to calculate the quote volume, but in reality, when it comes to LIFETIME values, only using the last two days of the BTC-USD average price would give us a better estimate.

In most cases this would not be that big of a problem, for example, with the BTC-ETH instrument, ETH-USD began trading after BTC-USD, we use the base volume of ETH (from ETH-USD) as the quote volume for the BTC-ETH pair for each of the periods we cover. This ETH volume is then converted to BTC using average period values for BTC-ETH and used as base volume. In the case of ETH-BTC, the ETH-USD base volume is used as the base for the ETH-BTC pair, we then do the conversion using average period values to calculate the quote volume (in BTC).

By using Methodology 2 we ensure a consistent and accurate depiction of trading activities in synthetic pairs, regardless of their configuration, be it BTC-ETH or ETH-BTC. It also aligns seamlessly with our objective to deliver a thorough analysis. This analysis effectively cushions the effects of short-term price fluctuations and provides adaptability in time frames. Crucially, it achieves this without necessitating excessive resource expenditure or data lookups, relying instead on precomputed values. By adopting this methodology, we can present a more stable and historically faithful perspective on the volume dynamics of synthetic pairs.

To implement Methodology 2, we have the quoteVolumeTransform function in our API:

quoteVolumeTransform: function (fieldName, baseData, quoteData, dividedInstrument, baseVolumeFieldName) {
   if (baseData[baseVolumeFieldName] === 0) {
       return 0;
   }
   if (quoteData[baseVolumeFieldName] === 0) {
       return utilTransformModule.multiplyWithPrecision(baseData[baseVolumeFieldName], dividedInstrument.VALUE);
   }
   const averageQuoteValue = quoteData[fieldName] / quoteData[baseVolumeFieldName];
   const averageBaseValue = baseData[fieldName] / baseData[baseVolumeFieldName];
   return utilTransformModule.multiplyWithPrecision(baseData[baseVolumeFieldName], averageBaseValue / averageQuoteValue);
},
//Example usage:
//ethBtcData.CURRENT_HOUR_QUOTE_VOLUME = quoteVolumeTransform('CURRENT_HOUR_QUOTE_VOLUME', ethUsdData, btcUsdData, ethBtcData, 'CURRENT_HOUR_VOLUME');

This function ensures that our users receive accurate and representative volume data for synthetic pairs across various time frames, capturing our volume conversion logic effectively.

It is worth noting that on the higher Enterprise plans we do offer a much more accurate volume figure even for synthetic instruments by using underlying exchange data and historical data instead of precomputed tick values, however, that comes at a much higher compute cost for us and we can’t offer it on the free or premium plans.

Here is an example of the methodology in action, this is the response of the API Latest Tick endpoint for the BTC-USD, ETH-USD and ETH-BTC instruments.

{
 "Data": {
   "BTC-USD": {
     "TYPE": "266",
     "MARKET": "cadli",
     "INSTRUMENT": "BTC-USD",
     "CCSEQ": 73403478,
     "VALUE": 37781.2242950387,
     "VALUE_FLAG": "UP",
     "VALUE_LAST_UPDATE_TS": 1700950612,
     "VALUE_LAST_UPDATE_TS_NS": 517000000,
     //......
     "CURRENT_WEEK_VOLUME": 1870373.14930039,
     "CURRENT_WEEK_QUOTE_VOLUME": 69665646938.1537,
     "CURRENT_WEEK_VOLUME_TOP_TIER": 1012977.4447722,
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER": 37723838932.5028,
     "CURRENT_WEEK_VOLUME_DIRECT": 210979.420329009,
     "CURRENT_WEEK_QUOTE_VOLUME_DIRECT": 7856824366.18103,
     "CURRENT_WEEK_VOLUME_TOP_TIER_DIRECT": 169993.88929574,
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER_DIRECT": 6330893012.14093,
     //......
   },
   "ETH-USD": {
     "TYPE": "266",
     "MARKET": "cadli",
     "INSTRUMENT": "ETH-USD",
     "CCSEQ": 56406252,
     "VALUE": 2082.69091011576,
     "VALUE_FLAG": "UP",
     "VALUE_LAST_UPDATE_TS": 1700950612,
     "VALUE_LAST_UPDATE_TS_NS": 872000000,
     //......
     "CURRENT_WEEK_VOLUME": 18956477.9222111,
     "CURRENT_WEEK_QUOTE_VOLUME": 38658997848.3347,
     "CURRENT_WEEK_VOLUME_TOP_TIER": 9818123.49899776,
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER": 20019060997.1869,
     "CURRENT_WEEK_VOLUME_DIRECT": 1517485.0510653,
     "CURRENT_WEEK_QUOTE_VOLUME_DIRECT": 3095647614.92368,
     "CURRENT_WEEK_VOLUME_TOP_TIER_DIRECT": 1503459.69216905,
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER_DIRECT": 3067071858.33926,
     //......
   },
   "ETH-BTC": {
     "TYPE": "985", //985 is a synthetic instrument calculated though DIVISON
     "MARKET": "cadli",
     "INSTRUMENT": "ETH-BTC",
     "CCSEQ": 129809730, // the sum of the constituent CCSEQ
     "VALUE": 0.0551250243732639, // ETH-USD.VALUE / BTC-USD.VALUE
     "VALUE_FLAG": "UP", // based on ETH-USD since it is the most recent value
     "VALUE_LAST_UPDATE_TS": 1700950612,
     "VALUE_LAST_UPDATE_TS_NS": 872000000,
     //......
     "CURRENT_WEEK_VOLUME": 18956477.9222111, // ETH-USD.CURRENT_WEEK_VOLUME
     "CURRENT_WEEK_QUOTE_VOLUME": 1037911.1474925, // ETH-USD.CURRENT_WEEK_VOLUME * ((ETH-USD.CURRENT_WEEK_QUOTE_VOLUME / ETH-USD.CURRENT_WEEK_VOLUME) / (BTC-USD.CURRENT_WEEK_QUOTE_VOLUME / BTC-USD.CURRENT_WEEK_VOLUME))
     "CURRENT_WEEK_VOLUME_TOP_TIER": 9818123.49899776, // same methodology as CURRENT_WEEK_VOLUME
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER": 537560.805832965, // same methodology as CURRENT_WEEK_QUOTE_VOLUME
     "CURRENT_WEEK_VOLUME_DIRECT": 0, // 0 since this is based on the index conversions, for enterprise plans we offer this value based on the direct ETH-BTC trading pair on exchnages
     "CURRENT_WEEK_QUOTE_VOLUME_DIRECT": 0, // 0 since this is based on the index conversions, for enterprise plans we offer this value based on the direct ETH-BTC trading pair on exchnages
     "CURRENT_WEEK_VOLUME_TOP_TIER_DIRECT": 0, // 0 since this is based on the index conversions, for enterprise plans we offer this value based on the direct ETH-BTC trading pair on exchnages
     "CURRENT_WEEK_QUOTE_VOLUME_TOP_TIER_DIRECT": 0, // 0 since this is based on the index conversions, for enterprise plans we offer this value based on the direct ETH-BTC trading pair on exchnages
     //......
   }
 },
 "Err": {}
}

‍

Calculating OPEN, HIGH, LOW for Synthetic Instruments

Calculating the OPEN, HIGH, and LOW values for synthetic divided index pairs, such as ETH-BTC, presents a very similar challenge to the volume one discussed above. The challenge intensifies when considering different historical periods, ranging from short-term intervals like hours and days to long-term frames such as months and the entire lifetime of the instruments.

When deciding how to tackle the issue of estimated historical OPEN, HIGH and LOW values based on limited access to data we investigated three methodologies.

‍

Methodology 1: Using the Latest Quote Asset Value

This method applies the latest quote asset to USD value to the OPEN, HIGH, and LOW values of the base asset to USD (e.g., ETH-BTC. CURRENT_MONTH_OPEN = ETH-USD.CURRENT_MONTH_OPEN / BTC-USD.VALUE).

  • Advantage: Offers immediate reflection of the latest market conditions, essential for short-term trading strategies.
  • Limitation: The crypto market’s volatility may cause significant skews in OPEN, HIGH, and LOW values due to short-term jumps or drops in the quote asset’s value.

‍

Methodology 2: Combination of Specific and Averaged Data

This approach combines the ETH-USD.{PERIOD}_OPEN value with the average price of BTC-USD over the same period (e.g., ETH-BTC. CURRENT_MONTH_OPEN = ETH-USD.CURRENT_MONTH_OPEN / (BTC-USD.CURRENT_MONTH_QUOTE_VOLUME / BTC-USD.CURRENT_MONTH_VOLUME)).

  • Advantage: Provides a comprehensive view, blending specific start points with long-term average trends.
  • Limitation: Risks smoothing out significant historical market movements, losing granular details. Also, it leads to always changing open values based on the trading activity of the quote asset to USD pair. Furthermore, it leads to unbalanced HIGHs and LOWs over different time-periods, for example the CURRENT_YEAR_HIGH could be lower than the CURRENT_MONTH_HIGH

‍

Methodology 3: Period-Specific Open Ratio Calculation

This approach calculates OPEN, HIGH, and LOW values using period-specific ratios of the base and quote assets (e.g., ETH-BTC. CURRENT_MONTH_OPEN = ETH-USD.CURRENT_MONTH_OPEN / BTC-USD.CURRENT_MONTH_OPEN).

  • Advantage: Offers consistent time-based comparisons, crucial for accurate period analysis. For example unlike Methodology 1 and 2 the open of each period will not change with the latest tick.
  • Limitation: Does not adequately capture intra-period trends and market evolutions. This method can significantly inflate HIGH values and reduce LOW values, as it’s unlikely for both instruments to reach their peaks or troughs simultaneously, leading to skewed representations in these metrics.

We decided to use two of the methodologies listed above instead of sticking to just one as we did with volume.

We decided to adopt Methodology 3 for calculating the OPEN value of synthetic instruments like ETH-BTC, which involves using period-specific ratios, such as ETH-USD.CURRENT_MONTH_OPEN / BTC-USD.CURRENT_MONTH_OPEN. This methodology was selected due to its ability to provide consistent time-based comparisons, crucial for an accurate and clear understanding of the market’s opening conditions at the beginning of each period. By effectively capturing the real market scenario at the start of a timeframe, it offers a reliable baseline for further analysis, ensuring that the OPEN value accurately reflects the initial market conditions.

We chose Methodology 2 for the HIGH and LOW calculations in synthetic instruments like ETH-BTC. This methodology entails blending specific period values, such as the HIGH of ETH-USD, with averaged data like the average price of BTC-USD over the same period. The strength of this approach lies in its comprehensive market view, capturing both the extremities and average trends of the market. By marrying specific period highs and lows with long-term averages, it encompasses the market’s peaks, troughs, and general behaviour over a period. This balanced perspective is particularly beneficial in the crypto market, known for its rapid and significant price fluctuations, as it helps smooth out volatility spikes or drops, providing a more stabilized and realistic view of market movements.

To calculate the OPEN values over multiple periods we use the following function:

openTransform: function (fieldName, baseData, quoteData) {
   return utilTransformModule.divideWithPrecision(baseData[fieldName], quoteData[fieldName]);
},
//Example usage:
//ethBtcData.CURRENT_HOUR_OPEN = openTransform('CURRENT_HOUR_OPEN', ethUsdData, btcUsdData);

  • This function applies Methodology 3 for calculating the OPEN value. It takes a field name (like ‘CURRENT_HOUR_OPEN’) and the respective data from both the base and quote instruments (ETH-USD and BTC-USD).
  • The function uses utilTransformModule.divideWithPrecision to divide the base data's OPEN value by the quote data's OPEN value, ensuring precision in the division.
  • Example: ethBtcData.CURRENT_HOUR_OPEN = openTransform('CURRENT_HOUR_OPEN', ethUsdData, btcUsdData);
  • This approach ensures time-aligned and period-specific OPEN values for the synthetic pair, reflecting accurate market opening conditions.

To calculate the HIGH and LOW values over multiple periods we use the following function:

highLowTransform: function (fieldName, baseData, quoteData, dividedInstrument, baseVolumeFieldName, quoteVolumeFieldName) {
   if (quoteData[baseVolumeFieldName] === 0) {
       return utilTransformModule.divideWithPrecision(baseData[fieldName], quoteData.VALUE);
   }
   const averageBaseQuotePrice = quoteData[quoteVolumeFieldName] / quoteData[baseVolumeFieldName];
   return utilTransformModule.divideWithPrecision(baseData[fieldName], averageBaseQuotePrice);
},
//Example usage:
//ethBtcData.CURRENT_HOUR_HIGH = highLowTransform('CURRENT_HOUR_HIGH', ethUsdData, btcUsdData, ethBtcData, 'CURRENT_HOUR_VOLUME', 'CURRENT_HOUR_QUOTE_VOLUME');

  • The highLowTransform function implements Methodology 2, applicable for calculating HIGH and LOW values.
  • It also takes a field name (e.g., ‘CURRENT_HOUR_HIGH’), base and quote data, and additional fields for volumes.
  • If the base volume of the quote data is zero, it divides the base data’s field value by the latest value of the quote asset. Otherwise, it calculates the average price from the quote data’s volume and uses this for division.
  • Example: ethBtcData.CURRENT_HOUR_HIGH = highLowTransform('CURRENT_HOUR_HIGH', ethUsdData, btcUsdData, ethBtcData, 'CURRENT_HOUR_VOLUME', 'CURRENT_HOUR_QUOTE_VOLUME');
  • This function effectively blends specific period values with averaged data, offering a more comprehensive view of market highs and lows.

For our Enterprise clients, we’ve developed a refined approach, Methodology 3+, to enhance the accuracy of OPEN, HIGH, and LOW value calculations for synthetic instruments. This methodology is particularly focused on instances where the opening times of the base and quote data do not align on the same dates. Instead of the standard calculation of dividing the base OPEN by the quote OPEN regardless of their timing, Methodology 3+ incorporates an alignment with the actual opening dates using data from our historical records. This ensures a more accurate reflection of market conditions, particularly for the OPEN value. Similarly, for the HIGH and LOW values, Methodology 3+ diverges from averaging over a period, opting instead to use the close of the specific date. This alignment with specific closing values provides a more precise capture of market highs and lows, offering a tailored and accurate representation for our Enterprise clients.

However, the implementation of Methodology 3+ comes with additional computational costs, hence it is only available for our Enterprise clients. This enhanced accuracy requires an extra lookup to our historical data storage, a step that is more resource-intensive and time-consuming compared to standard methodologies. Accessing specific historical data points, such as the exact open value from a previous date or the close of a particular date, involves querying and retrieving data from another database, increasing the processing time and demanding more from our data systems. While this approach offers significantly improved accuracy and detail, it does so at the expense of increased computational resources. For our Enterprise clients, who prioritize detailed and precise market analysis, this trade-off is often justified, as the value of the more aligned and accurate data provided by Methodology 3+ outweighs the additional resource requirements.

Here is an example methodologies 2 and 3 in action, this is the response of the API Latest Tick endpoint for the BTC-USD, ETH-USD and ETH-BTC pairs

{
 "Data": {
   "BTC-USD": {
     "TYPE": "266",
     "MARKET": "cadli",
     "INSTRUMENT": "BTC-USD",
     "CCSEQ": 73508998,
     "VALUE": 37781.2651134514,
     "VALUE_FLAG": "DOWN",
     "VALUE_LAST_UPDATE_TS": 1700960099,
     "VALUE_LAST_UPDATE_TS_NS": 597000000,
     //......
     "CURRENT_WEEK_VOLUME": 1879027.54239257,
     "CURRENT_WEEK_QUOTE_VOLUME": 69992660200.2589
     //......
     "CURRENT_WEEK_OPEN": 37388.3505763348,
     "CURRENT_WEEK_HIGH": 38407.8791777158,
     "CURRENT_WEEK_LOW": 35702.7363290693,
     "CURRENT_WEEK_TOTAL_INDEX_UPDATES": 8211850,
     "CURRENT_WEEK_CHANGE": 392.9145371166,
     "CURRENT_WEEK_CHANGE_PERCENTAGE": 1.0509009653004
      //......
   },
   "ETH-USD": {
     "TYPE": "266",
     "MARKET": "cadli",
     "INSTRUMENT": "ETH-USD",
     "CCSEQ": 56496535,
     "VALUE": 2082.5473810759,
     "VALUE_FLAG": "UP",
     "VALUE_LAST_UPDATE_TS": 1700960099,
     "VALUE_LAST_UPDATE_TS_NS": 249000000,
     //......
     "CURRENT_WEEK_VOLUME": 19063626.0641834,
     "CURRENT_WEEK_QUOTE_VOLUME": 38882198951.2168,
     //......
     "CURRENT_WEEK_OPEN": 2012.62574245691,
     "CURRENT_WEEK_HIGH": 2131.64857740821,
     "CURRENT_WEEK_LOW": 1932.53171758948,
     "CURRENT_WEEK_TOTAL_INDEX_UPDATES": 6963272,
     "CURRENT_WEEK_CHANGE": 69.92163861899,
     "CURRENT_WEEK_CHANGE_PERCENTAGE": 3.47415006893598
     //......
   },
   "ETH-BTC": {
     "TYPE": "985", //985 is a synthetic instrument calculated though DIVISON
     "MARKET": "cadli",
     "INSTRUMENT": "ETH-BTC",
     "CCSEQ": 130005533,
     "VALUE": 0.0551211658694415,
     "VALUE_FLAG": "UP",
     "VALUE_LAST_UPDATE_TS": 1700960099,
     "VALUE_LAST_UPDATE_TS_NS": 597000000,
     //......
     "CURRENT_WEEK_VOLUME": 19063626.0641834,
     "CURRENT_WEEK_QUOTE_VOLUME": 1043834.06101564,
     //......
     "CURRENT_WEEK_OPEN": 0.0538302896873663, // ETH-USD.CURRENT_WEEK_OPEN / BTC-USD.CURRENT_WEEK_VOLUME
     "CURRENT_WEEK_HIGH": 0.057226377397171, // ETH-USD.CURRENT_WEEK_HIGH / (BTC-USD.CURRENT_WEEK_QUOTE_VOLUME / BTC-USD.CURRENT_WEEK_OPEN)
     "CURRENT_WEEK_LOW": 0.0518808731302432, // ETH-USD.CURRENT_WEEK_LOW/ (BTC-USD.CURRENT_WEEK_QUOTE_VOLUME / BTC-USD.CURRENT_WEEK_OPEN)
     "CURRENT_WEEK_TOTAL_INDEX_UPDATES": 0, // 0 since this has not really had any index updates since it's calculated  synthetically
     "CURRENT_WEEK_CHANGE": 0.0012908761820752,
     "CURRENT_WEEK_CHANGE_PERCENTAGE": 2.3980479941168897
     //......
   }
 },
 "Err": {}
}

‍

Adapting the Forex Market Dynamics to Work With Synthetic Conversions

For forex (foreign exchange) conversions within the CADLI Data API endpoints, we employ a distinct methodology, tailored to the unique nature of forex markets. Unlike digital assets, forex conversions typically do not come with volume data. To address this, we have devised a system that ensures accurate and meaningful forex average rate calculations, despite the absence of traditional volume information.

To navigate the lack of volume data in forex conversions, we use a concept of synthetic volumes on the forex pairs. This is where fields like CURRENT_HOUR_VOLUME_SYNTHETIC_FOR_AVERAGE_VALUE_CALCULATION come into play. This field represents a synthetically generated base volume of the forex instrument for the current day. It's important to note that this volume is not derived from actual trading data but is instead created to facilitate the calculation of an average value (rate or price) over a specific period.

For each forex update we receive, since actual volume data is unavailable, we assume a base volume of 1. This synthetic base volume is then used alongsideCURRENT_HOUR_QUOTE_VOLUME_SYNTHETIC_FOR_AVERAGE_VALUE_CALCULATION. This latter field represents a synthetically generated quote volume of the forex instrument for the current hour, again not based on actual trading data but designed for calculating an average value over the hour.

The average value is then derived by dividing the CURRENT_DAY_QUOTE_VOLUME_SYNTHETIC_FOR_AVERAGE_VALUE_CALCULATION by the CURRENT_DAY_VOLUME_SYNTHETIC_FOR_AVERAGE_VALUE_CALCULATION.

This method allows us to maintain a consistent and rational approach to forex rate calculations, despite the lack of traditional volume data. By synthesizing volumes and employing them in our calculations, we ensure that the average values we present are not only logical but also representative of the forex market dynamics over the given period.

‍

Ensuring Accurate and Reliable Forex Rates

The CADLI API’s approach to forex conversions stands as a testament to our commitment to accuracy and reliability. Understanding the unique challenges posed by the forex market, we’ve tailored our system to ensure that the rates we provide are both meaningful and useful for our users. This method allows for a more nuanced and realistic representation of forex rates, crucial for those who rely on these conversions for their digital asset trading and analysis.

In this blog post, we’ve explored the intricate workings of the CADLI Data API, focusing on its sophisticated approach to converting and understanding digital assets in USD.

CADLI plays a crucial role in providing accurate USD conversion pathways, essential for both liquid and illiquid assets. The simplicity of direct index pairs like BTC-USD is contrasted with the complexity of synthetic pairs requiring additional conversion steps. We’ve detailed three distinct methods used in CADLI: Direct Index Pairs for straightforward conversions, Synthetic Inverted Index Pairs for inverted analysis, and Synthetic Divided Index Pairs for creating new synthetic pairs, useful in forex conversions and two-index combinations.

Particularly for synthetic instruments, we’ve delved into the methodologies for calculating historical metrics, focusing on OPEN, HIGH, and LOW values. We’ve discussed the merits and limitations of using the latest quote asset value, a combination of specific and averaged data, and period-specific ratios. Our approach caters to a wide user base, balancing precision and efficiency. We’ve highlighted the enhanced Methodology 3+ for our Enterprise clients, which, despite its higher computational cost, offers greater accuracy by aligning OPEN, HIGH, and LOW values with actual market dates. The implementation of this methodology is exemplified through functions like openTransform and highLowTransform in our API, showcasing our commitment to providing reliable and nuanced market insights.

Additionally, we’ve adapted our methodologies to the unique dynamics of the forex market within the CADLI Data API, employing synthetic volumes to calculate meaningful average rates despite the absence of traditional volume data. This innovative approach underscores our dedication to maintaining consistent and rational forex rate calculations, ensuring that our clients receive accurate and reliable market data. The CADLI API stands as a robust solution for digital asset conversions and analysis, reflecting our unwavering commitment to precision, reliability, and user-centric innovation in the digital asset sector.

‍

It Takes a Village to Build the Global Standard for Digital Asset Data and Indices

Before finishing and if you’re still around at the end of one of the longest blog posts we have ever published, I think it’s important to acknowledge the diverse and collaborative effort that underpins our ability to provide access to over 100 million potential asset combinations.

This undertaking truly exemplifies the notion that “it takes a village.” The journey begins with our exchange integrations teams, who play a crucial role in collecting comprehensive trade and swap data from over 300 markets, covering more than 1 million instruments. The seamless operation of this system is also heavily reliant on the great work of our internal tools team, who, among other things, provide indispensable support to our asset management and instrument mapping teams. These teams, in turn, diligently maintain over 10,000 assets and accurately map over 120,000 instruments ensuring we have a consistent view across all the industry.

Further enhancing our capabilities, our index team aggregates this vast data pool to create detailed indices, while our API team develops interfaces and additional functionalities, leveraging the rich base exchange data and the indices. The success of these technical endeavours is driven by our product and research teams, who ensure we are building the right products to meet market needs. Keeping us all on track, our data analysis and quality assurance teams maintain high standards of accuracy and reliability.

Our marketing team plays a pivotal role in helping people discover our products, maintaining a consistent and appealing brand image and making sure our sales team is well supported. In parallel, the sales team works tirelessly to onboard and guide both new and existing clients, while our support and account management team ensures that everyone receives the assistance they need.

Last but not least, the backbone of our smooth operation is our devops team, who ensure that everything runs seamlessly and securely. Together, these teams form the village that powers our service, enabling us to offer sophisticated, reliable, and accessible data analysis in the digital asset industry.

Stay Up To Date

Get our latest research, reports and event news delivered straight to your inbox.

By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.
Preferences
DenyAccept