CADLI Part 2: The Three Building Data Pillars

The strength, adaptability, and precision of CADLI come from the integrated interplay of its three core elements: Underlying Data, Conversion Data, and Component Data. These components enable CADLI to execute real-time recalculations, adjust to market fluctuations, and provide a comprehensive perspective on digital asset values.

Over the past three years, our extensive infrastructure improvements have laid the groundwork for CADLI’s impressive capabilities. We’ve integrated advanced features like deduplicated polling and streaming integrations via WebSockets and FIX, and real-time on-chain connections for DEFI activity. This technological arsenal not only elevates data quality but also significantly minimizes latency, allowing CADLI to update more than 50,000 times per second.

‍

Underlying Data

Underlying Data acts as CADLI’s foundational layer, encompassing all the spot and DeFi exchanges that CCData has integrated with. This data layer is critical for the accurate and timely price discovery of various digital assets. We load and store all the underlying data at index start and then we constantly apply streaming updates as soon as any of the underlying instruments record a new trade.

The advantage of keeping the underlying data in memory is twofold. First, it minimizes latency during index updates — a crucial feature for an asset class characterized by its volatility. Second, it optimizes the time complexity involved in fetching the data, making it instantly available for real-time recalculations as new ticks of underlying data arrive.

‍

Conversion Data

Conversion Data serves as the second pillar of CADLI, where all conversion values are stored in memory for real-time access. This design enables dynamic adjustments to the index and keeps the calculations current. Thus, when new conversion data data streams in, the conversion values are automatically updated to reflect these changes.

Real-time conversion data is essential for CADLI’s adaptability, especially given the diverse range of digital assets and their fluctuating values in various currencies. By keeping this data readily accessible, the index can rapidly adjust to new market conditions, ensuring its computations remain accurate and up-to-date.

‍

Component Data

Component Data, the final pillar, comprises underlying data values that have been mapped and converted. This layer is subjected to our Time-Adjusted 24-Hour Volume Weighted Price with Outlier Detection methodology, ensuring the index remains current and precise.

Component Data is computed under specific conditions to ensure the index remains up-to-date and accurate:

Upon New Underlying Data Update: Component Data is calculated when a new update is received for an underlying asset that satisfies two conditions: it has a 24-hour moving volume value greater than 0 and, if needed, a recent conversion (updated within the last 24 hours — 86400 seconds). This ensures that the index is continuously updated and reflects current market conditions. It filters out updates that are irrelevant due to no recent trading activity or outdated conversion rates.
Configuration Changes: When there’s a change in the configuration, such as the addition or removal of an underlying asset, Component Data is recalculated. This only happens if the newly added or removed underlying asset has 24-hour moving volume value greater than 0 and a recent conversion. The index needs to adapt to its new configuration, especially if a new asset has sufficient market activity and up-to-date conversion information. Recalculating Component Data ensures that the index immediately reflects these changes. Configuration changes are applied once per hour if the composition has changed.
No Recalculation on Restart: If the system restarts, the Component Data is not recalculated but it is loaded back to the state it was at when the server or service was asked to restart. Maintaining continuity and integrity of the index is crucial, especially for downstream applications that rely on the index for trading or analytics. Not recalculating on restart prevents unnecessary fluctuations and inconsistencies.
Unchanged Volume & Price: If an update in the underlying data results in no change in the price and the 24-hour calculated moving volume, then Component Data will not be recalculated. This would happen when we backfill trades on a currently trading instrument or when a quarantined trade is found to be legitimate and it is processed by our integration. Recalculating Component Data when there are no meaningful changes in underlying data would consume computational resources without providing additional value.

‍

Sharding Mechanism in CADLI

Sharding offers an essential scalability feature in CADLI’s architecture. Individual assets are assigned to different instances or “shards” based on their trading volume, number of constituents and market significance. Each shard operates on its internal data, as well as output data from other shards, for conversion to USD. For example if Shard 1 is in charge of calculating BTC-USD and USDT-USD, Shard 2 will mostly likely subscribe to these updates and use them for internal conversions.

This distributed system design does more than just offer scalability. It also ensures high availability and fault tolerance, critical factors for an index that aims to provide uninterrupted and accurate data feeds. This cohesive and integrated setup equips CADLI to adapt to the continuously evolving digital asset landscape.

‍

Volume Metrics in CADLI

Volume metrics in CADLI are designed to provide an in-depth view of market activity across all instruments and markets. While conventional systems primarily rely on just two volume fields — VOLUME and QUOTE_VOLUME — CADLI goes beyond these limitations to integrate eight distinct volume fields. This nuanced approach ensures a more comprehensive understanding of market conditions, and these eight volume fields are:

VOLUME: This metric represents the sum of all base quantities across all included components in the index. It provides a general snapshot of the overall trading activity, setting the stage for a holistic understanding of market behavior.
QUOTE_VOLUME: This complements the VOLUME metric by summing up all quote quantities across all included components. By showing the quote side, always denominated in USD, this metric provides a broader view of the asset’s trading landscape.
VOLUME_TOP_TIER: This field sums up all base quantities that are traded on top-tier markets. By focusing on data from reputable platforms, this metric emphasizes high-quality and reliable trading data, which is particularly useful for discerning market sentiment.
QUOTE_VOLUME_TOP_TIER: This is the sum of all quote quantities for components that are traded in top-tier markets. It complements the VOLUME_TOP_TIER metric by showing the quote side (USD-denominated) of those specific top-tier trades, providing an additional layer of market context.
VOLUME_DIRECT: This metric sums up all base quantities where the quote asset is denominated in USD, thus eliminating the need for any conversion. It gives a straightforward view of the market, highlighting the most direct trading activity.
QUOTE_VOLUME_DIRECT: This is the sum of all quote quantities that are already in USD. By providing a clean measurement of direct trading activity in terms of USD, this metric brings a level of precision and simplicity to the index’s analyses.
VOLUME_TOP_TIER_DIRECT: This field represents the sum of all base quantities traded in top-tier markets where the quote asset is in USD. In effect, it offers a focused snapshot of the best direct trading data, sourced exclusively from high-quality, reputable platforms.
QUOTE_VOLUME_TOP_TIER_DIRECT: Finally, this metric sums up all quote quantities traded in top-tier markets where the quote asset is USD. By completing the volume picture for top-tier markets, this metric offers a rounded understanding of high-quality, direct trading activity in terms of USD.

The granular approach to volume metrics offers multiple advantages. Firstly, the detailed nature of these metrics allows CADLI to provide a more accurate and nuanced snapshot of the market, catering to various trading scenarios and market tiers. Secondly, this level of granularity ensures that CADLI remains a powerful tool for traders, analysts, and investors. With its ability to offer high-resolution insights, CADLI becomes a vital asset in understanding the complexities of the ever-evolving digital asset landscape.

‍

Explore the CADLI REST endpoints:

Instrument Latest Tick — This endpoint retrieves the most recent index values and other index data for specified index instrument(s) on a selected market. Also delivers the current index value, along with OHLC (open, high, low, close) metrics aggregated across diverse time intervals.
Historical OHLCV+ Day — Get detailed index candlestick information at one-day intervals, encompassing open, high, low, and close values. This data enables a comprehensive understanding of the index’s performance over time, allowing for in-depth analysis and decision-making.
Historical OHLCV+ Hour — Get detailed index candlestick information at one-hour intervals, encompassing open, high, low, and close values. This data enables a comprehensive understanding of the index’s performance over time, allowing for in-depth analysis and decision-making.
Historical OHLCV+ Minute — Get detailed index candlestick information at one-minute intervals, encompassing open, high, low, and close values. This data enables a comprehensive understanding of the index’s performance over time, allowing for in-depth analysis and decision-making.
Historical Messages By Timestamp — Returns tick-level index data (every available update) for a selected instrument on a chosen market, starting from a specified timestamp.
Historical Messages Full Hour — You should use this endpoint to get a full hour of historical messages when catching up. For latest messages use the Historical Messages By Timestamp endpoint.
Markets + Instruments Mapped — Retrieves a dictionary with one or more mapped instruments across one or more markets that are in a given state/status. The dictionary uses the instrument ID as defined by our mapping team as the key, so it will only contain instruments that we have mapped

‍

Start your journey today with CCData’s API.