“If I have seen further, it is by standing on the shoulders of giants.” — Sir Isaac Newton

In the dynamic realm of decentralized finance (DeFi), platforms like Uniswap stand at the forefront of innovation, offering liquidity providers and traders unparalleled flexibility and efficiency. CCData’s integration with UniswapV3 exemplifies our commitment to harnessing the power of on-chain data to fuel the DeFi ecosystem’s growth. This detailed account unveils our journey through the integration process, the technical challenges we navigated, and how our solutions are empowering users with actionable insights.

Our journey with UniswapV3 began with the goal of providing our clients with a granular view of the DeFi market. The integration process involves several stages, starting from data ingestion to the final output where data becomes available on our REST APIs and WebSocket servers.

Data Ingestion and Parsing

At the heart of our integration lies the meticulous process of data ingestion. Leveraging our processes described in Part 1 and Part 2 we get immediate access to full block data. This ensures that our database is always up-to-date with the latest transactions on Uniswap V3, including swaps and liquidity events.

Our parsing engine is finely tuned to dissect complex block data, extracting key transaction details such as sender, receiver, amount, and contract interactions. This is particularly challenging given the sophisticated nature of Uniswap V3’s smart contracts, which introduce features like concentrated liquidity and multiple fee tiers.

The integration begins with listening for and parsing blockchain transactions including their associated logs and traces. This process is essential for identifying and extracting relevant data from the Ethereum blockchain.

onst erc20AbiSubSet = [
'event Initialize(uint160 sqrtPriceX96, int24 tick)',
'event PoolCreated(address indexed token0, address indexed token1, uint24 indexed fee, int24 tickSpacing, address pool)',
'event Collect(address indexed owner, address recipient, int24 indexed tickLower, int24 indexed tickUpper, uint128 amount0, uint128 amount1)',
'event Swap(address indexed sender, address indexed recipient, int256 amount0, int256 amount1, uint160 sqrtPriceX96, uint128 liquidity, int24 tick)',
'event Burn(address indexed owner, int24 indexed tickLower, int24 indexed tickUpper, uint128 amount, uint256 amount0, uint256 amount1)',
'event Mint(address sender, address indexed owner, int24 indexed tickLower, int24 indexed tickUpper, uint128 amount, uint256 amount0, uint256 amount1)',
];
const erc20Interface = ethersUtilsModule.createContractInterface(erc20AbiSubSet);

By defining a subset of the ERC-20 ABI (erc20AbiSubSet), the integration specifies which contract events it’s interested in. This is crucial for monitoring activities like swaps and liquidity changes on UniswapV3.

const discoverNewSwapsAndLiquidityUpdates = (blockchainData, blockchainAssetId) => {
  const transactions = blockchainData.TRANSACTIONS;
  const timestamp = blockchainData.TIMESTAMP;
  const blockNumber = blockchainData.NUMBER;
  const swapAndLiquidityUpdatesObject = {};

  if (!transactions?.length) {
    logModule.toConsole('Transactions List is empty.', logModule.INFO, scriptParams.LOG_LEVEL);
    return swapAndLiquidityUpdatesObject;
}

  for (const transaction of transactions) {
    if (!transaction?.LOGS?.length) {
      logModule.toConsole('Transaction Logs List is empty.', logModule.INFO, scriptParams.LOG_LEVEL);
      continue;
    }

    for (const transactionLog of transaction.LOGS) {
      const eventType = eventsSignatureHash[transactionLog.TOPICS[0]];
      if (Object.values(eventNames).includes(eventType)) {
        const header = unmappedAmmSwapOnChainModule.createHeaderObject(transaction, timestamp, blockNumber);
        const { error, instrumentID } = unmappedAmmSwapOnChainModule.createInstrumentId(transactionLog.ADDRESS, blockchainAssetId);
        if (error) {
          logModule.toConsole('Error creating instrumentID.', logModule.ERROR, scriptParams.LOG_LEVEL, error);
          continue;
        }
        if (!swapAndLiquidityUpdatesObject[instrumentID]) {
          swapAndLiquidityUpdatesObject[instrumentID] = [];
        }
        swapAndLiquidityUpdatesObject[instrumentID].push({ header, transactionLog, eventType });
      }
    }
  }
  return swapAndLiquidityUpdatesObject;
};

This is how we discover swaps and liquidity updates from the full block data that we get from our internal block streaming or polling (the data produced by the blockchain output service covered in On-Chain Series II: Streamlining Blockchain Data Output and Distribution). We use our internal distributor pools to get streaming data and the Data-API endpoint for getting the Ethereum full block by block number.

Handling Swap Events

After ingesting event logs, the system must identify and process transactions related to specific DeFi activities, such as swaps or liquidity events.

The raw data is then enriched, transforming it into a valuable resource for analysis and decision-making.

const getSide = (amount0, amount1) => {
  if (amount0.lt(0)) {
    return unmappedAmmSwapOnChainModule.SIDE.BUY;
  }
  if (amount1.lt(0)) {
    return unmappedAmmSwapOnChainModule.SIDE.SELL;
  }
  return unmappedAmmSwapOnChainModule.SIDE.UNKNOWN;
};

const convertExternalSwapToInternal = (instrumentsInMemoryCacheObject, externalTradeData, exchangeInternalName, instrumentID, receivedMS, fullBlockOnDataApi) => {
  const { header: headerData, transactionLog: swapEventLog } = externalTradeData;
  const decodedSwapEventLogs = erc20Interface.parseLog({ data: swapEventLog.DATA, topics: swapEventLog.TOPICS });
  const baseAmount = decodedSwapEventLogs.args['amount0'];
  const quoteAmount = decodedSwapEventLogs.args['amount1'];
  const quantity = BigInt(baseAmount) < 0n ? -BigInt(baseAmount) : BigInt(baseAmount);
  const quoteQuantity = BigInt(quoteAmount) < 0n ? -BigInt(quoteAmount) : BigInt(quoteAmount);
  const price = utilTransformModule.divideWithPrecision(quoteQuantity, quantity);
  const executedTS = headerData['TIMESTAMP'];
  const executedNS = utilTimestampModule.getNsFromTimestamp(executedTS, 'MILLI');
  const side = getSide(baseAmount, quoteAmount);
  const tradeID = unmappedAmmSwapOnChainModule.createTradeId(headerData, swapEventLog, 'TRANSACTION_HASH', 'INDEX');
  const from = headerData['FROM_ADDRESS'];
  const blockNumber = headerData['BLOCK_NUMBER'];
  const transactionHash = headerData['TRANSACTION_HASH'];
  const marketFeePercentage = instrumentsInMemoryCacheObject[instrumentID].instrumentMarketFeePercentage;
  const marketFeeValue = onchainFilter.getMarketFeeValue(baseAmount, quoteAmount, side, marketFeePercentage);
  
  const internalSwapObject = unmappedAmmSwapOnChainModule.createInternalObj(
  exchangeInternalName, fullBlockOnDataApi.ASSET_ID, instrumentID, side, tradeID, executedTS, executedNS, transactionHash, blockNumber,
  from, quantity, quoteQuantity, price, marketFeePercentage, marketFeeValue, fullBlockOnDataApi.PROVIDER_KEY, receivedMS
  );
  return internalSwapObject;
};

This snippet demonstrates the process of converting raw event data into a format that’s easier to work with, illustrating the enrichment phase where raw data is augmented with additional context.

Handling Liquidity Events

Dealing with the complexity of UniswapV3’s contracts, especially for features like concentrated liquidity, requires sophisticated data handling.

const parseLiquidityUpdateEventExternalFormat = (decodedLiquidityUpdateLog, eventType) => {
  if (eventType === eventNames.swapEventName) {
    return {
      changeInInvariant: 0,
      tick: parseFloat(decodedLiquidityUpdateLog.args['tick']),
      changeInBase: BigInt(decodedLiquidityUpdateLog.args['amount0']),
      changeInQuote: BigInt(decodedLiquidityUpdateLog.args['amount1']),
    };
  }
  if (eventType === eventNames.mintEventName) {
    return {
      changeInInvariant: BigInt(decodedLiquidityUpdateLog.args['amount']),
      tickUpper: parseFloat(decodedLiquidityUpdateLog.args['tickUpper']),
      tickLower: parseFloat(decodedLiquidityUpdateLog.args['tickLower']),
      changeInBase: BigInt(decodedLiquidityUpdateLog.args['amount0']),
      changeInQuote: BigInt(decodedLiquidityUpdateLog.args['amount1']),
    };
  }
  if (eventType === eventNames.burnEventName) {
    return {
      changeInInvariant: -BigInt(decodedLiquidityUpdateLog.args['amount']),
      tickUpper: parseFloat(decodedLiquidityUpdateLog.args['tickUpper']),
      tickLower: parseFloat(decodedLiquidityUpdateLog.args['tickLower']),
      changeInBase: 0,
      changeInQuote: 0,
    };
  }
  if (eventType === eventNames.collectEventName) {
    return {
      changeInInvariant: 0,
      tickUpper: parseFloat(decodedLiquidityUpdateLog.args['tickUpper']),
      tickLower: parseFloat(decodedLiquidityUpdateLog.args['tickLower']),
      changeInBase: -BigInt(decodedLiquidityUpdateLog.args['amount0']),
      changeInQuote: -BigInt(decodedLiquidityUpdateLog.args['amount1']),
    };
  }
};

This function parses liquidity update events, highlighting how the integration deals with the intricacies of UniswapV3’s smart contracts.

New Instrument Discovery

Converting on-chain transactions to financial instruments involves understanding how pools are created in UniswapV3.

const discoverNewInstruments = async (redisLocalClient, uninitializedInstrumentsCacheObject, blockchainData, factoryAddress) => {
  const newInstrumentsList = [];
  const transactionsList = blockchainData.TRANSACTIONS;

  if (transactionsList === undefined || transactionsList.length === 0) {
    logModule.toConsole('Transactions List is empty.', logModule.INFO, scriptParams.LOG_LEVEL);
    return newInstrumentsList;
  }

  for (const transaction of transactionsList) {
    const transactionLogsList = transaction.LOGS;
    if (transactionLogsList === undefined || transactionLogsList.length === 0) {
      logModule.toConsole('Transaction Logs List is empty.', logModule.INFO, scriptParams.LOG_LEVEL);
      continue;
    }

    let currentIndex = 0;
    while (currentIndex < transactionLogsList.length) {
      const transactionLog = transactionLogsList[currentIndex];
      if (transactionLog.ADDRESS === undefined) {
        logModule.toConsole('transactionLog.ADDRESS is undefined!', logModule.WARNING, scriptParams.LOG_LEVEL, transactionLog);
        currentIndex += 1;
        continue;
      }
      if (transactionLog.ADDRESS.toLowerCase() === factoryAddress.toLowerCase()) { // PoolCreated log
        const initializeTransactionLog = transactionLogsList[currentIndex + 1];
        // If there is no Initialize log immediately after the PoolCreated log, then we store the instrument address and later look for the Initialize log in a future block
        if (!initializeTransactionLog || eventsSignatureHash[initializeTransactionLog?.TOPICS[0]] !== 'Initialize') {
          await storeUninitializedInstrument(redisLocalClient, uninitializedInstrumentsCacheObject, transactionLog, transaction.HASH, blockchainData.NUMBER, blockchainData.TIMESTAMP);
          currentIndex += 1;
          continue;
        }
        newInstrumentsList.push({
          transactionLog,
          initializeTransactionLog,
          instrumentTransactionHash: transaction.HASH,
          instrumentBlockNumber: blockchainData.NUMBER,
          instrumentBlockNumberTs: blockchainData.TIMESTAMP,
        });
        currentIndex += 2;
        continue;
      }
      // Ideally, an Initialize log is found immediately after a PoolCreated log, but there are instances where these Initialize logs are in an entirely different transaction and block
      // so anytime we find an Initialize log that doesn't have a PoolCreated log before it, we check our list of un-initialized instruments and try to find a match
      if (eventsSignatureHash[transactionLog?.TOPICS[0]] === 'Initialize') {
        for (const instrumentAddress in uninitializedInstrumentsCacheObject) {
          const metadata = uninitializedInstrumentsCacheObject[instrumentAddress];
          if (instrumentAddress.toLowerCase() === transactionLog.ADDRESS.toLowerCase()) {
            logModule.toConsole(`Found an Initialize log with index ${transactionLog.INDEX} in transaction ${transaction.HASH} for an uninitialized instrument ${instrumentAddress}`, logModule.WARNING, scriptParams.LOG_LEVEL);
            newInstrumentsList.push({ ...metadata, initializeTransactionLog: transactionLog });
            await removeUninitializedInstrument(redisLocalClient, uninitializedInstrumentsCacheObject, instrumentAddress);
          }
        }
      }
      currentIndex += 1;
    }
  }
  return newInstrumentsList;
};

This code snippet outlines the process of discovering new instruments from blockchain data and converting them to our internal representations, crucial for accurately tracking and analysing DeFi market dynamics.

Unique DeFi challenges

Integrating UniswapV3’s data posed unique challenges, from handling the protocol’s complex smart contract interactions to ensuring data accuracy and timeliness, main issues that we needed to cover:

• Smart Contract Complexity: UniswapV3 introduces innovative features like concentrated liquidity, which necessitated a deeper understanding and sophisticated parsing logic to accurately interpret transaction data. UniswapV2 is much easier in this regard.
• Instrument Space: The high number of instruments listed on UniswapV3 required a general reassessment of our exchange integrations. Over the last 10 years we’ve built exchange integrations that focus on handling a lot of trades or order book updates on a small subset of instruments. With DeFi, we have the opposite problem, a much smaller number of swaps and liquidity updates and a much higher number of instruments. As an extreme example, PancakeSwapV2 has two million instruments and UniswapV2 has around three hundred thousand as of Feb 2024.
• Ensuring Data Integrity: Given the sometimes mutable nature of some of the blockchain data (e.g., reorgs), maintaining data integrity, especially for real-time analytics, was paramount. Our solutions involved advanced error checking, redundancy mechanisms, and real-time monitoring systems to ensure the highest data quality.

Conclusion

The integration of DeFi into CCData’s suite of REST endpoints, WebSocket servers and Reference Indices marks a significant milestone in our mission to empower the digital asset industry with precise, timely, and actionable data. By leveraging innovative solutions and our well known attention to data quality and reliability, we’ve opened new avenues for market analysis, strategy development, and operational efficiency for our clients.

As the DeFi landscape continues to evolve, CCData remains committed to staying at the cutting edge, continuously enhancing our systems and methodologies. Through our work, we aim to enable our clients to navigate the complexities of the DeFi market with confidence, backed by the most accurate and comprehensive data available.

This was part three of our On-Chain integrations series.

‍

Explore our On-Chain DEX endpoints:

Latest Tick (Swap) — Retrieves the latest amm swap information for any given instrument(s) for a chosen exchange. Also returns up-to-date price and volume metrics aggregated over various time periods.

Historical OHLCV+ (Swap) Day — Retrieves daily amm swap candlestick data, including open, high, low, close prices and trading volume in both base and quote currencies, for a selected instrument on a specified exchange.

Historical OHLCV+ (Swap) Hour — Retrieves hourly amm swap candlestick data, including open, high, low, close prices and trading volume in both base and quote currencies, for a selected instrument on a specified exchange.

Historical OHLCV+ (Swap) Minute — Retrieves amm swap candlestick data at a minute granularity, including open, high, low, close prices and trading volume in both base and quote currencies, for a selected instrument on a specified exchange.

Historical Messages (Swap) By Timestamp — Returns tick-level amm swap data (every executed transaction) for a specified instrument on a chosen exchange, starting from a specified timestamp.

Historical Messages (Swap) by Hour — Returns tick-level amm swap data (every executed transaction) for a selected instrument on a chosen exchange, for a specified hour. You should use this endpoint to get a full hour of historical amm swap messages when catching up.

Historical Messages (Liquidity) by Timestamp — Returns tick-level amm liquidity update data (every available update) for a selected instrument on a chosen exchange, starting from a specified timestamp.

Historical Messages (Liquidity) by Hour — Returns tick-level amm liquidity update data (every available update) for a selected instrument on a chosen exchange, for a specified hour. Use this endpoint to get a full hour of historical amm liquidity update messages when catching up.

Instrument Metadata — Returns key information about any given instrument(s) traded on a given exchange. Can be used to retrieve metadata for many different instruments for a chosen exchange

Markets — Returns information about a chosen market. If “market” parameter is left blank, will provide information for all available markets.

Markets + Instruments Mapped — Retrieves a dictionary with one or more mapped instruments across one or more markets that are in a given state/status. The dictionary uses the instrument ID as defined by our mapping team as the key, so it will only contain instruments that we have mapped.

Markets + Instruments Unmapped — Retrieves a list of one or more instruments across one or more markets that are in a given state/status. The dictionary has the instrument ID as defined by the individual market as the key, so it will contain both CCData.io mapped instruments and instruments that have not yet been mapped by us.

‍

‍

If you’re interested in learning more about CCData’s market-leading data solutions and indices, please contact us directly.