In our initial integrations from 7 years ago, each exchange integration was directly polling for the trades associated with the instruments listed. Not only is this not ideal for CPU usage (each REST request uses SSL and every request has quite a bit of overhead), it is also not ideal for rate limits.

Whilst we have contracts with almost all the exchanges we integrate with (there is nobody to sign a contract with on the DEX exchanges) and also have preferential limits and connections with most of them, on occasion due to very high trading activity either our server spends a lot of CPU cycles on sending and receiving requests, or we run into rate limits based on what the exchanges API permits.

Historically we handled this with a very simple ‘choose somebody else from this list’ approach. So instead of instance x making the request, it would cross-call to another instance and ask it to do the request instead.

While this approach solved our immediate rate limit issue, it did break one of our core Reliability rules: Once set up, an exchange integration should have all the information it needs to consume data in isolation. We knew we had to come up with a better approach and last year we spent a few months trying different methods, from reverse proxy servers to queues in NSQ or RabbitMQ to Redis and even PostgreSQL. Since we already had a lot of experience working with Redis, we did not need any of the guarantees provided by the more advanced queue systems and we had a local instance of Redis that we can use as a fallback, we decided Redis was the best optionfor now. We also tested Redis to work with over 10,000 requests per secondand the largest part of the data, the HTTP/HTTPS response goes directly from the node that makes the external request to the node that requested the data. In some cases, this is over 10MB in size, and requiring another network hop (going through another external queue system) would slow everything down.

In the end, after multiple iterations, we found the architecture described in this blog post to be the most resilient and reliable. It does not break any of our Reliability or Broad Data Set rules.

‍

‍

Each integration opens 3 connections to 3 Redis instances (Fig 1). One on the cc-db-redis-proxy-swarm-request-queue-01 server, one on the cc-db-redis-proxy-swarm-request-queue-02 server, and one on its own local Redis instance. It will use these connections to push requests for data (Fig 2). It will first try to push on the main connection, then on the backup connection and eventually, if both are down / unreachable it will push the request to the local queue.

‍

‍

On the proxy-node service, we’ll have 3 blocking Redis connections (Fig 3), one on each of the cc-db-redis-proxy-swarm-request-queue-01, cc-db-redis-proxy-swarm-request-queue-02, and local Redis instances. The proxy node will make the external request on behalf of the exchange integration and then put the response in a response queue on the local instance of the requesting service. As soon as it finishes serving the request it will put itself in blocking mode again until a new request comes through.

‍

The integration will then just keep one blocking connection (Fig 4) on its local queue to consume responses from the proxy swam.

‍

‍

Each request is represented by a JSON with the following fields

return {
version: moduleExports.currentVersion,
url: url || ‘’,
method: method || ‘GET’,
contentType: contentType || ‘’,
body: body || ‘’,
timeoutMs: timeoutMs || moduleExports.defaultTimeoutMs,
extraHeaders: extraHeaders || [],
timesRequested: 0,
destinationConnectionString: destinationConnectionString || ‘’,
destinationKey: destinationKey || ‘’,
serviceCategory: serviceCategory,
serviceName: serviceName,
serviceHostname: serviceHostname,
requestDataType: requestDataType,
metadata: metadata || {},
rateLimitDelayMs: rateLimitDelayMs || 0,
timeCreatedMs: common.UTIL.getCurrentTimestamp()
};

‍

The response is represented by a JSON with the following fields:

return {
version: moduleExports.currentVersion,
body: ‘’,
proxiedRESTRequest: {}, //the full request object as seen above.
responseTimeMs: 0,
responseCode: ‘’,
error: ‘’,
proxyNodeHostname: hostname,
responseHeaders: []
};

‍

With this structure, we can decide which instances make requests and which ones slow down on making requests because they have a big queue of responses to process. It also means that the instances that don’t need to process as much exchange data (because it’s a less active exchange) can use CPU cycles on getting data for the more active exchanges.

‍

‍

So an integration pushes a request to a queue, a proxy node somewhere on the network gets the request, makes the external call, packs the response of the external call and the original request together, and puts it in the local queue of the initiator of the request. This means all our integrations have access to each others’ Redis instances and can push data on a specific queue. Redis keeps track of the order the blocking pops came in and serves them in the order they were registered.

‍

‍

‍

Disclaimer: Please note that the content of this blog post was created prior to our company's rebranding from CryptoCompare to CCData.