Who monetizes traffic
AdsCompass is an intelligent platform for traffic monetizing. The company operates in the field of online advertising and serves as a link between publishers and advertisers when buying and selling traffic around the world. AdsCompass' competitive advantage is that instead of renting a third-party platform, which is common practice in online advertising, it develops its own. Starting with a partner base of a few dozen online companies, AdsCompass extended its reach over six years. Today, AdsCompass has over 5000 advertisers, and this number grows by about 10-15% every month. Since 2018, clients have been able to buy ads directly from their personal account on the platform, meaning the network has expanded by private advertisers as well.
Another billion records
To reach all its partners from over 200 countries, the services responsible for the load operate in two data centers in Europe and the US on their own hardware. Currently, processed traffic generates more than 4 billion statistics records per day. Records are made from several dozen servers to a single ClickHouse cluster. Traffic increases by an average of 6-10% per month.
The company has to constantly increase technical capacity and increase the cost of maintaining the ClickHouse cluster on its own hardware. Losses from cluster crashes also grew in direct proportion to the scale of the business.
At that point, we set the following goals:
- Increase cluster stability.
- Reduce infrastructure support and development costs, focusing directly on the product.
Migration and testing
Yandex Managed Service for ClickHouse pushed us to migrate to Yandex Cloud.
The cluster was moved to the Yandex Managed Service for ClickHouse by the head AdsCompass developers. The process, which included testing and the gradual full transfer of traffic, took less than a week.
During the first stage, they simultaneously wrote data to their cluster and Yandex Cloud in order to detect any unaccounted for limits. On those first days, several regular checks were also carried out, including:
- Testing to catch any possible traffic spikes.
- Checking how the cloud service operates when the volume of records increases significantly after pausing to catch up.
Cluster updates and data security
The initial result was an almost complete reduction of time spent on cluster support, not counting a brief discussion with Yandex Cloud support. Since moving, AdsCompass has also noticed growing cluster stability. Regular cluster version updates occur with minimal downtime. Yandex Cloud support already solved an issue we were having with hardware and kept all the data in tact. After migrating, we suddenly ran out of disk space. We quickly increased our quota and added space from the web interface.
AdsCompass also plans to use service of managed databases PostgreSQL, cloud computing service — Compute Cloud, docker management service — Container Registry, and Message Queue. The developers plan to write a script for clearing partitions and are counting on solutions in this area to come out.
Opinion
Increased stability and saving weren’t the only results of the project. There used to be cases where, when we stopped the ClickHouse cluster, we would collect statistics for a subsequent record. If we ran out of disk space and there wasn’t anywhere to write statistics files to, we’d have to delete them. Deleting statistics is a direct loss of financial and other data. Losing even a small amount of financial data might damage customer relations, which poses an immense reputational risk. With Yandex Managed Service for ClickHouse, we’ve forgotten about these risks.