Nielsen is the leading global measurement and data analytics tool that has been innovating in their industry for over 90 years. Headquartered in New York, Nielsen has operations in over 100 countries and cover more than 90% of the world’s population. The Nielsen Marketing Cloud combines Nielsen’s data analytics, media planning, marketing activation and data management platform capabilities in a single cloud platform. This allows clients to gain more accurate and validated data at scale. Nielsen Marketing Cloud use Hosted Graphite to understand what’s going on inside their systems.
Itai and the twenty member Big Data group collect, process, store and analyze all the data for Nielsen Marketing Cloud. As they offer both data-as-as-service (DaaS) and software-as-a-service (SaaS) for the AdTech industry, for Itai that means data, and lots of it.
The team monitors both system (e.g. memory usage) and application metrics (e.g. the number of records per processing cycle). However, as the company expanded, the limitations of its tech stack became more obvious and they began the process of updating their tools. As Itai puts it, “We started a process of transforming our entire data tech stack to one that would be able to support our growing business needs. We were looking for better tools and frameworks to replace our old tools.” After deciding on Apache Spark and Druid, the team set their mind to finding a new monitoring tool. In addition to seamless scaling, they needed one that would:
The solution
While Hosted Graphite met all the team's requirements, Itai and the Big Data group had one major concern: spiralling costs. They planned to integrate one of the new big data tools they were using, Apache Spark, to send application metrics. However the team were concerned that the metrics coming from Spark might drive costs up. Each time a Spark application starts, it generates an ID to be used as the application identifier, so a typical metric name would be:
application_<APPLICATION_ID>.driver.<APPLICATION_NAME>.<METRIC_NAME>
That way, if you have a batch application that runs every hour, it would send a metric: <METRIC_NAME> with the current <APPLICATION_ID>, so the team would end up with multiple metrics in Hosted Graphite, which are essentially the same metric. “Lucky for us, Hosted Graphite were able to solve this problem,” says Itai. The issue was fixed by removing the <APPLICATION_ID> part from the metric name coming into Graphite, such that,
application_<APPLICATION_ID>.driver.<APPLICATION_NAME>.<METRIC_NAME>
has become,
application.driver.<APPLICATION_NAME>.<METRIC_NAME>
This straightforward solution was put in place by Hosted Graphite’s technical support, without the need for Itai and the team to change anything in their code or configuration. This keeps their metric count under control, which put the team’s budgetary concerns to rest: “Hosted Graphite charges per metric, so it’s very easy to understand how to keep costs under control.” In addition, according to Itai, “when we saw how easy it was to use Hosted Graphite we were more than happy to choose it”.
The result
Since migrating to Hosted Graphite for their monitoring, Itai and the Big Data group have a much deeper insight into what’s happening with their apps and infrastructure. They also heavily rely on Hosted Graphite’s advanced alerts feature and get notifications through Slack to keep on top of unusual activity. This allows the team to keep track of when a certain threshold is violated, and act quickly to rectify problems before Nielsen’s customers are affected.
The team also use the metrics dashboard feature to help them see how much traffic they’re sending, how it varies over time and over what protocols and the metric treemap features, to explore what part of their infrastructure is creating a lot of metrics and is a cost driver. Hosted Graphite’s continuous improvement has been the cherry on the cake for Itai: “It’s great to know that Hosted Graphite are constantly improving the product and adding new features.
Hosted Graphite’s transparent pricing–no surprises and no hidden fees–has removed the team’s worry about spiralling costs. According to Itai, “there’s complete transparency with everything Hosted Graphite do which means we can accurately predict what we’ll be spending and comfortably keep within our budget.”
Read similar stories
