Analytics Innovation – The story of the first Netezza box
TEOCO’s portfolio of planning, assurance, analytics and optimization products has helped more than 300 of the largest service providers in over 100 countries run their networks more efficiently and profitably. We have done this through years of innovation in areas like data management, analytics and more recently NFV, SON and 5G. Below is an interview with Chuck Privitera, Vice President of TEOCO and one of its early innovators, who shares his insight and stories about the early days of analytics.
What were the early years of analytics like?
In the early 1990s we were working with a large Tier-1 US service provider, helping them process their call detail records (CDRs). Magnetic disks were prohibitively expensive at the time, so to store and process the data we built a solution that used optical jukeboxes. An optical jukebox was, in the most basic terms, an automatic disk changer, and all the data was stored on actual disks. Then in the mid-1990s our focus evolved to providing cost management services; auditing and analyzing wholesale costs to find areas where the CSP overpaid for interconnect voice traffic. This was still before the era of analytics and these new ‘automated audits’ required that we be able to store, process and analyze huge volumes of call detail records (CDRs).
Needless to say, these technological hurdles impacted our ability to deliver timely results for our clients and made the whole process really cumbersome, but it was the best anyone had at the time.
How did you overcome your scalability issues?
In the early 2000s, a venture company that we knew introduced us to a start-up company they were backing called Netezza. Their CEO actually came to our office in Rochester, New York, to pitch this new ‘data appliance’ idea they were developing. It was just a prototype at the time and it didn’t even really exist yet, but we were interested in their approach, or frankly any approach that would help solve our problem, so we engaged them in a proof of concept.
As a test, the first thing we did was load one week’s worth of call detail records, about 800 million rows of data, into Netezza’s prototype data appliance, and it performed reasonably well. Then we loaded another week’s worth of data, and that’s when things started to slow down. The more data we loaded the slower the system became. So, we went back to the Netezza team and told them they had to do better. To meet the needs of our Tier-1 customers, their solution had to be able to scale well beyond its current capacity.
How did they solve the problem?
It took a lot of time to convince the Netezza team that they had to scale to the levels we needed. In fact, we went back and forth for about six months, working with them and pushing for better processing speeds at higher capacity. In the end, the Netezza team devised a way to process data that was completely new and innovative. They created something they called ‘Zone Maps’, although initially they called it Nearly Ordered Tables, and this feature completely changed how data was processed. It basically allowed the system to avoid processing unnecessary data by collocating it into various ‘zones’. At query time, only the applicable zones required processing, which saved time and allowed the appliance to scale incredibly well. Zone maps helped put Netezza on the map, turning them into a true market disruptor. In fact, to this day, Zone Maps are still the secret-sauce of the company, and it might not have existed the way it does today had TEOCO not worked so closely with their team and helped push the envelope in terms of data processing power.
With the new Zone Maps architecture, the Netezza prototype worked so well that TEOCO committed to purchasing the first unit. Marked as serial number #001, it came with a set of release notes that were personally autographed by the entire Netezza design team. They were proud of their work, and rightly so. With the Netezza data appliance, TEOCO was able to process data 100 times faster. This ability then led to the development of our HyperAnalytics solution, one of the first telco-grade data analytics solutions on the market. We used it to perform cost and margin analytics, and our first Client on the system was Nextel’s IDEN (push-to-talk) network.
What new technologies are you using today?
Since then, IBM purchased Netezza in 2010, and we are still using their data appliances today, along with Teradata. But TEOCO has continued to push the envelope in terms of technology. Two years ago we began using Hadoop to address scalability and cost issues that we were encountering with Oracle and others. With Hadoop, we created our customer analytics application, which provides end-to-end Customer Experience Management (CEM) for mobile, fixed, IP & OTT services. It identifies & diagnoses problems via cross-dimensional drill downs and ad-hoc investigations that can be performed on the fly.
Based on Hadoop technology, TEOCO’s CEM solution allows CSPs to store large quantities of data for long periods of time, and then access it in near real time for key performance and quality of service queries. CEM also builds on TEOCO’s advanced geo-location capabilities to pinpoint problematic areas and correlate geo-location information with service quality metrics. These types of applications are very different from those that utilize a data appliance like Netezza or Teradata, yet are relevant for the latest LTE and 5G technologies where you have huge amounts of data that are costly to store.
For example, Hadoop is perfect for supporting ‘data lakes’ which are large object-based storage repositories that hold data in its native format until it is needed. Netezza will provide faster results, but for queries that don’t require near real-time processing, Hadoop is more cost effective than most other applications. It’s really about matching the technology to the need – there is no one size fits all approach.
This is an exciting time in the industry, and of course TEOCO is continuing to focus on innovation. 5G, the next generation of mobile communication technology, will offer far greater capacity and be faster, more energy-efficient and more cost-effective than anything that has gone before. TEOCO is one of the participants of the 5G Innovation Center in Surrey, England, where we are working to develop the networks of tomorrow. The 5G Innovation Center brings together leading academic experts and industry partners to define and develop the 5G infrastructure that will help enable the Internet of Things, and change the way we communicate, work and live our everyday lives in the future.
For over twenty years TEOCO has been helping to shape the face of analytics. We have some of the brightest data science minds in the industry, and we’ve always been committed to finding and creating innovative solutions to address the needs of our Clients. We’ll continue down this path, because it’s what our company was built on, and what our customers have come to expect.