Building a Distributed Data Solution
An enterprise data, analytics and marketing company had multiple data collection streams running on different platforms used by a number of departments, making it difficult to evaluate their business’s performance. Ultimately, they were losing money due to a lack of organization and marketing agility.
Business Challenge
Our client’s data was gathered from a wide variety of sources including online tracking (search and sales data), social media crawlers, telemarketing, brick & mortar retail sales, and contracted market research partners. All this data was used by various stakeholders within the company - everyone from ad campaign planners, to marketing copy writers, business analysts, and accountants. Our client’s marketing strategy was based on targeted marketing aimed at specific demographic cohorts in specific locations using their own proprietary distribution software.
The chief technical problems that needed to be addressed were:
- Unifying data formats throughout the company.
- Optimizing data processing.
- Integrating all data streams in a single browser-based platform with separate frontends - each being a marketed product specific for our client’s various departments.
- Designing a solution architecture which would provide for a high level of security, and solid reliability at peak loads.
Engineered Solution
We first began by investigating our client’s data structure, then modelling various data distribution schemes before settling on an optimal proposition for all stakeholders. Next, we began work on unifying data formats and making the necessary database conversions while preparing for migration to a new on-premise cloud service. Our solution used the following technologies: Bootstrap, NodeJS, Java, Cassandra, Amazon Web Services, and on-premise Openstack.
For performance and security reasons, we decided to store all data, and conduct all data processing on premises, while the API was hosted on a contracted cloud service.
Delivered Value
As a result of the project, our client saw an increase in profits as they were able to reduce their IT staff, more quickly appraise the marketing landscape and put decisions into action. Furthermore, with their new unified database and analytics software, they were able to reduce internal conflict and simplify their business processes.