THE PRODUCT PROBLEM
As Grab's data ecosystem expanded, the existing Hive Parquet infrastructure struggled to keep pace with the demands of high-concurrency data operations. The business needed a solution that could efficiently handle the growing data volume without compromising on performance or increasing operational costs. Users experienced delays and inefficiencies due to the centralized failure point of the Hive Metastore and the fragmentation of files, which led to increased API costs and slower data scans. Competitively, the market was moving towards more scalable and efficient data solutions, putting pressure on Grab to innovate and maintain its edge in data processing capabilities.
THE DECISION
The transition to a modern table-centric architecture using Apache Iceberg was a strategic product decision aimed at overcoming the limitations of the Hive Parquet model. This shift required alignment across multiple teams, including product management, engineering, and operations, to ensure a smooth transition and integration with existing data platforms like Slide and Hugo. The decision involved trading off the familiarity and established processes of Hive Parquet for the promise of improved performance and scalability with Iceberg. Aligning these diverse teams was challenging due to differing priorities and the inherent risks of adopting a new technology. In hindsight, while the transition was complex, it was necessary to meet the evolving needs of the business and users.
THE LESSON
This journey underscores the importance of proactive collaboration between PMs and engineering teams when addressing architectural bottlenecks. Unlike typical PM advice that often focuses on feature delivery, this case highlights the critical role of infrastructure decisions in product success. PMs must be deeply involved in understanding technical constraints and advocating for changes that align with long-term business goals. The transition to Apache Iceberg was not just a technical upgrade but a strategic move that required PMs to champion the initiative, ensuring that all stakeholders understood its value and were committed to its success. This experience reveals that PMs must be as invested in the technical foundations as they are in user-facing features to drive meaningful product evolution.