SHEET FIG.2 — /en/work/bigdata-saas
CASE STUDY § 01-2
Big data SaaS platform
2 years, continuous
TERM
migration · build · operations
SCOPE
GCP (GKE · BigQuery)
PLATFORM
Challenge
A SaaS data platform takes more operational effort as data volume and traffic grow. Cluster management, stable daily processing, and the data connection to the analytics platform each carry their own load.
The existing GKE cluster managed its nodes manually, so operational effort and scale tuning were a continuous burden. On top of that, the platform needed a path to deliver business data (MySQL) to the analytics platform (BigQuery) and stable operation of the daily processing pipeline.
Design decisions
- Considered migrating GKE from Manual operation to Autopilot, and carried it out after evaluating both operational load and cost
- Connected business data to the analytics platform via Datastream, linking MySQL→BigQuery, and included the connection’s latency and consistency in the scope of operations
- Built the daily pipeline with Cloud Composer (Airflow), managing dependencies and retries explicitly
- Evaluated Workflows and Cloud Run in a PoC before implementing new processing, confirming fit for the use case before deciding whether to adopt them
- Designed monitoring at each layer — platform, pipeline, and data connection
Hover, focus, or tap a node to see the design rationale for each element.
- GKE Autopilot — application runtime. Migrated from Manual operation, moved to Autopilot after evaluating operational load and cost.
- MySQL — source of business data. The origin for the connection to the analytics platform.
- Datastream — carries the MySQL→BigQuery connection. Its latency and consistency were included in the scope of operations.
- BigQuery — analytics platform. The destination where connected data is aggregated.
- Cloud Composer — daily pipeline. Dependencies and retries managed explicitly with Airflow.
Approach
- Evaluated the operational and cost impact of the migration during the review phase and carried it out in stages
- Carried the built platform straight into the operations phase, continuing monitoring and improvement over two years of continuous engagement
- Adopted new technology only after evaluating it in a PoC