Case Study — ETL & Data Engineering
Shoprenter to BigQuery ETL Sync Pipeline
A production-grade n8n ETL automation that synchronizes 80K+ Shoprenter products with Google BigQuery using scalable batch processing, staging tables, MERGE/UPSERT operations, soft delete reconciliation, metadata enrichment, and fault-tolerant workflows for reliable ecommerce data synchronization.
The Challenge
An 80,000-record product catalog was synced from Shoprenter to BigQuery through manual exports and ad-hoc scripts. The warehouse drifted out of date, deleted products lingered, and every sync required babysitting.
What We Built
- n8n ETL workflows orchestrating full Shoprenter API ingestion at scale
- Scalable batch processing moving 80K+ products in every sync cycle
- BigQuery staging tables with MERGE/UPSERT operations for consistent upserts
- Soft-delete reconciliation so removed products are reflected in the warehouse
- Metadata enrichment to make catalog data analytics-ready
- Fault-tolerant execution with retries and error handling for reliable nightly runs
n8nGoogle BigQueryShoprenter APIETL PipelinesREST APIsJavaScriptSQLWorkflow Automation
Results
Products synced
80K+ per cycle
Sync strategy
MERGE / UPSERT
Deletes reconciled
Soft-delete aware
More Work
Batch Disclosure — Price-Moving Catalyst Detection
An AI-powered system that scans ASX and SEC filings for forward-looking disclosures — contracts, M&A, guidance — that move micro-cap prices before the market catches on.
WikiGen — Autonomous, Bias-Aware Knowledge Synthesis
A multi-agent GraphRAG system that autonomously writes reliable, bias-aware Wikipedia-style articles with verifiable provenance on every claim.
Let's Work Together
Build Something Like This?
This system solved a real problem. We can build something similar for you.