The Challenge
RedCap needed to manage large volumes of clinical research data from multiple sources, transform it through complex ETL pipelines, and make it available for analytics at scale. Traditional relational databases couldn’t handle the analytical query workloads, and building data pipelines required significant engineering effort for each new data source. They needed a visual ETL tool that data engineers could use without writing code from scratch.
Our Approach
Daiviksoft built an enterprise healthcare data platform centered around VisualFlow — a visual ETL tool with a drag-and-drop interface for building data pipelines. The platform uses ClickHouse as an OLAP data warehouse for high-performance analytics, Kafka for real-time data streaming, and Kubernetes for orchestrating the microservices architecture. The system ingests clinical research data, transforms it through configurable pipelines, and serves it through analytics dashboards.
Key Deliverables
- VisualFlow visual ETL tool for building data pipelines without code
- ClickHouse OLAP data warehouse for high-performance clinical analytics
- Kafka-based real-time data streaming pipelines
- Kubernetes-orchestrated microservices for scalability
- Multiple data source connectors for clinical research systems
- REDCap (Research Electronic Data Capture) integration
- Separate frontend, backend, and jobs components for VisualFlow
Results & Impact
RedCap gained a scalable data platform capable of handling clinical research data at volume. VisualFlow reduced the time needed to build new ETL pipelines from weeks to hours, while ClickHouse provided sub-second query performance on analytical workloads that previously took minutes. The Kubernetes deployment ensures the platform scales with growing data volumes.