๐๐๐ถ๐น๐ฑ๐ถ๐ป๐ด ๐ฎ ๐ฆ๐ฎ๐น๐ฒ๐๐ณ๐ผ๐ฟ๐ฐ๐ฒ ๐๐ ๐๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป ๐ฃ๐ถ๐ฝ๐ฒ๐น๐ถ๐ป๐ฒ ๐ณ๐ฟ๐ผ๐บ ๐ฆ๐ฐ๐ฟ๐ฎ๐๐ฐ๐ต: ๐ฃ๐ฎ๐ฟ๐ ๐ญ โ ๐๐๐๐ต๐ฒ๐ป๐๐ถ๐ฐ๐ฎ๐๐ถ๐ผ๐ป ๐๐ถ๐๐ต ๐๐ต๐ฒ ๐ฆ๐ข๐๐ฃ ๐๐ฃ๐
Introduction Hello! Welcome to my new series of articles focused on building a Salesforce extraction pipeline from scratch, without relying on any special libraries or SaaS services. In this series, I will use Python to demonstrate the pipeline, but I aim to present it in a generalized way that should allow you to port it […]
Generalize to Optimize: Enhancing Data Pipelines for Future Growth
Introduction When designing a data pipelineโor any project, for that matterโit’s best to build it as generalized or abstracted as possible. While an exception might be made if it’s intended only for a single, specific data source or target, things rarely remain static in the long run. A well-built pipeline will likely be used by […]
Stateless Distributed ETL Pipelines: Using Message Brokers for Horizontal Scaling
Introduction Creating a stateless distributed ETL (Extract, Transform, Load) pipeline can be a daunting task, especially if you’re new to the concept. While there are many approaches to building such pipelines, one simple and effective method is to use message brokers. Tools like Kafka Streams, RabbitMQ, and even Redis Streams can simplify the process of constructing a stateless distributed pipeline. […]
Create ETL pipelines (almost)Entirely in Snowflake
In this article, I will demonstrate how both basic and slightly in-depth ETL can be facilitated almost entirely in Snowflake. The only outside elements would be some form of cloud storage (s3 bucket for example) and a cloud notification service (AWS SNS, or GCP pub/sub). These external items could be avoided, but it would require any of […]
How to Understand Project Stakeholder Needs
In this blog, I will convey some tips/strategies I have developed over the years to get the most out of project kickoff meetings. The main goal of these meetings should be to get a true understanding of what the stakeholders (the people who will use/benefit from the project in the end) need/want when all is […]