List: data engineering | Curated by Suman Kumar Gangopadhyay

Aug 11, 2024

38 stories

data engineering 
In
Snowflake Engineering
by
Maeda Kentaro
Snowflake Unveils Their Own Notebook: 6 Game-Changing FeaturesUse Snowflake Notebooks for data analysis on Snowflake
Jun 1, 2024
23
Jun 1, 2024
23
In
In the Pipeline
by
Dave Flynn
How to use a Makefile to speed up your dbt project workflowLearn how to use a makefile to reduce dbt command fatigue and group related commands for easy reuse and sharing.
Jun 13, 2024
112
2
Jun 13, 2024
112
2
In
Snowflake Builders Blog: Data Engineers, App Developers, AI/ML, & Data Science
by
Maja Ferle
Monitoring Data Quality Natively in SnowflakeUsing Snowflake’s data metric functions to monitor the quality of your data, such as data freshness, duplicate counts, number of NULL…
May 6, 2024
222
May 6, 2024
222
In
Snowflake Builders Blog: Data Engineers, App Developers, AI/ML, & Data Science
by
Somen Swain
9 Snowflake Tables- A briefing(as of 2023)In this blog I would be discussing about various Snowflake tables and also some of the use cases which each of these tables can solve. Post…
Dec 19, 2023
121
3
Dec 19, 2023
121
3
In
TDS Archive
by
LucianoSphere (Luciano Abriata, PhD)
The Most Advanced Libraries for Data Visualization and Analysis on the WebA careful selection looking for performance, flexibility, and richness of features.
Feb 13, 2024
533
2
Feb 13, 2024
533
2
Sahil Singla
The Optimal Way to (Automatic) Cluster in SnowflakeA simple strategy to maximize the ROI of automatic clustering
Feb 13, 2024
60
Feb 13, 2024
60
Sahil Singla
The fastest way to reduce Snowflake costs by 20%Stop letting your warehouses run idle.
Jan 3, 2024
25
Jan 3, 2024
25
In
Snowflake Builders Blog: Data Engineers, App Developers, AI/ML, & Data Science
by
Sudhendu
Data Sharing Patterns in SnowflakeData Sharing is a Snowflake feature that allows you to share your Snowflake Data with users in other Snowflake accounts.
Sep 14, 2023
23
1
Sep 14, 2023
23
1
Amar Kumar Gupta
Join Strategies in Apache Spark — A hands-on approach!Basics and Pre-requisites:
Aug 13, 2023
130
6
Aug 13, 2023
130
6
In
Data Engineer Things
by
Ravish Kumar
Supercharge Your Data Pipelines with Advanced Apache ArrowAdvanced features and real-world use cases of Apache Arrow.
Aug 7, 2023
165
2
Aug 7, 2023
165
2
Rasiksuhail
dbt Beyond the Basics: Advanced Strategies for ImplementationAdvanced dbt topics are useful for data professionals who are looking to enhance their data modeling and analytics capabilities. With…
May 11, 2023
87
May 11, 2023
87
In
SurrealDB
by
Pratim Bhosale
Understanding SurrealQL and how it is different from PostgreSQLPostgreSQL has been around for a while now and is the go-to choice for applications that need reliability, performance and scalability…
Apr 4, 2023
113
1
Apr 4, 2023
113
1
In
Astrafy
by
Charles Verleyen
Our top dbt packages pickEvery successful individual knows that his or her achievement depends on a community of persons working together. — Paul Ryan
Nov 11, 2022
70
1
Nov 11, 2022
70
1
Ashutosh Kumar
Spark Connect — Apache Spark 3.4During the last decade, Spark has been widely used by developers, researchers, and the community to create tens of thousands of data…
Apr 24, 2023
74
Apr 24, 2023
74
Alsathamhussain
Salting technique in Spark: A Thrilling Interview ExperienceJohn was a data engineer who had been studying PySpark for weeks in preparation for an upcoming interview. He had learned about many…
May 4, 2023
65
2
May 4, 2023
65
2
Dagster Blog
Deciphering arcane Kubernetes and ECS errors with Dagster | Dagster BlogRecent enhancements allow Dagster to surface clearer and more actionable errors to accelerate your development cycles.
May 18, 2023
35
May 18, 2023
35
Dagster Blog
Introducing Dynamic Definitions for Flexible Asset Partitioning | Dagster BlogPartitioning improves the performance and manageability of data. But what if you don’t know the set of partitions up-front?
May 19, 2023
39
May 19, 2023
39
Open Data Analytics
Top 10 Best Open Source Projects on GitHub 2023Open Source Software (OSS) has revolutionized the way software development is done today. With millions of Open Source GitHub projects…
Feb 6, 2023
3.4K
29
Feb 6, 2023
3.4K
29
In
Better Programming
by
Jack C
dbt v1.5 — the 3 Big New ThingsData contracts, model versions, and model access
Apr 28, 2023
341
6
Apr 28, 2023
341
6
Dagster Blog
Declarative Scheduling for Data Assets — a breakthrough in data orchestrationby Sandy Ryza
Dec 12, 2022
4
1
Dec 12, 2022
4
1

data engineering

Snowflake Unveils Their Own Notebook: 6 Game-Changing Features

Use Snowflake Notebooks for data analysis on Snowflake

How to use a Makefile to speed up your dbt project workflow

Learn how to use a makefile to reduce dbt command fatigue and group related commands for easy reuse and sharing.

Monitoring Data Quality Natively in Snowflake

Using Snowflake’s data metric functions to monitor the quality of your data, such as data freshness, duplicate counts, number of NULL…

9 Snowflake Tables- A briefing(as of 2023)

In this blog I would be discussing about various Snowflake tables and also some of the use cases which each of these tables can solve. Post…

The Most Advanced Libraries for Data Visualization and Analysis on the Web

A careful selection looking for performance, flexibility, and richness of features.

The Optimal Way to (Automatic) Cluster in Snowflake

A simple strategy to maximize the ROI of automatic clustering

The fastest way to reduce Snowflake costs by 20%

Stop letting your warehouses run idle.

Data Sharing Patterns in Snowflake

Data Sharing is a Snowflake feature that allows you to share your Snowflake Data with users in other Snowflake accounts.

Join Strategies in Apache Spark — A hands-on approach!

Basics and Pre-requisites:

Supercharge Your Data Pipelines with Advanced Apache Arrow

Advanced features and real-world use cases of Apache Arrow.

dbt Beyond the Basics: Advanced Strategies for Implementation

Advanced dbt topics are useful for data professionals who are looking to enhance their data modeling and analytics capabilities. With…

Understanding SurrealQL and how it is different from PostgreSQL

PostgreSQL has been around for a while now and is the go-to choice for applications that need reliability, performance and scalability…

Our top dbt packages pick

Every successful individual knows that his or her achievement depends on a community of persons working together. — Paul Ryan

Spark Connect — Apache Spark 3.4

During the last decade, Spark has been widely used by developers, researchers, and the community to create tens of thousands of data…

Salting technique in Spark: A Thrilling Interview Experience

John was a data engineer who had been studying PySpark for weeks in preparation for an upcoming interview. He had learned about many…

Deciphering arcane Kubernetes and ECS errors with Dagster | Dagster Blog

Recent enhancements allow Dagster to surface clearer and more actionable errors to accelerate your development cycles.

Introducing Dynamic Definitions for Flexible Asset Partitioning | Dagster Blog

Partitioning improves the performance and manageability of data. But what if you don’t know the set of partitions up-front?

Top 10 Best Open Source Projects on GitHub 2023

Open Source Software (OSS) has revolutionized the way software development is done today. With millions of Open Source GitHub projects…

dbt v1.5 — the 3 Big New Things

Data contracts, model versions, and model access

Declarative Scheduling for Data Assets — a breakthrough in data orchestration

by Sandy Ryza

Suman Kumar Gangopadhyay

better python

data engineering

ideas