Exploring the what, why, and how of design docs for data components — and why they matter. — Over the past few years, adopting software engineering best practices has become a common theme within the data engineering space. From dbt’s software-engineering-inspired capabilities to the rise of data observability, data engineers are getting increasingly accustomed to the software engineer’s toolset and principles.

A detailed guide to help you navigate the modern data stack and build your own platform using open-source technologies. — In the past few years, the data engineering field’s meteoric rise in importance opened the door for accelerated innovation and progress — since today more people than ever are thinking about data resources and how to better leverage them. …

A set of recommendations and starting points to efficiently run Superset at scale — When it comes to the Business Intelligence (BI) ecosystem, proprietary tools were the standard for a very long period. Tableau, Power BI, and more recently Looker, were the go-to solutions for enterprise BI use cases. But then, Apache Superset happened. Frustrated with the multiple inconveniences of relying on a proprietary…

In an ever-changing tech ecosystem, notebooks are steadily replacing BI dashboards as the standard tool when it comes to extracting value from data. Just a few years ago, notebooks were merely used for ad-hoc data exploration and analysis, while dashboards were the norm when it came to business-oriented analytics, KPIs…

An in-depth look into how an API-focused data catalog can help you ensure the success of your data platform via combining different types of metadata — Over the past few years, the data field went through a major shift in how data platforms are designed, built, and used. This change, which can be referred to as the “third wave” of data technologies, ushered in the era of the Modern Data Stack. An era in which scalability…

An in-depth look into the past and present of data validation, and how you can leverage today’s tools to ensure data quality at scale — With the rising interest in the Modern Data Stack (MDS) and its growing ecosystem of data technologies and SaaS offerings, data teams are heavily betting on it to open new doors of innovation and capabilities. …

Thanks for sharing this - the architecture is really impressive and it's great to see Livy leveraged in this way! I experimented with a similar design a couple of years ago but one problem I had with Livy was the fact that it doesn't support concurrent queries within the same session (https://issues.apache.org/jira/browse/LIVY-544), even when the FAIR scheduler is activated. Is this something you also modified? Because for interactive use-cases (especially in a notebook-based environment) users might want to run more than one query at a time.

A guide to building an automated MLOps pipeline by leveraging the trusted DevOps toolset. — Throughout the past few years, Machine Learning (ML) saw an exponential rise in popularity thanks to the advancements led by companies like Google and Facebook, along with the contributions of the open-source community. …

After more than a decade of using Big Data systems, data professionals have a set of new challenges to face in the upcoming year. — Apache Spark, the open-source project that fuels most of the world’s data pipelines, turned 10 years old in 2020. …