Channel Public Channel

DataEng

@dataeng

Data Engineering & Distributed Systems

Contact @adilkhash

Open in Telegram

Subscribers: 4.39K
Photos: 43
Videos: 0
Links: 547

Showing posts older than #230 · Back to latest

Older Posts 20 shown

Post #229 4.37K Nov 8, 2020, 05:01 UTC

Отличный материал про сравнение самых популярных облачных хранили: BigQuery, Amazon Redshift и Snowflake — https://poplindata.com/data-warehouses/2020-database-showdown-bigquery-vs-redshift-vs-snowflake/

Snowplow Snowplow Behavioral Data Platform - Fuel AI, Analytics, Marketing | Snowplow Snowplow empowers organizations to unlock the value of its customer behavioral data in their cloud data warehouse to fuel next-gen AI, analytics, and marketing.

Post #228 800 Nov 6, 2020, 04:45 UTC

Для Redshift выпущен коннектор под Python: https://github.com/aws/amazon-redshift-python-driver

GitHub GitHub - aws/amazon-redshift-python-driver: Redshift Python Connector. It supports Python Database API Specification v2.0. Redshift Python Connector. It supports Python Database API Specification v2.0. - aws/amazon-redshift-python-driver

Post #227 1.19K Nov 1, 2020, 11:26 UTC

https://a16z.com/2020/10/15/the-emerging-architectures-for-modern-data-infrastructure/

Andreessen Horowitz Emerging Architectures for Modern Data Infrastructure This is an updated version of a post we originally published in 2020. You can read the original version here. The growth of the data infrastructure industry has continued unabated since we published a set of reference architectures in …

Post #226 1.17K Oct 31, 2020, 15:01 UTC

Внезапно! От издательства Packt Publishing вышла книга Data Engineering with Python: https://www.packtpub.com/product/data-engineering-with-python/9781839214189
В книге акцент уделён построению дата пайплайнов на Apache Airflow и Apache NiFi. Также есть главы, посвященные Kafka и Spark.

Packt Data Engineering with Python | Data | Paperback Work with massive datasets to design data models and automate data pipelines using Python. 24 customer reviews. Top rated Data products.

Post #225 985 Oct 31, 2020, 10:43 UTC

В Dagster появилась интеграция с dbt: https://dagster.io/blog/dagster-dbt

dagster.io Dagster and dbt: Better Together | Dagster Blog People sometimes ask us — should I use Dagster, or should I use dbt? We view Dagster and dbt as complementary technologies, not competing technologies.

Post #224 900 Oct 30, 2020, 16:19 UTC

https://www.astronomer.io/blog/introducing-airflow-2-0/

www.astronomer.io What’s Apache Airflow® 2.0 All About? Get to know the highlights of Apache Airflow® 2.0 and see hundreds of new features it includes. Have a look at how Airflow 2.0 compares to Airflow 1.10.

Post #223 1.09K Oct 27, 2020, 16:03 UTC

Serverless Data Lake Framework Workshop :: Serverless Data Lake Framework (SDLF) Workshop
https://sdlf.workshop.aws/

sdlf.workshop.aws Serverless Data Lake Framework (SDLF) Workshop

Post #222 1.11K Oct 26, 2020, 12:22 UTC

Про Apache Beam за 12 минут: https://www.youtube.com/watch?v=yZUe4th9gwY

YouTube Apache Beam Explained in 12 Minutes Apache Beam is a popular parallel processing framework. In this video, Alexandra will give you an overview of Apache Beam and by the end of the video you will hopefully have the skills that you need to write a simple pipeline. Source code - https://gith…

Post #221 1.17K Oct 16, 2020, 03:48 UTC

Кейс нашей компании по использованию облачного сервиса AWS: https://aws.amazon.com/ru/solutions/case-studies/playrix/

Amazon Playrix Case Study Playrix uses AWS to produce a 10-times increase in analytic querying speed and store and process 5 TB of data with zero bottlenecks to garner strong data insights that improve the end user experience.

Post #220 1K Oct 15, 2020, 07:07 UTC

https://www.freecodecamp.org/news/scalable-data-analytics-pipeline/

freeCodeCamp.org How to Build a Scalable Data Analytics Pipeline By Priyanka Vergadia Every application generates data, but what do those data mean? This is a question all data scientists are hired to answer. There is no doubt that this information is the most precious commodity for a business. But making sense o...

Post #219 987 Oct 13, 2020, 10:58 UTC

Хорошая вводная статья про Amazon DynamoDB: https://medium.com/@thomas.storm/amazon-dynamodb-deep-dive-chapter-1-essentials-cb4df19356a0

Medium Amazon DynamoDB Deep Dive. Chapter 1: Essentials The walkthrough of the one of the world’s fastest databases in a human-friendly format

Post #218 838 Oct 12, 2020, 07:09 UTC

https://youtu.be/RGslyyVpLQE

YouTube Apache Arrow Flight: Accelerating Columnar Dataset Transport (Wes McKinney, Ursa Labs) CMU Database Group - Quarantine Tech Talks (2020) Speaker: Wes McKinney (Ursa Labs) Apache Arrow Flight: Accelerating Columnar Dataset Transport October 5, 2020 https://db.cs.cmu.edu/seminar2020/#db21 Sponsored by the Steven Moy Foundation for Keeping it…

Post #217 935 Oct 10, 2020, 04:28 UTC

Гайд про XComs в Apache Airflow: https://www.youtube.com/watch?v=zw9Vy42IwG0

YouTube Getting started with XComs in Apache Airflow Welcome in Getting started with XComs in Apache Airflow My name is Marc Lamberti and I'm super excited to see you there. Indeed, you are going to discover how to get started with XComs in Apache Airflow. As you can see from the curriculum below, you are…

Post #216 1.21K Oct 7, 2020, 14:58 UTC

Ищите проект в области data engineering на котором можно потренироваться и чтобы не было стыдно показать при устройстве на работу?

Тогда эта статья для вас: https://www.startdataengineering.com/post/data-engineering-project-for-beginners-stream-edition/

Start Data Engineering Data Engineering Project: Stream Edition – Start Data Engineering Stream processing differs from batch; one needs to be mindful of the system’s memory, event order, and system recovery in case of failures. However, understanding the fundamental concepts of time attributes, cluster memory, time-bounded joins, and system…

Post #215 1.52K Oct 7, 2020, 06:12 UTC

Хороший доклад про базы данных от сотрудницы Яндекса: https://www.youtube.com/watch?v=YjSIdz8DnAo

YouTube 04. Базы Данных – Татьяна Денисова Во время лекции мы поговорим о том, что такое данные, какие базы данных бывают и чем они отличаются. Вы узнаете, какие особенности работы с базами данных нужно иметь в виду разработчику.Обсудим, как характеризовать, структурировать и хранить данные — с расчетом…

Post #214 908 Oct 5, 2020, 10:37 UTC

Доклады с последнего DE or DIE митапа:

- Своя песочница – как сделать кластер для инженера данных
- Data governance – что это, зачем, и с чего начать

YouTube DE or DIE #4. Артем Селезнев – Своя песочница – как сделать кластер для инженера данных Материалы всех наших митапов: https://deordie.com Наш чат в Telegram: https://t.me/deordie_chat Новые события сообщества DE or DIE: https://deordie.timepad.ru/events/ Автор доклада: Артем Селезнев, Senior Data Engineer, Сбербанк. Отличные коробочные решения…

Post #213 843 Oct 4, 2020, 12:10 UTC

Смотрите "B-tree indexes - learn more about the heart of PostgreSQL" на YouTube
https://youtu.be/n5-xEEQFqPY

YouTube B-tree indexes - learn more about the heart of PostgreSQL Talk by Anastasia Lubennikova https://www.socallinuxexpo.org/scale/18x/presentations/b-tree-indexes-learn-more-about-heart-postgresql This talk will offer a deep dive into the B-tree indexes architecture. It covers the newest major features of PostgreSQL…

Post #212 999 Oct 3, 2020, 02:58 UTC

Митап о Clickhouse: https://www.youtube.com/watch?v=1LVJ_WcLgF8

YouTube ClickHouse: онлайн-встреча, 1 октября 2020 Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

Post #211 1.05K Oct 1, 2020, 14:38 UTC

Forwarded from Artem Pichugin

Всем привет! Уже меньше, чем через час начнется наш 4-й митап. Ссылку на трансляцию мы отправили на почту зарегистрировавшимся на Timepad, но на всякий случай дублируем и здесь: https://youtu.be/0IC9vHNZTNk

Напомню, что план мероприятия следующий:
18:00-19:00. Своя песочница – как сделать кластер для инженера данных
Артем Селезнев, Senior Data Engineer, Сбербанк

19:00-19:15. Перерыв

19:15-20:15. Data governance – что это, зачем, и с чего начать
Андрей Вихров, Главный системный аналитик, Связной

Вопросы можно будет задавать на трансляции в YouTube, либо здесь в чате. До встречи!

Post #210 801 Oct 1, 2020, 06:15 UTC

В AWS появилась в общем доступе Time series база данных Amazon Timestream: https://aws.amazon.com/ru/about-aws/whats-new/2020/09/amazon-timestream-now-generally-available/

Amazon Web Services, Inc. Amazon Timestream is now Generally Available

Older posts →