Channel Public Channel

DataEng

@dataeng

Data Engineering & Distributed Systems

Contact @adilkhash

Open in Telegram

Subscribers: 4.39K
Photos: 43
Videos: 0
Links: 547

Showing posts older than #52 · Back to latest

Older Posts 20 shown

Post #51 619 Apr 23, 2019, 07:00 UTC

Нашел очень интересный доклад от Chief Data Engineer компании Paypal Sid Anand про data инфраструктуру компании. К слову, Paypal ежедневно обрабатывает более 1 миллиарда долларов транзакций в 200 странах. Как всё это работает и не падает смотрите в докладе

YouTube Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

Post #50 5.18K Apr 17, 2019, 09:54 UTC

В блоге AirBnB вышла полезная статья про идемпотентный фреймворк для построения распределенных систем.

В самой статье идёт речь про кейс реализации системы предотвращающей двойное списание денег у клиентов сервиса.

Читаем: https://medium.com/airbnb-engineering/avoiding-double-payments-in-a-distributed-payments-system-2981f6b070bb

Medium Avoiding double payments in a distributed payments system How we built a generic idempotency framework to achieve eventual consistency and correctness across our payments micro-service…

Post #49 602 Apr 10, 2019, 07:01 UTC

Если вы пользуетесь Apache Airflow для построения batch-процессов и хотите знать тонкости его работы, то рекомендую к прочтению пост: How Apache Airflow Distributes Jobs on Celery workers

www.sicara.ai How Apache Airflow Distributes Jobs on Celery workers Discover what happens when Apache Airflow performs task distribution on Celery workers through RabbitMQ queues.

Post #48 571 Apr 9, 2019, 10:01 UTC

Принципы построения дата пайплайнов из презентации:

- Understand the data consumer
- Understand the data
- Keep data in its raw form
- Do not delete or move your raw data
- Validate your extracted data before saving
- Transform your data over all time
- Separate you E-TL
- Minimize the number of data and compute nodes
- Store all you data of practically feasible to do so
- Make your ETL acyclical
- Validate your data before its given to consumers
- Join your data at the database level
- Monitor your data

Post #47 530 Apr 9, 2019, 09:31 UTC

Хороший доклад про принципы построения пайплайнов без привязки к каким-либо инструментам: Data Engineering Principles - Build frameworks not pipelines

YouTube Data Engineering Principles - Build frameworks not pipelines - Gatis Seja PyData London Meetup #54 Tuesday, March 5, 2019 Data pipelines are necessary for the flow of information from its source to its consumers, typically data scientists, analysts and software developers. Managing data flow from many sources is a complex task…

Post #46 2.09K Apr 7, 2019, 06:01 UTC

В техническом блоге DataDog вышел крутой материал про построение надёжных дата пайплайнов: Building Highly Reliable Data Pipelines at Datadog

Datadog Building Highly Reliable Data Pipelines at Datadog | Datadog A look at how Datadog builds and operates data pipelines reliably at scale.

Post #44 62 Mar 26, 2019, 06:00 UTC

Forwarded from DevBrain

Собрал свои мысли в одно место и таким образом появилась статья Как статья Data Engineer. Область Data Engineering активно растёт, надеюсь материал будет полезен!

Khashtamov Как стать Data Engineer Сейчас специализация в области data engineering активно набирает обороты. Судя по отчёту компании hired.com, спрос на data engineer специалистов вырос на 38%, и рост продолжится. Средняя зарплата у...

Post #43 913 Mar 24, 2019, 10:23 UTC

В подкасте Software Engineering Daily вышел эпизод про Apache Superset с его создателем Maxime Beauchemin: https://softwareengineeringdaily.com/2019/03/22/apache-superset-with-maxime-beauchemin/

Software Engineering Daily Apache Superset with Maxime Beauchemin - Software Engineering Daily Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Data engineering touches every area of an organization. Engineers need a data platform to build search indexes and microservices.…

Post #42 603 Mar 6, 2019, 08:39 UTC

Пост в блоге про новый nano degree: https://blog.udacity.com/2019/03/learn-data-engineering-nanodegree-program.html

Udacity Introducing the Udacity Data Engineering Nanodegree Program | Udacity The ratio of data engineer to data scientist job openings is four-to-one. Get skills to qualify for these roles in the Data Engineering Nanodegree program.

Post #41 487 Mar 6, 2019, 08:17 UTC

На Udacity вышел Nano degree про Data Engineering: https://www.udacity.com/course/data-engineer-nanodegree--nd027, цена правда заоблачная - $999

Udacity Data Engineering Training Course | Become a Data Engineer | Udacity Data Engineering is the foundation of Big Data. Enroll in our data engineering with AWS training course and learn essential skills to become a data engineer.

Post #40 77 Feb 28, 2019, 09:28 UTC

Forwarded from DevBrain

Post #39 560 Feb 26, 2019, 07:00 UTC

История развития Apache Flink в блоге Alibaba Tech: https://medium.com/@alitech_2017/a-brief-history-of-flink-tracing-the-big-data-engines-open-source-development-87464fd19e0f

Medium A Brief History of Flink: Tracing the Big Data Engine’s Open-source Development From version 1.1.0 to 1.6.0, Apache Flink’s relentless improvement exemplifies open-source development.

Post #38 521 Feb 25, 2019, 15:00 UTC

Неплохое введение в распределённую БД FoundationDB от компании Apple: https://tech.marksblogg.com/minimalist-guide-tutorial-foundationdb.html

Marksblogg A Minimalist Guide to FoundationDB Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More...

Post #37 517 Feb 25, 2019, 07:01 UTC

Классный доклад про DB Event Streaming на Qcon: https://www.infoq.com/presentations/wepay-database-streaming

InfoQ The Whys and Hows of Database Streaming Joy Gao talks about how database streaming is essential to WePay's infrastructure and the many functions that database streaming serves. She provides information on how the database streaming infrastructure was created & managed so that others can leverage…

Post #36 5.34K Feb 7, 2019, 06:32 UTC

Вебинары на тему RabbitMQ vs. Kafka:

Part I
Part II

YouTube RabbitMQ vs Kafka - Jack Vanlightly x Erlang Solutions webinar RabbitMQ vs Kafka Messaging is at the core of many architectures and two giants in the messaging space are RabbitMQ and Apache Kafka. In this webinar we'll take a look at RabbitMQ and Kafka within the context of real-time event-driven architectures. In this…

Post #34 616 Jan 31, 2019, 07:00 UTC

На сайте baseDS тем временем вышли 2 новые статьи:

- Transparency: Illusions of a Single System (Part 1)
- Transparency: Illusions of a Single System (Part 2)

Medium Transparency: Illusions of a Single System (Part 1) Even though we might be new to distributed systems, by now we can see that, by definition, they involve many moving parts. Those moving…

Post #33 588 Jan 29, 2019, 08:15 UTC

Внутреннее устройство PostgreSQL: http://www.interdb.jp/pg/index.html

Post #32 709 Jan 12, 2019, 15:39 UTC

Пару дней назад от программы Insight Data Engineering прошел вебинар на тему Transitioning to Data & DevOps Engineering. Его цель - познакомить начинающих Data/DevOps инженеров со сферой и помочь в неё плавно окунуться.

Помимо вебинара вам также может быть полезна их статья Preparing for the Transition to Data Engineering

YouTube Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

Post #31 689 Jan 9, 2019, 09:00 UTC

Does my Startup Data Team Need a Data Engineer?

Отличный пост, где автор рассуждает о роли Data Engineer в современных data организациях. Основная идея в том, что привычные ETL задачи легко автоматизируются благодаря сервисам типа Stitch, поэтому роль Data Engineer смещается в сторону построения data infrastructure со всеми вытекающими (reliability, data consistency, monitoring и т.д.), активной работе с командой аналитиков (data scientists, data analysts). Автор утверждает, что дата инженер это командный игрок роль которого всячески оказывать поддержку людям, формирующим выводы из данных.

А ещё мне понравилась фраза: data engineers don’t provide direct business value—their value comes in making your data analysts and scientists more productive.

Must read!

Fishtown Analytics Does my Startup Data Team Need a Data Engineer? The role of the data engineer in a startup data team is changing rapidly. Are you thinking about it the right way?

Post #30 533 Jan 8, 2019, 13:59 UTC

Apache Airflow стал полноценным членом Apache Software Foundation: https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces44

Older posts →