Smartface, technologies and decay
@antonrevyako
Post #1142
2.11K
Channel Public Channel
TH There will be no singularity
@nosingularity
- Subscribers
- 1.96K
- Photos
- 258
- Videos
- 15
- Links
- 1K
Showing posts older than #1143 · Back to latest
Older Posts 20 shown
Post #1141
1.94K
Post #1140
1.82K
Post #1139
1.54K
☕️ Мерлин заваривает τσάι 🐌 Сейчас с трепетом обнаружил что база данных пакетов репозитория brew – это директория, в которой лежат тысячи небольших руби файлов с манифестами пакетов. Теперь не удивлён, что он такой тормозной
Post #1138
1.48K
Post #1137
1.5K
Post #1136
1.57K
Post #1135
1.64K
Post #1134
1.71K
There will be no singularity https://supabase.com/blog/postgres-wasm
Post #1133
2.21K
Post #1132
2.37K
Post #1131
4.93K
Post #1130
2.2K
Post #1129
2.4K
Hey, it's been a while since we had any updates on our projects...
Just to refresh your memory, all of our projects dwh.dev, holistic.dev, and parsers.dev only work with raw SQL code, without any database connection.
But, dwh.dev now needs a database connection. We need to connect to Snowflake, dump the schema into PostgreSQL, and regularly check for schema changes.
The task does not seem to be difficult. But if it doesn't, you need to make it difficult, otherwise, it will be boring :)
So, the part that transfers data from one database to another was decided to be open-sourced.
"Anton invented the open-source Fivetran!", - shouts are heard from the auditorium.
There are many data transfer tools out there, but with a catch... They either no-code approach (Fivetran and others) or something in python.
But, we love SQL here, right?
Let's do it like Snowflake: everything in a SQL-like language!
Snowflake has this construction called PIPE:
The desire was to make something similar, but more universal.
Today's no details, it's all in progress :)
I want to talk about something else. Connecting to a server that'll understand and do everything we want can be done in different ways. One of them is pg-protocol.
Supporting pg-protocol opens up a lot of possibilities, but I'll talk about that later :)
Suddenly, it turned out that there's no proper implementation of pg-protocol in the js-ecosystem (yes, yes, all in typescript again). Had to do it ourselves...
Not like it was super complicated until we tried connecting to our server from the IDE.
It'd be cool to connect from any IDE with pg-support, right?
When connecting, the IDE makes a lot of requests: set encoding, ask for schemas, types, etc.
And without proper responses to these requests, the IDE refuses to connect.
It looks like we'll have to route these requests to a special pg-instance, filtering out calls that we don't need...
Or maybe use postgres-wasm?
Waiting for your ideas in the comments.
Just to refresh your memory, all of our projects dwh.dev, holistic.dev, and parsers.dev only work with raw SQL code, without any database connection.
But, dwh.dev now needs a database connection. We need to connect to Snowflake, dump the schema into PostgreSQL, and regularly check for schema changes.
The task does not seem to be difficult. But if it doesn't, you need to make it difficult, otherwise, it will be boring :)
So, the part that transfers data from one database to another was decided to be open-sourced.
"Anton invented the open-source Fivetran!", - shouts are heard from the auditorium.
There are many data transfer tools out there, but with a catch... They either no-code approach (Fivetran and others) or something in python.
But, we love SQL here, right?
Let's do it like Snowflake: everything in a SQL-like language!
Snowflake has this construction called PIPE:
create pipe mypipe2 as copy into mytable(c1, c2) from (select $5, $4 from @mystage);And then logs can be taken from a special table.
The desire was to make something similar, but more universal.
Today's no details, it's all in progress :)
I want to talk about something else. Connecting to a server that'll understand and do everything we want can be done in different ways. One of them is pg-protocol.
Supporting pg-protocol opens up a lot of possibilities, but I'll talk about that later :)
Suddenly, it turned out that there's no proper implementation of pg-protocol in the js-ecosystem (yes, yes, all in typescript again). Had to do it ourselves...
Not like it was super complicated until we tried connecting to our server from the IDE.
It'd be cool to connect from any IDE with pg-support, right?
When connecting, the IDE makes a lot of requests: set encoding, ask for schemas, types, etc.
And without proper responses to these requests, the IDE refuses to connect.
It looks like we'll have to route these requests to a special pg-instance, filtering out calls that we don't need...
Or maybe use postgres-wasm?
Waiting for your ideas in the comments.
Post #1128
1.85K
Post #1127
2.69K
Post #1126
2.99K
Post #1125
2.38K
There will be no singularity В эфире наша постоянная рубрика «шо там у рубистов» https://t.me/oleg_log/2697 So let’s talk about what we spend on our programming language and web framework. It’s about 15% Дальше идет сухой расчет, что это всего какие-то $450k в год, «у нас в долине…
Post #1124
2.15K
Post #1123
2.05K
everything is better with bluetooth rust. even a python…
https://twitter.com/aberges4/status/1614226340349108224
Twitter Processing 800k lines CSV file in #pandas vs #duckdb vs #polars, involving a fairly long regex:
pandas: 9.32s
DuckDB: 3.05s
polars: 120ms
wow. https://twitter.com/aberges4/status/1614226340349108224