Lets say you have 5 TB of data stored in your Amazon S3 bucket consisting of 500 million records and 100 columns.

Now, suppose there are 100 cities and you want to get the data for a particular city, and you want to retrieve only 10 columns.

~ considering each city has equal amount of records,
we want to get 1% of data in terms of number of rows
and 10% in terms of columns

thats roughly 0.1% of the actual data which might be 5 GB roughly.

Now lets the pricing if you are using serverless technology like AWS Athena

- the worst case you end up having the data in a csv format (row based) with no compression. you end up scanning the entire 5 TB data and you pay $25 for this query. (The charges are $5 for each TB of data scanned)

Now lets try to improve it..

- use a columnar file format like parquet with snappy compression which takes lesser space so your 5 TB data might roughly become 2 TB (actually it will be even lesser)

- partition this based on city so that we have 1 folder for each city.

This way you have 2 TB data sitting across 100 folders, but you have to scan just one folder which is 20 GB,

Not just this you need 10 columns out of 100 so roughly you scan 10% of 20 GB (as we are using columnar file format)

This comes out to be 2 GB only.

so how much do we pay?
just $.01 which is 2500 times lesser than what you paid earlier.

This is how you save cost.

what we did?

- using columnar file formats for column pruning
- using partitioning for row pruning
- using efficient compression techniques

Join our WhatsApp channel for more data engineering resources
👇👇
https://whatsapp.com/channel/0029Vaovs0ZKbYMKXvKRYi3C