lakehouse lab short-form video show has formally launched; a new episode will be published each week and tool, process, and topic suggestions are encouraged
Tag Archives: data_engineering
well, shoot… (i guess i do like writing queries via chat)
i actually like writing code and did not imagine i would enjoy just asking my business data analysis questions with natural language, but i’m nothing if not flexible and open to reevaluating my opinions
my iceberg book (early release 1)
super stoked to announce the first early release of my upcoming o’reilly book, optimizing your apache iceberg lakehouse, has been published — pull down the pdf and let me know what you think
understanding iceberg deletion vectors (and enjoying some humble pie)
for or a given iceberg snapshot, there can be 0 or 1 deletion vector per data file & a deletion vector cannot span more than one data file
building trino data pipelines (with sql or python)
trino is well-known as a fast query engine, but it is also a robust transformation processing engine that allows data engineers to developer in sql and/or python
yarp: yet another rag post (this time using sql)
you don’t have to know python or bother your data scientists to start exploring genai concepts like rag; you just need a tool that offers these features in a familiar sql interface
trino query plan analysis (video series)
query plan analysis is critical for getting every single ounce of performance & scalability out of your trino cluster; my 3-part video series will get you started with the basics
delta lake time-travel (just reference the version)
trino’s delta lake connector offers features around versioning to include comparing versions and time-travel querying
unstructured docs in ai (the wild west)
rag ai apps can only be as good as the parsed and chunked data that fuels them – testing, testing, and more testing the outputs of all the various available libraries with the front-end apps is critical
apache spark (yet another overview)
an overview of apache spark presented from 20,000 feet, on the surface, and below the waterline