trino is well-known as a fast query engine, but it is also a robust transformation processing engine that allows data engineers to developer in sql and/or python
Tag Archives: software_development
yarp: yet another rag post (this time using sql)
you don’t have to know python or bother your data scientists to start exploring genai concepts like rag; you just need a tool that offers these features in a familiar sql interface
unstructured docs in ai (the wild west)
rag ai apps can only be as good as the parsed and chunked data that fuels them – testing, testing, and more testing the outputs of all the various available libraries with the front-end apps is critical
apache spark (yet another overview)
an overview of apache spark presented from 20,000 feet, on the surface, and below the waterline
google codelabs (my go-to tutorial authoring framework)
i’ve been creating technical hands-on tutorials for years, mostly in blog posts, but for standing up a site focused on self-paced learning modules i haven’t found a better tool than google codelabs
develop, deploy, execute & monitor in one tool (welcome to apache nifi)
for those not familiar with apache nifi, come on a short overview of how this framework rather uniquely spans so many of the phases of the typical software development lifecycle
exploring ai data pipelines (hands-on with datavolo)
after explaining what rag ai apps are all about & showing what a typical ai data engineering pipeline looks like, i wanted to offer a hands-on lab exercise actually building a simple pipeline use datavolo cloud
understanding rag ai apps (and the pipelines that feed them)
i’m learning all about rag ai apps and wanted to try to explain, at a high-level, what these are all about plus do the same for the etl pipelines that are key to their success
joining spark dataframes with identical column names (an easier way)
presenting an easier solution to the problem of colliding column names when joining spark dataframes than i previously offered in my most popular post that just happens to be four years old — some things do age well
pystarburst in 90 seconds (try it)
still thinking about trying to get a pystarburst code stub up/n/running? starburst galaxy makes it pain free and you can even get your first dataframe created via python in under 90 seconds — why not give it a try?