a high-level overview of how functional programming with immutable datasets is a great partner with big data processing frameworks — code examples with spark rdds using scala
Tag Archives: data_engineering
building a spark sql udf with scala (using multiple arguments)
a short & sweet code-focused tutorial declaring a scala function as a spark sql udf that can be leveraged via the api approach or in a formal sql statement
joining spark dataframes with identical column names (not just in the join condition)
a quick walkthru of spark sql dataframe code showing joining scenarios when both tables have columns with the same name; this includes when they are used in the join condition as well as when they are not
topology supervision features of streaming frameworks (or lack thereof)
a smackdown of sort pitting kafka streams, spark streaming, and storm against each other — not for the features they give developers, but for the features they offer the operations side of the devops formula
presenting at hadoop summit (archiving evolving databases in hive)
overview of, and links to related artifacts for, my presentation at hadoop summit about strategies to handle changing data in hive’s immutable architecture
how do i load a fixed-width formatted file into hive? (with a little help from pig)
presents a couple of options for converting a fixed-width formatted file a a delimited one to prepare it to be exposed as a hive table