topology supervision features of streaming frameworks (or lack thereof)

This blog post introduces the three streaming frameworks that are bundled in the Hortonworks Data Platform (HDP) – Apache Storm, Spark Streaming, and Kafka Streams – and focuses on the supervision features offered to the topologies (aka workflows) running with, or within, these particular frameworks. This post does not attempt to fully described each framework nor does it provide examples of their usage via working code. The goal is to develop an understanding of what, if any, services are available to help with lifecyle events, scalability, management, and monitoring.

The Frameworks

Kafka Streams

Kafka Streams is a client library for building applications and microservices, where the input and output data are stored in Kafka clusters. It combines the simplicity of writing and deploying standard Java and Scala applications on the client side with the benefits of Kafka’s server-side cluster technology. –http://kafka.apache.org/documentation/streams/

Kafka Streams are tightly coupled with Kafka’s messaging platform; especially the streaming input data. Kafka Streams is intentionally designed to fit into any Java or Scala application which gives it plenty of flexibility, but offers no inherent lifecycle, scaling, management, or monitoring features.

Spark Streaming

Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Kinesis, or TCP sockets, and can be processed using complex algorithms expressed with high-level functions like map, reduce, join and window. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. –http://spark.apache.org/docs/latest/streaming-programming-guide.html

Apache Spark’s streaming frameworks allow for a variety of input and output data technologies. Spark Streaming apps are themselves Spark applications who, in a Hadoop cluster at least, run under YARN which provides coverage for many of the lifecycle and management features. The Spark framework addresses a number of the scaling and monitoring needs.

Apache Storm

Apache Storm is a … distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, … Storm is simple, can be used with any programming language, … Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate. –http://storm.apache.org/

Storm topologies are assembled with components that can be built with any language, but primarily Java is used. It includes ready to use components for many queueing and database technologies. Storm is a comprehensive framework that solely focuses on streaming applications and has rich solutions addressing lifecyle events, scalability, management, and topology monitoring.

Feature Analysis

Summary & Recommendations

Let me start by pointing out that it looks like Kafka Streams is “all bad”, but that’s not the case. It is build around the concept of writing and deploying standard applications and consciously does not want be part of a runtime framework. Due to that and the focus of this blog post, it should be obvious why it scored so low on these features. The RYO (Roll Your Own – aka “custom”) callouts I gave are likely a badge of honor to the folks who are bringing us this framework.

Kafka Streams also has a lot of early interest and I surely would not discount it for a second. The biggest issue for those teams who stand up a decent sized Hadoop/Spark cluster is that you don’t get to take advantage of all those nodes to run your Kafka Streams apps on. You’ll need to size out what is needed for each application and ensure that needed resources are available to run your apps on.

On the other end of the spectrum, one would think that will an almost perfect green checkmark score on the features identified that Storm would be a no-brainer. Storm is the grandpa of the streaming engines and its event-level isolation provide something the other microbatch frameworks can’t do. This maturity shines through in all of these supervision features, but on the other hand it is the least “exciting” of the frameworks for folks starting their streaming journey in 2019. If you need to get something into production asap and you just need to know it works – all day long and every day… then go with Storm!

This brings me to my personal recommendation of Spark Streaming. Note that this comes from a guy who really does love Apache Storm and values the simplicity & flexibility of Kafka Streams. There is simply too much excitement & focus around Spark in general and the ability to transition applications between batch and streaming paradigms with minimal coding close the case. It is still maturing, but its alignment with YARN help it score high on many of these supervision-oriented features.

presenting at hadoop summit (archiving evolving databases in hive)

4/13/2020 UPDATE: see hive’s merge statement (it drops a lot of acid) as a much better way to solve this problem.

What a humbling experience to have the opportunity to present at the 2015 Hadoop Summit conference in San Jose.  I’ve done a decent number of user group presentations over the years, and even have presented Hadoop topics to audiences as big as 500, but this is the first time I have talked at a major industry conference and I had a blast.  It was just cool to have a “presenter” badge and to have my name in all of the conference literature.

My topic was Mutable Data in Hive’s Immutable World and here is the synopsis that is visible from the agenda.

Going beyond Hive`s sweet spot of time-series immutable data, another popular utilization is replicating RDBMS schemas. This “active archive” use case`s intention is not to capture every single change, but to update the current view of the source system at regular intervals. This breakout session will compare/contrast full-refresh & delta-processing approaches as well as present advanced strategies for truly “big” data. Those strategies not only parallelize their processing, but leverage Hive`s partitioning to intelligently target the smallest amount of data as possible to improve performance and scalability. Hive 14`s INSERT, UPDATE, and DELETE statements will also be explored.

I’ve loaded my slides up on SlideShare.  Please use http://www.slideshare.net/lestermartin/mutable-data-in-hives-immutable-world if the preview below is having troubles.

I was lucky enough to find out that Jennifer Knight took some “action shots” of myself during my presentation.

Yep, as that last one shows, I was talking about BIG data!

The conference coordinators posted my presentation to YouTube, too.

small files and hadoop’s hdfs (bonus: an inode formula)

Many people have heard of the “small files” concern with HDFS.  Most think it is related to the Namenode (NN) and its memory utilization, but the NN really doesn’t care much if the files it is managing are big or small — it really is concerned about how many there are.

This topic is a fairly detailed and better described via sources such as this HDFS Scalability whitepaper, but basically it comes down to the NN needing to keep track of “objects” (i.e. in memory).  These objects are directories, files and blocks.  For example; if you had a tree that only had 3 top-level directories *and* each of these had 3 files *and* each of these took up 3 blocks, then your NN would need to keep track of 39 objects as detailed below.

  • 3 for each of the directories
  • 9 for the total number of files
  • 27 for all the blocks

Now… wouldn’t it be nice if Hadoop could tell you how many objects you have in place?  Of course it would… and of course it does…  You can run the count FS Shell operation,  but that will only tell you how many directories and files are present.

[hdfs@sandbox ~]$ hdfs dfs -count /
         248          510          560952712 /

One way to get the full accounting is to pull up the NN web UI.

So how much memory is required to keep track of all of these objects?  It is in the neighborhood of 150-200 bytes per object, but most folks are thinking about the total number of files on a filesystem like HDFS.  So if you start applying some heuristics (such as the average number of blocks per files and average files per directory) then you find yourself with the following de facto sizing guesstimate for NN memory heap size:

1 GB of heap size for every 1 million files

Again, that’s not a one-size-fits-all formula, but it is a one-size-fits-MOST one that has been used rather universally with great success.  Obviously, many things to consider when setting this up and just as many strategies for reducing the number of files in the first place such as concatenating files together, compressing files into a single binary (this one is most useful when the files are purely for long-term storage), leveraging Hadoop Archives, HDFS Federation, and so on.  All that said, it is not uncommon to see a 96GB heap sized NN managing 100+ million files.

Not to prescribe a setting for your cluster, but at a minimum, even for early POC efforts, increase your NN heap size to, at least, 4096 MB.  If using Ambari, the following screenshot shows you where to find this setting.

So, if we are actively monitoring our NN heap size and keeping it inline with the number of objects the NN is managing we can more accurately fine-tune our expectations for each cluster.

On the flip side, it seems easy enough to manage the amount of disk space we have on HDFS by all the inherent reporting abilities of HDFS (and tools like Ambari), not to mention some very simple math.  I recently did get asked about how inodes themselves can prevent the cluster from allowing new files to be added.  As a refresher, inodes keep track of all the files on a given (physical, not HDFS) file system.  Here is an example of a partition that has 99% of the space allocated showing free, but the inodes are all used up.

Filesystem    1K-blocks     Used  Available Use% Mounted on
/dev/sdf1    3906486416 10586920 3700550300   1% /data04

Filesystem          Inodes  IUsed    IFree IUse% Mounted on
/dev/sdf1           953856 953856        0  100% /data04

I’ll save a richer description on inodes, pros/cons to increasing this value, and the actual procedure(s) to utilize for a stronger linux administrator than I, but again, the question raised to me was; “will the inode size matter on the (underlying) hdfs file system mounts“.  The short answer is that, in a cluster with more than a trivial amount of Datanodes (DNs) then inodes should not cause an issue.  Basically, this is because *all* of the files will be spread across all the DNs *and* each of these will have multiple physical spindles to further spread the love.

Of course, things are often more complicated when we dive deeper.  First, each file will be broken up into appropriately sized blocks and then each of these will have three copies.  So, our example file above with 3 blocks will need to have 9 separate physical files stored by the DNs.  As you peel back the onion, you’ll see there really are two files for each block stored; the block itself and then a second file that contains metadata & checksum values.  In fact, it gets a tiny bit more complicated than that by the DNs needing to have a directory structure so they don’t overrun a flat directory.  Chapter 10 of Hadoop: The Definitive Guide (3rd Edition) has a good write up on this as you can see here which is further visualized by the abbreviated directory listing from a DN’s block data below.

[hdfs@sandbox finalized]$ pwd
/hadoop/hdfs/data/current/BP-1200952396-10.0.2.15-1398089695400/current/finalized
[hdfs@sandbox finalized]$ ls -al
total 49940
drwxr-xr-x 66 hdfs hadoop    12288 Apr 21 07:18 .
drwxr-xr-x  4 hdfs hadoop     4096 Jul  3 12:33 ..
-rw-r--r--  1 hdfs hadoop        7 Apr 21 07:16 blk_1073741825
-rw-r--r--  1 hdfs hadoop       11 Apr 21 07:16 blk_1073741825_1001.meta
-rw-r--r--  1 hdfs hadoop       42 Apr 21 07:16 blk_1073741826
-rw-r--r--  1 hdfs hadoop       11 Apr 21 07:16 blk_1073741826_1002.meta
-rw-r--r--  1 hdfs hadoop   392124 Apr 21 07:18 blk_1073741887
-rw-r--r--  1 hdfs hadoop     3071 Apr 21 07:18 blk_1073741887_1063.meta
-rw-r--r--  1 hdfs hadoop  1363159 Apr 21 07:18 blk_1073741888
-rw-r--r--  1 hdfs hadoop    10659 Apr 21 07:18 blk_1073741888_1064.meta
drwxr-xr-x  2 hdfs hadoop     4096 Apr 21 07:22 subdir0
drwxr-xr-x  2 hdfs hadoop     4096 Jun  3 08:45 subdir1
drwxr-xr-x  2 hdfs hadoop     4096 Apr 21 07:21 subdir63
drwxr-xr-x  2 hdfs hadoop     4096 Apr 21 07:18 subdir9

To generalize (and let’s be clear… that’s what I’m doing here — just creating a rough formula to make sure we are in the right ballpark), we can say that we’ll need 2.1 times the amount of inodes for each copy of a block; that’s 6.3 times the number of blocks we’d expect to be loaded into HDFS to account for replication.

To start the math, let’s calculate the NbrOfAvailBlocksPerCluster which is simply the inode limit per disk TIMES the number of disks in a DN TIMES the number of DNs in the cluster and then DIVIDE that number by the 6.3 value described above.  For example, the following values surface for a cluster that has DNs with 10 disks each whose JBOD file system partitions can support 1 million inodes.

  • 3 Datanodes = 4.7 MM blocks
  • 30 Datanodes = 47.6 MM blocks
  • 300 Datanodes = 476.2 MM blocks

To then know about how many files our cluster can store before running out of inodes, we need to simply divide the NbrOfAvailBlocksPerCluster by the AvgNbrBlocksPerFile we expect for a given cluster.  The following numbers are coupled with the 30 node cluster identified above.

  • 1 block/file = 47.6 MM files
  • 2.5 blocks/file = 19.0 MM files
  • 10 blocks/file = 4.7 MM files

Now, if you are immediately worried that these calculations suggest that a cluster with 30 DNs can only hold 20 MM files (using 2.5 blocks on average), then simply raise the inodes to something larger.  If the inodes setting was raised to 100 MM then that same 30 DN cluster would allow for 1.9 BILLION files.  If that cluster’s block size was 128 MB then it would have twice as much data as the cluster could hold.

So, let’s see if we can write that down in an algebraic formula.  Remember, we’re just thinking about inodes here — the disk sizing calculation is MUCH easier.

NbrOfFilesInodeSettingCanSupport = ( ( InodeSettingPerDisk * NbrDisksPerDN * NbrDNs ) / ( ReplFactor * 2.1 ) )  /  AvgNbrBlocksPerFile

As I’m doing this from the beach while enjoying my last full day of vacation, I challenge everyone to double-check my numbers AND MY LOGIC as the sun and the beverages are surely getting to me.

how do i load a fixed-width formatted file into hive? (with a little help from pig)

FEB 5, 2015 UPDATE: See the comments section in the original Confluence posting that this was ported from which identifies a better way to do this using FixedWidthLoader.

I was talking with a client earlier this week that is using the Hortonworks Sandbox to jumpstart his Hadoop learning (hey, that’s exactly why we put it out there). 

Thanks to all the Sandbox tutorials out there he already knew how to take a delimited file, get it into HDFS, use HCatalog to build a table, and then run Hive queries against it.  What he didn’t know how to do was to load up a good old-fashioned fixed-width formatted file into a table.  My first hunch was to use a simple Pig script to convert it into a delimited file and I told him I’d pull together a simple set of instructions to help him with his Sandbox self-training.  Robert, here ya go!!

Create & Upload a Test File

To try this all out we first need a little test file.  Create a file called emps-fixed.txt on your workstation with the following contents (yes… the first two lines are part of the exploration, but they don’t break anything either).

12345678901234567890123456789012345678901234567890123456789012345678901234567890
EMP-ID    FIRST-NAME          LASTNAME            JOB-TITLE           MGR-EMP-ID
12301     Johnny              Begood              Programmer          12306     
12302     Ainta               Listening           Programmer          12306     
12303     Neva                Mind                Architect           12306     
12304     Joseph              Blow                Tester              12308     
12305     Sallie              Mae                 Programmer          12306     
12306     Bilbo               Baggins             Development Manager 12307     
12307     Nuther              One                 Director            11111     
12308     Yeta                Notherone           Testing Manager     12307     
12309     Evenmore            Dumbnames           Senior Architect    12307     
12310     Last                Sillyname           Senior Tester       12308

Then upload the file into HDFS and just land it in your home directory on the Sandbox.

I’m taking some creative liberty and assuming that you’ve worked through (or at least understand the concepts of) some/most/all of the Sandbox tutorials.  For example, check out Tutorial 1 to see how to upload this file.  If at any point in this blog posting you are unfamiliar with how to do an activity, check back with the tutorials for help.  If you can’t find which one can help you, let me know in the comments section and I’ll reply with more info.

At this point, you should have a /user/hue/emps-fixed.txt file sitting in HDFS.

Build a Converter Script

Now we just need to build a simple converter script.  There are plenty of good resources out there on Pig including the tutorials; I’m even trying to build up a Pig Cheat Sheet.  For the one you’ll see below, I basically stripped down the info found at https://bluewatersql.wordpress.com/2013/04/17/shakin-bacon-using-pig-to-process-data/ (remember, a lazy programmer is a good programmer & imitation is the sincerest form of flattery) which is worth checking out.

Using the Sandbox lets create a Pig script called convert-emp which will start out with some plumbing.  Yes, that base “library” is called the piggy bank – I love the name, too!!

REGISTER piggybank.jar;
define SUBSTRING org.apache.pig.piggybank.evaluation.string.SUBSTRING();

fixed_length_input = load '/user/hue/emps-fixed.txt' as (row:chararray);

Then build a structure using the SUBSTRING function to pluck the right values out of each line.  Lastly, just write this new structure into a file on HDFS.

employees = foreach fixed_length_input generate
    (int)TRIM(SUBSTRING(row, 0,   9)) AS emp_id,
         TRIM(SUBSTRING(row, 10, 29)) AS first_name,
         TRIM(SUBSTRING(row, 30, 49)) AS last_name,
         TRIM(SUBSTRING(row, 50, 69)) AS job_title,
    (int)TRIM(SUBSTRING(row, 70, 79)) AS mgr_emp_id;
    
store employees into '/user/hue/emps-delimited';

Verify the Output File

Now you can navigate into /user/hue/emps-delimited on HDFS and you’ll see the familiar (or not so familiar?) “part” file via the Sandbox’s File Browser which shows the following tab-delimited content.

You’re welcome to rerun the exercise without the first two rows, but again, they won’t cause any real harm due to the forgiving nature of Pig who tries its best to do what you tell it to do.  For the “real world”, you’d probably want to write some validation logic to make sure things like the value “EMP-ID” (from the source file) fails miserable when trying to be cast to an integer.

Load a Table and Run a Query

Now you need to create a table based on this delimited file (hint, see Tutorial 1 again for one way to do that) and then run a simple query to verify all works.  I ran the following HiveQL.

select emp_id, last_name, job_title
  from employees
 where mgr_emp_id = 12306;

It returned the following in the Sandbox.

There you have it – a super simple way to get a fixed-width file into a table that you can query from hive.

visiting the computer history museum (yes, i’m a geek)

In the effort to move to WordPress in 2020, I could not find a suitable conversion program to move my Confluence-hosted technical blog for all occasions. For this blog post, I’m including the following links.

Original Confluence blog post as well as a PDF copy of it.

fruITion and recrEAtion (a double-header book review)

Several years ago, Chris Potts wrote his two-part Information Technology and Enterprise Architecture story (or was it a warning?) in FruITion and RecrEAtion.  These stories are intended to express Chris’ belief of how IT/EA teams should be fully-engaged in the Strategy of the companies they belong to, and not just be a “cost center”.  He uses a novel format in both books as well as follows a character from the first novel into the second.

I have to admit I read them in the wrong order.  That said, I’m glad I did as I’m not sure I would have picked up recrEAtion if I finished fruITion first.  That’s not to say that it is dramatically different than the other one, but I really identified with recrEAtion in the first place because of my involvement in the EA function at my employer.  I wanted to get some additional viewpoints on EA which is often the most misunderstood (or least organized?) team within a technology shop.  

The first book, fruITion is centered on a CIO whose CEO went ballistic when she was presented a 78 page IT Strategy document full of business process maps, architecture blueprints and technology roadmaps.  The CEO really had already decided to dramatically shake-up the IT organization and the rest of the book was about how the CIO decided to respond to an approach that seemed totally against everything he had done, and known, his entire career.  I’ll let you read it to find out how he did.

The second book, recrEAtion focused on a character that left the company in the first book.  His role at a new company was to lead the Enterprise Architecture function.  He didn’t buy into the CEO’s thoughts at his prior company and was thrown another curve ball when his new CEO said, “sounds like you and I have the same job.”  Needless to say, this really caused the main character to question what was going on around him for having two strange experiences back to back.

The underlying theme in both of these novels is about IT/EA’s involvement in the “strategy” of the company – not just having an IT strategy that was centered on what was in place to day and IT’s understanding of what needed to be done next.  Both throw in some rather drastic changes to how IT/EA shops should be structured and aligned.  It goes past the well known “partner with the business” mantra and suggests to the reader to “be the business”.  In both stories, there is a strong push for measuring success by metrics – almost always financial.

While I’m not so sure many organizations are ready for the sweeping changes presented in these two stories, the push for technology teams to not be isolated, but very tightly integrated, is a good topic for discussion in almost any organization.  

I can recommend this book to senior technologists and formal technology leaders in your organizations, but would not recommend it to less seasoned and/or more narrowly focused employees, or even folks in “the business”, because the book overly simplifies (for brevity’s sake, not necessarily that the author does not understand) the complexity of a technology team of any size.  I’d hate for someone to get the impression that all IT/EA needs to do is talk “business strategy” and not be responsible for the normal “care and feeding” activities that are the responsibility of the non-innovation aspects of an IT organization.

As for me personally, the biggest thing I take away from these books (especially recrEAtion) is that we should stop imagining that we know where we need to be in five years (i.e. the “end-state architecture”) and focusing so hard on continuing to maintain an accurate roadmap of how to get there.  The author doesn’t suggest we should stop planning, but what I took from it was that we need to be constantly preparing multiple options of what we might do next and focus on the financial reasons our companies might want, or need, to go there.  If these options are truly aligned with our business needs then we will be more respected at the decision-making table and have a much better chance of not only being successful, but working on the “right” things for our enterprises.

are you a mort, elvis or einstein (or are these labels nonsense)?

I was recently discussing the old Microsoft personas of Mort, Elvis and Einstein to some folks at work and was shocked that most had never heard of them.  You can easily google these three magical names and find several articles such as this one, here and yet another one (well… if you can read German).  Here they are in a nutshell.

  • Mort, the opportunist developer, likes to create quick-working solutions for immediate problems.  He focuses on productivity and learns as needed.
  • Elvis, the pragmatic programmer, likes to create long-lasting solutions addressing the problem domain, and learning while working on the solution.
  • Einstein, the paranoid programmer, likes to create the most efficient solution to a given problem, and typically learns in advance before working on the solution.

Well, which best describes you?  Can you use these labels to describe folks that you work with?  Or… are these just junk?

enterprise 2.0 book review (using web 2.0 technologies within organizations)

My prior employer had a company-wide MediaWiki instance along with other social/collaborative tools (blogs, forums, etc) in addition to the expected deployment of SharePoint.  To help span the various notifications, an internally-developed aggregator was created to roll up the various activity feeds that are produced into a social-oriented view.  At my current organization, this level of open authoring tooling is not as prevalent.

Coupled with a rollout of JIRA/Greenhopper, I introduced Confluence to help development projects with their documentation.  That said, we’ve seen an explosion of grass-roots content creation and collaboration.  Looking for information on this phenomenon led me to read Andrew McAfee’s Enterprise 2.0 which really addresses this subject matter well.  What’s “Enterprise 2.0”?  Well, check out this three minute video from the author.

If that sparked your interest, or you just have trouble falling asleep at night, check out my review of the book.

You can also grab the slides.

I’m continuing my push to enable & encourage further collaboration and I’d love to hear about your experiences of how collaborative tools such as wikis and blogs have made a difference in your organizations.

give as few orders as possible (encourage autonomy and responsibility)

I came across a great quote the other day; “Give as few orders as possible. Once you’ve given orders on a subject, you must always give orders on that subject.“  Brownie points to anyone who can source that quote (hint: it is from a SciFi book).

I’m not talking about Saint Philip Neri’s (paraphrased) quote of “he who wishes to be perfectly obeyed should give few orders” which itself was directed at government and its influence on citizens.  I’m talking about encouraging autonomy and responsibility by not micro-managing your team.

Working by this approach simply means to tell folks what you want done and get out of their way to allow them to find a workable solution by themselves.  Many of us might think we follow this approach, but it falls apart in a hurry when someone comes back with a solution that isn’t exactly what we were thinking the implementation would be.  It takes a solid leader to hold their tongue and simply ask questions that validate the solution presented actually solves the problem.  If it meets the requirements and was completed within whatever constraints have been established (company standards, team’s best practices, etc) then its a job well done.

If it doesn’t meet the requirements I highly encourage you to coach your team member in such a way that they are still in control of much of their destiny and how they choose to solve the problem.  Every time we tell a person how to solve a problem, we’re telling them to stop thinking on a problem like this as the official (aka “boss approved”) way to handle this particular case has been set.

I’m not just talking about what tools, patterns, frameworks, etc to use; I’m talking about how to give a coarse-grained problem to someone and letting them decompose the work effort into the needed tasks.  If you have to break down the problem into discrete tasks then that person will start looking for you to break it down that way for them in the future.  They’ll do this because they now feel like you want to micro-manage them in this way and they will now wait for you to give them direction on similar work in the future.  Your team is not going to scale like that.

This doesn’t mean we give our teams complete autonomy.  Again, there are constraints that make sense in your environment and this varies, but on the things that allow flexibility — give that flexibility back to your team members.  When you eliminate the flexibility by making choices for them, then the team will stop searching for a better way and just use your way going forward.  Worse yet, folks will start waiting for you to do the work you really want them to tackle and get better at.

Your team is bright and they want to help you. Give them an assignment and encourage them to tackle even tougher assignments by offering them as much flexibility as you can give and being careful to coach, not direct, them through the tough spots.