Setting up Heron Cluster with Apache Aurora Locally

In this post we will be looking at how we can setup Heron steam processing engine in Apache Aurora in our local machine. Oh Boy this is going to be a long post :D. I am doing this on Ubuntu 14.04 and these steps should be similar to any Linux machine. Heron supports deployment in Apache Aurora out of the box. Apache Aurora will act as the Scheduler for Heron after the setup is complete. In order to do this first you will have to setup Apache Zookeeper and allow Heron to communicate with it. Here Apache Zookeeper will act as the State Manager of the Heron deployment. if you just want to setup a local cluster without the hassle of  installing aurora take a look at my previous blog post - Getting started with Heron stream processing engine in Ubuntu 14.04

Setting Up Apache Aurora Cluster locally 

First thing we need to do is to setup Apache Aurora locally. I will try to explain as much of the configurations as i can as we go on. First lets get a Apache Aurora cluster running on our local machine. The steps are extracted from here.

Step 1: Install VirtualBox and Vagrant

Download and install VirtualBox and Vagrant in your machine. If vagrant is successfully installed in your machine  the following command should list several common commands for this tool
$ vagrant
Step 2: Clone the Aurora repository

You can get the source repository for Aurora with the following command
$ dgit clone git:// 
Once the clone is complete cd into the aurora folder

$ cd aurora

Step 3: Starting Local Aurora Cluster

To start the local cluster all you have to do is execute the following command. It will install all the needed dependencies like Apache Mesos and Zookeeper in the VM. 
$ vagrant up
Additionally to get rid of some of the warning messages that you get during up command execute the following command 
$ vagrant plugin install vagrant-vbguest
You can verify that the Aurora cluster is properly running by opening the following links in your web-browser 
If you go into you can notice that the name of the default cluster that is setup in aurora is named "devcluster" this will be important to note when submitting typologies from heron.

Installing Heron within VM

Now that we have the Aurora cluster setup we need to install heron within the cluster VM in order to be able to create get our Heron deployment working. Since this is a fresh VM instance you will have to install the basic software such as "unzip" and set the JAVA_HOME path as an environmental variable ( Just need to add this to .bashrc file). After you have the basic stuff working follow the following steps to install Heron in the VM. You can ssh into the VM with the following command

$ vagrant ssh

Step 1.a : Download installation script files

You can download the script files that match to Ubuntu from

For the 0.14.0 release the files you need to download will be the following.

Optionally - You want need the following for the steps in the blog post

Step 1.b: Execute the client and tools shell scripts

$ chmod +x
$ ./ --user
Heron client installer

Heron is now installed!

Make sure you have "/home/vagrant/bin" in your path.

After this you need to add the path "/home/vagrant/bin". You can just execute the following command or add it to the end of  .bashrc file ( which is more convenient ).
$ export PATH=$PATH:/home/vagrant/bin
Install the following packages to make sure that you have all the needed dependencies in the VM. You might have to do sudo apt-get update before you execute the following.
$ sudo apt-get install git build-essential automake cmake libtool zip libunwind-setjmp0-dev zlib1g-dev unzip pkg-config -y

Configuring State Manager ( Apache Zookeeper )

Since Heron only uses Apache Zookeeper for coordination the load on the Zookeeper node is minimum. Because of this it is sufficient to use a single Zookeeper node or if you have an Zookeeper instance running for some other task you can simply use that. Since Apache Aurora already uses an Zookeeper instance we can directly use that instance to execute State Manager tasks of Heron. First you need to configure Heron to work with the Zookeeper instance. You can find meanings of each attribute here. Configurations for State manager are located in the directory /home/vagrant/.heron/conf/aurora

Open the file statemgr.yaml using vim ( or some other text editor you prefer ) and add/edit the file to include the following. 

# local state manager class for managing state in a persistent fashion
heron.class.state.manager: com.twitter.heron.statemgr.zookeeper.curator.CuratorStateManager

# local state manager connection string
heron.statemgr.connection.string:  ""

# path of the root address to store the state in a local file system
heron.statemgr.root.path: "/heronroot"

# create the zookeeper nodes, if they do not exist True

# timeout in ms to wait before considering zookeeper session is dead 30000

# timeout in ms to wait before considering zookeeper connection is dead 30000

# timeout in ms to wait before considering zookeeper connection is dead
heron.statemgr.zookeeper.retry.count: 10

# duration of time to wait until the next retry 10000

Creating Paths in Zookeeper

Next we need to create some paths within Zookeeper since some of the paths are not created by Heron automatically. So we need to create them manually. Since Aurora installation already installed Zookeeper we can use the Zookeeper cli to create the manual paths.
$ sudo ./usr/share/zookeeper/bin/
This will connect to the Zookeeper instance running locally. Then execute the following commands from within the client to create paths /heronroot/topologies and /heron/topologies. Later in "Associating new Aurora cluster into Heron UI"  we will see that we only need to create /heronroot/topologies but for now lets create both to make sure we don't get any errors when we run things.

create /heronroot null
create /heronroot/topologies null
create /heron null
create /heron/topologies null

Configuring Scheduler ( Apache Aurora )

Next we need to configure Apache Aurora to be used as the Scheduler for our Heron local cluster.
In order to do this we need to edit the scheduler.yaml file that is also located in /home/vagrant/.heron/conf/aurora. Add/Edit the file to include the following. More information regarding parameters can be found here

# scheduler class for distributing the topology for execution
heron.class.scheduler: com.twitter.heron.scheduler.aurora.AuroraScheduler

# launcher class for submitting and launching the topology
heron.class.launcher: com.twitter.heron.scheduler.aurora.AuroraLauncher

# location of the core package
heron.package.core.uri: file:///home/vagrant/.heron/dist/heron-core.tar.gz

# location of java - pick it up from shell environment /usr/lib/jvm/java-1.8.0-openjdk-amd64/

# Invoke the IScheduler as a library directly False

Additionally edit the client.yaml file and change the core uri to make it consistant.
# location of the core package
heron.package.core.uri: file:///home/vagrant/.heron/dist/heron-core.tar.gz

Important Step: Change folder name aurora to devcluster

Next we need to change the folder name of /home/vagrant/.heron/conf/aurora to /home/vagrant/.heron/conf/devcluster. This is because the name of our aurora cluster is devcluster as we noted in a previous step. You can do this with the following commands
$ cd /home/vagrant/.heron/conf/
$ mv aurora devcluster
Now we are almost done :).

Submitting Example Topology to Aurora cluster

Now we can submit a topology to the aurora cluster. this can be done with the following command.
$ heron submit devcluster/pulasthi/devel --config-path ~/.heron/conf/ ~/.heron/examples/heron-examples.jar com.twitter.heron.examples.ExclamationTopology ExclamationTopology

Now you should be able to see the topology in the Aurora UI ( ) .

Understanding the parameters 

I will try to explain some of the important parameters that are used in this command. the first parameter "devcluster/pulasthi/devel" defines cluster, role and env ( env can have values prod | devel | test | staging ). The cluster is the name of the aurora cluster which is devcluster in our case. You can give something like your name for the role name and for env you need to choose from one of the env values. 

--config-path points to the config folder. the program will automatically look for a folder with the cluster name. This is why we had to change the name of the aurora conf folder to devcluster.

Now that everything is working we need to perform one last step to be able to see the typologies that we can see in Aurora UI in Heron UI.

Associating new Aurora cluster into Heron UI.

Heron UI uses information that is gets from the heron tracker when displaying the information in the heron UI interface. So in-order to allow the Heron UI to show Aurora cluster information we need to modify configuration of the Heron tracker so that it can identify the Aurora Cluster. 

Heron Tracker configurations are located at /home/vagrant/.herontools/conf the configuration file is named heron_tracker.yaml. By default you should see the following in the file

    type: "file"
    name: "local"
    rootpath: "~/.herondata/repository/state/local"
    tunnelhost: "localhost"
    type: "zookeeper"
    name: "localzk"
    hostport: "localhost:2181"
    rootpath: "/heron"
    tunnelhost: "localhost"

 You can see that there already two entries. Before we had to create paths in Zookeeper for /heron/topologiesthis is because the entry named localzk in this file. If we remove this we will not need to create that path in Zookeeper. Not all we have to is to add a new entry for the aurora cluster into this file ( we will comment out localzk ). Then the file would look like below.

    type: "file"
    name: "local"
    rootpath: "~/.herondata/repository/state/local"
    tunnelhost: "localhost"
   #type: "zookeeper"
   # name: "localzk"
   # hostport: "localhost:2181"
   # rootpath: "/heron"
   # tunnelhost: "localhost"
    type: "zookeeper"
    name: "devcluster"
    hostport: "localhost:2181"
    rootpath: "/heronroot"
    tunnelhost: "localhost"

Now you can start Heron tracker and then Heron UI, Now you will be able to see the aurora cluster from the Heron UI ( ) as below
$ heron-tracker
$ heron-ui

Now you should have everything working. I hope this post was helpful. And leave a comment and share your thoughts and if you notice any mistakes in the post please let me know. I will make sure to correct the post.


  1. keep up the good work. this is an Assam post. this to helpful, i have reading here all post. i am impressed. thank you. this is our digital marketing training center. This is an online certificate course
    digital marketing training in bangalore |

  2. Your work is very good and I appreciate you and hopping for some more informative posts
    Please check this ExcelR Courses in Digital Marketing in Pune

  3. This is an awesome motivating article. I am practically satisfied with your great work. You put truly extremely supportive data. Keep it up. Continue blogging. Hoping to be perusing your next post
    Courses in Digital Marketing in Pune

  4. Wonderful blog found to be very impressive to come across such an awesome blog. I should really appreciate the blogger for the efforts they have put in to develop such an amazing content for all the curious readers who are very keen of being updated across every corner. Ultimately, this is an awesome experience for the readers. Anyways, thanks a lot and keep sharing the content in future too.
    Digital Marketing Training

  5. Extraordinary blog went amazed with the content that they have developed in a very descriptive manner. This type of content surely ensures the participants to explore themselves. Hope you deliver the same near the future as well. Gratitude to the blogger for the efforts.

    Data Science Training

  6. Stupendous blog huge applause to the blogger and hoping you to come up with such an extraordinary content in future. Surely, this post will inspire many aspirants who are very keen in gaining the knowledge. Expecting many more contents with lot more curiosity further.

    Data Science Certification in Bhilai

  7. Become a Data Science Expert with us. we provide Classroom training on Data Science course in Hyderabad for the individuals who believe hand-held training. We teach as per the Indian Standard Time (IST) with In-depth practical Knowledge on each topic in classroom training, 80 – 90 Hrs of Real-time practical training classes. There are different slots available on weekends or weekdays according to your choices.

  8. Data Science Training in Hyderabad program by AI Patasala, where you can learn practical knowledge with real-time experience.
    Data Science Training and Placements in Hyderabad

  9. Congratulations on your article, it was very helpful and successful. 34c49f87992975e1d6e56bf32beca663
    website kurma
    website kurma
    numara onay

  10. Thank you for your explanation, very good content. 837081866fb5ceb2650319a3712fc07f
    altın dedektörü


Post a Comment

Popular posts from this blog

Reading and Writing Binary files in java - Converting Endianness ( between Big-endian byte order and Little-endian byte order)

Writing Unit Tests to test JMS Queue listener code with ActiveMQ