Access Apache Spark Web UI when cluster is running on closed port server machines

When you have a Apache spark cluster running on a server were ports are closed you cannot simply access the Spark master web UI by localhost:8080. The solution to this is to use SSH Tunnels. Which is pretty straight forward.

Note: You can checkout my blog post on how to setup a Spark standalone cluster locally  (The steps are pretty much the same when you are setting it up on a server) - How to set up a Apache Spark cluster in your local machine

Scenario 1:

The first most basic scenario would be if you have direct ssh access to the server where the Apache spark master is running on. The all you have to do is run the following command in a terminal window on your local machine ( Laptop or desktop that you use) after you start the master in the server machine.

 $ ssh -L 8080:localhost:8080 username@your.server.name

Once you have run this command you can access the Spark Web UI by simply going to "http://localhost:8080/" on your web browser. Likewise you might want to create SSH tunnels for other ports that are needed when using the Spark Web UI, such as 4040.

Scenario 2:

If you do not have direct access to the server that is running the Apache Spark master you can do a multilevel SSH Tunnel. This might the case if you are running the cluster in a super computer where you only have access to the login node of the super computer and you are running the Spark master on a compute node, which you can only access through the login node. When this is the case you can simply do the same SSH tunnel in two steps. First after you start the Spark master on the compute node run the following command to create a SSH tunnel for port 8080

 $ ssh -L 8080:localhost:8080 username@compute.node.name

Then run the following command from your local machine ( Laptop or desktop that you use).

 $ ssh -L 8080:localhost:8080 username@your.server.name

Then you can access the Web UI just as before by going to "http://localhost:8080/".

Update: Based on Saliya's comment you can do the two steps in a single command. An example of the command is as follows. You just need to run this from your local machine

 $ ssh -L 8080:compute.node.name:8080 username@your.server.name


Comments

Post a Comment

Popular posts from this blog

Writing Unit Tests to test JMS Queue listener code with ActiveMQ

During my GSoC project for OpenNMS i did some work with JMS and wanted to write test classes to make sure that my code was working well. In this article i will try to explain how you can use ActiveMQ to write test cases for your JMS code. This is very handy when your code has a part that listens to a JMS queue. Apache ActiveMQ  Apache ActiveMQ is a extremely popular and very powerful open source messaging and Integration Patterns server. You can check it out here . The part we will be using to run our test classes is the embedded broker that is provided by ActiveMQ. This allows you to create a temporary broker for test purposes to create a JMS queue in our case. If you want to learn more about the embedded broker check out this article . The easiet way to create a embedded broker is through the following code line. This will automatically create a embedded broker. ConnectionFactory connectionFactory = new ActiveMQConnectionFactory("vm://localhost?broker.pe...
Read more

Reading and Writing Binary files in java - Converting Endianness ( between Big-endian byte order and Little-endian byte order)

Image
In this post we will look into working with binary files in java. The end goal of the post will be to create simple java application that reads a binary file and write it back to a different binary file. The difference between the two files would be the byte order. If we read in a file with Big-endian byte order we will write it back with a Little-endian byte order and vice-versa. Lets first understand what is meant by byte order and what Big-endian and Little-endian are. If you are already familiar with this you can skip this section of the post You can find the complete code for Binary format converter that is explained in this post at GitHub -  https://github.com/pulasthi/binary-format-converter Understanding byte order and Endianness The endianness refers to the order that bytes are stored when storing multi byte values such as Integers and Doubles this is also known as the byte order. The endianness does not have any meaning when you consider a single byte ...
Read more

Setting up Heron Cluster with Apache Aurora Locally

Image
In this post we will be looking at how we can setup Heron steam processing engine in Apache Aurora in our local machine. Oh Boy this is going to be a long post :D. I am doing this on Ubuntu 14.04 and these steps should be similar to any Linux machine. Heron supports deployment in Apache Aurora out of the box. Apache Aurora will act as the Scheduler for Heron after the setup is complete. In order to do this first you will have to setup Apache Zookeeper and allow Heron to communicate with it. Here Apache Zookeeper will act as the State Manager of the Heron deployment. if you just want to setup a local cluster without the hassle of  installing aurora take a look at my previous blog post -  Getting started with Heron stream processing engine in Ubuntu 14.04 Setting Up Apache Aurora Cluster locally  First thing we need to do is to setup Apache Aurora locally. I will try to explain as much of the configurations as i can as we go on. First lets get a Apache Aurora cluster ...
Read more