How To

This post part 2 from a series of posts that are extracted from a technical report i wrote for a research study and Indiana University. The full paper can be found in the  digital science center  publications page, or you can get the pdf link  here . I thought of posting the content here so more people can access the content which i think might be helpful to some. I am breaking this down into several posts because of the length of the document. The content is based on my knowledge and the current technologies as of day of writing the report and i might not have got everything correct. Please point out what you think in the comments if you think i have got something wrong, Will try to respond and update the document if needed. Will also include the list of references in each post for completeness. Part 1:   Similarities and Differences Between Parallel Systems and Distributed Systems Part 1 This post will give the introduction and compare and contrast the tw...

This post part 1 from a series of posts that are extracted from a technical report i wrote for a research study and Indiana University. The full paper can be found in the digital science center publications page, or you can get the pdf link here . I thought of posting the content here so more people can access the content which i think might be helpful to some. I am breaking this down into several posts because of the length of the document. The content is based on my knowledge and the current technologies as of day of writing the report and i might not have got everything correct. Please point out what you think in the comments if you think i have got something wrong, Will try to respond and update the document if needed. Will also include the list of references in each post for completeness. This post will give the introduction and compare and contrast the two domains with regards to the following Fault tolerance  Support of collectives  Dynamic resources utilization...

When you have a Apache spark cluster running on a server were ports are closed you cannot simply access the Spark master web UI by localhost :8080. The solution to this is to use SSH Tunnels. Which is pretty straight forward. Note: You can checkout my blog post on how to setup a Spark standalone cluster locally  (The steps are pretty much the same when you are setting it up on a server) -  How to set up a Apache Spark cluster in your local machine Scenario 1: The first most basic scenario would be if you have direct ssh access to the server where the Apache spark master is running on. The all you have to do is run the following command in a terminal window on your local machine ( Laptop or desktop that you use) after you start the master in the server machine. $ ssh -L 8080:localhost:8080 username@your.server.name Once you have run this command you can access the Spark Web UI by simply going to "http://localhost:8080/" on your web browser. Likewise you might want ...

In this post we will try to redo the sample that we did in my previous post Simple log analysis with Apache Spark , using the Spark JAVA api and since i am more accustomed to maven we will create a simple maven project to accomplish this task. For this post i will not use a separate Spark cluster, we will just use an embedded Spark that will automatically startup a server when running the code. First lets create a new maven project. I am using Intellij Idea. I will name my project LogAnalizer. The first thing you would need to do is add the maven dependencies to your pom file. To do this add the following lines into the pom.xml file in your newly created Maven project. You will also need to add the repository. <repositories> <repository> <id>Akka repository</id> <url>http://repo.akka.io/releases</url> </repository> </repositories> <dependencies> <dependency...

In this post we will try to learn to run some simple commands with Spark, some simple transformations and actions. We will do some simple log analysis using Spark. I will be using the local Spark cluster that i setup on my laptop. if you haven't read my first post on how to setup an Spark cluster on your local machine i recommend you read the post  How to set up a Apache Spark cluster in your local machine . First we will connect the the cluster with the following command. MASTER= spark://pulasthi-laptop:7077 ./spark-shell "spark://pulasthi-laptop:7077" is the URL of the master that can be found in the Spark web ui. After connecting successfully you should be able to see an Scala console where you can execute commands. Also you should be able to see your application listed in the web-ui under running applications. To run this scenarios i am using a set of log files generated from various WSO2 products as sample data. Any set of log files or even just a set of ...

The past few days i grew some interest in Apache Spark and thought of playing around with it a little bit. If you haven't heard about it go an take a look its a pretty cool project it claims to be around 40x faster than Hadoop in some situation. The incredible increase in performance is gained by leveraging in-memory computing technologies. I want go into details about Apache Spark here if you want to get a better look at Spark just check out there web site -  Apache Spark . In this post we will be going through the steps to setup an Apache Spark cluster on your local machine. we will setup one master node and two worker nodes. If you are completely new to Spark i recommend you to go through  First Steps with Spark - Screencast #1  it will get you started with spark and tell you how to install Scala and other stuff you need. We will be using the  launch scripts that are provided by Spark to make our lives more easier. First of all there are a couple of conf...