I've recently been investigating how to make Apache Spark a lot more user friendly and ways in which you can harness the power of Spark whilst still keeping traditional ETL work flows. What do I mean by a traditional ETL...

With the release of Spark 1.6 came the feature to allow your streaming applications to apply back pressure, which is a form of rate limiting that ensures that your streaming application can handle spikes in events without flooding your clusters...

One of the great features of Apache Spark is the Spark Shell. It allows for interactive analysis of data from a variaty of data sources, be that HDFS, S3 or just your local file system. The spark shell allows you...

In this post I will explain the steps that I took in order to setup a development environment on my macbook, in order to run Spark jobs using HDFS and YARN.

Install Hadoop & Spark

brew...