However, the rapid adoption and development of Apache Spark, coupled with an increasing research on using it for big data analytics, make it difficult for beginners to comprehend the full body of development and research behind it. Other good sources are the official blog Footnote 3 at Databricks and Spark Hub Footnote 4 where you can find Spark’s news, events, resources, etc. In addition, Databricks, the company founded by the creators of Apache Spark, has developed a set of reference applications Footnote 2 to demonstrate how Apache Spark can be used for different workloads. While the official programming guide Footnote 1 is the most up-to-date source about Apache Spark, several books (e.g., ) have been published to show how Apache Spark can be used to solve big data problems. It has become the most active big data open source project and one of the most active projects in the Apache Software Foundation.Īs an evolving project in the big data community, having good references is a key need to get the most of the Apache Spark and contribute effectively to its progress. Following its advanced programming model, Apache Spark has been adopted as a fast and scalable framework in both academia and industry. It has introduced a new approach for data science and engineering where a wide range of data problems can be solved using a single processing engine with general-purpose languages. In this regard, Apache Spark has emerged as a unified engine for large-scale data analysis across a variety of workloads. To fulfill the computational requirements of massive data analysis, an efficient framework is essential to design, implement and manage the required pipelines and algorithms. In addition, we highlight some research and development directions on Apache Spark for big data analytics.īig data analytics is one of the most active research areas with a lot of challenges and needs for new innovations that affect a wide range of industries.

More specifically, it shows what Apache Spark has for designing and implementing big data algorithms and pipelines for machine learning, graph analysis and stream processing. This review focuses on the key components, abstractions and features of Apache Spark. In this paper, we present a technical review on big data analytics using Apache Spark. As a rapidly evolving open source project, with an increasing number of contributors from both academia and industry, it is difficult for researchers to comprehend the full body of development and research behind Apache Spark, especially those who are beginners in this area.

It is a general-purpose cluster computing framework with language-integrated APIs in Scala, Java, Python and R. Apache Spark has emerged as the de facto framework for big data analytics with its advanced in-memory programming model and upper-level libraries for scalable machine learning, graph analysis, streaming and structured data processing.

Big data analysis with apache spark