Posts

Showing posts with the label Spark

Featured Post

8 Ways to Optimize AWS Glue Jobs in a Nutshell

-
Image
  Improving the performance of AWS Glue jobs involves several strategies that target different aspects of the ETL (Extract, Transform, Load) process. Here are some key practices. 1. Optimize Job Scripts Partitioning : Ensure your data is properly partitioned. Partitioning divides your data into manageable chunks, allowing parallel processing and reducing the amount of data scanned. Filtering : Apply pushdown predicates to filter data early in the ETL process, reducing the amount of data processed downstream. Compression : Use compressed file formats (e.g., Parquet, ORC) for your data sources and sinks. These formats not only reduce storage costs but also improve I/O performance. Optimize Transformations : Minimize the number of transformations and actions in your script. Combine transformations where possible and use DataFrame APIs which are optimized for performance. 2. Use Appropriate Data Formats Parquet and ORC : These columnar formats are efficient for storage and querying, signif
Post a Comment
Read more

Spark SQL Query how to write it in Ten steps

-
Image
Spark SQL example The post tells how to write SQL query in Spark and explained in ten steps.This example demonstrates how to use sqlContext.sql to create and load two tables and select rows from the tables into two DataFrames. The next steps use the DataFrame API to filter the rows for salaries greater than 150,000 from one of the tables and shows the resulting DataFrame. Then the two DataFrames are joined to create a third DataFrame. Finally the new DataFrame is saved to a Hive table. 1. At the command line, copy the Hue sample_07 and sample_08 CSV files to HDFS: $ hdfs dfs -put HUE_HOME/apps/beeswax/data/sample_07.csv /user/hdfs $ hdfs dfs -put HUE_HOME/apps/beeswax/data/sample_08.csv /user/hdfs where HUE_HOME defaultsto /opt/cloudera/parcels/CDH/lib/hue (parcel installation) or /usr/lib/hue (package installation). 2. Start spark-shell: $ spark-shell 3. Create Hive tables sample_07 and sample_08: scala> sqlContext.sql("CREATE TABLE sample_07 (code string
Post a Comment
Read more

SPARK is Replacement for MapReduce in Bigdata Real Analytics!

-
Image
Apache Spark is among the Hadoop ecosystem technologies acting as catalysts for broader adoption of big data infrastructure. Now, Looker -- a vendor of business intelligence software -- has announced support for Spark and other Hadoop technologies. The goal? To speed up access to the data that fuels business decision making. SPARK Jobs Hadoop's arrival on the scene 10 years ago may have started the big data revolution, but only recently did adoption of this technology begin spreading to a wider audience. Apache Spark is one of the catalysts for the growing adoption rates. Spark can be used as a replacement for MapReduce, a component of Hadoop implementations, to speed up the processing and analytics of big data by 100x in memory, according to the Apache Software Foundation. In today's business environment, in which real-time analytics is the goal and organizations don't want to wait for data warehouses and analysts to provide batch intelligence back to business u
Post a Comment
Read more

Hot Skills: Spark Self Study Materials

-
Image
Spark: With job postings up 120% year-over-year on Dice, demand for this open-source cluster-computing framework is broad-based. Government contractors and financial-services firms are just a few of the groups eager to find candidates with this skillset. 2015 Average Salary: $113,214 Related: SPARK Self Study Materials Spark Big Data and Cloud:  As companies expand their tech infrastructures, they need cloud and Big Data services such as Azure (#2), Hive (#8), and Cassandra (#9) for data storage, analysis, and security. Big Data and cloud-related skills dominated the Highest-Paid Skills list on Dice’s salary survey for the second straight year.  2015 Average Salary: Big Data—$121,328 Azure — $110,207 Salesforce: This customer-service platform serves as the bedrock for many companies’ customer service departments. Demand for Salesforce professionals seems unlikely to decline anytime soon. Employers are even willing to offer telecommuting options to lure Salesforce talent. 2
Post a Comment
Read more