Posts

Showing posts with the label Spark SQL

Featured Post

How to Compare Text in Python

Image
Python like other programming languages supports comparison operators. These primarily use to test the condition in the program. Here's a list of operators that you can use in python programs. Comparison operators List of operators < <= > >= == != Is is not How to use comparison operators Here, I have assigned 23 to a and 11 to b. Then, I did apply all the comparison operators. The output is self-explanatory, and If you are in doubt while programming, remember to visit this page. Examples a = 23 b = 11 print("Is a greater than b?", a > b)           #greater than print("Is a less than b?", a < b)              #less than print("Is a greater or equal to b?", a >= b)   #greater or equal print("Is a less or equal to b?", a <= b)      #less or equal print("Is a equal to b (option 1)?", a == b)         #test for equality print("Is a equal to b (option 2)?", a is b)         #test for equality print("I

Spark SQL Query how to write it in Ten steps

Image
Spark SQL example The post tells how to write SQL query in Spark and explained in ten steps.This example demonstrates how to use sqlContext.sql to create and load two tables and select rows from the tables into two DataFrames. The next steps use the DataFrame API to filter the rows for salaries greater than 150,000 from one of the tables and shows the resulting DataFrame. Then the two DataFrames are joined to create a third DataFrame. Finally the new DataFrame is saved to a Hive table. 1. At the command line, copy the Hue sample_07 and sample_08 CSV files to HDFS: $ hdfs dfs -put HUE_HOME/apps/beeswax/data/sample_07.csv /user/hdfs $ hdfs dfs -put HUE_HOME/apps/beeswax/data/sample_08.csv /user/hdfs where HUE_HOME defaultsto /opt/cloudera/parcels/CDH/lib/hue (parcel installation) or /usr/lib/hue (package installation). 2. Start spark-shell: $ spark-shell 3. Create Hive tables sample_07 and sample_08: scala> sqlContext.sql("CREATE TABLE sample_07 (code string