TheBeArsenal

Spark - The Elite List

Filter rows by column value in PySpark.

# Showing non-empty rows;
df.where(F.col("col_name").like("% %") == False).show()
Comment: Filtering out rows with column value spaces(" ").

Create new column in Spark Dataframe from existing column using withColumn method.

df = df.withColumn('new_col', df.old_col[1:2])
Comment: A new column 'new_col' is added to dataframe 'df' containing first two characters of column 'old_col'.

Groupby and aggregate example in PySpark.

df.groupby("col").agg(F.min('agg_col').alias('agg')).orderBy('col', ascending=True)
Comment: Here, rows in 'df' are grouped by 'col' column values and minimum value of 'agg_col' is found out for each group.

Save Spark Dataframe as Apache Parquet storage in HDFS.

df.write.parquet("hdfs:///user/abc123/outputs/df.parquet")
Comment: Above statement will save dataframe 'df' to given HDFS location in parquet format.

Load Fixed Width Text file as Dataframe in PySpark.

df = spark.read.text("hdfs:///path/to/dir")
df = df.select( df.value.substr(1,2).alias('ID'), df.value.substr(4,57).alias('NAME') )
Comment: Using 'spark.read' the file is loaded as undefined Spark Dataframe . Later using substr() data is split and assigned to two columns. (4,57) implies 57 characters starting from 4th position.