Monday, 18 June 2018

SSIS Folder Traversing in SPARK SQL


Introduction
Here in this article, we are trying to demonstrate Folder Traversing of SSIS ForEach loop container for searching a specified file.
Hope it will be interesting

Scenario
We have a folder named “Sample”. Under this folder, we have three other folder named “Sample-1”, “Sample-2” and “Sample-3”. For each folder there is a flat file named “Student-1.txt”,”Student-2.txt” and “Student-3.txt”.

We need to read the entire file from different folder location
The folder and file structure is displayed by DOS TREE command



Scala Code
//---------------------------------------
// Scala for SPARK to Read Flat File form Different Folder
// Implementation Folder Traversing of SSIS in Spark
// Creation Date: 06/18/2018
//-----------------------------------------
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.sql.Encoder
import spark.implicits._


case class Student(roll: Long, name: String)

val employeeDF = spark.sparkContext.textFile("d:/spark/bin/examples/src/main/resources/sample/*/student-*.txt").map(_.split(",")).map(attributes=>Student(attributes(0).trim.toInt, attributes(1).trim)).toDF()


employeeDF.createOrReplaceTempView("student")


val youngstersDF = spark.sql("SELECT roll, name FROM student")

youngstersDF.show

Output






Hope you like it.



Posted By: MR. JOYDEEP DAS

No comments:

Post a Comment