SSIS Folder Traversing in SPARK SQL

Introduction

Here in this article, we are trying to demonstrate Folder Traversing of SSIS ForEach loop container for searching a specified file.

Hope it will be interesting

Scenario

We have a folder named “Sample”. Under this folder, we have three other folder named “Sample-1”, “Sample-2” and “Sample-3”. For each folder there is a flat file named “Student-1.txt”,”Student-2.txt” and “Student-3.txt”.

We need to read the entire file from different folder location

The folder and file structure is displayed by DOS TREE command

Scala Code

//---------------------------------------

// Scala for SPARK to Read Flat File form Different Folder

// Implementation Folder Traversing of SSIS in Spark

// Creation Date: 06/18/2018

//-----------------------------------------

import org.apache.spark.sql.SparkSession

import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder

import org.apache.spark.sql.Encoder

import spark.implicits._

case class Student(roll: Long, name: String)

val employeeDF = spark.sparkContext.textFile("d:/spark/bin/examples/src/main/resources/sample/*/student-*.txt").map(_.split(",")).map(attributes=>Student(attributes(0).trim.toInt, attributes(1).trim)).toDF()

employeeDF.createOrReplaceTempView("student")

val youngstersDF = spark.sql("SELECT roll, name FROM student")

youngstersDF.show

Output

Hope you like it.

Posted By: MR. JOYDEEP DAS