Sunday, 24 June 2018

Spark working with Unstructured data


Introduction
In my previous article with Spark, we worked with structure and semi structure data source. Here in this article we are trying to work with unstructured data source.
Hope it will be interesting.

Case Study
We have a note book and we want to find number of work count in it.



Scala Code

val dfsFilename = "D:/spark/bin/examples/src/main/resources/notebook.txt"

val text = sc.textFile(dfsFilename)
val counts = text.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)
counts.collect.foreach(println)

Output






Hope it will be interesting.

Posted by: MR. JOYDEEP DAS

13 comments:

  1. I love this blog . This is one of the best blog i ever seen. It's all about what i'm searching for. I love to read this blog again and again . Every time i enter this blog i get something new. This blog inspire me to write new blog. I write a blog name http://tutorialabc.com. It's about sql,c#,net etc

    ReplyDelete
  2. We at Coepd declared Data Science Internship Programs (Self sponsored) for professionals who want to have hands on experience. We are providing this program in alliance with IT Companies in COEPD Hyderabad premises. This program is dedicated to our unwavering participants predominantly acknowledging and appreciating the fact that they are on the path of making a career in Data Science discipline. This internship is designed to ensure that in addition to gaining the requisite theoretical knowledge, the readers gain sufficient hands-on practice and practical know-how to master the nitty-gritty of the Data Science profession. More than a training institute, COEPD today stands differentiated as a mission to help you "Build your dream career" - COEPD way.

    http://www.coepd.com/AnalyticsInternship.html

    ReplyDelete
  3. I feel satisfied to read your blog, you have been delivering a useful & unique information to our vision even you have explained the concept as deep clean without having any uncertainty, keep blogging. SQL server dba Online Training

    ReplyDelete

  4. It's so nice article thank you for sharing a valuable content. SQL server dba Online Training Bangalore

    ReplyDelete
  5. Hello Author,
    while I visit your blog I found that your content is so clear which attracts user attention. thanks for sharing a worthwhile article. See useful Information on Shrug Ransomware.

    ReplyDelete
  6. We at Coepd declared Data Science Internship Programs (Self sponsored) for professionals who want to have hands on experience. We are providing this program in alliance with IT Companies in COEPD Hyderabad premises. This program is dedicated to our unwavering participants predominantly acknowledging and appreciating the fact that they are on the path of making a career in Data Science discipline. This internship is designed to ensure that in addition to gaining the requisite theoretical knowledge, the readers gain sufficient hands-on practice and practical know-how to master the nitty-gritty of the Data Science profession. More than a training institute, COEPD today stands differentiated as a mission to help you "Build your dream career" - COEPD way.

    http://www.coepd.com/AnalyticsInternship.html

    ReplyDelete
  7. thanks for sharing this Informative content. Well explained. Got to learn new things from your Blog on. SQL server dba Online Training

    ReplyDelete
  8. Are you tired of seeking loans and Mortgages,have you been turned down constantly By your banks and other financial institutions,We offer any form of loan to individuals and corporate bodies at low interest rate.If you are interested in taking a loan,feel free to contact us today,we promise to offer you the best services ever.Just give us a try,because a trial will convince you.What are your Financial needs?Do you need a business loan?Do you need a personal loan?Do you want to buy a car?Do you want to refinance?Do you need a mortgage loan?Do you need a huge capital to start off your business proposal or expansion? Have you lost hope and you think there is no way out, and your financial burdens still persists? Contact us (gaincreditloan1@gmail.com)

    Your Name:...............
    Your Country:...............
    Your Occupation:...............
    Loan Amount Needed:...............
    Loan Duration...............
    Monthly Income:...............
    Your Telephone Number:.....................
    Business Plan/Use Of Your Loan:...............
    Contact Us At : gaincreditloan1@gmail.com

    ReplyDelete
  9. Nice blog! Thanks for sharing such an amazing blog
    Bulletproof Offshore VPS

    ReplyDelete
  10. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this. best sap simple finance online training institute in hyderabad

    ReplyDelete