Apache Spark 3 for Data Engineering and Analytics with Python - Challenge - XYZ Research Part 1

This video tutorial guides viewers through the process of analyzing research project data over three years using Spark. It begins with creating a new heading in a markdown cell, followed by loading data from an attached file. The tutorial then demonstrates how to create RDDs for each year, combine them using union operations, and ensure uniqueness with distinct operations. Finally, it concludes by counting the total number of unique research projects, revealing that 12 projects were conducted over the three years.

What is the first step in creating a new heading in markdown?

• Add a semicolon at the end

• Run the cell without any changes

• Change the cell mode to code

• Insert a hash symbol

What is the purpose of loading the attached file in the lesson?

• To format the data for presentation

• To copy the entire contents for analysis

• To create a backup of the data

• To delete unnecessary data

How are RDDs created for each year in the lesson?

• By using a pre-built function

• By using a for loop

• By copying and modifying existing lines

• By manually entering data

What transformation is used to combine data from different years?

• Filter transformation

• Map transformation

• Union transformation

• Reduce transformation

What is the result of using the distinct function on the data set?

• It duplicates the data

• It sorts the data

• It removes duplicate entries

• It merges the data with another set

How many research projects were conducted in the first three years?

• 18

• 10

• 12

• 15

What is the final step to reveal the answer to the first question?

• Run a sort function

• Use a filter to refine data

• Perform a count on the data

• Export the data to a file