Apache Spark 3 for Data Engineering and Analytics with Python - Challenge - XYZ Research

The video tutorial introduces a project to explore the use of RDDS through a research challenge. The challenge involves analyzing data from XYZ Research, which conducts projects over a span of up to three years. The tutorial outlines specific questions to address, such as the number of projects initiated and completed within certain timeframes. It also introduces data transformations like union and subtract to solve the challenge. Viewers are encouraged to download the data file and follow along in the next lecture.

What is the maximum duration a research project can take at XYZ Research?

• Three years

• Two years

• Four years

• One year

How many research projects were conducted in 2001?

• 3 projects

• 10 projects

• 7 projects

• 5 projects

Which years had approximately five research projects each?

• 2000 and 2004

• 2002 and 2005

• 2001 and 2003

• 2003 and 2006

What is the first task in the challenge related to the research projects?

• Identify the projects that took more than two years

• Calculate the total duration of all projects

• Find out how many projects were initiated in three years

• Determine the number of projects completed in the first year

Which transformations are introduced to tackle the problem?

• Union and Subtract

• Map and Filter

• Reduce and Collect

• Join and GroupBy