Advanced DF PySpark

Which method is used to create a new column in a DataFrame based on a condition?

• withColumn()

• select()

• filter()

• groupBy()

The lit function in PySpark can be used to create a column with a constant value, but it cannot be used within expressions involving other columns.

• True

• False

Pandas UDFs (also known as vectorized UDFs) in PySpark are generally faster than regular PySpark UDFs because they operate on a single row at a time.

• True

• False

In PySpark, MapType can be used to create a column containing key-value pairs, and both the keys and values must be of the same data type.

• True

• False

Which method is used to create an alias for a column in PySpark?

• alias()

• withColumn()

• select()

• groupBy()

What does the collect_list() function do in PySpark?

• Collects all elements into a list and removes duplicates

• Collects all elements into a list without removing duplicates

• Collects all elements into a set and removes duplicates

• Collects all elements into a set without removing duplicates

Which of the following is a valid transformation operation in PySpark?

• collect()

• show()

• filter()

• count()

What does the sort_array() function do in PySpark?

• Sorts the elements of an array in ascending or descending order

• Sorts the rows of a DataFrame based on an array column

• Sorts the columns of a DataFrame based on an array column

• Sorts the elements of an array and removes duplicates

The rank function in PySpark assigns the same rank to rows with identical values, but it skips subsequent ranks, resulting in gaps in the ranking sequence. For example, if two rows are tied for rank 1, the next rank assigned will be 3, not 2.

• True

• False

PySpark’s selectExpr method allows you to use SQL expressions to select columns, but it cannot be used to perform complex transformations such as aggregations or window functions within those expressions.

• True

• False