Certified Associate Developer for Apache Spark practice test

1. Question 1 · 1

   Which of the following describes the Spark driver?

   Choose one answer.

   • The Spark driver is responsible for performing all execution in all execution modes – it is the entire Spark application.
   • The Spare driver is fault tolerant – if it fails, it will recover the entire Spark application.
   • The Spark driver is the coarsest level of the Spark execution hierarchy – it is synonymous with the Spark application.
   • The Spark driver is the program space in which the Spark application’s main method runs coordinating the Spark entire application.
   • The Spark driver is horizontally scaled to increase overall processing throughput of a Spark application.

2. Question 2 · 1

   Which of the following DataFrame operations is classified as a wide transformation?

   Choose one answer.

   • DataFrame.filter()
   • DataFrame.join()
   • DataFrame.select()
   • DataFrame.drop()
   • DataFrame.union()

3. Question 3 · 1

   The code block shown below contains an error. The code block is intended to return the exact number of distinct values in column division in DataFrame storesDF. Identify the error. Code block: storesDF.agg(approx_count_distinct(col(“division”)).alias(“divisionDistinct”))

   Choose one answer.

   • The approx_count_distinct() operation needs a second argument to set the rsd parameter to ensure it returns the exact number of distinct values.
   • There is no alias() operation for the approx_count_distinct() operation's output.
   • There is no way to return an exact distinct number in Spark because the data Is distributed across partitions.
   • The approx_count_distinct()operation is not a standalone function - it should be used as a method from a Column object.
   • The approx_count_distinct() operation cannot determine an exact number of distinct values in a column.

4. Question 4 · 1

   Which of the following code blocks returns the number of rows in DataFrame storesDF for each distinct combination of values in column division and column storeCategory?

   Choose one answer.

   • storesDF.groupBy(Seq(col(“division”), col(“storeCategory”))).count()
   • storesDF.groupBy(division, storeCategory).count()
   • storesDF.groupBy(“division”, “storeCategory”).count()
   • storesDF.groupBy(“division”).groupBy(“StoreCategory”).count()
   • storesDF.groupBy(Seq(“division”, “storeCategory”)).count()

5. Question 5 · 1

   The code block shown below contains an error. The code block is intended to return a collection of summary statistics for column sqft in Data Frame storesDF. Identify the error. Code block: storesDF.describes(col(“sgft”))

   Choose one answer.

   • The column sqft should be subsetted from DataFrame storesDF prior to computing summary statistics on it alone.
   • The describe() operation does not accept a Column object as an argument outside of a sequence — the sequence Seq(col(“sqft”)) should be specified instead.
   • The describe()operation doesn’t compute summary statistics for a single column — the summary() operation should be used instead.
   • The describe()operation doesn't compute summary statistics for numeric columns — the summary() operation should be used instead.
   • The describe()operation does not accept a Column object as an argument — the column name string “sqft” should be specified instead.

6. Question 6 · 1

   The code block shown below contains an error. The code block is intended to create and register a SQL UDF named “ASSESS_PERFORMANCE” using the Scala function assessPerformance() and apply it to column customerSatisfaction in the table stores. Identify the error. Code block: spark.udf.register(“ASSESS_PERFORMANCE”, assessPerforance) spark.sql(“SELECT customerSatisfaction, assessPerformance(customerSatisfaction) AS result FROM stores”)

   Choose one answer.

   • The customerSatisfaction column cannot be called twice inside the SQL statement.
   • Registered UDFs cannot be applied inside of a SQL statement.
   • The order of the arguments to spark.udf.register() should be reversed.
   • The wrong SQL function is used to compute column result - it should be ASSESS_PERFORMANCE instead of assessPerformance.
   • There is no sql() operation - the DataFrame API must be used to apply the UDF assessPerformance().

7. Question 7 · 1

   The code block shown below contains an error. The code block is intended to create the Scala UDF assessPerformanceUDF() and apply it to the integer column customers1t1sfaction in Data Frame storesDF. Identify the error. Code block:

   Choose one answer.

   

   • The input type of customerSatisfaction is not specified in the udf() operation.
   • The return type of assessPerformanceUDF() must be specified.
   • The withColumn() operation is not appropriate here - UDFs should be applied by iterating over rows instead.
   • The assessPerformanceUDF() must first be defined as a Scala function and then converted to a UDF.
   • UDFs can only be applied via SQL and not through the Data Frame API.

8. Question 8 · 1

   Which of the following describes the difference between cluster and client execution modes?

   Choose one answer.

   • The cluster execution mode runs the driver on a worker node within a cluster, while the client execution mode runs the driver on the client machine (also known as a gateway machine or edge node).
   • The cluster execution mode is run on a local cluster, while the client execution mode is run in the cloud.
   • The cluster execution mode distributes executors across worker nodes in a cluster, while the client execution mode runs a Spark job entirely on one client machine.
   • The cluster execution mode runs the driver on the cluster machine (also known as a gateway machine or edge node), while the client execution mode runs the driver on a worker node within a cluster.
   • The cluster execution mode distributes executors across worker nodes in a cluster, while the client execution mode submits a Spark job from a remote machine to be run on a remote, unconfigurable cluster.

9. Question 9 · 1

   The code block shown below contains an error. The code block intended to return a new DataFrame that is the result of an inner join between DataFrame storesDF and DataFrame employeesDF on column storeId. Identify the error. Code block: StoresDF.join(employeesDF, Seq("storeId")

   Choose one answer.

   • The key column storeId needs to be a string like “storeId”.
   • The key column storeId needs to be specified in an expression of both Data Frame columns like storesDF.storeId ===employeesDF.storeId.
   • The default argument to the joinType parameter is “inner” - an additional argument of “left” must be specified.
   • There is no DataFrame.join() operation - DataFrame.merge() should be used instead.
   • The key column storeId needs to be wrapped in the col() operation.

10. Question 10 · 1

    Which of the following pairs of arguments cannot be used in DataFrame.join() to perform an inner join on two DataFrames, named and aliased with "a" and "b" respectively, to specify two key columns column1 and column2?

    Choose one answer.

    • joinExprs = col(“a.column1”) === col(“b.column1”) and col(“a.column2”) === col(“b.column2”)
    • usingColumns = Seq(col(“column1”), col(“column2”))
    • All of these options can be used to perform an inner join with two key columns.
    • joinExprs = storesDF(“column1”) === employeesDF(“column1”) and storesDF(“column2”) === employeesDF (“column2”)
    • usingColumns = Seq(“column1”, “column2”)

11. Question 11 · 1

    The code block shown below contains an error. The code block is intended to return a new DataFrame that is the result of a position-wise union between DataFrame storesDF and DataFrame acquiredStoresDF.

    Choose one answer.

    • concat(storesDF, acquiredStoresDF)
    • storesDF.unionByName(acquiredStoresDF)
    • union(storesDF, acquiredStoresDF)
    • unionAll(storesDF, acquiredStoresDF)
    • storesDF.union(acquiredStoresDF)

12. Question 12 · 1

    Which of the following code blocks writes DataFrame storesDF to file path filePath as parquet overwriting any existing files in that location?

    Choose one answer.

    • storesDF.write(filePath, mode = “overwrite”)
    • storesDF.write().mode(“overwrite”).parquet(filePath)
    • storesDF.write.mode(“overwrite”).parquet(filePath)
    • storesDF.write.option(“parquet”, “overwrite”).path(filePath)
    • storesDF.write.mode(“overwrite”).path(filePath)

13. Question 13 · 1

    Which of the following code blocks reads a CSV at the file path filePath into a Data Frame with the specified schema schema?

    Choose one answer.

    • spark.read().csv(filePath)
    • spark.read().schema(“schema”).csv(filePath)
    • spark.read.schema(schema).csv(filePath)
    • spark.read.schema(“schema”).csv(filePath)
    • spark.read().schema(schema).csv(filePath)

14. Question 14 · 1

    Which of the following code blocks returns a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 AND the value in column customerSatisfaction is greater than or equal to 30?

    Choose one answer.

    • storesDF.filter(col("sqft") <= 25000 and col("customerSatisfaction") >= 30)
    • storesDF.filter(col("sqft") <= 25000 or col("customerSatisfaction") >= 30)
    • storesDF.filter(sqft) <= 25000 and customerSatisfaction >= 30)
    • storesDF.filter(col("sqft") <= 25000 & col("customerSatisfaction") >= 30)
    • storesDF.filter(sqft <= 25000) & customerSatisfaction >= 30)

15. Question 15 · 1

    Which of the following sets of DataFrame methods will both return a new DataFrame only containing rows that meet a specified logical condition?

    Choose one answer.

    • drop(), where()
    • filter(), select()
    • filter(), where()
    • select(), where()
    • filter(), drop()

Paid access to all 132 questions

Each purchase applies to this certification. Prices are one-time payments, not monthly subscriptions.