Databricks Associate-Developer-Apache-Spark-3.5 問題集 : Databricks Certified Associate Developer for Apache Spark 3.5 - Python

A Spark engineer must select an appropriate deployment mode for the Spark jobs.
What is the benefit of using cluster mode in Apache Spark™?

• A. In cluster mode, resources are allocated from a resource manager on the cluster, enabling better performance and scalability for large jobs
• B. In cluster mode, the driver is responsible for executing all tasks locally without distributing them across the worker nodes.
• C. In cluster mode, the driver program runs on one of the worker nodes, allowing the application to fully utilize the distributed resources of the cluster.
• D. In cluster mode, the driver runs on the client machine, which can limit the application's ability to handle large datasets efficiently.

An engineer wants to join two DataFrames df1 and df2 on the respective employee_id and emp_id columns:
df1: employee_id INT, name STRING
df2: emp_id INT, department STRING
The engineer uses:
result = df1.join(df2, df1.employee_id == df2.emp_id, how='inner')
What is the behaviour of the code snippet?

• A. The code works as expected because the join condition explicitly matches employee_id from df1 with emp_id from df2
• B. The code fails to execute because the column names employee_id and emp_id do not match automatically
• C. The code fails to execute because it must use on='employee_id' to specify the join column explicitly
• D. The code fails to execute because PySpark does not support joining DataFrames with a different structure

A developer is working with a pandas DataFrame containing user behavior data from a web application.
Which approach should be used for executing a groupBy operation in parallel across all workers in Apache Spark 3.5?
A)
Use the applylnPandas API
B)

C)

• A. Use a Pandas UDF:
  @pandas_udf("double")
  def mean_func(value: pd.Series) -> float:
  return value.mean()
  df.groupby("user_id").agg(mean_func(df["value"])).show()
• B. Use the mapInPandas API:
  df.mapInPandas(mean_func, schema="user_id long, value double").show()
• C. Use a regular Spark UDF:
  from pyspark.sql.functions import mean
  df.groupBy("user_id").agg(mean("value")).show()
• D. Use the applyInPandas API:
  df.groupby("user_id").applyInPandas(mean_func, schema="user_id long, value double").show()

A developer is running Spark SQL queries and notices underutilization of resources. Executors are idle, and the number of tasks per stage is low.
What should the developer do to improve cluster utilization?

• A. Enable dynamic resource allocation to scale resources as needed
• B. Increase the size of the dataset to create more partitions
• C. Reduce the value of spark.sql.shuffle.partitions
• D. Increase the value of spark.sql.shuffle.partitions

How can a Spark developer ensure optimal resource utilization when running Spark jobs in Local Mode for testing?
Options:

• A. Increase the number of local threads based on the number of CPU cores.
• B. Set the spark.executor.memory property to a large value.
• C. Use the spark.dynamicAllocation.enabled property to scale resources dynamically.
• D. Configure the application to run in cluster mode instead of local mode.