[Q28-Q52] Best Quality Databricks Associate-Developer-Apache-Spark Exam Questions DumpsTests Realistic Practice Exams [2023]

Best Quality Databricks Associate-Developer-Apache-Spark Exam Questions DumpsTests Realistic Practice Exams [2023]

Critical Information To Databricks Certified Associate Developer for Apache Spark 3.0 Exam Pass the First Time

NEW QUESTION 28
The code block displayed below contains an error. The code block should produce a DataFrame with color as the only column and three rows with color values of red, blue, and green, respectively.
Find the error.
Code block:
1.spark.createDataFrame([("red",), ("blue",), ("green",)], "color")
Instead of calling spark.createDataFrame, just DataFrame should be called.

A. Instead of color, a data type should be specified.
B. The colors red, blue, and green should be expressed as a simple Python list, and not a list of tuples.
C. The "color" expression needs to be wrapped in brackets, so it reads ["color"].
D. The commas in the tuples with the colors should be eliminated.

Answer: C

Explanation:
Explanation
Correct code block:
spark.createDataFrame([("red",), ("blue",), ("green",)], ["color"])
The createDataFrame syntax is not exactly straightforward, but luckily the documentation (linked below) provides several examples on how to use it. It also shows an example very similar to the code block presented here which should help you answer this question correctly.
More info: pyspark.sql.SparkSession.createDataFrame - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2

NEW QUESTION 29
Which of the following describes the characteristics of accumulators?

A. Accumulators are immutable.
B. If an action including an accumulator fails during execution and Spark manages to restart the action and complete it successfully, only the successful attempt will be counted in the accumulator.
C. Accumulators are used to pass around lookup tables across the cluster.
D. All accumulators used in a Spark application are listed in the Spark UI.
E. Accumulators can be instantiated directly via the accumulator(n) method of the pyspark.RDD module.

Answer: B

Explanation:
Explanation
If an action including an accumulator fails during execution and Spark manages to restart the action and complete it successfully, only the successful attempt will be counted in the accumulator.
Correct, when Spark tries to rerun a failed action that includes an accumulator, it will only update the accumulator if the action succeeded.
Accumulators are immutable.
No. Although accumulators behave like write-only variables towards the executors and can only be read by the driver, they are not immutable.
All accumulators used in a Spark application are listed in the Spark UI.
Incorrect. For scala, only named, but not unnamed, accumulators are listed in the Spark UI. For pySpark, no accumulators are listed in the Spark UI - this feature is not yet implemented.
Accumulators are used to pass around lookup tables across the cluster.
Wrong - this is what broadcast variables do.
Accumulators can be instantiated directly via the accumulator(n) method of the pyspark.RDD module.
Wrong, accumulators are instantiated via the accumulator(n) method of the sparkContext, for example: counter
= spark.sparkContext.accumulator(0).
More info: python - In Spark, RDDs are immutable, then how Accumulators are implemented? - Stack Overflow, apache spark - When are accumulators truly reliable? - Stack Overflow, Spark - The Definitive Guide, Chapter 14

NEW QUESTION 30
Which of the following code blocks reads JSON file imports.json into a DataFrame?

A. spark.read().json("/FileStore/imports.json")
B. spark.read.json("/FileStore/imports.json")
C. spark.read.format("json").path("/FileStore/imports.json")
D. spark.read().mode("json").path("/FileStore/imports.json")
E. spark.read("json", "/FileStore/imports.json")

Answer: B

Explanation:
Explanation
Static notebook | Dynamic notebook: See test 1
(https://flrs.github.io/spark_practice_tests_code/#1/25.html ,
https://bit.ly/sparkpracticeexams_import_instructions)

NEW QUESTION 31
Which of the following describes the role of the cluster manager?

A. The cluster manager allocates resources to the DataFrame manager.
B. The cluster manager schedules tasks on the cluster in client mode.
C. The cluster manager allocates resources to Spark applications and maintains the executor processes in client mode.
D. The cluster manager schedules tasks on the cluster in local mode.
E. The cluster manager allocates resources to Spark applications and maintains the executor processes in remote mode.

Answer: C

Explanation:
Explanation
The cluster manager allocates resources to Spark applications and maintains the executor processes in client mode.
Correct. In cluster mode, the cluster manager is located on a node other than the client machine. From there it starts and ends executor processes on the cluster nodes as required by the Spark application running on the Spark driver.
The cluster manager allocates resources to Spark applications and maintains the executor processes in remote mode.
Wrong, there is no "remote" execution mode in Spark. Available execution modes are local, client, and cluster.
The cluster manager allocates resources to the DataFrame manager
Wrong, there is no "DataFrame manager" in Spark.
The cluster manager schedules tasks on the cluster in client mode.
No, in client mode, the Spark driver schedules tasks on the cluster - not the cluster manager.
The cluster manager schedules tasks on the cluster in local mode.
Wrong: In local mode, there is no "cluster". The Spark application is running on a single machine, not on a cluster of machines.

NEW QUESTION 32
Which of the following code blocks returns a copy of DataFrame transactionsDf that only includes columns transactionId, storeId, productId and f?
Sample of DataFrame transactionsDf:
1.+-------------+---------+-----+-------+---------+----+
2.|transactionId|predError|value|storeId|productId| f|
3.+-------------+---------+-----+-------+---------+----+
4.| 1| 3| 4| 25| 1|null|
5.| 2| 6| 7| 2| 2|null|
6.| 3| 3| null| 25| 3|null|
7.+-------------+---------+-----+-------+---------+----+

A. transactionsDf.drop(["predError", "value"])
B. transactionsDf.drop(value, predError)
C. transactionsDf.drop("predError", "value")
D. transactionsDf.drop([col("predError"), col("value")])
E. transactionsDf.drop(col("value"), col("predError"))

Answer: C

Explanation:
Explanation
Output of correct code block:
+-------------+-------+---------+----+
|transactionId|storeId|productId| f|
+-------------+-------+---------+----+
| 1| 25| 1|null|
| 2| 2| 2|null|
| 3| 25| 3|null|
+-------------+-------+---------+----+
To solve this question, you should be fmailiar with the drop() API. The order of column names does not matter
- in this question the order differs in some answers just to confuse you. Also, drop() does not take a list. The *cols operator in the documentation means that all arguments passed to drop() are interpreted as column names.
More info: pyspark.sql.DataFrame.drop - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2

NEW QUESTION 33
Which of the following describes a narrow transformation?

A. A narrow transformation is a process in which data from multiple RDDs is used.
B. A narrow transformation is an operation in which data is exchanged across the cluster.
C. A narrow transformation is a process in which 32-bit float variables are cast to smaller float variables, like 16-bit or 8-bit float variables.
D. A narrow transformation is an operation in which no data is exchanged across the cluster.
E. narrow transformation is an operation in which data is exchanged across partitions.

Answer: D

Explanation:
Explanation
A narrow transformation is an operation in which no data is exchanged across the cluster.
Correct! In narrow transformations, no data is exchanged across the cluster, since these transformations do not require any data from outside of the partition they are applied on. Typical narrow transformations include filter, drop, and coalesce.
A narrow transformation is an operation in which data is exchanged across partitions.
No, that would be one definition of a wide transformation, but not of a narrow transformation. Wide transformations typically cause a shuffle, in which data is exchanged across partitions, executors, and the cluster.
A narrow transformation is an operation in which data is exchanged across the cluster.
No, see explanation just above this one.
A narrow transformation is a process in which 32-bit float variables are cast to smaller float variables, like
16-bit or 8-bit float variables.
No, type conversion has nothing to do with narrow transformations in Spark.
A narrow transformation is a process in which data from multiple RDDs is used.
No. A resilient distributed dataset (RDD) can be described as a collection of partitions. In a narrow transformation, no data is exchanged between partitions. Thus, no data is exchanged between RDDs.
One could say though that a narrow transformation and, in fact, any transformation results in a new RDD being created. This is because a transformation results in a change to an existing RDD (RDDs are the foundation of other Spark data structures, like DataFrames). But, since RDDs are immutable, a new RDD needs to be created to reflect the change caused by the transformation.
More info: Spark Transformation and Action: A Deep Dive | by Misbah Uddin | CodeX | Medium

NEW QUESTION 34
The code block displayed below contains an error. The code block should save DataFrame transactionsDf at path path as a parquet file, appending to any existing parquet file. Find the error.
Code block:

A. transactionsDf.format("parquet").option("mode", "append").save(path)
B. The code block is missing a bucketBy command that takes care of partitions.
C. The code block is missing a reference to the DataFrameWriter.
D. Given that the DataFrame should be saved as parquet file, path is being passed to the wrong method.
E. save() is evaluated lazily and needs to be followed by an action.
F. The mode option should be omitted so that the command uses the default mode.

Answer: C

Explanation:
Explanation
Correct code block:
transactionsDf.write.format("parquet").option("mode", "append").save(path)

NEW QUESTION 35
Which of the following code blocks returns a 2-column DataFrame that shows the distinct values in column productId and the number of rows with that productId in DataFrame transactionsDf?

A. transactionsDf.groupBy("productId").count()
B. transactionsDf.count("productId")
C. transactionsDf.groupBy("productId").agg(col("value").count())
D. transactionsDf.count("productId").distinct()
E. transactionsDf.groupBy("productId").select(count("value"))

Answer: A

Explanation:
Explanation
transactionsDf.groupBy("productId").count()
Correct. This code block first groups DataFrame transactionsDf by column productId and then counts the rows in each group.
transactionsDf.groupBy("productId").select(count("value"))
Incorrect. You cannot call select on a GroupedData object (the output of a groupBy) statement.
transactionsDf.count("productId")
No. DataFrame.count() does not take any arguments.
transactionsDf.count("productId").distinct()
Wrong. Since DataFrame.count() does not take any arguments, this option cannot be right.
transactionsDf.groupBy("productId").agg(col("value").count())
False. A Column object, as returned by col("value"), does not have a count() method. You can see all available methods for Column object linked in the Spark documentation below.
More info: pyspark.sql.DataFrame.count - PySpark 3.1.2 documentation, pyspark.sql.Column - PySpark
3.1.2 documentation
Static notebook | Dynamic notebook: See test 3

NEW QUESTION 36
Which of the following code blocks generally causes a great amount of network traffic?

A. DataFrame.count()
B. DataFrame.coalesce()
C. DataFrame.collect()
D. DataFrame.select()
E. DataFrame.rdd.map()

Answer: C

Explanation:
Explanation
DataFrame.collect() sends all data in a DataFrame from executors to the driver, so this generally causes a great amount of network traffic in comparison to the other options listed.
DataFrame.coalesce() just reduces the number of partitions and generally aims to reduce network traffic in comparison to a full shuffle.
DataFrame.select() is evaluated lazily and, unless followed by an action, does not cause significant network traffic.
DataFrame.rdd.map() is evaluated lazily, it does therefore not cause great amounts of network traffic.
DataFrame.count() is an action. While it does cause some network traffic, for the same DataFrame, collecting all data in the driver would generally be considered to cause a greater amount of network traffic.

NEW QUESTION 37
The code block displayed below contains an error. The code block should create DataFrame itemsAttributesDf which has columns itemId and attribute and lists every attribute from the attributes column in DataFrame itemsDf next to the itemId of the respective row in itemsDf. Find the error.
A sample of DataFrame itemsDf is below.

Code block:
itemsAttributesDf = itemsDf.explode("attributes").alias("attribute").select("attribute", "itemId")

A. explode() is not a method of DataFrame. explode() should be used inside the select() method instead.
B. The explode() method expects a Column object rather than a string.
C. The split() method should be used inside the select() method instead of the explode() method.
D. Since itemId is the index, it does not need to be an argument to the select() method.
E. The alias() method needs to be called after the select() method.

Answer: A

Explanation:
The correct code block looks like this:

Then, the first couple of rows of itemAttributesDf look like this:

explode() is not a method of DataFrame. explode() should be used inside the select() method instead.
This is correct.
The split() method should be used inside the select() method instead of the explode() method.
No, the split() method is used to split strings into parts. However, column attributs is an array of strings. In this case, the explode() method is appropriate.
Since itemId is the index, it does not need to be an argument to the select() method.
No, itemId still needs to be selected, whether it is used as an index or not.
The explode() method expects a Column object rather than a string.
No, a string works just fine here. This being said, there are some valid alternatives to passing in a string:

The alias() method needs to be called after the select() method.
No.
More info: pyspark.sql.functions.explode - PySpark 3.1.1 documentation (https://bit.ly/2QUZI1J) Static notebook | Dynamic notebook: See test 1 (https://flrs.github.io/spark_practice_tests_code/#1/22.html ,
https://bit.ly/sparkpracticeexams_import_instructions)

NEW QUESTION 38
The code block displayed below contains an error. When the code block below has executed, it should have divided DataFrame transactionsDf into 14 parts, based on columns storeId and transactionDate (in this order). Find the error.
Code block:
transactionsDf.coalesce(14, ("storeId", "transactionDate"))

A. Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .select() needs to be appended to the code block.
B. The parentheses around the column names need to be removed and .select() needs to be appended to the code block.
C. Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .count() needs to be appended to the code block.
(Correct)
D. Operator coalesce needs to be replaced by repartition and the parentheses around the column names need to be replaced by square brackets.
E. Operator coalesce needs to be replaced by repartition.

Answer: C

Explanation:
Explanation
Correct code block:
transactionsDf.repartition(14, "storeId", "transactionDate").count()
Since we do not know how many partitions DataFrame transactionsDf has, we cannot safely use coalesce, since it would not make any change if the current number of partitions is smaller than 14.
So, we need to use repartition.
In the Spark documentation, the call structure for repartition is shown like this:
DataFrame.repartition(numPartitions, *cols). The * operator means that any argument after numPartitions will be interpreted as column. Therefore, the brackets need to be removed.
Finally, the question specifies that after the execution the DataFrame should be divided. So, indirectly this question is asking us to append an action to the code block. Since .select() is a transformation. the only possible choice here is .count().
More info: pyspark.sql.DataFrame.repartition - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1

NEW QUESTION 39
Which of the following statements about Spark's configuration properties is incorrect?

A. The default number of partitions to use when shuffling data for joins or aggregations is 300.
B. The maximum number of tasks that an executor can process at the same time is controlled by the spark.executor.cores property.
C. The maximum number of tasks that an executor can process at the same time is controlled by the spark.task.cpus property.
D. The default value for spark.sql.autoBroadcastJoinThreshold is 10MB.
E. The default number of partitions returned from certain transformations can be controlled by the spark.default.parallelism property.

Answer: A

Explanation:
Explanation
The default number of partitions to use when shuffling data for joins or aggregations is 300.
No, the default value of the applicable property spark.sql.shuffle.partitions is 200.
The maximum number of tasks that an executor can process at the same time is controlled by the spark.executor.cores property.
Correct, see below.
The maximum number of tasks that an executor can process at the same time is controlled by the spark.task.cpus property.
Correct, the maximum number of tasks that an executor can process in parallel depends on both properties spark.task.cpus and spark.executor.cores. This is because the available number of slots is calculated by dividing the number of cores per executor by the number of cores per task. For more info specifically to this point, check out Spark Architecture | Distributed Systems Architecture.
More info: Configuration - Spark 3.1.2 Documentation

NEW QUESTION 40
The code block shown below should read all files with the file ending .png in directory path into Spark.
Choose the answer that correctly fills the blanks in the code block to accomplish this.
spark.__1__.__2__(__3__).option(__4__, "*.png").__5__(path)

A. 1. open
2. as
3. "binaryFile"
4. "pathGlobFilter"
5. load
B. 1. read
2. format
3. "binaryFile"
4. "pathGlobFilter"
5. load
C. 1. open
2. format
3. "image"
4. "fileType"
5. open
D. 1. read
2. format
3. binaryFile
4. pathGlobFilter
5. load
E. 1. read()
2. format
3. "binaryFile"
4. "recursiveFileLookup"
5. load

Answer: B

Explanation:
Explanation
Correct code block:
spark.read.format("binaryFile").option("recursiveFileLookup", "*.png").load(path) Spark can deal with binary files, like images. Using the binaryFile format specification in the SparkSession's read API is the way to read in those files. Remember that, to access the read API, you need to start the command with spark.read. The pathGlobFilter option is a great way to filter files by name (and ending). Finally, the path can be specified using the load operator - the open operator shown in one of the answers does not exist.

NEW QUESTION 41
Which of the following describes Spark's standalone deployment mode?

A. Standalone mode means that the cluster does not contain the driver.
B. Standalone mode is a viable solution for clusters that run multiple frameworks, not only Spark.
C. Standalone mode is how Spark runs on YARN and Mesos clusters.
D. Standalone mode uses a single JVM to run Spark driver and executor processes.
E. Standalone mode uses only a single executor per worker per application.

Answer: E

Explanation:
Explanation
Standalone mode uses only a single executor per worker per application.
This is correct and a limitation of Spark's standalone mode.
Standalone mode is a viable solution for clusters that run multiple frameworks.
Incorrect. A limitation of standalone mode is that Apache Spark must be the only framework running on the cluster. If you would want to run multiple frameworks on the same cluster in parallel, for example Apache Spark and Apache Flink, you would consider the YARN deployment mode.
Standalone mode uses a single JVM to run Spark driver and executor processes.
No, this is what local mode does.
Standalone mode is how Spark runs on YARN and Mesos clusters.
No. YARN and Mesos modes are two deployment modes that are different from standalone mode. These modes allow Spark to run alongside other frameworks on a cluster. When Spark is run in standalone mode, only the Spark framework can run on the cluster.
Standalone mode means that the cluster does not contain the driver.
Incorrect, the cluster does not contain the driver in client mode, but in standalone mode the driver runs on a node in the cluster.
More info: Learning Spark, 2nd Edition, Chapter 1

NEW QUESTION 42
The code block displayed below contains an error. The code block should use Python method find_most_freq_letter to find the letter present most in column itemName of DataFrame itemsDf and return it in a new column most_frequent_letter. Find the error.
Code block:
1. find_most_freq_letter_udf = udf(find_most_freq_letter)
2. itemsDf.withColumn("most_frequent_letter", find_most_freq_letter("itemName"))

A. Spark is not using the UDF method correctly.
B. UDFs do not exist in PySpark.
C. The "itemName" expression should be wrapped in col().
D. Spark is not adding a column.
E. The UDF method is not registered correctly, since the return type is missing.

Answer: A

Explanation:
Explanation
Correct code block:
find_most_freq_letter_udf = udf(find_most_frequent_letter)
itemsDf.withColumn("most_frequent_letter", find_most_freq_letter_udf("itemName")) Spark should use the previously registered find_most_freq_letter_udf method here - but it is not doing that in the original codeblock. There, it just uses the non-UDF version of the Python method.
Note that typically, we would have to specify a return type for udf(). Except in this case, since the default return type for udf() is a string which is what we are expecting here. If we wanted to return an integer variable instead, we would have to register the Python function as UDF using find_most_freq_letter_udf = udf(find_most_freq_letter, IntegerType()).
More info: pyspark.sql.functions.udf - PySpark 3.1.1 documentation

NEW QUESTION 43
Which of the following code blocks reads in the two-partition parquet file stored at filePath, making sure all columns are included exactly once even though each partition has a different schema?
Schema of first partition:
1.root
2. |-- transactionId: integer (nullable = true)
3. |-- predError: integer (nullable = true)
4. |-- value: integer (nullable = true)
5. |-- storeId: integer (nullable = true)
6. |-- productId: integer (nullable = true)
7. |-- f: integer (nullable = true)
Schema of second partition:
1.root
2. |-- transactionId: integer (nullable = true)
3. |-- predError: integer (nullable = true)
4. |-- value: integer (nullable = true)
5. |-- storeId: integer (nullable = true)
6. |-- rollId: integer (nullable = true)
7. |-- f: integer (nullable = true)
8. |-- tax_id: integer (nullable = false)

A. spark.read.option("mergeSchema", "true").parquet(filePath)
B. 1.nx = 0
2.for file in dbutils.fs.ls(filePath):
3. if not file.name.endswith(".parquet"):
4. continue
5. df_temp = spark.read.parquet(file.path)
6. if nx == 0:
7. df = df_temp
8. else:
9. df = df.join(df_temp, how="outer")
10. nx = nx+1
11.df
C. spark.read.parquet(filePath)
D. 1.nx = 0
2.for file in dbutils.fs.ls(filePath):
3. if not file.name.endswith(".parquet"):
4. continue
5. df_temp = spark.read.parquet(file.path)
6. if nx == 0:
7. df = df_temp
8. else:
9. df = df.union(df_temp)
10. nx = nx+1
11.df
E. spark.read.parquet(filePath, mergeSchema='y')

Answer: A

Explanation:
Explanation
This is a very tricky question and involves both knowledge about merging as well as schemas when reading parquet files.
spark.read.option("mergeSchema", "true").parquet(filePath)
Correct. Spark's DataFrameReader's mergeSchema option will work well here, since columns that appear in both partitions have matching data types. Note that mergeSchema would fail if one or more columns with the same name that appear in both partitions would have different data types.
spark.read.parquet(filePath)
Incorrect. While this would read in data from both partitions, only the schema in the parquet file that is read in first would be considered, so some columns that appear only in the second partition (e.g. tax_id) would be lost.
nx = 0
for file in dbutils.fs.ls(filePath):
if not file.name.endswith(".parquet"):
continue
df_temp = spark.read.parquet(file.path)
if nx == 0:
df = df_temp
else:
df = df.union(df_temp)
nx = nx+1
df
Wrong. The key idea of this solution is the DataFrame.union() command. While this command merges all data, it requires that both partitions have the exact same number of columns with identical data types.
spark.read.parquet(filePath, mergeSchema="y")
False. While using the mergeSchema option is the correct way to solve this problem and it can even be called with DataFrameReader.parquet() as in the code block, it accepts the value True as a boolean or string variable. But 'y' is not a valid option.
nx = 0
for file in dbutils.fs.ls(filePath):
if not file.name.endswith(".parquet"):
continue
df_temp = spark.read.parquet(file.path)
if nx == 0:
df = df_temp
else:
df = df.join(df_temp, how="outer")
nx = nx+1
df
No. This provokes a full outer join. While the resulting DataFrame will have all columns of both partitions, columns that appear in both partitions will be duplicated - the question says all columns that are included in the partitions should appear exactly once.
More info: Merging different schemas in Apache Spark | by Thiago Cordon | Data Arena | Medium Static notebook | Dynamic notebook: See test 3

NEW QUESTION 44
The code block shown below should add column transactionDateForm to DataFrame transactionsDf. The column should express the unix-format timestamps in column transactionDate as string type like Apr 26 (Sunday). Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__(__2__, from_unixtime(__3__, __4__))

A. 1. withColumn
2. "transactionDateForm"
3. "MMM d (EEEE)"
4. "transactionDate"
B. 1. withColumnRenamed
2. "transactionDate"
3. "transactionDateForm"
4. "MM d (EEE)"
C. 1. withColumn
2. "transactionDateForm"
3. "transactionDate"
4. "MM d (EEE)"
D. 1. withColumn
2. "transactionDateForm"
3. "transactionDate"
4. "MMM d (EEEE)"
E. 1. select
2. "transactionDate"
3. "transactionDateForm"
4. "MMM d (EEEE)"

Answer: D

Explanation:
Explanation
Correct code block:
transactionsDf.withColumn("transactionDateForm", from_unixtime("transactionDate", "MMM d (EEEE)")) The question specifically asks about "adding" a column. In the context of all presented answers, DataFrame.withColumn() is the correct command for this. In theory, DataFrame.select() could also be used for this purpose, if all existing columns are selected and a new one is added.
DataFrame.withColumnRenamed() is not the appropriate command, since it can only rename existing columns, but cannot add a new column or change the value of a column.
Once DataFrame.withColumn() is chosen, you can read in the documentation (see below) that the first input argument to the method should be the column name of the new column.
The final difficulty is the date format. The question indicates that the date format Apr 26 (Sunday) is desired. The answers give "MMM d (EEEE)" and "MM d (EEE)" as options. It can be hard to know the details of the date format that is used in Spark. Specifically, knowing the differences between MMM and MM is probably not something you deal with every day. But, there is an easy way to remember the difference: M (one letter) is usually the shortest form: 4 for April. MM includes padding: 04 for April. MMM (three letters) is the three-letter month abbreviation: Apr for April. And MMMM is the longest possible form: April. Knowing this four-letter sequence helps you select the correct option here.
More info: pyspark.sql.DataFrame.withColumn - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3

NEW QUESTION 45
Which of the following code blocks returns approximately 1000 rows, some of them potentially being duplicates, from the 2000-row DataFrame transactionsDf that only has unique rows?

A. transactionsDf.sample(True, 0.5, force=True)
B. transactionsDf.take(1000).distinct()
C. transactionsDf.sample(True, 0.5)
D. transactionsDf.sample(False, 0.5)
E. transactionsDf.take(1000)

Answer: C

Explanation:
Explanation
To solve this question, you need to know that DataFrame.sample() is not guaranteed to return the exact fraction of the number of rows specified as an argument. Furthermore, since duplicates may be returned, you should understand that the operator's withReplacement argument should be set to True. A force= argument for the operator does not exist.
While the take argument returns an exact number of rows, it will just take the first specified number of rows (1000 in this question) from the DataFrame. Since the DataFrame does not include duplicate rows, there is no potential of any of those returned rows being duplicates when using take(), so the correct answer cannot involve take().
More info: pyspark.sql.DataFrame.sample - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2

A. 1. length
2. regexp_extract
3. upper
4. col("itemName")
5. as
B. 1. size
2. regexp_replace
3. lower
4. "itemName"
5. alias
C. 1. length
2. regexp_replace
3. lower
4. col("itemName")
5. alias
D. 1. lower
2. regexp_replace
3. length
4. "itemName"
5. alias
E. 1. size
2. regexp_extract
3. lower
4. col("itemName")
5. alias

Answer: C

Explanation:
Explanation
Correct code block:
itemsDf.select(length(regexp_replace(lower(col("itemName")), "a|e|i|o|u|\s", "")).alias("consonant_ct")) Returned DataFrame:
+------------+
|consonant_ct|
+------------+
| 19|
| 16|
| 10|
+------------+
This question tries to make you think about the string functions Spark provides and in which order they should be applied. Arguably the most difficult part, the regular expression "a|e|i|o|u|
\s", is not a numbered blank. However, if you are not familiar with the string functions, it may be a good idea to review those before the exam.
The size operator and the length operator can easily be confused. size works on arrays, while length works on strings. Luckily, this is something you can read up about in the documentation.
The code block works by first converting all uppercase letters in column itemName into lowercase (the lower() part). Then, it replaces all vowels by "nothing" - an empty character "" (the regexp_replace() part). Now, only lowercase characters without spaces are included in the DataFrame. Then, per row, the length operator counts these remaining characters. Note that column itemName in itemsDf does not include any numbers or other characters, so we do not need to make any provisions for these. Finally, by using the alias() operator, we rename the resulting column to consonant_ct.
More info:
- lower: pyspark.sql.functions.lower - PySpark 3.1.2 documentation
- regexp_replace: pyspark.sql.functions.regexp_replace - PySpark 3.1.2 documentation
- length: pyspark.sql.functions.length - PySpark 3.1.2 documentation
- alias: pyspark.sql.Column.alias - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2

NEW QUESTION 47
Which of the following code blocks uses a schema fileSchema to read a parquet file at location filePath into a DataFrame?

A. spark.read().schema(fileSchema).parquet(filePath)
B. spark.read.schema(fileSchema).format("parquet").load(filePath)
C. spark.read.schema("fileSchema").format("parquet").load(filePath)
D. spark.read().schema(fileSchema).format(parquet).load(filePath)
E. spark.read.schema(fileSchema).open(filePath)

Answer: B

Explanation:
Explanation
Pay attention here to which variables are quoted. fileSchema is a variable and thus should not be in quotes.
parquet is not a variable and therefore should be in quotes.
SparkSession.read (here referenced as spark.read) returns a DataFrameReader which all subsequent calls reference - the DataFrameReader is not callable, so you should not use parentheses here.
Finally, there is no open method in PySpark. The method name is load.
Static notebook | Dynamic notebook: See test 1

NEW QUESTION 48
Which of the following describes the difference between client and cluster execution modes?

A. In cluster mode, the driver runs on the worker nodes, while the client mode runs the driver on the client machine.
B. In cluster mode, the driver runs on the edge node, while the client mode runs the driver in a worker node.
C. In cluster mode, each node will launch its own executor, while in client mode, executors will exclusively run on the client machine.
D. In client mode, the cluster manager runs on the same host as the driver, while in cluster mode, the cluster manager runs on a separate node.
E. In cluster mode, the driver runs on the master node, while in client mode, the driver runs on a virtual machine in the cloud.

Answer: A

Explanation:
Explanation
In cluster mode, the driver runs on the master node, while in client mode, the driver runs on a virtual machine in the cloud.
This is wrong, since execution modes do not specify whether workloads are run in the cloud or on-premise.
In cluster mode, each node will launch its own executor, while in client mode, executors will exclusively run on the client machine.
Wrong, since in both cases executors run on worker nodes.
In cluster mode, the driver runs on the edge node, while the client mode runs the driver in a worker node.
Wrong - in cluster mode, the driver runs on a worker node. In client mode, the driver runs on the client machine.
In client mode, the cluster manager runs on the same host as the driver, while in cluster mode, the cluster manager runs on a separate node.
No. In both modes, the cluster manager is typically on a separate node - not on the same host as the driver. It only runs on the same host as the driver in local execution mode.
More info: Learning Spark, 2nd Edition, Chapter 1, and Spark: The Definitive Guide, Chapter 15. ()

NEW QUESTION 49
The code block displayed below contains an error. The code block should return DataFrame transactionsDf, but with the column storeId renamed to storeNumber. Find the error.
Code block:
transactionsDf.withColumn("storeNumber", "storeId")

A. The withColumn operator should be replaced with the copyDataFrame operator.
B. Instead of withColumn, the withColumnRenamed method should be used and argument "storeId" should be the first and argument "storeNumber" should be the second argument to that method.
C. Instead of withColumn, the withColumnRenamed method should be used.
D. Argument "storeId" should be the first and argument "storeNumber" should be the second argument to the withColumn method.
E. Arguments "storeNumber" and "storeId" each need to be wrapped in a col() operator.

Answer: B

Explanation:
Explanation
Correct code block:
transactionsDf.withColumnRenamed("storeId", "storeNumber")
More info: pyspark.sql.DataFrame.withColumnRenamed - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1

NEW QUESTION 50
Which of the following code blocks returns a single-row DataFrame that only has a column corr which shows the Pearson correlation coefficient between columns predError and value in DataFrame transactionsDf?

A. transactionsDf.select(corr(["predError", "value"]).alias("corr")).first()
B. transactionsDf.select(corr(col("predError"), col("value")).alias("corr")) (Correct)
C. transactionsDf.select(corr(predError, value).alias("corr"))
D. transactionsDf.select(corr("predError", "value"))
E. transactionsDf.select(corr(col("predError"), col("value")).alias("corr")).first()

Answer: B

Explanation:
Explanation
In difficulty, this question is above what you can expect from the exam. What this question NO:
wants to teach you, however, is to pay attention to the useful details included in the documentation.
pyspark.sql.corr is not a very common method, but it deals with Spark's data structure in an interesting way.
The command takes two columns over multiple rows and returns a single row - similar to an aggregation function. When examining the documentation (linked below), you will find this code example:
a = range(20)
b = [2 * x for x in range(20)]
df = spark.createDataFrame(zip(a, b), ["a", "b"])
df.agg(corr("a", "b").alias('c')).collect()
[Row(c=1.0)]
See how corr just returns a single row? Once you understand this, you should be suspicious about answers that include first(), since there is no need to just select a single row. A reason to eliminate those answers is that DataFrame.first() returns an object of type Row, but not DataFrame, as requested in the question.
transactionsDf.select(corr(col("predError"), col("value")).alias("corr")) Correct! After calculating the Pearson correlation coefficient, the resulting column is correctly renamed to corr.
transactionsDf.select(corr(predError, value).alias("corr"))
No. In this answer, Python will interpret column names predError and value as variable names.
transactionsDf.select(corr(col("predError"), col("value")).alias("corr")).first() Incorrect. first() returns a row, not a DataFrame (see above and linked documentation below).
transactionsDf.select(corr("predError", "value"))
Wrong. Whie this statement returns a DataFrame in the desired shape, the column will have the name corr(predError, value) and not corr.
transactionsDf.select(corr(["predError", "value"]).alias("corr")).first() False. In addition to first() returning a row, this code block also uses the wrong call structure for command corr which takes two arguments (the two columns to correlate).
More info:
- pyspark.sql.functions.corr - PySpark 3.1.2 documentation
- pyspark.sql.DataFrame.first - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3

NEW QUESTION 51
The code block displayed below contains an error. The code block is intended to return all columns of DataFrame transactionsDf except for columns predError, productId, and value. Find the error.
Excerpt of DataFrame transactionsDf:
transactionsDf.select(~col("predError"), ~col("productId"), ~col("value"))

A. The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value wrapped in the col operator so they should be expressed like drop(col(predError), col(productId), col(value)).
B. The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value as strings.
(Correct)
C. The column names in the select operator should not be strings and wrapped in the col operator, so they should be expressed like select(~col(predError), ~col(productId), ~col(value)).
D. The select operator should be replaced by the drop operator.
E. The select operator should be replaced with the deselect operator.

Answer: B

Explanation:
Explanation
Correct code block:
transactionsDf.drop("predError", "productId", "value")
Static notebook | Dynamic notebook: See test 1

NEW QUESTION 52
......

Associate-Developer-Apache-Spark EXAM DUMPS WITH GUARANTEED SUCCESS: https://freetorrent.dumpstests.com/Associate-Developer-Apache-Spark-latest-test-dumps.html

[Q28-Q52] Best Quality Databricks Associate-Developer-Apache-Spark Exam Questions DumpsTests Realistic Practice Exams [2023]

Related Articles

Contact Us