Databricks Databricks-Certified-Data-Engineer-Professional : Databricks Certified Data Engineer Professional Exam

A Databricks job has been configured with 3 tasks, each of which is a Databricks notebook. Task A does not depend on other tasks. Tasks B and C run in parallel, with each having a serial dependency on Task A.
If task A fails during a scheduled run, which statement describes the results of this run?

• A. Tasks B and C will be skipped; task A will not commit any changes because of stage failure.
• B. Tasks B and C will be skipped; some logic expressed in task A may have been committed before task failure.
• C. Tasks B and C will attempt to run as configured; any changes made in task A will be rolled back due to task failure.
• D. Because all tasks are managed as a dependency graph, no changes will be committed to the Lakehouse until all tasks have successfully been completed.
• E. Unless all tasks complete successfully, no changes will be committed to the Lakehouse; because task A failed, all commits will be rolled back automatically.

A data governance team at a large enterprise is improving data discoverability across its organization. The team has hundreds of tables in their Databricks Lakehouse with thousands of columns that lack proper documentation. Many of these tables were created by different teams over several years, with missing context about column meanings and business logic. The data governance team needs to quickly generate comprehensive column descriptions for all existing tables to meet compliance requirements and improve data literacy across the organization. They want to leverage modern capabilities to automatically generate meaningful descriptions rather than manually documenting each column, which would take months to complete. Which approach should the team use in Databricks to automatically generate column comments and descriptions for existing tables?

• A. Write custom PySpark code using df.describe() and df.schema to programmatically generate basic statistical descriptions for each column.
• B. Use the DESCRIBE TABLE command to extract existing schema information and manually write descriptions based on column names and data types.
• C. Use Delta Lake's DESCRIBE HISTORY command to analyze table evolution and infer column purposes from historical changes.
• D. Navigate to the table in Databricks Catalog Explorer, select the table schema view, and use the AI Generate option which leverages artificial intelligence to automatically create meaningful column descriptions based on column names, data types, sample values, and data patterns.

Which approach demonstrates a modular and testable way to use DataFrame transform for ETL code in PySpark?

• A.
  
• B.
  
• C.
  
• D.
  

A junior data engineer has been asked to develop a streaming data pipeline with a grouped aggregation using DataFrame df. The pipeline needs to calculate the average humidity and average temperature for each non-overlapping five-minute interval. Events are recorded once per minute per device.
Streaming DataFrame df has the following schema:
"device_id INT, event_time TIMESTAMP, temp FLOAT, humidity FLOAT"
Code block:

Choose the response that correctly fills in the blank within the code block to complete this task.

• A. to_interval("event_time", "5 minutes").alias("time")
• B. "event_time"
• C. lag("event_time", "10 minutes").alias("time")
• D. window("event_time", "10 minutes").alias("time")
• E. window("event_time", "5 minutes").alias("time")

A data engineer is working on a Databricks notebook that requires several third-party Python libraries. Some of these are available on PyPI, while others are custom-developed and stored as local.wheel (.whl) and source (.tar.gz) files in an S3 bucket. The goal is to ensure all dependencies are installed and correctly available across multiple jobs running on any automated cluster in a Unity Catalog-enabled workspace. The engineer needs to install the required dependencies in a way that ensures a consistent environment setup across interactive notebooks and jobs and complies with workspace security policies (no internet access). Which approach should the engineer use to install and manage these dependencies while also ensuring reproducibility and compliance?

• A. Use an init script on the cluster to install all dependencies using pip, referencing the local file system.
• B. Create a Python wheel file for the entire project, upload it to the Databricks Workspace Files or Volumes, and install it using a Cluster Library or pip install in a requirements.txt declared within a Databricks Asset Bundle.
• C. Install all dependencies manually in the driver node of an interactive cluster, then export the environment and reimport on job clusters using %conda.
• D. Use %pip install in every notebook and job to install packages directly from PyPl and custom S3 paths.