Databricks Certified-Data-Engineer-Professional : Databricks Certified Data Engineer Professional

A data engineer is tasked with ensuring that a Delta table in Databricks continuously retains deleted files for 15 days (instead of the default 7 days), in order to permanently comply with the organization's data retention policy. Which code snippet correctly sets this retention period for deleted files?

• A. spark.conf.set("spark.databricks.delta.deletedFileRetentionDuration", "15 days")
• B. from delta.tables import *
  deltaTable = DeltaTable.forPath(spark, "/mnt/data/my_table")
  deltaTable.deletedFileRetentionDuration = "interval 15 days"
• C. spark.sql("ALTER TABLE my_table SET TBLPROPERTIES
  ('delta.deletedFileRetentionDuration' = 'interval 15 days')")
• D. spark.sql("VACUUM my_table RETAIN 15 HOURS")

A data engineer is building a customer data pipeline in Lakeflow Spark Declarative Pipelines. The source is a cloud-based event stream with limited retention containing inserts, updates, and deletes for customer records. These changes are being applied using the AUTO CDC INTO syntax to maintain an SCD Type 1 table as the target table, customer_dim. How should the data engineer build a downstream job that streams from the customer_dim table to only act on updates and delete events, processing data incrementally?

• A. Use ignoreChanges flag while streaming from customer_dim to avoid breaking the pipeline during updates and deletes.
• B. When stored as SCD 1, the target of AUTO CDC INTO includes updates and deletes. Streaming from customer_dim can fail due to these operations. Instead, build another stream from the original source.
• C. Read change data feed from customer_dim table and apply filters to incrementally act on the change events.
• D. Streaming from customer_dim table would only be possible in the case of SCD 2 retention.

A CHECK constraint has been successfully added to the Delta table named activity_details using the following logic:

A batch job is attempting to insert new records to the table, including a record where latitude =
45.50 and longitude = 212.67.
Which statement describes the outcome of this batch insert?

• A. The write will insert all records except those that violate the table constraints; the violating records will be reported in a warning log.
• B. The write will fail when the violating record is reached; any records previously processed will be recorded to the target table.
• C. The write will insert all records except those that violate the table constraints; the violating records will be recorded to a quarantine table.
• D. The write will fail completely because of the constraint violation and no records will be inserted into the target table.
• E. The write will include all records in the target table; any violations will be indicated in the boolean column named valid_coordinates.

The data governance team has instituted a requirement that all tables containing Personal Identifiable Information (PH) must be clearly annotated. This includes adding column comments, table comments, and setting the custom table property "contains_pii" = true.
The following SQL DDL statement is executed to create a new table:

Which command allows manual confirmation that these three requirements have been met?

• A. SHOW TBLPROPERTIES dev.pii test
• B. DESCRIBE HISTORY dev.pii test
• C. DESCRIBE DETAIL dev.pii test
• D. DESCRIBE EXTENDED dev.pii test
• E. SHOW TABLES dev

A data engineer is using Structured Streaming to read in transaction data from a bronze Delta table. It was discovered that the data has quality issues where sometimes the transaction value is negative, and when that occurs, the rows need to be routed to a separate quarantine table. They have low latency requirements for the good data since it is used by downstream systems, but the bad data will only be analyzed periodically and has no production dependencies. The quarantine job needs to be implemented so that it cannot affect the production processes that depend on the good data, and the cost of the job needs to be minimized. How should the quarantine process be implemented in order to satisfy these requirements?

• A. The streaming job for the good data needs to be modified to filter out records with a transaction value less than 0 before writing, and should not share compute with other processes. The streaming job for the quarantine data needs to filter out records with a transaction value greater than or equal to 0 before writing, and should be implemented on a separate small cluster and only run once a day to minimize cost.
• B. The existing streaming job for the good data should be updated to incorporate the quarantining of the bad data. Inside a foreachBatch function, the dataframe should be filtered so that records with a transaction value greater than or equal to 0 are written to the good data table and records with a transaction value less than 0 are written to a quarantine table. Try/Catch can be added around the writes in the foreachBatch function so that the stream can't fail.
• C. The streaming job for the good data needs to be modified to filter out records with a transaction value less than 0 before writing. The streaming job for the quarantine data needs to filter out records with a transaction value greater than or equal to 0 before writing. Both should run as separate streams on the same cluster to minimize cost.
• D. The existing streaming job for the good data should be updated to incorporate the quarantining of the bad data. A new boolean column called "quarantine" should be added to the dataframe, and its value should be set to true if the transaction value is less than 0 and false if the transaction value is greater than or equal to 0. Processing and storing all the data together will save costs.