Certified Data Engineer Professional practice test

1. Question 1 · 1

   A Delta table of weather records is partitioned by date and has the below schema: date DATE, device_id INT, temp FLOAT, latitude FLOAT, longitude FLOAT To find all the records from within the Arctic Circle, you execute a query with the below filter: latitude > 66.3 Which statement describes how the Delta engine identifies which files to load?

   Choose one answer.

   • All records are cached to an operational database and then the filter is applied
   • The Parquet file footers are scanned for min and max statistics for the latitude column
   • All records are cached to attached storage and then the filter is applied
   • The Delta log is scanned for min and max statistics for the latitude column
   • The Hive metastore is scanned for min and max statistics for the latitude column

2. Question 2 · 1

   Which indicators would you look for in the Spark UI’s Storage tab to signal that a cached table is not performing optimally? Assume you are using Spark’s MEMORY_ONLY storage level.

   Choose one answer.

   • Size on Disk is < Size in Memory
   • The RDD Block Name includes the “*” annotation signaling a failure to cache
   • Size on Disk is > 0
   • The number of Cached Partitions > the number of Spark Partitions
   • On Heap Memory Usage is within 75% of Off Heap Memory Usage

3. Question 3 · 1

   What is the first line of a Databricks Python notebook when viewed in a text editor?

   Choose one answer.

   • %python
   • // Databricks notebook source
   • # Databricks notebook source
   • -- Databricks notebook source
   • # MAGIC %python

4. Question 4 · 1

   Which statement describes a key benefit of an end-to-end test?

   Choose one answer.

   • Makes it easier to automate your test suite
   • Pinpoints errors in the building blocks of your application
   • Provides testing coverage for all code paths and branches
   • Closely simulates real world usage of your application
   • Ensures code is optimized for a real-life workflow

5. Question 5 · 1

   The Databricks CLI is used to trigger a run of an existing job by passing the job_id parameter. The response that the job run request has been submitted successfully includes a field run_id. Which statement describes what the number alongside this field represents?

   Choose one answer.

   • The job_id and number of times the job has been run are concatenated and returned.
   • The total number of jobs that have been run in the workspace.
   • The number of times the job definition has been run in this workspace.
   • The job_id is returned in this field.
   • The globally unique ID of the newly triggered run.

6. Question 6 · 1

   The data science team has created and logged a production model using MLflow. The model accepts a list of column names and returns a new column of type DOUBLE. The following code correctly imports the production model, loads the customers table containing the customer_id key column into a DataFrame, and defines the feature columns needed for the model. Which code block will output a DataFrame with the schema "customer_id LONG, predictions DOUBLE"?

   Choose one answer.

   

   • df.map(lambda x:model(x[columns])).select("customer_id, predictions")
   • df.select("customer_id", model(*columns).alias("predictions"))
   • model.predict(df, columns)
   • df.select("customer_id", pandas_udf(model, columns).alias("predictions"))
   • df.apply(model, columns).select("customer_id, predictions")

7. Question 7 · 1

   A nightly batch job is configured to ingest all data files from a cloud object storage container where records are stored in a nested directory structure YYYY/MM/DD. The data for each date represents all records that were processed by the source system on that date, noting that some records may be delayed as they await moderator approval. Each entry represents a user review of a product and has the following schema: user_id STRING, review_id BIGINT, product_id BIGINT, review_timestamp TIMESTAMP, review_text STRING The ingestion job is configured to append all data for the previous date to a target table reviews_raw with an identical schema to the source system. The next step in the pipeline is a batch write to propagate all new records inserted into reviews_raw to a table where data is fully deduplicated, validated, and enriched. Which solution minimizes the compute costs to propagate this batch of data?

   Choose one answer.

   • Perform a batch read on the reviews_raw table and perform an insert-only merge using the natural composite key user_id, review_id, product_id, review_timestamp.
   • Configure a Structured Streaming read against the reviews_raw table using the trigger once execution mode to process new records as a batch job.
   • Use Delta Lake version history to get the difference between the latest version of reviews_raw and one version prior, then write these records to the next table.
   • Filter all records in the reviews_raw table based on the review_timestamp; batch append those records produced in the last 48 hours.
   • Reprocess all records in reviews_raw and overwrite the next table in the pipeline.

8. Question 8 · 1

   Which statement describes Delta Lake optimized writes?

   Choose one answer.

   • Before a Jobs cluster terminates, OPTIMIZE is executed on all tables modified during the most recent job.
   • An asynchronous job runs after the write completes to detect if files could be further compacted; if yes, an OPTIMIZE job is executed toward a default of 1 GB.
   • Data is queued in a messaging bus instead of committing data directly to memory; all data is committed from the messaging bus in one batch once the job is complete.
   • Optimized writes use logical partitions instead of directory partitions; because partition boundaries are only represented in metadata, fewer small files are written.
   • A shuffle occurs prior to writing to try to group similar data together resulting in fewer files instead of each executor writing multiple files based on directory partitions.

9. Question 9 · 1

   Which statement describes the default execution mode for Databricks Auto Loader?

   Choose one answer.

   • Cloud vendor-specific queue storage and notification services are configured to track newly arriving files; the target table is materialized by directly querying all valid files in the source directory.
   • New files are identified by listing the input directory; the target table is materialized by directly querying all valid files in the source directory.
   • Webhooks trigger a Databricks job to run anytime new data arrives in a source directory; new data are automatically merged into target tables using rules inferred from the data.
   • New files are identified by listing the input directory; new files are incrementally and idempotently loaded into the target Delta Lake table.
   • Cloud vendor-specific queue storage and notification services are configured to track newly arriving files; new files are incrementally and idempotently loaded into the target Delta Lake table.

10. Question 10 · 1

    A Delta Lake table representing metadata about content posts from users has the following schema: user_id LONG, post_text STRING, post_id STRING, longitude FLOAT, latitude FLOAT, post_time TIMESTAMP, date DATE Based on the above schema, which column is a good candidate for partitioning the Delta Table?

    Choose one answer.

    • post_time
    • latitude
    • post_id
    • user_id
    • date

11. Question 11 · 1

    The data engineering team has configured a job to process customer requests to be forgotten (have their data deleted). All user data that needs to be deleted is stored in Delta Lake tables using default table settings. The team has decided to process all deletions from the previous week as a batch job at 1am each Sunday. The total duration of this job is less than one hour. Every Monday at 3am, a batch job executes a series of VACUUM commands on all Delta Lake tables throughout the organization. The compliance officer has recently learned about Delta Lake's time travel functionality. They are concerned that this might allow continued access to deleted data. Assuming all delete logic is correctly implemented, which statement correctly addresses this concern?

    Choose one answer.

    • Because the VACUUM command permanently deletes all files containing deleted records, deleted records may be accessible with time travel for around 24 hours.
    • Because the default data retention threshold is 24 hours, data files containing deleted records will be retained until the VACUUM job is run the following day.
    • Because Delta Lake time travel provides full access to the entire history of a table, deleted records can always be recreated by users with full admin privileges.
    • Because Delta Lake's delete statements have ACID guarantees, deleted records will be permanently purged from all storage systems as soon as a delete job completes.
    • Because the default data retention threshold is 7 days, data files containing deleted records will be retained until the VACUUM job is run 8 days later.

12. Question 12 · 1

    A large company seeks to implement a near real-time solution involving hundreds of pipelines with parallel updates of many tables with extremely high volume and high velocity data. Which of the following solutions would you implement to achieve this requirement?

    Choose one answer.

    • Use Databricks High Concurrency clusters, which leverage optimized cloud storage connections to maximize data throughput.
    • Partition ingestion tables by a small time duration to allow for many data files to be written in parallel.
    • Configure Databricks to save all data to attached SSD volumes instead of object storage, increasing file I/O significantly.
    • Isolate Delta Lake tables in their own storage containers to avoid API limits imposed by cloud vendors.
    • Store all tables in a single database to ensure that the Databricks Catalyst Metastore can load balance overall throughput.

13. Question 13 · 1

    Which describes a method of installing a Python package scoped at the notebook level to all nodes in the currently active cluster?

    Choose one answer.

    • Run source env/bin/activate in a notebook setup script
    • Use b in a notebook cell
    • Use %pip install in a notebook cell
    • Use %sh pip install in a notebook cell
    • Install libraries from PyPI using the cluster UI

14. Question 14 · 1

    A Delta Lake table in the Lakehouse named customer_churn_params is used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream sources. Currently, the data engineering team populates this table nightly by overwriting the table with the current valid values derived from upstream data sources. Immediately after each update succeeds, the data engineering team would like to determine the difference between the new version and the previous version of the table. Given the current implementation, which method can be used?

    Choose one answer.

    • Execute a query to calculate the difference between the new version and the previous version using Delta Lake’s built-in versioning and lime travel functionality.
    • Parse the Delta Lake transaction log to identify all newly written data files.
    • Parse the Spark event logs to identify those rows that were updated, inserted, or deleted.
    • Execute DESCRIBE HISTORY customer_churn_params to obtain the full operation metrics for the update, including a log of all records that have been added or modified.
    • Use Delta Lake’s change data feed to identify those records that have been updated, inserted, or deleted.

15. Question 15 · 1

    A data team’s Structured Streaming job is configured to calculate running aggregates for item sales to update a downstream marketing dashboard. The marketing team has introduced a new promotion, and they would like to add a new field to track the number of times this promotion code is used for each item. A junior data engineer suggests updating the existing query as follows. Note that proposed changes are in bold. Original query: Proposed query: Which step must also be completed to put the proposed query into production?

    Choose one answer.

    

    

    • Specify a new checkpointLocation
    • Remove .option('mergeSchema', 'true') from the streaming write
    • Increase the shuffle partitions to account for additional aggregates
    • Run REFRESH TABLE delta.‛/item_agg‛

Paid access to all 171 questions

Each purchase applies to this certification. Prices are one-time payments, not monthly subscriptions.