Databricks-Certified-Data-Engineer-Associate Databricks Certified Data Engineer Associate Exam Questions and Answers

 Create a table using SQL | Databricks on AWS, Create a table using SQL - Azure Databricks, Delta Lake Quickstart - Azure Databricks

 Option E is the correct answer because it is the only mandatory requirement when creating a new Delta Live Tables pipeline. A pipeline is a data processing workflow that contains materialized views and streaming tables declared in Python or SQL source files. Delta Live Tables infers the dependencies between these tables and ensures updates occur in the correct order. To create a pipeline, you need to specify at least one notebook library to be executed, which contains the Delta Live Tables syntax. You can also specify multiple libraries of different languages within your pipeline. The other options are optional or not applicable for creating a pipeline. Option A is not required, but you can optionally provide a key-value pair configuration to customize the pipeline settings, such as the storage location, the target schema, the notifications, and the pipeline mode. Option B is not applicable, as the DBU/hour cost is determined by the cluster configuration, not the pipeline creation. Option C is not required, but you can optionally specify a storage location for the output data from the pipeline. If you leave it empty, the system uses a default location. Option D is not required, but you can optionally specify a location of a target database for the written data, either in the Hive metastore or the Unity Catalog.

 Tutorial: Run your first Delta Live Tables pipeline, What is Delta Live Tables?, Create a pipeline, Pipeline configuration.

A left anti-join returns rows from the left-side relation for which no matching row exists in the right-side relation. By placing customers on the left and orders on the right and joining by customer_id, the result contains precisely the customers who have no corresponding order. A left anti-join also returns only columns from the left relation, which satisfies the requirement to exclude all order-related fields. A left outer join could produce the desired result only with an additional NULL filter and correct table order. An inner join returns customers who have placed orders, which is the opposite requirement. A full outer join can introduce order-only rows and unnecessary columns. Therefore, the left anti-join described in option C is the most exact and efficient expression for this requirement.

================

Option B preserves the historical evidence needed to diagnose the regression. The Lakeflow Jobs UI provides run history, duration information, task details, links to the compute resource, and task metrics. Comparing several slow and normal runs shows whether the increase is consistent or intermittent. Cluster metrics such as CPU utilization, memory pressure, disk I/O, and network activity can then reveal resource spikes or bottlenecks that correlate with the longer runs. Automatically increasing cluster size before diagnosis could hide the cause and increase cost. Archiving old runs or resetting a baseline removes useful comparison data. Rerunning the job once does not make seven days of historical information irrelevant and cannot establish a trustworthy new baseline. Therefore, comparing run-history and cluster metrics, as stated in option B, is the appropriate troubleshooting action.

================

 An external table is a table that is defined in the metastore and points to an existing location in the storage system. When you drop an external table, only the metadata is deleted from the metastore, but the data files are not deleted from the storage system. This is because external tables are meant to be shared by multiple applications and users, and dropping them should not affect the data availability. On the other hand, a managed table is a table that is defined in the metastore and also managed by the metastore. When you drop a managed table, both the metadata and the data files are deleted from the metastore and the storage system, respectively. This is because managed tables are meant to be exclusive to the application or user that created them, and dropping them should free up the storage space. Therefore, the correct answer is C, because the table was external and only the metadata was deleted when the table was dropped. References: Databricks Documentation - Managed and External Tables, Databricks Documentation - Drop Table

This question’s options do not include the most precise Unity Catalog answer, which would normally be the MANAGE privilege on the schema, because Databricks documents that MANAGE allows a principal to grant and revoke privileges on an object without automatically granting all data-access privileges on that object. However, since MANAGE is not one of the answer choices, the best available answer is D, Schema Owner . Databricks states that schema owners can grant and revoke permissions on the schema, and schema-level administration is a documented responsibility of the schema owner. Option A is incorrect because USE CATALOG and USE SCHEMA only allow object access/navigation and do not confer privilege-management rights. Option B is too broad because a catalog owner governs the entire catalog scope, not just one schema. Option C is too narrow because a table owner controls only one table, not various objects across a schema. So, among the choices provided, Schema Owner is the closest Databricks-aligned answer, though the cleanest real-world implementation would be MANAGE on the schema .

=========

The --target prod option explicitly selects the prod target from the bundle configuration. Databricks then resolves ${var.dataset_catalog} using the target-level value catalog_prod, which takes precedence over the top-level default of catalog_dev. This allows the same dashboard resource definition to be promoted across environments without editing the resource itself. The --var option can override an individual variable, but it does not automatically select the production target. The --profile option selects an authentication profile from .databrickscfg; it does not select a bundle target or automatically apply the target’s variables. bundle execute is not the deployment command. Consequently, option D correctly combines production-target selection with the environment-specific variable override already defined in the YAML configuration.

================

With schema evolution enabled, Auto Loader detects source fields that are not present in the current schema and evolves the destination schema to include them. The new userAgent field is therefore added to the Delta table as a column, and values from newly processed records are written to it. Existing rows cannot be retroactively populated because the earlier source records did not contain that field, so they receive NULL for the newly added column. The column is consequently nullable unless a later transformation explicitly enforces another constraint. Option A describes behavior without suitable schema evolution. Option B incorrectly claims that historical rows receive a default value and that the column becomes non-nullable. Option D contradicts schema-evolution behavior. Therefore, option C correctly describes how the new JSON field is handled.

Option B reads the relational source through JDBC and writes the resulting Spark DataFrame directly as a registered Unity Catalog table using its three-level name: catalog, schema, and table. Once written with saveAsTable, the data is immediately discoverable and queryable through Databricks SQL, subject to Unity Catalog permissions. Because the read and write operations occur through supported Databricks workloads, Unity Catalog can capture lineage between the source operation, notebook or job, and destination table. Writing files only to DBFS or an external path does not by itself create the required governed table. Introducing a separate staging and loading job adds unnecessary processing and operational complexity. A Pandas extraction also reduces scalability and does not satisfy the direct managed-table requirement. Therefore, option B is correct.

================

The metrics indicate data skew because one task is taking substantially longer than most tasks in the stage. Databricks advises checking the maximum task duration against the 75th-percentile duration: when the maximum is at least 50% higher, the stage may be skewed. Here, the 75th percentile is 50 seconds and the maximum is 80 seconds, so the maximum is 60% higher. The 25th percentile, median, and 75th percentile are tightly grouped between 40 and 50 seconds, while the maximum is a clear outlier. That pattern means a small number of tasks are processing disproportionately expensive partitions. These values alone do not prove cluster under-provisioning or excessive garbage collection; those diagnoses require resource, executor, spill, and GC metrics. Therefore, option A is correct.

================

Deploying a bundle creates or updates the resources defined in its configuration, but deployment does not automatically execute those resources. The databricks bundle run command starts a deployed job or pipeline by using its resource key from the bundle configuration. Because the job is keyed as etl_job, the correct command is databricks bundle run etl_job. The command bundle deploy run is not valid CLI syntax, and bundle execute is not the command used to run a bundle resource. databricks jobs submit creates and runs a one-time job from a supplied JSON definition rather than executing the job already managed by the bundle. Therefore, option D correctly triggers the deployed job while preserving the bundle-managed resource configuration.

================

Databricks Repos is a visual Git client and API in Databricks that supports common Git operations such as commit, pull, push, branch management, and visual comparison of diffs when committing1. However, merge is not supported in the Git dialog2. You need to use the Repos UI or your Git provider to merge branches3. Merge is a way to combine the commit history from one branch into another branch1. During a merge, a merge conflict is encountered when Git cannot automatically combine code from one branch into another. Merge conflicts require manual resolution before a merge can be completed1. References: 4: Run Git operations on Databricks Repos4, 1: CI/CD techniques with Git and Databricks Repos1, 3: Collaborate in Repos3, 2: Databricks Repos - What it is and how we can use it2.

Databricks Repos is a visual Git client and API in Databricks that supports common Git operations such as commit, pull, push, merge, and branch management. However, to clone a remote Git repository to a Databricks repo, you must use the Databricks UI or API. You cannot clone a Git repo using the CLI through a cluster’s web terminal, as the files won’t display in the Databricks UI1. References: 1: Run Git operations on Databricks Repos | Databricks on AWS2

Delta Sharing is an open protocol developed by Databricks that enables secure, real-time sharing of data across organizations without requiring recipients to have a Databricks account. It is specifically designed for scenarios where external partners need read-only access to data stored in Delta format. With Delta Sharing, data providers create a share and grant access to recipients via a secure mechanism (such as a profile file or URL-based access), ensuring that the data cannot be modified by the consumer. This approach eliminates the need for data duplication or manual file transfers, maintaining data freshness and governance. Option B is incorrect because Unity Catalog governs internal access within an organization, not external sharing without accounts. Option C is inefficient and breaks data lineage and freshness. Option D violates security best practices by granting excessive permissions. Delta Sharing ensures fine- grained access control, auditability, and secure external collaboration , making it the correct solution for this requirement.

=========

https://spark.apache.org/docs/latest/api/python/reference/pyspark.ss/api/pyspark.sql.streaming.DataStreamWriter.trigger.html

The command databricks bundle deploy deploys the bundle’s local files, artifacts, and resource definitions to the configured remote Databricks workspace. Databricks documents this command as the deployment operation for Declarative Automation Bundles, and a specific target can be selected with the -t or --target option. The validate command checks whether the bundle configuration is valid but does not create or update workspace resources. The run command executes a resource that has already been deployed; it is not the deployment step. The init command creates a new bundle project from a template, while bundle deployment is not a valid command combination. Because the question asks for the single command that sends local bundle assets to the target workspace, option C is correct.

================

A Gold table is a table that contains highly refined and aggregated data that powers analytics, machine learning, and production applications. It represents data that has been transformed into knowledge, rather than just information. A Gold table is typically the final output of a medallion lakehouse architecture, where data flows from Bronze to Silver to Gold tables, with each layer improving the structure and quality of data. A job that queries aggregated data designed to feed into a dashboard is an example of a data workload that will utilize a Gold table as its source, as it requires data that is ready for consumption and analysis. The other options are either data workloads that will use a Bronze or Silver table as their source, or data workloads that will produce a Gold table as their output. References: Databricks Documentation - What is the medallion lakehouse architecture?, Databricks Documentation - What is a Medallion Architecture?, K21Academy - Delta Lake Architecture & Azure Databricks Workspace.

A data engineer can create a multi-task job in Databricks that consists of multiple tasks that run in a specific order. Each task can have one or more dependencies, which are other tasks that must run before the current task. The Depends On field of a new Databricks Job Task allows the data engineer to specify the dependencies of the task. The data engineer should select a task in the Depends On field when they want the new task to run only after the selected task has successfully completed. This can help the data engineer to create a logical sequence of tasks that depend on each other’s outputs or results. For example, a data engineer can create a multi-task job that consists of the following tasks:

Task A: Ingest data from a source using Auto Loader

Task B: Transform the data using Spark SQL

Task C: Write the data to a Delta Lake table

Task D: Analyze the data using Spark ML

Task E: Visualize the data using Databricks SQL

In this case, the data engineer can set the dependencies of each task as follows:

Task A: No dependencies

Task B: Depends on Task A

Task C: Depends on Task B

Task D: Depends on Task C

Task E: Depends on Task D

This way, the data engineer can ensure that each task runs only after the previous task has successfully completed, and the data flows smoothly from ingestion to visualization.

The other options are incorrect because they do not describe valid scenarios for selecting a task in the Depends On field. The Depends On field does not affect the following aspects of a task:

Whether the task needs to be replaced by another task

Whether the task needs to fail before another task begins

Whether the task has the same dependency libraries as another task

Whether the task needs to use as little compute resources as possible References: Create a multi-task job, Run tasks conditionally in a Databricks job, Databricks Jobs.

 Databricks SQL allows users to schedule queries to run automatically at a specified frequency and time zone. This can help users to keep their dashboards or alerts updated with the latest data. To schedule a query, users need to do the following steps:

In the Query Editor, click Schedule > Add schedule to open a menu with schedule settings.

Choose when to run the query. Use the dropdown pickers to specify the frequency, period, starting time, and time zone. Optionally, select the Show cron syntax checkbox to edit the schedule in Quartz Cron Syntax.

Choose More options to show optional settings. Users can also choose a name for the schedule, and a SQL warehouse to power the query.

Click Create. The query will run automatically according to the schedule.

The other options are incorrect because they do not refer to the correct location or frequency to schedule the query. The query’s page in Databricks SQL is the place where users can edit, run, or schedule the query. The SQL endpoint’s page in Databricks SQL is the place where users can manage the SQL warehouses and SQL endpoints. The Jobs UI is the place where users can create, run, or schedule jobs that execute notebooks, JARs, or Python scripts. References: Schedule a query, What are Databricks SQL alerts?, Jobs.

dbfs:/user/hive/warehouse Thereby showing " dbfs:/user/hive/warehouse/customer360.db

 The location of the customer360 database depends on the value of the spark.sql.warehouse.dir configuration property, which specifies the default location for managed databases and tables. If the property is not set, the default value is dbfs:/user/hive/warehouse. Therefore, the customer360 database will be located in dbfs:/user/hive/warehouse/customer360.db. However, if the property is set to a different value, such as dbfs:/user/hive/database, then the customer360 database will be located in dbfs:/user/hive/database/customer360.db. Thus, more information is needed to determine the correct response.

Option A is not correct, as dbfs:/user/hive/database/customer360 is not the default location for managed databases and tables, unless the spark.sql.warehouse.dir property is explicitly set to dbfs:/user/hive/database.

Option B is not correct, as dbfs:/user/hive/warehouse is the default location for the root directory of managed databases and tables, not for a specific database. The database name should be appended with .db to the directory path, such as dbfs:/user/hive/warehouse/customer360.db.

Option C is not correct, as dbfs:/user/hive/customer360 is not a valid location for a managed database, as it does not follow the directory structure specified by the spark.sql.warehouse.dir property.

Databases and Tables

[Databricks Data Engineer Professional Exam Guide]

Unity Catalog attribute-based access control policies provide centralized, tag-driven governance. Administrators define policies at a catalog or schema scope and use governed tags to identify the tables or columns to which the policies apply. When matching tags are assigned to current or future objects, the applicable policy is evaluated automatically, avoiding repetitive manual attachment of filters and masks to individual tables. This approach improves scalability and consistency when many objects require the same protection rules. It is important that the policy applies to objects matching its tag conditions, rather than indiscriminately to every object. The other choices incorrectly describe supported object types, multiple-filter behavior, or implementation languages. Databricks specifically positions attribute-based access control as the centralized approach for consistently applying row filtering and column masking across many governed tables. Therefore, option A is correct.

================

To read from a stream source, the data engineer needs to use the spark.readStream method instead of the spark.read method. The spark.readStream method returns a DataStreamReader object that can be used to specify the details of the input source, such as the format, the schema, the path, and the options. The spark.read method is only suitable for batch processing, not streaming processing. The other changes are not necessary or correct for reading from a stream source. References: Structured Streaming Programming Guide, Read a stream, Databricks Data Sources

To grant full privileges on a table such as ' sales ' to a group like ' team ' , the correct SQL command in Databricks is:

GRANT ALL PRIVILEGES ON TABLE sales TO team;

This command assigns all available privileges, including SELECT, INSERT, UPDATE, DELETE, and any other data manipulation or definition actions, to the specified team. This is typically necessary when a team needs full control over a table to manage and manipulate it as part of a project or ongoing maintenance.

https://www.databricks.com/blog/2021/10/20/introducing-sql-user-defined-functions.html

 To grant full permissions on a database to a user, group, or service principal, the GRANT ALL PRIVILEGES ON DATABASE command can be used. This command grants all the applicable privileges on the database, such as CREATE, SELECT, MODIFY, and USAGE. The other options are either incorrect or incomplete, as they do not grant all the privileges or specify the wrong database or principal. References:

GRANT

Privileges

Option C extracts each JSON field into a separate Silver-table column and casts age to the appropriate integer type. get_json_object accepts a JSON string and a JSONPath expression rooted at $; paths such as $.user_id and $.age therefore identify the required top-level fields correctly. This produces analytics-ready columns and prevents downstream queries from repeatedly parsing the original JSON string. Option A returns one nested struct and incorrectly defines age as STRING, so further projection or casting would still be required. Option B attempts field notation directly on a STRING column, which is not valid until the JSON has been parsed into a structured type. Option D omits the required $ JSONPath root and leaves age uncast. Therefore, option C most completely standardizes the source data for the Silver layer.

================

Databricks Asset Bundles provide a modern, declarative approach to defining, testing, and deploying Databricks workflows as version-controlled code artifacts . Using YAML configuration files, engineers can describe jobs, pipelines, dependencies, and environments in a structured and reproducible way. Asset Bundles integrate well with CI/CD pipelines and external schedulers, enabling teams to package workflows, run integration tests, and deploy consistently across environments such as development, staging, and production. This aligns directly with the requirement for version control and external orchestration. The Databricks CLI (A) is primarily used as a deployment interface but does not provide the structured packaging and workflow definition capabilities of Asset Bundles. Databricks Connect (C) is intended for local development and testing of Spark code, not workflow deployment. The Databricks SDK (D) provides programmatic access to APIs but lacks the declarative, packaged deployment model required here. Databricks documentation positions Asset Bundles as the recommended solution for production-grade workflow deployment and testing within modern data engineering practices.

Workspace-specific deployment settings should be defined under each environment in the targets section of databricks.yml. A bundle target represents a deployment environment such as development, staging, or production. Each target can contain its own workspace mapping, including the workspace host and an optional root_path. This structure allows one bundle codebase to be deployed predictably to different environments without embedding infrastructure details in notebooks or pipeline source code. A README file is documentation and is not interpreted as bundle configuration. Hardcoding workspace details in Python couples transformation logic to a particular environment. Global local environment variables are machine-specific and do not provide a complete, version-controlled bundle definition. Therefore, option C correctly places the workspace-specific settings in the declarative configuration used during validation and deployment.

================

For large-scale batch ETL workloads , Databricks documentation recommends using job clusters that are created specifically for a job run and terminated automatically when the job completes. A job cluster with autoscaling enabled provides the best balance of performance, reliability, and cost efficiency for long-running nightly pipelines. Autoscaling allows the cluster to dynamically add or remove worker nodes based on the workload demands, ensuring sufficient parallelism to process very large JSON datasets efficiently while avoiding overprovisioning when demand decreases. This is especially important when processing raw data into Delta tables, where shuffle-heavy transformations and writes benefit from multiple workers. Single-node clusters (option B) are not suitable for very large volumes of data and risk excessive runtimes or job failures. High-concurrency clusters (option C) are optimized for interactive and concurrent SQL queries, not long-running batch ETL jobs. Always-on all-purpose clusters (option D) increase costs unnecessarily and are intended for interactive development, not scheduled production pipelines. Databricks best practices clearly position autoscaling job clusters as the preferred solution for reliable, cost-effective batch ETL in production on Databricks.

Databricks separates workspace-management services from data processing. Notebook source files stored as workspace objects are managed in the Databricks control plane and are encrypted at rest. When the scheduled job starts, Databricks orchestrates the run from the control plane, but the notebook commands execute on compute resources in the compute plane. The compute plane is where Spark transformations, Python code, and other data-processing operations run. Unity Catalog governs data and AI assets but is not the execution environment or the general storage location for workspace notebook source. Delta Lake is a table-storage layer and does not execute notebook code. Option D is also incorrect because Databricks does not describe workspace notebook source as unencrypted. Therefore, option C accurately represents the separation between notebook storage and workload execution.

Databricks Asset Bundles are a Databricks CLI feature that lets teams define Databricks projects “as code” using YAML configuration files . In a bundle, metadata is expressed in YAML (for example, a required databricks.yml at the root), and the configuration can declare resources (such as Databricks Jobs/workflows and other supported assets), artifacts , and deployment targets . After defining these YAML manifests, teams use the Databricks CLI to validate, deploy, and run the bundle into a Databricks workspace, which matches the scenario described (YAML + CLI deployment). This approach standardizes how workflows and related resources are packaged and promoted across environments (dev/test/prod) while keeping definitions version-controlled and reproducible. GitHub is only a hosting/version-control platform (not the YAML deployment feature itself), Terraform is a separate IaC tool, and “DataOps” is a methodology rather than a specific Databricks feature. ( Databricks Documentation )

Option A enables predictive optimization, which is designed to eliminate manual maintenance scheduling for Unity Catalog managed tables. Databricks evaluates table activity and automatically runs operations such as OPTIMIZE, VACUUM, and ANALYZE when beneficial, using serverless jobs compute. Predictive optimization can be enabled at multiple hierarchy levels, allowing tables to inherit the applicable setting. Auto Loader schema evolution handles changes in incoming file schemas but does not schedule table optimization. Liquid clustering improves data organization and query pruning, but it does not independently provide automated stale-file removal. Optimized writes and auto compaction reduce the creation of small files during write operations, but they do not replace adaptive VACUUM scheduling and the broader maintenance capabilities of predictive optimization. Therefore, option A directly satisfies the requirement for platform-managed table maintenance.

================

In Unity Catalog, managed tables and external tables have different lifecycle behavior. When you drop a managed table , Databricks removes the table metadata and also schedules deletion of the underlying managed data files according to the managed-table retention behavior. By contrast, when you drop an external table , Unity Catalog removes only the table metadata and does not delete the underlying files in cloud storage. This distinction exists specifically so that external data can remain available to systems outside Databricks. In the scenario described, ephemeral staging outputs that should disappear completely are best represented by managed tables, while tables backed by shared cloud storage should be external tables so that dropping them does not remove files used by downstream jobs. Therefore, option C is the correct answer. Options A, B, and D contradict documented Unity Catalog behavior because external table files are not automatically deleted, while managed table data is lifecycle-managed by Databricks.

=========

Option D is the correct answer because it enables the Serverless feature for the SQL endpoint, which allows the endpoint to automatically scale up and down based on the query load. This way, the endpoint can handle more concurrent queries and reduce the time it takes to return results. The Serverless feature also reduces the cold start time of the endpoint, which is the time it takes to start the cluster when a query is submitted to a non-running endpoint. The Serverless feature is available for both AWS and Azure Databricks platforms.

 Databricks SQL Serverless, Serverless SQL endpoints, New Performance Improvements in Databricks SQL

Pipeline A has a driver-side failure because collect() transfers every row to the driver process. Databricks states that collect() should be used only when the result is expected to be small. For a 100 GB dataset, the preferred correction is to remove the full collection and use distributed processing, aggregation, sampling, or limit; increasing driver capacity alone is not a durable solution. Pipeline B is an executor-side problem during a shuffle-intensive join. Increasing spark.sql.shuffle.partitions can divide the shuffled data into smaller task partitions, while workers with more memory per core can provide additional executor memory. Increasing the number of executors does not solve Pipeline A’s driver bottleneck, and increasing driver memory does not address Pipeline B’s executor failures. Therefore, option D correctly distinguishes the affected Spark components and appropriate remedies.

In Databricks, when creating a table, you can add a comment to columns or the entire table to provide more information about the data it contains. In this case, since it’s organization policy to indicate that the new table includes personally identifiable information (PII), option D is correct. The line of code would be added after defining the table structure and before closing with a semicolon. References: Data Engineer Associate Exam Guide, CREATE TABLE USING (Databricks SQL)

In Databricks Asset Bundles , the databricks.yml file located at the repository root serves as the single authoritative configuration file for the entire project. This file defines key elements such as the bundle name, deployment targets (for example, dev, test, prod), workspace configurations, permissions, and resource definitions including jobs and pipelines. Databricks documentation specifies that there must be one primary databricks.yml file at the root , ensuring consistency and simplifying deployment processes. However, to support modularity and maintainability, this root configuration file can reference additional YAML files using the include mapping. This allows teams to organize configurations across directories while still maintaining a single source of truth. Option A is incorrect because multiple root-level configuration files are not supported as independent authorities. Option C is invalid since configuration files are not restricted to hidden directories. Option D introduces ambiguity and contradicts the requirement for a single authoritative configuration. Therefore, using one root-level configuration file with optional includes is the correct and recommended approach.

Option D matches the heterogeneous sources and governance requirements. A relational database, object storage, and Kafka require source-appropriate ingestion configurations rather than a single connector attempting to handle every source. Database ingestion can use incremental or change-data-capture processing, Amazon S3 files can use file-oriented ingestion, and Kafka requires a streaming connector. Publishing the outputs into Unity Catalog schemas provides centralized permissions, discovery, and governance boundaries. Supported Databricks ingestion workloads also record lineage between source and destination objects in Unity Catalog. Delta table history and time travel provide access to earlier table versions when the required transaction-log and data-file retention settings are maintained. Disabling version history or repeatedly performing complete reloads would weaken auditability and increase processing overhead. Therefore, the separate, incrementally processed, Unity Catalog-governed design in option D is correct.

================

The File Arrival trigger and Auto Loader perform separate, complementary functions. The trigger monitors the configured storage location and starts the Lakeflow Job after detecting new files. After the notebook starts, Auto Loader examines its own checkpoint and file-tracking state to determine which files have not yet been committed successfully. This separation provides incremental processing and exactly-once ingestion behavior. With availableNow=True, Auto Loader processes all data available when the query starts and then stops, allowing the job compute to terminate and control costs. File Notification mode is not required solely because a File Arrival trigger is used. Disabling checkpoints would remove progress tracking and could cause duplicate processing. The trigger also does not eliminate the need to configure the Auto Loader source path. Therefore, option A is correct.

================

Option A uses the supported ALTER TABLE ... SET MANAGED conversion for a Unity Catalog external Delta table. Databricks documents that this conversion retains the table name, permissions, dependent objects, settings, and table history while minimizing downtime for readers and writers. After conversion, Databricks manages the table’s storage lifecycle, satisfying the requirement for platform-controlled cleanup when the managed table is dropped. Dropping the original table with PURGE destroys the object instead of migrating it. SET LOCATION changes the storage path referenced by an external table but does not convert the table into a managed table. Creating a replacement table with CREATE TABLE AS SELECT introduces a new object and requires manual migration of permissions, dependencies, and workloads. Therefore, ALTER TABLE catalog.schema.sales SET MANAGED is the correct action.

================

 A data lakehouse is a data management architecture that combines the flexibility, cost-efficiency, and scale of data lakes with the data management and ACID transactions of data warehouses, enabling business intelligence (BI) and machine learning (ML) on all data12. By using a data lakehouse, both the data analysis and data engineering teams can access the same data sources and formats, ensuring data consistency and quality across their reports. A data lakehouse also supports schema enforcement and evolution, data validation, and time travel to old table versions, which can help resolve data conflicts and errors1. References: 1: What is a Data Lakehouse? - Databricks 2: What is a data lakehouse? | IBM

One of the features of DLT is that it provides data quality metrics for each dataset in the pipeline, such as the number of records that pass or fail expectations, the number of records that are dropped, and the number of records that are written to the target. These metrics can be accessed from the DLT pipeline page, where the data engineer can click on each table and view the data quality statistics for the latest update or any previous update. This way, they can identify which table is dropping the records and why. References:

Monitor Delta Live Tables pipelines

Manage data quality with Delta Live Tables

Unity Catalog organizes data objects through a three-level namespace consisting of catalog, schema, and object. To query sales_catalog.market_data.summary, the analyst requires USE CATALOG on sales_catalog, USE SCHEMA on market_data, and SELECT on the summary view. Granting SELECT on the view alone is insufficient when either parent usage privilege is missing. The analyst does not normally require access to the underlying tables queried by the view because the view owner’s privileges are used to resolve those objects. A workspace or cluster is not part of the Unity Catalog securable hierarchy for resolving the view name. Storage credentials and external locations govern cloud-storage access rather than ordinary view-query permissions. Therefore, the engineer should verify privileges in the order catalog, schema, and view, making option B correct.

================

An If/else condition task is specifically designed to branch a Lakeflow Job based on a Boolean expression. The condition can reference the upstream value as {{tasks.check_volume.values.record_count}} and compare it with 1000000. The true branch can invoke high_volume_pipeline, while the false branch invokes normal_pipeline. Run if conditions evaluate task outcomes such as success or failure; they do not compare a calculated record count. A For each task iterates over an input array and is not appropriate for selecting one of two execution paths. Retries are intended to recover from unsuccessful task attempts, not to wait for a business metric to cross a threshold. Therefore, option A implements the required value-based branching. This question was retained after comparison with the existing Word question bank.

================

Databricks intentionally stops existing streaming reads and writes after an external Delta table is converted to a Unity Catalog managed table. This protects data consistency because the table’s managed storage location and access path have changed. The documented recovery action is to restart the stream with the same configuration. The existing checkpoint can then resume processing from the last committed offset, and supported path-based access is redirected to the converted managed table. REFRESH TABLE invalidates cached metadata but does not repair a stopped Structured Streaming query or modify its checkpoint. Deleting the checkpoint would discard progress information and could cause unnecessary reprocessing or duplicate output. Additional permissions are not the indicated solution when the failure is specifically caused by managed-table conversion. Therefore, option A is correct.

================

 The keyword that can be used to compact the small files associated with a Delta table is OPTIMIZE. The OPTIMIZE command performs file compaction on a Delta table by rewriting a set of small files into a set of larger files1. This can improve the performance of queries that scan the table by reducing the number of files that need to be read and the amount of metadata that needs to be processed1. The OPTIMIZE command can also optionally sort the data within each file by a given set of columns, which can further improve the query performance by enabling data skipping and predicate pushdown1. The OPTIMIZE command can be applied to the whole table or to a specific partition of the table1.

The other keywords are not suitable for compacting the small files associated with a Delta table. REDUCE is a keyword used in the SQL syntax for aggregating data using a user-defined function2. COMPACTION is not a valid keyword in SQL or Python. REPARTITION is a keyword used in the Python syntax for changing the number of partitions of a DataFrame or an RDD3. VACUUM is a keyword used to remove files that are no longer referenced by a Delta table and are older than a retention threshold4.

1: OPTIMIZE | Databricks on AWS

2: REDUCE | Databricks on AWS

3: repartition | Databricks on AWS

4: VACUUM | Databricks on AWS

The query performing a streaming hop from raw data to a Bronze table is identified by using the Spark streaming read capability and then writing to a Bronze table. Let ' s analyze the options:

Option A: Utilizes .writeStream but performs a complete aggregation which is more characteristic of a roll-up into a summarized table rather than a hop into a Bronze table.

Option B: Also uses .writeStream but calculates an average, which again does not typically represent the raw to Bronze transformation, which usually involves minimal transformations.

Option C: This uses a basic .write with .mode( " append " ) which is not a streaming operation, and hence not suitable for real-time streaming data transformation to a Bronze table.

Option D: It employs spark.readStream.load() to ingest raw data as a stream and then writes it out with .writeStream, which is a typical pattern for streaming data into a Bronze table where raw data is captured in real-time and minimal transformation is applied. This approach aligns with the concept of a Bronze table in a modern data architecture, where raw data is ingested continuously and stored in a more accessible format.

A table is a data entity that is stored in a physical location and can be accessed by other data engineers in other sessions. A table can be created from one or more tables using the CREATE TABLE or CREATE TABLE AS SELECT commands. A table can also be registered from an existing DataFrame using the spark.catalog.createTable method. A table can be queried using SQL or DataFrame APIs. A table can also be updated, deleted, or appended using the MERGE INTO command or the DeltaTable API. References:

Create a table

Create a table from a query result

Register a table from a DataFrame

[Query a table]

[Update, delete, or merge into a table]

In Delta Live Tables (DLT), when configured to run in Continuous Pipeline Mode, particularly in a production environment, the system is designed to continuously process and update data as it becomes available. This mode keeps the compute resources active to handle ongoing data processing and automatically updates all datasets defined in the pipeline at predefined intervals. Once the pipeline is manually stopped, the compute resources are terminated to conserve resources and reduce costs. This mode is suitable for production environments where datasets need to be kept up-to-date with the latest data.

 In a medallion architecture, a common data design pattern for lakehouses, data flows from Bronze to Silver to Gold layer tables, with each layer progressively improving the structure and quality of data. Bronze tables store raw data ingested from various sources, while Silver tables apply minimal transformations and cleansing to create an enterprise view of the data. Silver tables can also join and enrich data from different Bronze tables to provide a more complete and consistent view of the data. Therefore, option D is the correct answer, as Silver tables contain a more refined and cleaner view of data than Bronze tables. Option A is incorrect, as it is the opposite of the correct answer. Option B is incorrect, as Silver tables do not necessarily contain aggregates, but can also store detailed records. Option C is incorrect, as Silver tables may contain less data than Bronze tables, depending on the transformations and cleansing applied. Option E is incorrect, as Silver tables may contain more data than Bronze tables, depending on the joins and enrichments applied. References: What is a Medallion Architecture?, Transforming Bronze Tables in Silver Tables, What is the medallion lakehouse architecture?

A temporary view is a relational object that is defined in the metastore and points to an existing DataFrame. It does not copy or store any physical data, but only saves the query that defines the view. The lifetime of a temporary view is tied to the SparkSession that was used to create it, so it does not persist across different sessions or applications. A temporary view is useful for accessing the same data multiple times within the same notebook or session, without incurring additional storage costs. The other options are either materialized (A, E), persistent (B, C), or not relational objects ©. References: Databricks Documentation - Temporary View, Databricks Community - How do temp views actually work?, Databricks Community - What’s the difference between a Global view and a Temp view?, Big Data Programmers - Temporary View in Databricks.

 Data Explorer is a graphical interface that allows you to browse, create, and manage data objects such as databases, tables, and views in your workspace. You can also review and modify the permissions on these data objects using Data Explorer. To access Data Explorer, you can click on the Data icon in the sidebar, or use the %sql magic command in a notebook. You can then select a database and a table, and click on the Permissions tab to view and edit the access control lists (ACLs) for the table. You can also use SQL commands such as SHOW GRANT and GRANT to query and modify the permissions on a Delta table. References:

Data Explorer

Access control for Delta tables

SHOW GRANT

[GRANT]