DP-750 Implementing Data Engineering Solutions Using Azure Databricks (beta) Questions and Answers

First dropdown: Year

Second dropdown: (2019, 2020, 2021)

The UNPIVOT operator converts the separate 2019, 2020, and 2021 columns into rows. Value becomes the output column containing the values previously stored in those year columns. Year becomes the output name column that identifies the original column from which each value came. Therefore, the expression must use Value FOR Year IN (2019, 2020, 2021). The Salesperson and Item columns are not included in the IN list because they remain identifier columns and are repeated for every resulting year row. A single source row consequently produces three output rows—one for each listed year. Selecting (Year) would reference an output name rather than the source columns that must be rotated.

The correct answers are C and D. Together they deliver cost-efficient autoscaling:

D (Enable autoscaling) allows the cluster to grow when workloads increase and shrink when they ease off. This satisfies both scale-up and scale-down requirements without manual intervention.

C (Auto-termination after 30 minutes of inactivity) ensures the cluster stops entirely when no work is running, eliminating the cost of an idle cluster. This is the cheapest possible state.

Option A (disable Photon) reduces compute acceleration — that ' s a performance regression with no meaningful cost benefit for autoscaling. Option B (compute policy that lets users manage settings) adds governance overhead and doesn ' t address scaling behaviour. Option E (fixed number of workers) is the opposite of autoscaling — a static worker count that either over-provisions during quiet periods or under-provisions during peaks.

Branch1 must first incorporate the current changes from main. During that merge, the conflicting sections of Notebook1 can be reviewed and resolved so that the resulting feature-branch version contains the required work from both branches. After committing the resolution, Branch1 can be merged into main normally. Cloning either branch into another Git folder creates another working copy but does not resolve the conflicting histories. Applying changes directly to main bypasses the controlled feature-branch workflow and risks omitting or overwriting one contributor’s work. Resolving the conflict on Branch1 also allows the combined notebook to be tested before updating the shared main branch. Therefore, merging main into Branch1, resolving the conflict, and completing the final merge is the correct workflow.

Two task-level settings solve this:

Task1 retry policy: configure Task1 with a maximum number of retries and a retry interval. When Task1 fails, Lakeflow Jobs automatically re-runs it up to the retry limit without any manual intervention. This handles transient infrastructure failures transparently.

Task2 run condition set to ' All succeeded ' with Task1 as its dependency: this means Task2 only starts when Task1 has succeeded. If Task1 fails and exhausts all retries, Task2 remains blocked — it never runs on data from a failed upstream ingestion. The dependency is declared in Task2 ' s ' Depends on ' setting in the job configuration.

These two settings are independent and composable. Task1 ' s retry policy gives it multiple chances to recover. Task2 ' s dependency and run condition ensure the downstream transformation only runs on clean, successfully ingested data.

The correct answer is B — No.

A Databricks Access Connector is an Azure resource that provides a managed identity for Databricks to authenticate to Azure Storage services. It is an authentication bridge — nothing more. Creating one doesn ' t register any external database, doesn ' t create any Unity Catalog objects representing DB1, and doesn ' t enable querying of SQL Server data from Databricks.

An Access Connector is a prerequisite for setting up storage credentials and external locations in Unity Catalog (to access ADLS Gen2, for example), but it has no role in Lakehouse Federation or foreign catalog creation. It cannot expose an external database ' s schemas and tables in Unity Catalog alongside native objects.

The correct solution for this scenario is a foreign catalog (Q47), which requires a connection object pointing to DB1 — not an Access Connector.

The solution is a Key Vault-backed secret scope combined with dbutils.secrets.get() in the notebook.

An Azure Key Vault-backed secret scope does not copy the secret value into Databricks. Every call to dbutils.secrets.get() fetches the live current value directly from Key Vault at runtime. This means when the cloud security team rotates storageKey in Key Vault, the notebook automatically picks up the new value on its next run — no Databricks configuration changes needed. That satisfies ' always retrieve the latest value. '

The principle of least privilege is upheld because only the secret scope has read access to Key Vault through a service principal or managed identity, and individual users never see the raw secret value — Databricks redacts it in logs automatically.

A Databricks-backed scope stores a copy of the secret inside Databricks, so it wouldn ' t automatically reflect Key Vault updates. Hardcoding or using environment variables would expose the secret in plain text.

The solution requires spark.readStream with format( ' cloudFiles ' ) for Auto Loader, paired with .writeStream using mergeSchema=true and a checkpointLocation.

Auto Loader ' s cloudFiles source incrementally processes new JSON files without rescanning the entire directory. The mergeSchema option handles schema drift — when sensors add new fields, the target Delta table schema expands automatically instead of throwing a parse error. This directly addresses Contoso ' s requirement to ' support schema drift. '

The checkpointLocation is what gives the pipeline its resilience. Databricks writes the stream ' s committed offset and schema state to that path. If the cluster restarts, the engine reads the checkpoint and picks up exactly where it left off — no events are reprocessed, satisfying ' exactly-once semantics ' and ' resume processing after failures without reprocessing the data. '

Without a checkpoint, the stream would restart from the beginning on every cluster bounce, which is precisely the problem Contoso is trying to eliminate.

The right ingestion tool depends on the source characteristics:

File-based telemetry and maintenance data → Auto Loader (cloudFiles). It monitors ADLS Gen2 for new arrivals, handles schema inference and evolution for the frequent schema drift, and requires minimal operational effort.

Real-time telemetry from Event Hubs → Spark Structured Streaming with the azure-eventhubs-spark connector. This provides exactly-once semantics and checkpoint-based recovery, satisfying ' resume processing after failures without reprocessing. '

Structured maintenance data from PostgreSQL → JDBC connector. Databricks supports direct JDBC reads from relational databases with pushdown predicates.

Daily CSV ERP extracts (50–100 GB) → COPY INTO or Auto Loader. Both support idempotent incremental batch loading into Delta tables with minimal code.

The correct answer is A. Photon is Azure Databricks ' native vectorized query engine, written in C++, designed to accelerate data ingestion and SQL-heavy workloads significantly over the standard Spark JVM path. Enabling it on a job compute cluster directly addresses Contoso ' s requirement for ' fast and consistent performance for BI workloads ' and ' production ingestion workloads that can scale automatically during telemetry spikes. '

Photon integrates transparently — no code changes are needed — and pairs well with autoscaling job clusters to handle the bursty 40,000-sensor telemetry load.

Option B contradicts the isolation requirement: Contoso explicitly needs production and development separated, not merged onto shared compute. Option C with a fixed large node gives peak capacity at all times, driving up costs even during quiet periods. Option D disabling autoscaling is the opposite of what ' s needed — telemetry spikes require elastic scaling, not a locked node count.

The correct answer is D. Breaking the pipeline into separate tasks for ingestion, cleansing, and curation is the foundation of well-designed Lakeflow Jobs pipelines. Each task should own one responsibility — when a task does too much, debugging a failure becomes a hunt through unrelated code, and retry logic becomes expensive because you re-execute work that already succeeded.

Contoso ' s planned changes explicitly call for ' a clear execution order and dependencies ' and ' orchestrate multi-step ingestion and transformation workflows. ' Separate tasks map directly to those goals: Lakeflow Jobs tracks each task ' s status independently, so if cleansing fails, ingestion doesn ' t re-run.

Option A bundles everything into one notebook, which means a curation bug forces a full re-ingestion. Option B copies logic three times — any future change must be applied in triplicate, which is a maintenance hazard. Option C forces everything through SQL MERGE, which is the wrong tool for raw-event ingestion and doesn ' t address cleansing or schema drift.

The correct mapping is SCD Type 1 for equipment metadata and SCD Type 2 for IoT sensor ownership history.

SCD Type 1 overwrites the existing record whenever an attribute changes — no history is kept. Contoso ' s requirement for equipment metadata (name, manufacturer, model, commissioning date) states ' historical values are NOT required, ' which is the textbook definition of Type 1. A MERGE INTO with WHEN MATCHED THEN UPDATE handles this cleanly in Delta Lake.

SCD Type 2 creates a new row for each change, preserving the full history through effective-date or version columns. Contoso requires that ' analysts must track the full history of ownership ' as sensors change hands over time — that full audit trail is only possible with Type 2. Type 3 (keeping just the previous value in an extra column) would lose earlier ownership records, so it doesn ' t satisfy the ' full history ' requirement.