Key ADF Features for Row-Level Fault Tolerance

 


  1. Fault Tolerance in Copy Activity

    • You can configure Skip incompatible rows or Redirect rows with errors into a separate log/sink (like Blob or SQL).

    • This allows you to capture failed records without breaking the entire batch.

  2. Mapping Data Flows – Error Row Handling

    • Data Flows allow “Error Row Handling”, so you can redirect specific problematic rows (e.g., invalid GUID, missing lookup) to a quarantine table.

    • Example:

      • Valid rows → Dataverse

      • Invalid rows → Error table with reason

  3. Retry Policies

    • Copy activity has built-in retry policies with exponential backoff for transient errors.

    • Helpful when Dataverse throttles or API timeouts happen.

  4. Logging & Monitoring

    • ADF can now output error details (like column name, error code, error description) to another sink.

    • This is row-level granularity for diagnostics.

🔹 So Why Use Logic Apps Then?

Even though ADF now supports row-level fault tolerance, Logic Apps still makes sense in scenarios like:

  • Complex branching/business rules (e.g., conditional routing, multiple Dataverse tables, related lookups).

  • Fine-grained API control (Dataverse throttling management with adaptive pacing).

  • Event-driven workflows (send Teams notifications, trigger Power Automate, call 3rd party APIs).

  • Per-record Dataverse actions beyond simple insert/update (e.g., trigger a plugin, send approval).

🔹 Updated View

  • ADF Only → Best when:

    • Large volumes of clean data.

    • You need speed + row-level error capture is “good enough.”

  • ADF + Logic Apps → Best when:

    • Complex transformations, lookups, business rules.

    • Need for API throttling control beyond ADF’s retry.

    • Integration with other Azure/M365 services.

👉 So, ADF has definitely closed the gap on error handling — meaning in many migration scenarios you no longer need Logic Apps just for row-level fault tolerance. But if your scenario goes beyond “data movement” into “workflow orchestration,” Logic Apps still adds value.

Do you want me to draw up a decision tree flow (ADF only vs ADF+Logic Apps) so you can quickly decide which pattern to use in a given project?

Comments

Post a Comment

Popular posts from this blog

🔍 Dataverse + Azure Integration: Choosing Between Synapse Link and Microsoft Fabric

  Managing large-scale data in Microsoft Dataverse ? You're likely choosing between Azure Synapse Link and the newer Microsoft Fabric Link . Both unlock powerful analytics and reporting options—but with different trade-offs in performance, setup, and long-term cost. Let’s break it down 👇 📊 Power BI and Dataverse: Great, But With Limits Power BI uses TDS (Tabular Data Stream) to connect directly to Dataverse tables. Real-time querying works well for smaller datasets (tens of thousands of records). However, complex queries and relationships may cause timeouts after 5 minutes , making it unreliable for enterprise-scale analytics. 🔗 Azure Synapse Link: Power + Complexity ✅ Strengths: Designed for large datasets , converting Dataverse tables to CSV format for efficient querying and reporting. Highly customizable with Synapse Studio , Azure Data Lake , and Data Factory for deeper ETL/ELT capabilities. ⚠️ Watchouts: Requires technical setup : Azur...
Read more

⚡ Example: Rate Limiting in Azure API Management

  Scenario: You have a public API and want to limit each user to 5 calls per minute to prevent abuse or accidental overload. 🛡️ What is Rate Limiting? Rate limiting controls how many requests a client can make to your API within a specific time window . It helps: Prevent API abuse Protect backend systems Fairly distribute API usage 💡 Using the rate-limit-by-key Policy Azure APIM uses a policy called rate-limit-by-key . It limits requests based on a unique identifier — usually the caller's subscription key or IP address. 🔧 Example Policy (XML) Here’s how you can add this to your inbound policy : < inbound > < base /> < rate-limit-by-key calls = "5" renewal-period = "60" counter-key = "@(context.Subscription?.Key)" /> </ inbound > Explanation: Attribute Meaning calls="5" Maximum 5 calls allowed renewal-period="60" Time window of 60 seconds (1 minute...
Read more

👤 Anonymous Role in Power Pages – What It Is and When to Use It

  When building public-facing websites with Power Pages , not every visitor will be a logged-in user. Some might just be browsing, some might want to ask a question, and others might be using your portal for the very first time. So how do we allow unregistered users to interact with our portal safely ? That’s where the Anonymous Users role comes in. 🚪 Who Are Anonymous Users? Anonymous users are visitors who haven’t logged in to your Power Pages portal. They: Have not signed in May not have an account Might be visiting for the first time And here’s the best part — Power Pages automatically assigns them the “Anonymous Users” web role. You don’t have to do anything special to give it. 🔐 What Can They Do? By default, anonymous users can see pages marked as "anonymous access allowed" , but they can't create or edit data unless you let them . To let anonymous users interact with your data (e.g., submit a form), you need to: Create table permissi...
Read more