{"id":539,"date":"2026-10-16T12:06:45","date_gmt":"2026-10-16T10:06:45","guid":{"rendered":"https:\/\/dataguide.tech\/?p=539"},"modified":"2026-11-12T23:17:43","modified_gmt":"2026-11-12T22:17:43","slug":"agentic-data-engineering-in-microsoft-fabric-with-claude-code","status":"publish","type":"post","link":"https:\/\/dataguide.tech\/index.php\/2026\/10\/16\/agentic-data-engineering-in-microsoft-fabric-with-claude-code\/","title":{"rendered":"Agentic Data Engineering in Microsoft Fabric with Claude Code"},"content":{"rendered":"\n
\n

Status check, verified October 2026.<\/strong> The Fabric Core MCP Server is generally available since the September 2026 Fabric update, but the pieces around it move fast. The Fabric skills are open source and get new versions often, the Fabric CLI ships regularly, and the VS Code extension changes its UI from time to time. If something here doesn’t behave the way I describe, check the Microsoft Learn docs<\/a> before assuming your setup is broken. I date-stamp every post in this series and update them when things move.<\/p>\n<\/blockquote>\n\n\n\n

Most of a data engineer’s day in Fabric goes into notebooks and pipelines. And most of the AI help I see people using for that work stops at a chat window: copy the code out, ask a question, paste the answer back into the notebook, run it, repeat. It works. Kind of. But it’s a long way from an agent that can open your notebook, change it, run it and tell you what broke.<\/p>\n\n\n\n

In this post I’ll go through the full Data Engineering flow in Microsoft Fabric the way I actually work: notebooks, Fabric skills, pipelines, Dataflow Gen2, and committing all of it to Git at the end.<\/p>\n\n\n\n

Two prerequisites. The tool setup from Agentic Development in Microsoft Fabric, Part 1: Setup<\/a>, and the workspace we built in the infrastructure post, One Prompt, One Workspace: Agentic Microsoft Fabric Admin<\/a>. If you haven’t done them yet, go there first.<\/p>\n\n\n\n

The core of my setup is Visual Studio Code with the Fabric Data Engineering extension. It connects you to your Fabric notebooks, sure. More importantly, it lets your LLM (in my case Claude Code) see and edit those notebooks, and run and test them before the code goes back to the workspace.<\/p>\n\n\n\n

Which tool does what<\/h2>\n\n\n\n

Same as in the infrastructure post, the agent switches between tools depending on the job. This time there’s a fourth one in play: the VS Code extension. Quick map, so the rest of the post makes sense:<\/p>\n\n\n\n

Work<\/th>Tool<\/th>Why this one<\/th><\/tr><\/thead>
Edit, run and publish notebooks<\/td>Fabric Data Engineering VS Code extension<\/td>Puts the notebook on your disk as a real file, so any LLM can edit it. It has its own sign-in, separate from the MCP and both CLIs.<\/td><\/tr>
Notebook code and runs via the spark-authoring-cli<\/code> skill<\/td>Azure CLI (az rest<\/code>)<\/td>The skill calls the Fabric REST API directly, so it works for every endpoint, MCP or not.<\/td><\/tr>
Warehouse \/ SQL DB via sqldw-authoring-cli<\/code><\/td>sqlcmd<\/code> + Azure CLI<\/td>T-SQL runs over a SQL connection, not REST. az login<\/code> provides the identity.<\/td><\/tr>
Create a pipeline from a prompt<\/td>Fabric MCP<\/td>Built-in create item<\/em> tool, nothing to install.<\/td><\/tr>
Export, edit and re-import a pipeline, schedules<\/td>Fabric CLI (fab<\/code>)<\/td>Repeatable, file-based, works with service principals.<\/td><\/tr>
Dataflow Gen2 via dataflows-cli<\/code><\/td>Azure CLI (az rest<\/code>)<\/td>Definitions and M preview (executeQuery<\/code>) are REST endpoints the MCP doesn’t expose.<\/td><\/tr>
Publish an environment<\/td>az rest<\/code> \/ fab api<\/code><\/td>Staging and publish are separate API calls the MCP doesn’t have.<\/td><\/tr>
Commit to Git<\/td>git-integration-operations-cli<\/code> \u2192 fab api<\/code> (falls back to az rest<\/code>)<\/td>No Git tools in the MCP.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

The rule of thumb hasn’t changed: MCP when there’s a ready-made tool for it, fab<\/code> for repeatable work, az<\/code> whenever a skill needs a REST endpoint the MCP doesn’t cover. In practice that means you want all three logged in before you start. \/mcp<\/code> shows the MCP, fab auth status<\/code> shows fab<\/code>, and az account show<\/code> shows az<\/code>.<\/p>\n\n\n\n

Notebooks in VS Code<\/h2>\n\n\n\n

\ud83d\udd27 VS Code extension<\/strong>, not MCP and not a CLI. This whole notebook section runs through the extension and its own sign-in.<\/p>\n\n\n\n

Signing in to the extension<\/h3>\n\n\n\n

First, authorize yourself in the Fabric Data Engineering extension:<\/p>\n\n\n\n

    \n
  1. Open VS Code<\/li>\n\n\n\n
  2. Open the Command Palette (usually Ctrl+Shift+P, or Cmd+Shift+P on macOS)<\/li>\n\n\n\n
  3. Type “Fabric Data Engineering” and select the Sign In option:<\/li>\n<\/ol>\n\n\n\n
    \"Pasted<\/figure>\n\n\n\n

    Your default browser opens with the standard Microsoft login window. Enter your credentials and sign in. After that, the extension can connect to your workspaces and items:<\/p>\n\n\n\n

    \"Pasted<\/figure>\n\n\n\n
      \n
    1. Choose the extension<\/li>\n\n\n\n
    2. Click the select workspace icon<\/li>\n\n\n\n
    3. Pick your workspace from the list<\/li>\n<\/ol>\n\n\n\n

      At the bottom (point 4 in the picture) you can check which account is currently logged in to the extension. Once you’re connected to a workspace, the left-hand menu shows all its Data Engineering items: notebooks, Spark job definitions, lakehouses and environments. There’s also a section where you can save your prompts, so you don’t have to type the same thing twice (skills can do something similar).<\/p>\n\n\n\n

      \"Pasted<\/figure>\n\n\n\n

      Live or local?<\/h3>\n\n\n\n

      Expand Notebooks and you’ll see every notebook in the workspace. You can open one in two ways:<\/p>\n\n\n\n

        \n
      • Live connection<\/strong> – you work directly against the workspace, so every change shows up in the portal immediately. The catch: since it all happens online, Claude Code can’t see or edit the notebook. Claude works on local files.<\/li>\n\n\n\n
      • Local mode<\/strong> – my preferred one. You download the notebook, make your changes locally, and still run it from VS Code on a Spark cluster in Fabric. When you’re done, you publish the updated version back to Fabric.<\/li>\n<\/ul>\n\n\n\n

        Live connection<\/h3>\n\n\n\n

        To open a notebook live, click the “Open notebook folder” icon:<\/p>\n\n\n\n

        \"Pasted<\/figure>\n\n\n\n

        The notebook opens in exactly the state it’s in online. Here’s mine:<\/p>\n\n\n\n

        \"Pasted<\/figure>\n\n\n\n

        I changed the script to display only US holidays and ran it. When you run it, VS Code asks you to pick a kernel. Choose Microsoft Fabric Runtime, otherwise it won’t connect to Fabric:<\/p>\n\n\n\n

        \"Pasted<\/figure>\n\n\n\n

        Then pick the language, PySpark in my case:<\/p>\n\n\n\n

        \"Pasted<\/figure>\n\n\n\n

        The notebook runs, and for me it produced this output:<\/p>\n\n\n\n

        \"Pasted<\/figure>\n\n\n\n

        Save the notebook and the changes are immediately visible in Fabric:<\/p>\n\n\n\n

        \"Pasted<\/figure>\n\n\n\n

        And since the script actually executes in Fabric, the run shows up in the Monitoring tab:
        <\/p>\n\n\n\n

        \"\"<\/figure>\n\n\n\n

        Now the big caveat of this mode. The only AI that can edit your notebook here is the one native to VS Code notebooks: GitHub Copilot. If that’s what you use, you’re fine. Open the chat window on the right and ask it for whatever you need:<\/p>\n\n\n\n

        \"Pasted<\/figure>\n\n\n\n

        Copilot updates the notebook, and you’re all set for an agentic data engineering workflow:<\/p>\n\n\n\n

        \"Pasted<\/figure>\n\n\n\n

        Using a different AI? That’s where it gets complicated. Other LLMs, even ones installed as VS Code extensions, can’t see that directory, and you might get an error like this:<\/p>\n\n\n\n

        \"Pasted<\/figure>\n\n\n\n

        Local mode<\/h3>\n\n\n\n

        The workaround is the extension’s second mode. In local mode the extension physically copies the notebook to your disk, which means any LLM can get to it. When you’re done, you sync it back to the Fabric workspace.<\/p>\n\n\n\n

        Start by clicking a different button on the notebook, Download notebook<\/em>:<\/p>\n\n\n\n

        \"Pasted<\/figure>\n\n\n\n

        The icon turns green, meaning your local copy matches the one in the Fabric workspace. A new icon appears too, Publish resource folder<\/em>, which uploads all the files and subfolders under the builtin<\/strong> folder to the notebook resources:<\/p>\n\n\n\n

        \"Pasted<\/figure>\n\n\n\n

        To open the downloaded notebook, use the same “Open notebook folder” icon as before. This time it asks which mode you want:<\/p>\n\n\n\n

        \"Pasted<\/figure>\n\n\n\n
          \n
        • Virtual workspace<\/strong> – the live connection from the previous section<\/li>\n\n\n\n
        • Local folder<\/strong> – the downloaded copy (pick this one)<\/li>\n<\/ul>\n\n\n\n

          The notebook opens just like last time:<\/p>\n\n\n\n

          \"Pasted<\/figure>\n\n\n\n

          One difference: changes don’t sync to the workspace automatically. You push them with a button. But the file is now on your disk, so any<\/strong> LLM can help you write the notebook, including external ones like Claude Code or Codex.<\/p>\n\n\n\n

          From the Claude Code extension<\/h4>\n\n\n\n

          The easiest route is the Claude Code extension in VS Code. Open it and it sees the notebook context straight away:<\/p>\n\n\n\n

          \"Pasted<\/figure>\n\n\n\n

          And it worked:
          <\/p>\n\n\n\n

          \"\"<\/figure>\n\n\n\n

          I like this approach a lot. The notebook is right there, so you can correct things, execute cells and check the output as you go. One thing to remember: notebooks don’t save automatically while you edit them. Hit save.<\/strong> The LLM only sees the last saved version.<\/p>\n\n\n\n

          From the terminal<\/h4>\n\n\n\n

          Running the LLM from a terminal works too. First, find where the notebook is stored. Easiest way: right-click the notebook in the VS Code menu and choose Reveal in Finder<\/em> (or File Explorer<\/em>, depending on your OS):<\/p>\n\n\n\n

          \"Pasted<\/figure>\n\n\n\n

          This is the path to my file: ~\/50be2564-b12a-419d-93e8-fce57529a5b2\/SynapseNotebook\/c6be7bf2-3f60-4be6-a573-ffca78078b7f\/NB_AgenticWorkflow\/NB_AgenticWorkflow.ipynb<\/code><\/p>\n\n\n\n

          Which translates to:
          \/<WorkspaceID>\/<Item Type>\/<ItemID>\/<ItemName>\/<items><\/code><\/p>\n\n\n\n

          Other items land under the same main folder (unless you change that in the extension settings), so with the workspace and item IDs you can find anything quickly. The structure is similar to what you see in the code repository when it’s linked to your Fabric workspace.<\/p>\n\n\n\n

          From there it’s simple. Open a terminal, navigate to the folder, and tell the LLM what to do:<\/p>\n\n\n\n

          \"Pasted<\/figure>\n\n\n\n

          When it’s done, check the result by opening the file, or open it in VS Code like before:<\/p>\n\n\n\n

          \"Pasted<\/figure>\n\n\n\n

          Publishing back to Fabric<\/h3>\n\n\n\n

          Time to get the code into Fabric. Go back to the extension and click the Publish notebook<\/em> icon. That’s it, the whole notebook gets published.<\/p>\n\n\n\n

          Well, usually. Two things can get in the way.<\/p>\n\n\n\n

          \u26a0\ufe0f Caveat: 400 Bad Request<\/code> when publishing notebooks from VS Code<\/strong><\/p>\n\n\n\n

          If the extension throws Error running command synapse.publishNotebook: HTTP Error Response: 400 Bad Request<\/code>, in my experience your local .ipynb<\/code> JSON has drifted from the format Fabric expects. It usually happens after editing notebooks with the Jupyter\/NotebookEdit tooling in VS Code, which writes source<\/code> as a single string and injects id<\/code>\/execution_count<\/code> fields Fabric doesn’t want.<\/p>\n\n\n\n

          What to check:<\/strong><\/p>\n\n\n\n

            \n
          • source<\/code> must be a list of strings (one per line), not a single string<\/li>\n\n\n\n
          • Code and markdown cells shouldn’t carry id<\/code> or execution_count<\/code><\/li>\n\n\n\n
          • Code cell metadata<\/code> needs {\"microsoft\": {\"language\": \"python\", \"language_group\": \"synapse_pyspark\"}}<\/code><\/li>\n\n\n\n
          • The top-level notebook metadata<\/code> needs the Fabric block (kernel_info.name: synapse_pyspark<\/code>, spark_compute<\/code>, dependencies.lakehouse<\/code>), not a generic kernelspec: python3<\/code><\/li>\n\n\n\n
          • Close and reopen the .ipynb<\/code> tab in VS Code before republishing.<\/strong> The extension publishes its in-memory buffer, so it won’t see fixes made on disk otherwise<\/li>\n<\/ul>\n\n\n\n

            A prompt you can hand your AI assistant to fix it:<\/strong><\/p>\n\n\n\n

            My Fabric notebook <filename>.ipynb fails to publish from VS Code with HTTP Error Response: 400 Bad Request. Compare it structurally against <a sibling notebook that publishes successfully> and rewrite it to match Fabric's canonical format: strip any id\/execution_count keys from cells, convert source to a list of strings (one per line) if it's a single string, ensure code cells have metadata.microsoft.language = \"python\" and language_group = \"synapse_pyspark\" (preserving existing tags\/cellStatus), and fix the top-level metadata to use the Fabric kernel block instead of a generic python3\/ipykernel one. Validate the JSON loads and each code cell compiles after the rewrite.<\/code><\/pre>\n\n\n\n
            \u26a0\ufe0f Caveat 2: Editing the same notebook in two places<\/strong><\/p>\n\n\n\n
            If the notebook changes both locally through the extension and online in the workspace (it happens a lot when several people work in the same workspace), you get a conflict:<\/p>\n\n\n\n
            \"Pasted<\/figure>\n\n\n\n
            Click Update<\/em> and a comparison window opens, where you can review the conflict and choose which version to keep. You can also open it with the comparison button highlighted below:<\/p>\n\n\n\n
            \"Pasted<\/figure>\n\n\n\n
            Microsoft Fabric skills<\/h2>\n\n\n\n
            So far the LLM has been writing code with whatever it already knows about Fabric. Skills make it a lot better at that. In my setup post I showed how to install the Fabric skills: [[Starting with agentic workflow in Fabric – Polished]]. Once they’re installed your LLM can use them, but what do they actually do, and which ones help with notebook code?<\/p>\n\n\n\n
            Microsoft quietly open-sourced a whole plugin of Fabric skills, and spark-authoring-cli<\/code> is the one I keep coming back to. It does two jobs.<\/p>\n\n\n\n
            Job one: control-plane work.<\/strong> Spinning up workspaces, creating schema-enabled lakehouses, deploying notebooks, kicking off runs. All of it through az rest<\/code> against the Fabric REST API, with the long-running-operation dance (poll the Location<\/code> header, wait for Succeeded<\/code>) already figured out for you.<\/p>\n\n\n\n
            The notebook round-trip is what saved me the most time: getDefinition<\/code> \u2192 base64-decode \u2192 edit the cell array \u2192 re-encode \u2192 updateDefinition<\/code>. Sounds simple. It isn’t. The skill spells out the bits that actually bite. Every line in a cell’s source<\/code> array has to end with \\n<\/code>, or your code silently collapses into a single line. And the default lakehouse binding lives in the .ipynb<\/code> metadata but moves to a # METADATA<\/code> comment block if you’re working in the .py<\/code> format. Sound familiar? Same class of formatting drift as the 400 Bad Request<\/code> above.<\/p>\n\n\n\n
            Job two: the code inside<\/em> those cells.<\/strong> PySpark, Scala, SparkR, %%sql<\/code>, plus notebookutils<\/code> patterns for pipeline-context detection, connection credentials, and runMultiple<\/code> DAGs when you need fan-out\/fan-in orchestration instead of hand-rolled threads. Delta MERGE, OPTIMIZE, V-Order tuned per medallion layer. Materialized Lake Views too, including a readiness check that flags which SQL constructs will quietly force a full refresh instead of an incremental one, and why Change Data Feed has to be enabled on every source table before incremental refresh is even possible.<\/p>\n\n\n\n
            The part I didn’t expect: the guardrails.<\/strong> Check for a job instance from the last five minutes before submitting a new run. Capture the job ID immediately, then only ever GET<\/code> it, never retry the POST<\/code>. Never hardcode a workspace or lakehouse ID. Those are rules you normally learn the hard way, around 11pm, after stacking three duplicate Spark sessions on a paused capacity. Here they’re just written down.<\/p>\n\n\n\n
            Putting it to work<\/h3>\n\n\n\n
            Let’s use it. Open a terminal, or VS Code with an empty session, and create a notebook from scratch that does an extra transformation.<\/p>\n\n\n\n
            My prompt:<\/p>\n\n\n\n
            Markdown<\/span>