{"id":266,"date":"2025-12-16T13:38:37","date_gmt":"2025-12-16T12:38:37","guid":{"rendered":"https:\/\/dataguide.tech\/?p=266"},"modified":"2026-01-13T17:13:33","modified_gmt":"2026-01-13T16:13:33","slug":"automated-alerts-for-fabric-part-2","status":"publish","type":"post","link":"https:\/\/dataguide.tech\/index.php\/2025\/12\/16\/automated-alerts-for-fabric-part-2\/","title":{"rendered":"How to Build Microsoft Fabric Monitoring with Automated Alerts &#8211; Part 2"},"content":{"rendered":"\n<p>Microsoft Fabric monitoring is essential when you&#8217;re running production pipelines \u2013 you need to know when something breaks before your users do. In my previous post, I walked through how to connect and ingest logs using a notebook. Now let&#8217;s take it a step further &#8211; automating the entire process with built-in alerting.<\/p>\n\n\n\n<p>By automation, I mean creating a pipeline with a notification system that fires off alerts the moment something goes wrong. This setup enables early detection of anomalies and issues, giving you time to resolve them before they snowball into bigger problems.<\/p>\n\n\n\n<p>To achieve this, I extended my previous notebook and integrated it with Fabric Data Factory pipeline orchestration. The goal was to log every entry from the tenant to monitor pipeline failures, and support team alerts.<\/p>\n\n\n\n<p>Here&#8217;s what we&#8217;ll cover:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ingesting logs from the Fabric API endpoint<\/li>\n\n\n\n<li>Loading them into Lakehouse<\/li>\n\n\n\n<li>Setting up alerts when issues are detected<\/li>\n<\/ul>\n\n\n\n<p>I scheduled the flow to run every 30 minutes, but you can run it more frequently depending on your capacity. I&#8217;m using the trial version which corresponds to F64, and the entire flow takes around 3-5 minutes. Below you can find an overall picture of the architecture.<\/p>\n\n\n\n<p><img decoding=\"async\" src=\"https:\/\/dataguide.tech\/wp-content\/uploads\/2025\/12\/Pasted-image-20251216112108.png\" alt=\"Pasted image 20251216112108.png\"><br>Ok, let&#8217;s now go to the Fabric workspace to present the technical details.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong class=\"font-semibold\">Setting Up Microsoft Fabric Monitoring Notebook<\/strong><\/h2>\n\n\n\n<p>Starting from the beginning &#8211; I&#8217;ve extended the notebook to ingest and load data into Lakehouse as follows:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Add Pipeline Run ID Parameter<\/h3>\n\n\n\n<p>Start by adding a new cell at the top to capture the <code>pipeline_run_id<\/code> from the Fabric Pipeline. By default, you can set it to any GUID for testing purposes.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>pipeline_rund_id = \"1caf67a5-2bba-4085-920c-6b3f07c90523\" #You can add any GUID<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #A6ACCD\">pipeline_rund_id <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">1caf67a5-2bba-4085-920c-6b3f07c90523<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\"> <\/span><span style=\"color: #767C9DB0; font-style: italic\">#You can add any GUID<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<p><strong>Heads up:<\/strong> You need to set that cell as a toggle parameter:<br><img decoding=\"async\" src=\"https:\/\/dataguide.tech\/wp-content\/uploads\/2025\/12\/Pasted-image-20251216085608.png\" alt=\"Pasted image 20251216085608.png\"><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Import Libraries<\/h3>\n\n\n\n<p>Import the necessary libraries: <code>requests<\/code> for making HTTP requests, <code>PySpark (col, max)<\/code> for column selection and aggregation, <code>pandas<\/code> for data manipulation, <code>json<\/code> for handling JSON data, <code>datetime<\/code> for date and time manipulation, and <code>UUID<\/code> for unique identifier generation.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly># Import all libraries\n\nfrom pyspark.sql.functions import col, max\nfrom datetime import datetime, timedelta\nimport requests\nimport pandas as pd\nimport json\nimport uuid<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #767C9DB0; font-style: italic\"># Import all libraries<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #5DE4C7\">from<\/span><span style=\"color: #A6ACCD\"> pyspark.sql.functions <\/span><span style=\"color: #5DE4C7\">import<\/span><span style=\"color: #A6ACCD\"> col, max<\/span><\/span>\n<span class=\"line\"><span style=\"color: #5DE4C7\">from<\/span><span style=\"color: #A6ACCD\"> datetime <\/span><span style=\"color: #5DE4C7\">import<\/span><span style=\"color: #A6ACCD\"> datetime, timedelta<\/span><\/span>\n<span class=\"line\"><span style=\"color: #5DE4C7\">import<\/span><span style=\"color: #A6ACCD\"> requests<\/span><\/span>\n<span class=\"line\"><span style=\"color: #5DE4C7\">import<\/span><span style=\"color: #A6ACCD\"> pandas <\/span><span style=\"color: #5DE4C7\">as<\/span><span style=\"color: #A6ACCD\"> pd<\/span><\/span>\n<span class=\"line\"><span style=\"color: #5DE4C7\">import<\/span><span style=\"color: #A6ACCD\"> json<\/span><\/span>\n<span class=\"line\"><span style=\"color: #5DE4C7\">import<\/span><span style=\"color: #A6ACCD\"> uuid<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Execute API Call<\/h3>\n\n\n\n<p>Make the API call using the <code>requests.get<\/code> method and store the response.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly># Execute API call and save a response to variable\ntoken = mssparkutils.credentials.getToken(\"https:\/\/api.fabric.microsoft.com\")\n<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #767C9DB0; font-style: italic\"># Execute API call and save a response to variable<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">token <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> mssparkutils.credentials.getToken(<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">https:\/\/api.fabric.microsoft.com<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">)<\/span><\/span>\n<span class=\"line\"><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Define Time Range for Log Ingestion<\/h3>\n\n\n\n<p>Define the start and end date for log ingestion. The default setting is to take all logs starting from the last logged item till now. If there&#8217;s no table yet (initial run), do ingestion for the last 24 hours.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>try:\n\tfabric_logs_main_df = spark.read.format(\"delta\").load(f'Tables\/fabric_logs')\n\tstart_time = fabric_logs_main_df.agg(max(\"jobStartTimeUtc\")).collect()&#91;0&#93;&#91;0&#93;\nexcept:\n\tprint('No Table Found - setting yesterday as time to get the logs.')\n\tstart_time = datetime.today() - timedelta(hours=24, minutes=0)\n\nend_time = datetime.today()<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #5DE4C7C0\">try<\/span><span style=\"color: #A6ACCD\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">\tfabric_logs_main_df <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> spark.read.format(<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">delta<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">).load(<\/span><span style=\"color: #91B4D5\">f<\/span><span style=\"color: #5DE4C7\">&#39;Tables\/fabric_logs&#39;<\/span><span style=\"color: #A6ACCD\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">\tstart_time <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> fabric_logs_main_df.agg(<\/span><span style=\"color: #E4F0FBD0\">max<\/span><span style=\"color: #A6ACCD\">(<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">jobStartTimeUtc<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">)).collect()&#91;<\/span><span style=\"color: #5DE4C7\">0<\/span><span style=\"color: #A6ACCD\">&#93;&#91;<\/span><span style=\"color: #5DE4C7\">0<\/span><span style=\"color: #A6ACCD\">&#93;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #5DE4C7C0\">except<\/span><span style=\"color: #A6ACCD\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">\t<\/span><span style=\"color: #E4F0FBD0\">print<\/span><span style=\"color: #A6ACCD\">(<\/span><span style=\"color: #A6ACCD\">&#39;<\/span><span style=\"color: #5DE4C7\">No Table Found - setting yesterday as time to get the logs.<\/span><span style=\"color: #A6ACCD\">&#39;<\/span><span style=\"color: #A6ACCD\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">\tstart_time <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> datetime.today() <\/span><span style=\"color: #91B4D5\">-<\/span><span style=\"color: #A6ACCD\"> timedelta(<\/span><span style=\"color: #E4F0FB\">hours<\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #5DE4C7\">24<\/span><span style=\"color: #A6ACCD\">, <\/span><span style=\"color: #E4F0FB\">minutes<\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #5DE4C7\">0<\/span><span style=\"color: #A6ACCD\">)<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">end_time <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> datetime.today()<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Prepare API Request<\/h3>\n\n\n\n<p>Define the URL and parameters for the API request. The parameters include filters for the data you want to retrieve &#8211; in this case, taking all data from the latest saved logs (based on <code>start_time<\/code> variable) till current time (based on <code>end_time<\/code> variable). It also limits the statuses to only finished ones, excluding in-progress, pending, and queued processes.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>url = \"&lt;YOUR URL>\"\nparams = {\n    \"limit\": 10000, # Maximum number of rows to be retrieved\n    \"endTime\": f\"{end_time}\", # Limiting range of StartTime of the Job\n    \"startTime\": f\"{start_time}\", # Limiting range of StartTime of the Job\n    \"status\":\"4,6,2,3,5,7,8\", # All processed statuses. Excluded all in_progress or pending.\n    \"Accept\": \"application\/json\", # Do not manipulate to get json output\n    \"accept-encoding\": \"gzip, deflate, br, zstd\", # Do not manipulate\n    \"activityid\": \"d6678b61-1c72-4e43-9498-54b715064543\", # It could be any GUID\n    \"accept-language\": \"en-US,en;q=0.9\" # Do not manipulate\n}\nheaders = {\n    \"Authorization\": f\"Bearer {token}\" # Authorization using the generated token\n}<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #A6ACCD\">url <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">&lt;YOUR URL&gt;<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">params <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> {<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">limit<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">: <\/span><span style=\"color: #5DE4C7\">10000<\/span><span style=\"color: #A6ACCD\">, <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Maximum number of rows to be retrieved<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">endTime<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">: <\/span><span style=\"color: #91B4D5\">f<\/span><span style=\"color: #5DE4C7\">&quot;<\/span><span style=\"color: #5DE4C7\">{<\/span><span style=\"color: #A6ACCD\">end_time<\/span><span style=\"color: #5DE4C7\">}<\/span><span style=\"color: #5DE4C7\">&quot;<\/span><span style=\"color: #A6ACCD\">, <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Limiting range of StartTime of the Job<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">startTime<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">: <\/span><span style=\"color: #91B4D5\">f<\/span><span style=\"color: #5DE4C7\">&quot;<\/span><span style=\"color: #5DE4C7\">{<\/span><span style=\"color: #A6ACCD\">start_time<\/span><span style=\"color: #5DE4C7\">}<\/span><span style=\"color: #5DE4C7\">&quot;<\/span><span style=\"color: #A6ACCD\">, <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Limiting range of StartTime of the Job<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">status<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">:<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">4,6,2,3,5,7,8<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">, <\/span><span style=\"color: #767C9DB0; font-style: italic\"># All processed statuses. Excluded all in_progress or pending.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">Accept<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">: <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">application\/json<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">, <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Do not manipulate to get json output<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">accept-encoding<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">: <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">gzip, deflate, br, zstd<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">, <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Do not manipulate<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">activityid<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">: <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">d6678b61-1c72-4e43-9498-54b715064543<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">, <\/span><span style=\"color: #767C9DB0; font-style: italic\"># It could be any GUID<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">accept-language<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">: <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">en-US,en;q=0.9<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\"> <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Do not manipulate<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">}<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">headers <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> {<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">Authorization<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">: <\/span><span style=\"color: #91B4D5\">f<\/span><span style=\"color: #5DE4C7\">&quot;Bearer <\/span><span style=\"color: #5DE4C7\">{<\/span><span style=\"color: #A6ACCD\">token<\/span><span style=\"color: #5DE4C7\">}<\/span><span style=\"color: #5DE4C7\">&quot;<\/span><span style=\"color: #A6ACCD\"> <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Authorization using the generated token<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">}<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Execute the API Call<\/h3>\n\n\n\n<p>Make the API call and store the response.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly># Execute API call and save a response to variable\nresponse = requests.get(url, headers=headers, params=params)<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #767C9DB0; font-style: italic\"># Execute API call and save a response to variable<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">response <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> requests.get(url, <\/span><span style=\"color: #E4F0FB\">headers<\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\">headers, <\/span><span style=\"color: #E4F0FB\">params<\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\">params)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Process JSON Data<\/h3>\n\n\n\n<p>Extract the JSON data from the response and convert it into a pandas DataFrame for easy manipulation and analysis.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly># Take json data from response and save it to variable\ndata = response.json()\n# Open JSON and save it as pandas DF\ndf = pd.json_normalize(data)<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #767C9DB0; font-style: italic\"># Take json data from response and save it to variable<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">data <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> response.json()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #767C9DB0; font-style: italic\"># Open JSON and save it as pandas DF<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">df <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> pd.json_normalize(data)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Add Pipeline Run ID Column<\/h3>\n\n\n\n<p>Add <code>pipeline_run_id<\/code> to recognize which pipeline run collected which logs.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>df&#91;'pipeline_run_id'&#93; = pipeline_rund_id<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #A6ACCD\">df&#91;<\/span><span style=\"color: #A6ACCD\">&#39;<\/span><span style=\"color: #5DE4C7\">pipeline_run_id<\/span><span style=\"color: #A6ACCD\">&#39;<\/span><span style=\"color: #A6ACCD\">&#93; <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> pipeline_rund_id<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Check for Captured Logs<\/h3>\n\n\n\n<p>Check if the DataFrame is empty and set up the <code>df_processing<\/code> flag to determine whether to save to the Lakehouse table.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>if not df.empty:\n    df_processing = True\nelse:\n    df_processing = False<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #5DE4C7C0\">if<\/span><span style=\"color: #A6ACCD\"> <\/span><span style=\"color: #91B4D5\">not<\/span><span style=\"color: #A6ACCD\"> df.empty:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    df_processing <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> <\/span><span style=\"color: #5DE4C7\">True<\/span><\/span>\n<span class=\"line\"><span style=\"color: #5DE4C7C0\">else<\/span><span style=\"color: #A6ACCD\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    df_processing <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> <\/span><span style=\"color: #5DE4C7\">False<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Load DataFrame to Table<\/h3>\n\n\n\n<p>Check if the <code>df_processing<\/code> flag is True, then convert all columns to string. This isn&#8217;t the most elegant solution, but since we&#8217;re lacking proper documentation from Microsoft to rely on, I&#8217;d suggest starting with this approach and then adjusting columns to proper datatypes later. The logs are loaded in append mode with <code>mergeSchema<\/code> option set to true so you don&#8217;t lose any new columns that might appear.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>if df_processing:\n    for col_name in df.columns:\n        df&#91;col_name&#93; = df&#91;col_name&#93;.astype('str')\n    df = spark.createDataFrame(df)\n    df.write.format(\"delta\").mode(\"append\").option('mergeSchema', 'true').save('Tables\/fabric_logs')<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #5DE4C7C0\">if<\/span><span style=\"color: #A6ACCD\"> df_processing:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #5DE4C7C0\">for<\/span><span style=\"color: #A6ACCD\"> col_name <\/span><span style=\"color: #5DE4C7C0\">in<\/span><span style=\"color: #A6ACCD\"> df.columns:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">        df&#91;col_name&#93; <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> df&#91;col_name&#93;.astype(<\/span><span style=\"color: #A6ACCD\">&#39;<\/span><span style=\"color: #5DE4C7\">str<\/span><span style=\"color: #A6ACCD\">&#39;<\/span><span style=\"color: #A6ACCD\">)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    df <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> spark.createDataFrame(df)<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    df.write.format(<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">delta<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">).mode(<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">append<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">).option(<\/span><span style=\"color: #A6ACCD\">&#39;<\/span><span style=\"color: #5DE4C7\">mergeSchema<\/span><span style=\"color: #A6ACCD\">&#39;<\/span><span style=\"color: #A6ACCD\">, <\/span><span style=\"color: #A6ACCD\">&#39;<\/span><span style=\"color: #5DE4C7\">true<\/span><span style=\"color: #A6ACCD\">&#39;<\/span><span style=\"color: #A6ACCD\">).save(<\/span><span style=\"color: #A6ACCD\">&#39;<\/span><span style=\"color: #5DE4C7\">Tables\/fabric_logs<\/span><span style=\"color: #A6ACCD\">&#39;<\/span><span style=\"color: #A6ACCD\">)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Create Failures DataFrame<\/h3>\n\n\n\n<p>Filter only failed execution logs and store them in a new DataFrame called <code>df_failures<\/code>.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>df_failures = df.filter(df.isSuccessful == False)<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #A6ACCD\">df_failures <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> df.filter(df.isSuccessful <\/span><span style=\"color: #91B4D5\">==<\/span><span style=\"color: #A6ACCD\"> <\/span><span style=\"color: #5DE4C7\">False<\/span><span style=\"color: #A6ACCD\">)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Check for Failures<\/h3>\n\n\n\n<p>Check if <code>df_failures<\/code> is empty and set up the <code>df_failures_processing<\/code> flag to determine whether to create a list of errors for Teams\/email notifications.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>if not df_failures.rdd.isEmpty():\n    df_failures_processing = True\nelse:\n    df_failures_processing = False<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #5DE4C7C0\">if<\/span><span style=\"color: #A6ACCD\"> <\/span><span style=\"color: #91B4D5\">not<\/span><span style=\"color: #A6ACCD\"> df_failures.rdd.isEmpty():<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    df_failures_processing <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> <\/span><span style=\"color: #5DE4C7\">True<\/span><\/span>\n<span class=\"line\"><span style=\"color: #5DE4C7C0\">else<\/span><span style=\"color: #A6ACCD\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    df_failures_processing <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> <\/span><span style=\"color: #5DE4C7\">False<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Prepare Array for Notifications<\/h3>\n\n\n\n<p>Select and rename informative columns from <code>df_failures<\/code> (these work for me but you can adjust based on your needs), convert the DataFrame to a JSON list, and assign it to <code>result_array<\/code>. If <code>df_failures_processing<\/code> is false, <code>result_array<\/code> is set to an empty list.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>if df_failures_processing:\n    from pyspark.sql import functions as F\n    result_df = df_failures.select(\n        df_failures&#91;\"workspaceName\"&#93;, # Name of the workspace\n        df_failures&#91;\"workspaceObjectId\"&#93;.alias(\"WorkspaceId\"), # Workspace ID\n        df_failures&#91;\"artifactName\"&#93;.alias(\"FailedItemName\"), # Name of the item failed\n        df_failures&#91;\"artifactType\"&#93;.alias(\"FailedItemType\"), # Type of the failed item (like pipeline, notebook etc.\n        df_failures&#91;\"serviceExceptionJson\"&#93;.alias(\"ErrorDetails\"), # Json with ErrorDetails\n        df_failures&#91;\"`ownerUser.name`\"&#93;.alias(\"OwnerUserName\"), # Owner user name of failed item\n        df_failures&#91;\"`ownerUser.userPrincipalName`\"&#93;.alias(\"OwnerUserEmail\"), # Owner email of failed item\n        df_failures&#91;\"jobScheduleTimeUtc\"&#93;, # Scheduled time\n        df_failures&#91;\"jobStartTimeUtc\"&#93;, # Start time\n        df_failures&#91;\"jobEndTimeUtc\"&#93; # End time\n    )\n    # collect failures and format to json\n    result_json = result_df.toJSON().collect()\n    # format json to a list, to send them separetely.\n    result_array = &#91;json.loads(item) for item in result_json&#93;\nelse:\n    result_array=[]<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #5DE4C7C0\">if<\/span><span style=\"color: #A6ACCD\"> df_failures_processing:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #5DE4C7\">from<\/span><span style=\"color: #A6ACCD\"> pyspark.sql <\/span><span style=\"color: #5DE4C7\">import<\/span><span style=\"color: #A6ACCD\"> functions <\/span><span style=\"color: #5DE4C7\">as<\/span><span style=\"color: #A6ACCD\"> F<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    result_df <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> df_failures.select(<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">        df_failures&#91;<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">workspaceName<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">&#93;, <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Name of the workspace<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">        df_failures&#91;<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">workspaceObjectId<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">&#93;.alias(<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">WorkspaceId<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">), <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Workspace ID<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">        df_failures&#91;<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">artifactName<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">&#93;.alias(<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">FailedItemName<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">), <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Name of the item failed<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">        df_failures&#91;<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">artifactType<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">&#93;.alias(<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">FailedItemType<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">), <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Type of the failed item (like pipeline, notebook etc.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">        df_failures&#91;<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">serviceExceptionJson<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">&#93;.alias(<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">ErrorDetails<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">), <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Json with ErrorDetails<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">        df_failures&#91;<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">`ownerUser.name`<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">&#93;.alias(<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">OwnerUserName<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">), <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Owner user name of failed item<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">        df_failures&#91;<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">`ownerUser.userPrincipalName`<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">&#93;.alias(<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">OwnerUserEmail<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">), <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Owner email of failed item<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">        df_failures&#91;<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">jobScheduleTimeUtc<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">&#93;, <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Scheduled time<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">        df_failures&#91;<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">jobStartTimeUtc<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">&#93;, <\/span><span style=\"color: #767C9DB0; font-style: italic\"># Start time<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">        df_failures&#91;<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #5DE4C7\">jobEndTimeUtc<\/span><span style=\"color: #A6ACCD\">&quot;<\/span><span style=\"color: #A6ACCD\">&#93; <\/span><span style=\"color: #767C9DB0; font-style: italic\"># End time<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    )<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #767C9DB0; font-style: italic\"># collect failures and format to json<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    result_json <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> result_df.toJSON().collect()<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    <\/span><span style=\"color: #767C9DB0; font-style: italic\"># format json to a list, to send them separetely.<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    result_array <\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\"> &#91;json.loads(item) <\/span><span style=\"color: #5DE4C7C0\">for<\/span><span style=\"color: #A6ACCD\"> item <\/span><span style=\"color: #5DE4C7C0\">in<\/span><span style=\"color: #A6ACCD\"> result_json&#93;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #5DE4C7C0\">else<\/span><span style=\"color: #A6ACCD\">:<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">    result_array<\/span><span style=\"color: #91B4D5\">=<\/span><span style=\"color: #A6ACCD\">[]<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Close Notebook with Output<\/h3>\n\n\n\n<p>Pass the array as output from the notebook, which will be used in the Fabric Pipeline to send email\/Teams notifications.<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(1 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">Python<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>mssparkutils.notebook.exit(json.dumps(result_array))<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #A6ACCD\">mssparkutils.notebook.exit(json.dumps(result_array))<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n<h2 class=\"wp-block-heading\"><strong class=\"font-semibold\">Creating the Microsoft Fabric Monitoring Pipeline<\/strong><\/h2>\n\n\n\n<p>The next thing to prepare is the Fabric Pipeline for orchestrating the whole process.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Create Pipeline and Add Notebook Activity<\/h3>\n\n\n\n<p>First, create a new pipeline. In the pipeline, add the Notebook activity and select your prepared notebook:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/dataguide.tech\/wp-content\/uploads\/2025\/12\/Pasted-image-20251216090222.png\" alt=\"Pasted image 20251216090222.png\"\/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Configure the Notebook Activity<\/h3>\n\n\n\n<p>Configure the notebook to be executed:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/dataguide.tech\/wp-content\/uploads\/2025\/12\/Pasted-image-20251216090238.png\" alt=\"Pasted image 20251216090238.png\"\/><\/figure>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Choose the workspace where you have the notebook created<\/li>\n\n\n\n<li>Choose the notebook for logs ingestion<\/li>\n\n\n\n<li>Add the name of the parameter from the toggle parameter cell &#8211; in my case it&#8217;s <code>pipeline_run_id<\/code><\/li>\n\n\n\n<li>Choose <code>String<\/code> data type<\/li>\n\n\n\n<li>Click on <code>Add dynamic content<\/code> and add the following value: <code>@pipeline().RunId<\/code>. This takes the system variable which is the unique identifier of the running pipeline &#8211; you can check the details here: <a href=\"https:\/\/learn.microsoft.com\/en-us\/fabric\/data-factory\/parameters\" target=\"_blank\" rel=\"noopener\">Parameters &#8211; Microsoft Fabric | Microsoft Learn<\/a><\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\">Configuring Alert Notifications<\/h2>\n\n\n\n<p>Now it&#8217;s time to set up the notifications. First, add a ForEach loop activity, which will execute notifications for each item from the <code>result_array<\/code>.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/dataguide.tech\/wp-content\/uploads\/2025\/12\/Pasted-image-20251216090304.png\" alt=\"Pasted image 20251216090304.png\"\/><\/figure>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Add ForEach loop activity<\/li>\n\n\n\n<li>Link it by <code>On Success<\/code> path<\/li>\n\n\n\n<li>Name it as you wish<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/dataguide.tech\/wp-content\/uploads\/2025\/12\/Pasted-image-20251216090335.png\" alt=\"Pasted image 20251216090335.png\"\/><\/figure>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Go to the <code>Settings<\/code> tab<\/li>\n\n\n\n<li>Leave sequential option unchecked &#8211; this allows ForEach to run in parallel<\/li>\n\n\n\n<li>Set up batch count &#8211; this limits the number of parallel runs (I&#8217;ve set it to 10)<\/li>\n\n\n\n<li>Add array output from Notebook using: <code>@json(activity('Logs Ingestion Notebook').output.result.exitValue)<\/code><\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Add Email Notification<\/h3>\n\n\n\n<p>Now click on &#8220;Edit&#8221; in the ForEach loop and add Teams or Outlook activities. I&#8217;ll set it up for emails.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/dataguide.tech\/wp-content\/uploads\/2025\/12\/Pasted-image-20251216090435.png\" alt=\"Pasted image 20251216090435.png\"\/><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Add Outlook activity<\/li>\n\n\n\n<li>Name it as you wish<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/dataguide.tech\/wp-content\/uploads\/2025\/12\/Pasted-image-20251216090457.png\" alt=\"Pasted image 20251216090457.png\"\/><\/figure>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Go to the Settings tab<\/li>\n\n\n\n<li>Sign in using your account or a prepared technical email for notifications. Once signed in, the activity will show you the menu to set up the email<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/dataguide.tech\/wp-content\/uploads\/2025\/12\/Pasted-image-20251216090526.png\" alt=\"Pasted image 20251216090526.png\"\/><\/figure>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Here the account you&#8217;re logged in with will be displayed &#8211; this will be used for email notifications<\/li>\n\n\n\n<li>Add the email address where you want to send the notification<\/li>\n\n\n\n<li>Subject of the email &#8211; I&#8217;ve added an expression to build the subject &#8220;Issue spotted in workspace &lt;workspace_name>&#8221;. Expression: <code>Issue spotted in workspace @{item().WorkspaceName}<\/code><\/li>\n\n\n\n<li>Body of the email &#8211; I&#8217;ve listed all items from the <code>result_array<\/code> like this:<\/li>\n<\/ol>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro cbp-has-line-numbers\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;--cbp-line-number-color:#a6accd;--cbp-line-number-width:calc(2 * 0.6 * .875rem);line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:flex;align-items:center;padding:10px 0px 10px 16px;margin-bottom:-2px;width:100%;text-align:left;background-color:#252937;color:#969dc4\">HTML<\/span><span role=\"button\" tabindex=\"0\" style=\"color:#a6accd;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><pre class=\"code-block-pro-copy-button-pre\" aria-hidden=\"true\"><textarea class=\"code-block-pro-copy-button-textarea\" tabindex=\"-1\" aria-hidden=\"true\" readonly>&lt;p>Hello,&lt;\/p>\n&lt;p>The issue have been spotted during jobs execution, you can find a details below:&lt;\/p>\n&lt;ul>\n  &lt;li>Workspace Name: @{item().WorkspaceName}&lt;\/li>\n  &lt;li>Workspace Id: @{item().WorkspaceId}&lt;\/li>\n  &lt;li>Failed Item Name: @{item().FailedItemName}&lt;\/li>\n  &lt;li>Failed Item Type: @{item().FailedItemType}&lt;\/li>\n  &lt;li>ErrorMessage: @{item().ErrorMessage}&lt;\/li>\n  &lt;li>Owner User Name: @{item().OwnerUserName}&lt;\/li>\n  &lt;li>Owner User Email: @{item().OwnerUserEmail}&lt;\/li>\n  &lt;li>Scheduled On: @{item().jobScheduleTimeUtc}&lt;\/li>\n  &lt;li>Started On: @{item().jobStartTimeUtc}&lt;\/li>\n  &lt;li>Finished On: @{item().jobEndTimeUtc}&lt;\/li>\n&lt;\/ul>\n<\/textarea><\/pre><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M4.5 12.75l6 6 9-13.5\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M16.5 8.25V6a2.25 2.25 0 00-2.25-2.25H6A2.25 2.25 0 003.75 6v8.25A2.25 2.25 0 006 16.5h2.25m8.25-8.25H18a2.25 2.25 0 012.25 2.25V18A2.25 2.25 0 0118 20.25h-7.5A2.25 2.25 0 018.25 18v-1.5m8.25-8.25h-6a2.25 2.25 0 00-2.25 2.25v6\"><\/path><\/svg><\/span><pre class=\"shiki poimandres\" style=\"background-color: #1b1e28\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">p<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><span style=\"color: #A6ACCD\">Hello,<\/span><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">p<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">p<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><span style=\"color: #A6ACCD\">The issue have been spotted during jobs execution, you can find a details below:<\/span><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">p<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">ul<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">  <\/span><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><span style=\"color: #A6ACCD\">Workspace Name: @{item().WorkspaceName}<\/span><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">  <\/span><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><span style=\"color: #A6ACCD\">Workspace Id: @{item().WorkspaceId}<\/span><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">  <\/span><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><span style=\"color: #A6ACCD\">Failed Item Name: @{item().FailedItemName}<\/span><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">  <\/span><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><span style=\"color: #A6ACCD\">Failed Item Type: @{item().FailedItemType}<\/span><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">  <\/span><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><span style=\"color: #A6ACCD\">ErrorMessage: @{item().ErrorMessage}<\/span><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">  <\/span><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><span style=\"color: #A6ACCD\">Owner User Name: @{item().OwnerUserName}<\/span><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">  <\/span><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><span style=\"color: #A6ACCD\">Owner User Email: @{item().OwnerUserEmail}<\/span><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">  <\/span><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><span style=\"color: #A6ACCD\">Scheduled On: @{item().jobScheduleTimeUtc}<\/span><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">  <\/span><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><span style=\"color: #A6ACCD\">Started On: @{item().jobStartTimeUtc}<\/span><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #A6ACCD\">  <\/span><span style=\"color: #E4F0FB\">&lt;<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><span style=\"color: #A6ACCD\">Finished On: @{item().jobEndTimeUtc}<\/span><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">li<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><span style=\"color: #E4F0FB\">&lt;\/<\/span><span style=\"color: #5DE4C7\">ul<\/span><span style=\"color: #E4F0FB\">&gt;<\/span><\/span>\n<span class=\"line\"><\/span><\/code><\/pre><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">GitHub Repository<\/h2>\n\n\n\n<p>The whole solution is available on my GitHub Repository: <a href=\"https:\/\/github.com\/LukaszLubinaDataGuide\/DataGuide-Blog\/tree\/main\/Fabric_Automated_Monitoring_Alerts\" target=\"_blank\" rel=\"noopener\">GitHub<\/a>. You can sync the content directly to your Fabric workspace.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Wrapping Up<\/h2>\n\n\n\n<p>Automating log ingestion and monitoring using Fabric Data Factory pipeline orchestration can significantly enhance your ability to detect and resolve issues swiftly. By setting up a robust alerting system, you ensure that anomalies are promptly addressed, minimizing potential disruptions.<\/p>\n\n\n\n<p>The key steps involve:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ingesting logs from the Fabric API endpoint<\/li>\n\n\n\n<li>Loading them into Lakehouse<\/li>\n\n\n\n<li>Setting up alerts for detected issues<\/li>\n<\/ul>\n\n\n\n<p>Scheduling the flow to run at regular intervals ensures continuous monitoring and timely detection of any problems.<\/p>\n\n\n\n<p>I hope this guide helps you streamline your log management process and improve your overall system efficiency. If you have any questions or need further assistance, feel free to reach out.<\/p>\n\n\n\n<p>Happy automating!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Microsoft Fabric monitoring is essential when you&#8217;re running production pipelines \u2013 you need to know when something breaks before your users do. In my previous post, I walked through how to connect and ingest logs using a notebook. Now let&#8217;s take it a step further &#8211; automating the entire process with built-in alerting. By automation,&#8230;<\/p>\n","protected":false},"author":1,"featured_media":281,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_kadence_starter_templates_imported_post":false,"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"default","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[9,8],"tags":[],"class_list":["post-266","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-microsoft-fabric","category-administration"],"jetpack_featured_media_url":"https:\/\/dataguide.tech\/wp-content\/uploads\/2025\/12\/Gemini_Generated_Image_nhfc73nhfc73nhfc-scaled.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/dataguide.tech\/index.php\/wp-json\/wp\/v2\/posts\/266","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataguide.tech\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataguide.tech\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataguide.tech\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dataguide.tech\/index.php\/wp-json\/wp\/v2\/comments?post=266"}],"version-history":[{"count":2,"href":"https:\/\/dataguide.tech\/index.php\/wp-json\/wp\/v2\/posts\/266\/revisions"}],"predecessor-version":[{"id":279,"href":"https:\/\/dataguide.tech\/index.php\/wp-json\/wp\/v2\/posts\/266\/revisions\/279"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dataguide.tech\/index.php\/wp-json\/wp\/v2\/media\/281"}],"wp:attachment":[{"href":"https:\/\/dataguide.tech\/index.php\/wp-json\/wp\/v2\/media?parent=266"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataguide.tech\/index.php\/wp-json\/wp\/v2\/categories?post=266"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataguide.tech\/index.php\/wp-json\/wp\/v2\/tags?post=266"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}