{"id":373,"date":"2026-02-24T11:37:08","date_gmt":"2026-02-24T10:37:08","guid":{"rendered":"https:\/\/dataguide.tech\/?p=373"},"modified":"2026-02-24T13:04:29","modified_gmt":"2026-02-24T12:04:29","slug":"custom-environment-whats-that-what-is-the-reason","status":"publish","type":"post","link":"https:\/\/dataguide.tech\/index.php\/2026\/02\/24\/custom-environment-whats-that-what-is-the-reason\/","title":{"rendered":"Custom Environment – what’s that, what is the reason"},"content":{"rendered":"\n

If you’re working with Microsoft Fabric and running complex workloads – multiple notebooks executing at the same time, or needing Python libraries that aren’t built-in – you’ve probably asked yourself: “How do I manage this properly?”<\/p>\n\n\n\n

The answer is Fabric Environments<\/strong>. In this post, I’ll walk you through what they are, how to configure them, and how to use them in your notebooks. For the full official reference, you can check the Microsoft documentation on Fabric Environments<\/a>. Let’s get started.<\/p>\n\n\n\n

What Is a Fabric Environment?<\/h1>\n\n\n\n

An Environment in Microsoft Fabric lets you manage and reconfigure both hardware and software settings for your Spark workloads. Think of it as a centralized configuration layer where you can control:<\/p>\n\n\n\n

    \n
  • Spark runtimes<\/strong> – Choose your runtime version<\/li>\n\n\n\n
  • Compute resources<\/strong> – Adjust driver\/executor cores and memory<\/li>\n\n\n\n
  • Libraries<\/strong> – Install additional packages (from public repos or your own custom ones)<\/li>\n<\/ul>\n\n\n\n

    Instead of scattering pip install<\/code> commands across your notebooks, you define everything once in the Environment and it’s available wherever you need it.<\/p>\n\n\n\n

    Creating and Configuring Your Environment<\/h1>\n\n\n\n

    The Environment is a standard Fabric item. To create one:<\/p>\n\n\n\n

      \n
    1. Navigate to the workspace where you want to create it<\/li>\n\n\n\n
    2. Click New Item<\/strong> and search for Environment<\/strong><\/li>\n\n\n\n
    3. Give it a meaningful name (e.g., ENV_DataEngineering<\/code>) and hit Create<\/strong><\/li>\n<\/ol>\n\n\n\n

      You should see the following screen:<\/p>\n\n\n\n

      \"Pasted<\/figure>\n\n\n\n

      On the top menu, you can choose the Runtime<\/strong> – with details about Spark and Delta release versions. Need a newer or older version? You can switch it right there.<\/p>\n\n\n\n

      On the left pane, you’ll find three sections: Libraries<\/strong>, Spark Compute<\/strong>, and Storage<\/strong>. Let’s walk through each one.<\/p>\n\n\n\n

      Libraries<\/h2>\n\n\n\n

      The Libraries section has three tabs:<\/p>\n\n\n\n

        \n
      • Built-in<\/strong> – Check what libraries are installed by default with your runtime<\/li>\n\n\n\n
      • External repositories<\/strong> – Install libraries from pip or conda repositories (the process is the same for both). You can add them through the UI or via a YML file<\/li>\n\n\n\n
      • Custom<\/strong> – Upload your own libraries in .jar<\/code>, .whl<\/code>, or .tar.gz<\/code> format<\/li>\n<\/ul>\n\n\n\n

        Once published, chosen libraries are available during the Spark session in any connected notebook right away – no more pip install<\/code> cluttering your code.<\/p>\n\n\n\n

        Spark Compute<\/h2>\n\n\n\n

        Spark Compute contains three tabs:<\/p>\n\n\n\n

          \n
        • Acceleration<\/strong> – Enable Spark Native Execution Engine, which is particularly helpful for larger data processing workloads<\/li>\n\n\n\n
        • Compute<\/strong> – Adjust your Spark compute configuration: driver and executor cores, memory allocation, dynamic allocation, and executor instances<\/li>\n\n\n\n
        • Spark properties<\/strong> – Customize Spark job definitions and set properties like resource profiles to optimize compute for your workload<\/li>\n<\/ul>\n\n\n\n

          Storage<\/h2>\n\n\n\n

          Within Storage, you can add Resources<\/strong> – additional files you’d like to have available during a Spark session. Think of it as a shared folder across your team’s notebooks, unlike the notebook’s built-in resources folder which is scoped to a single notebook.<\/p>\n\n\n\n

          To access these files from your notebook code, use the following utility:<\/p>\n\n\n\n

          Python<\/span>