There are two strategies I lean on, and I switch between them depending on the setup. Neither is “the right one” – they solve slightly different problems.<\/p>\n\n\n\n
Two Ways to Manage Libraries Across Workspaces<\/h1>\n\n\n\n
Before we get into the wiring, here’s the short version of both approaches:<\/p>\n\n\n\n
- \n
- One shared environment<\/strong> – you create an environment in a central workspace and point notebooks from other workspaces at it. Update it once, everyone gets the change.<\/li>\n\n\n\n
- One shared library, many environments<\/strong> – you keep the
.whl<\/code> in a single place and use a script to push it out to environments living in each workspace. More moving parts, but far more flexible on the compute side.<\/li>\n<\/ul>\n\n\n\nLet’s take each one in turn.<\/p>\n\n\n\n
Architecture 1: A Single Shared Environment<\/h1>\n\n\n\n
This is the simpler of the two, and honestly it’s where I’d start most teams.<\/p>\n\n\n\n
You spin up a workspace that holds shareable items, and one of those items is a shareable environment (or a few of them, if you have different workload profiles). Because everything points back to one place, maintenance is painless – any update to the environment happens in a single spot, and every notebook connected to it immediately picks up the new Spark cluster, library, or resource.<\/p>\n\n\n\n
The trade-off is configurability. If you want a cluster defined independently while<\/em> reusing the same shared library, this isn’t the way to go – that’s exactly the gap the second architecture fills. But if you’re after simplicity and a single org-wide standard, this gets you there fast.<\/p>\n\n\n\n
Architecture with a single shared environment<\/strong>
\n![\"Pasted](\"https:\/\/dataguide.tech\/wp-content\/uploads\/2026\/07\/Pasted-image-20260525193245.png\")
<\/p>\n\n\n\nYou can see it in the diagram above: multiple notebooks all pointing at the same shared workspace, which holds the shared library the developers use. Wiring a notebook to it takes two clicks:<\/p>\n\n\n\n
- \n
- Open the notebook.<\/li>\n\n\n\n
- Expand the environment list and pick the shared environment. That’s it – your custom libraries from that environment are now available.<\/li>\n<\/ol>\n\n\n\n
![\"Pasted](\"https:\/\/dataguide.tech\/wp-content\/uploads\/2026\/07\/Pasted-image-20260531114732-1.png\")
<\/figure>\n\n\n\nThere’s a natural extension here. Instead of one shared environment, create a few – one per workload type. Small jobs get a lightweight Spark config, heavy loads get something beefier. The setup is identical; the only difference is that your shared workspace now holds more than one environment. Pick whichever one matches the job.<\/p>\n\n\n\n
Architecture 2: One Library, Pushed to Every Workspace<\/h1>\n\n\n\n
The second approach is the one I reach for on larger tenants. You still want reusable code sitting in one place, and you still want every workload running the latest version – but now you also want full control over each environment’s Spark engine and resources. A single shared environment can’t give you both.<\/p>\n\n\n\n
![\"Pasted](\"https:\/\/dataguide.tech\/wp-content\/uploads\/2026\/07\/Pasted-image-20260627135418-1.png\")
<\/figure>\n\n\n\nHere’s the idea. On a shared workspace you store the
.whl<\/code> library file (add versioning here if you like), plus a script that publishes it out to environments in other workspaces. You decide which environments are in scope. In my sample script I push it across the whole tenant – personal workspaces excluded – wherever I find an environment with aDE<\/code> suffix.<\/p>\n\n\n\nLet me walk through the setup.<\/p>\n\n\n\n
First, create the target environments in your workspaces using a consistent naming convention. I use a
DE<\/code> suffix. You can use your own rule, but heads up – the script matches on_DE<\/code>, so if you change the convention you’ll need to adjust it to match.<\/p>\n\n\n\n![\"Pasted](\"https:\/\/dataguide.tech\/wp-content\/uploads\/2026\/07\/Pasted-image-20260704140443-1.png\")
<\/figure>\n\n\n\nNext, create a notebook in your “shared tool” workspace. It can technically live anywhere, but I keep all reusable items in one workspace so the team always knows where to look. Once the notebook exists, attach your library to it as a built-in resource:<\/p>\n\n\n\n
Open the notebook \u2192 go to the Resources<\/strong> tab in the explorer \u2192 click the three dots next to Built-in<\/strong> \u2192 click Upload files<\/strong> \u2192 select your library.<\/p>\n\n\n\n
![\"Pasted](\"https:\/\/dataguide.tech\/wp-content\/uploads\/2026\/07\/Pasted-image-20260704140738-1.png\")
<\/figure>\n\n\n\nWith the prerequisites in place, here’s the script (This scripts exists as well in my GH Repo – Github<\/a><\/em>):<\/p>\n\n\n\n
Python<\/span>
- One shared library, many environments<\/strong> – you keep the