Setting up nix development enviroments

No, you spend all your time bike-shedding your setup
nix
linux
Published

June 30, 2024

Introduction

For the past little while now I’ve been changing how I manage my dotfiles and development environments to use nix. I haven’t gone full bore and switched to NixOS itself, but otherwise I’m pretty into it. This post will go over what problems I was trying to solve by switching, share some resources I found helpful, and document what I did to get my python environment working the way I like, because it took a ton of time and also does the best job of showing some of the different ways you can do things in nix.

What problem I’m trying to solve

Previously I was deploying my dotfiles with dotbot and setting up my development environment with devcontainers, specifically through devpod. Nothing against either of these projects, they’re great at what they’re for, but I did have some issues that left me wondering if there was a better solution.

First, while my dotfiles contained all the config for my apps, they didn’t have any capability to check for and fix installation of those apps if they weren’t present. So for example I could have my starship config in my dotfiles, but that wouldn’t do me any good if the environment I was developing in didn’t have it. For actual hosts I could take care of this with ansible, and for devcontainers I built a base image that had the tools I liked that I would then build language or project specific containers on top of. This worked, but it felt disjointed. Ansible feels heavy to just install a couple packages, especially when you have to deal with either setting up connectivity from an existing host to a WSL system or whatever, or you have to bootstrap installing ansible on the machine to run a playbook against a local host. For the devcontainers it meant that whenever I updated my base image I then had to rebuild and repull any child images based on it, which meant if I was working on a project and realized there was a cool tool I wanted to add to my base image I was either going into my devcontainer repo, making changes, pushing up, rebuilding the base image, rebuilding the child image I needed, pulling it down into my environment and resuming what I was actually working on, or I just hacked something into the devcontainer that was lost the next time I restarted and then I had to go through the process above.

Regarding devcontainers, they’re an awesome idea, and I think maybe if my dev environments involved a much larger stack, like if I was standing up a cache and a database and some other tooling along with whatever I was developing then being able to orchestrate all that with docker would appeal. What I generally actually want out of devcontainers though is just for some tools to be available. I can definitely get this from a devcontainer, but at the cost of a lot more isolation than I actually want. Any config files I want to mount in I have to add, ssh-agent seems to hate forwarding into devcontainers for me (I’m pretty sure this is because I have some weird stack with WSL but still), and configuring a Dockerfile is a bit of a hassle. If I realize I need something else in the container do I carefully rewrite the file to keep my layers small and related actions close together? Or do I jam it in at the end of the file so I don’t have to rebuild all those layers I already made so I can get back to work quickly?

To address these issues I decided to try nix. Home Manager lets me bundle common apps along with their config, and then nix-shell lets me declaratively define a development environment that I can bundle (along with pinned dependencies) with a project. It took a fair bit of work and reading to get my head around the tooling, but in the end I’m quite pleased with my setup.

Resources that were helpful

For the most part I don’t have anything original to add to this discussion. I’m not going to try and reiterate a worse version of the tutorials and guides I followed to configure my setup, that doesn’t seem helpful. Instead in this section I’ll list the links that were most useful for me when trying to figure this out. In the next section I will go through all the things I tried to set up a python environment, since I didn’t see anything that explored all the options I tried while I was searching, so it might actually be useful to others.

  • zero to nix. Great guide. Covers the basics and gets you set up with an environment where you can actually test other things out. Highly recommend starting here.
  • nix from first principles: flake edition Much more comprehensive. Very solid description of nix, how it works, some of the patterns it uses, and some nice example development environments. I referred back to this the most.
  • vimjoyer channel. Lots of good videos going over things in nix. Includes NixOS specific stuff that isn’t currently relevant to me and video format makes it harder to review, but a very nice way to get an overview of what’s possible in nix. Watched a lot of these while doing dishes.

Python environment

For the most part after going through the guides above I was able to get what I wanted working fairly easily. I would have to ask ChatGPT about some particular syntax, or refer to the home-manager appendices to figure out a specific configuration, but as I mentioned above I don’t think I have a lot of wisdom to contribute for those scenarios. Read the guides above, bang your head against the code a bit until it starts making sense and you’ll be good to go.

The exception to this is python environments. I did a lot of messing around with setting up my environment and I think describing what I tried will be helpful to others.

In the interest of not burying the lede for those who just want to know what worked I built an environment for each version of python I wanted to test against using poetry2nix.

You can see what I’m currently doing by checking out my stats_can project.

I resisted this approach at first because I was thinking of nix in terms of docker as something that should just give me a python runtime, poetry, and nox and then get out of my way. It turns out there are problems with that approach.

Log of what I tried before finally getting things working

I wanted to be able to just tell nix to give me an environment with a python runtime of my choice, poetry, and nox (not to be confused with nix, just an unfortunately very similarly named project). I think I probably could have gotten this working if my projects only used pure python dependencies, but stuff with c bindings like numpy and pandas had all sorts of issues since the c libraries of the nix packages didn’t match my OS, and installing c libraries into nix didn’t fix the fact that the… links? were wrong? To be honest I was pretty confused by all this. Suffice to say, it didn’t work. I also tried out devenv and devshell which are built on top of nix and intended to make things easier, but in the case of python environments I found they ended up adding complexity. The section below is a record of the things I tried, only loosely edited. I’m keeping it in for reference and also so maybe someone else trying these things will find it in google and be able to skip to the setup I actually got working.

First official attempt

I did a lot of fumbling around trying to just drop some flakes off the internet into existing python projects, but that quickly turned out to be too complicated for testing.

Instead I started a fresh git repo with the intent of making a minimum viable package and then extending from there as required.

I’ll start with the development environment described in nix from first principles. It’s fairly understandable, gives me multiple python versions, and will have a different version of python as the default than I’m using in poetry so I can test that.

It looks like this to start:


{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";

  outputs = { self, nixpkgs }:
    let
      pkgs = import nixpkgs { system = "x86_64-linux"; };
      # A list of shell names and their Python versions
      pythonVersions = {
        python38 = pkgs.python38;
        python39 = pkgs.python39;
        python310 = pkgs.python310;
        default = pkgs.python310;
      };
      # A function to make a shell with a python version
      makePythonShell = shellName: pythonPackage:
        pkgs.mkShell {
          # You could add extra packages you need here too
          packages = [ pythonPackage ];
          # You can also add commands that run on shell startup with shellHook
          shellHook = ''
            echo "Now entering ${shellName} environment."
          '';
        };
    in {
      # mapAttrs runs the given function (makePythonShell) against every value
      # in the attribute set (pythonVersions) and returns a new set
      devShells.x86_64-linux = builtins.mapAttrs makePythonShell pythonVersions;
    };
}

After staging this in my repo I run nix develop to load the default python 3.10 shell. Well, first off, I’m not in my default shell anymore, no fish, no starship. That’s annoying, but I think there’s a fix around that, or maybe I swap this over to devenv later since it handles it well. Put that on the todo for now. It loads pretty fast, I can start up fish and get all my good stuff back, and it’s got me a vanilla python. Not much more to test here to be honest. I’m not trying to add other packages currently so let’s call this step one and see what we can extend it to.

Try just sticking poetry in

Still not getting fancy with devenv or poetry2nix. Let’s try the suggestion in this post. I’m honestly not super optimistic this will work since it has this lovely warning:

Using LD_LIBRARY_PATH may lead to weird errors if the glibc version of the shell doesn’t match the one of the system. For a devshell that uses <nixpkgs> it shouldn’t be an issue, but otherwise I’d recommend using nix-ld.

As briefly mentioned above I can’t use nix-ld since I’m not doing this on NixOS. Well, we can’t know how bad it is until we try.

Let’s see if I can just put the diff in here and have it look reasonable:

       makePythonShell = shellName: pythonPackage:
         pkgs.mkShell {
           # You could add extra packages you need here too
-          packages = [ pythonPackage ];
+          packages = [ pythonPackage pkgs.poetry ];
           # You can also add commands that run on shell startup with shellHook
           shellHook = ''
             echo "Now entering ${shellName} environment."
           '';
+          env = {
+            LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath [ pkgs.stdenv.cc.cc ];
+            POETRY_VIRTUALENVS_IN_PROJECT = "true";
+            POETRY_VIRTUALENVS_PATH = "{project-dir}/.venv";
+            POETRY_VIRTUALENVS_PREFER_ACTIVE_PYTHON = "true";
+          };
         };
     in {
       # mapAttrs runs the given function (makePythonShell) against every value

That’s not so bad. Gives a little context.

Ok, that got so far as installing poetry and I created a project. Let’s try installing some stuff.

Add a basic dependency with poetry add requests and then a dev dependency with poetry add -G ruff.

Running python by default does not see either of these libraries. I guess that’s fair, usually with poetry you have to run poetry shell to activate the environment. Running that gives me some issues with my shell:

Spawning shell within /home/ipreston/nixpy/.venv
Unsupported use of '='. In fish, please use 'set VIRTUAL_ENV '/home/ipreston/nixpy/.venv''.
VIRTUAL_ENV='/home/ipreston/nixpy/.venv'

Ok, I can probably figure out my way around that, although I’m starting to really understand the appeal of using devenv where they’ve figured all this out.

It looks like this happens because I start my nix shell in bash, but when I run poetry shell it wants to activate in fish, but since I’m in bash it uses the bash activate script from the venv? That doesn’t really make sense to me but if I go into my fish shell by running fish and then run poetry shell it seems to work. Weird.

Running poetry install then ran into an error because I hadn’t actually created a package so I made a quick stub README and a src folder with a hello world function.

After doing that I was able to import requests successfully and run ruff on my stub code.

Basics are working, let’s try and deal with weird c dependencies now.

Ok sure, but how about with pandas?

‘poetry add pandas’, ‘python’, ‘import pandas as pd’, ’print(pd.__version__)’ all work? Cool I guess, but why is this working when so many of my other attempts failed miserably? Should I just be happy? I mean this works pretty well.

Work on different python versions

One thing I noticed is that although I specified my python package to be verion 3.10 and I set the environment variable to prefer the active python, when I enter poetry shell and call python I end up with 3.11, which I assume is what’s bundled with poetry. Why would that be? I do need the ability to specify my python versions so I can test my library against new and old releases.

From the docs it seems like the setting I created should have worked for this.

To double check I hadn’t done something stupid on the initial install I blew away the virtual environment with rm -rf .venv, confirmed that I was in a nix shell with python 3.10 as the active python and then ran poetry install. Same deal, outside the venv I’m on python 3.10, as soon as I activate it I’m in 3.11.

Devenv seemed to handle this really nicely. I wonder if I can couple that LD_LIBRARY_PATH argument and have the best of both worlds?

Try to bring in devenv again

I convert my flake back to a devenv config I was using in earlier testing.

The changes look like this:


diff --git a/flake.nix b/flake.nix
index 4fc6451..d3c26c2 100644
--- a/flake.nix
+++ b/flake.nix
@@ -1,31 +1,56 @@
 {
-  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
+  inputs = {
+    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+    nixpkgs-python.url = "github:cachix/nixpkgs-python";
+    nixpkgs-python.inputs = { nixpkgs.follows = "nixpkgs"; };
+    devenv.url = "github:cachix/devenv";
+  };
 
-  outputs = { self, nixpkgs }:
+  nixConfig = {
+    extra-trusted-public-keys =
+      "devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw=";
+    extra-substituters = "https://devenv.cachix.org";
+  };
+
+  outputs = { self, nixpkgs, devenv, ... }@inputs:
     let
-      pkgs = import nixpkgs { system = "x86_64-linux"; };
+      system = "x86_64-linux";
+      pkgs = import nixpkgs { system = system; };
       # A list of shell names and their Python versions
       pythonVersions = {
-        python38 = pkgs.python38;
-        python39 = pkgs.python39;
-        python310 = pkgs.python310;
-        default = pkgs.python310;
+        python39 = "3.9";
+        python310 = "3.10";
+        python311 = "3.11";
+        default = "3.10";
       };
       # A function to make a shell with a python version
-      makePythonShell = shellName: pythonPackage:
-        pkgs.mkShell {
-          # You could add extra packages you need here too
-          packages = [ pythonPackage pkgs.poetry ];
-          # You can also add commands that run on shell startup with shellHook
-          shellHook = ''
-            echo "Now entering ${shellName} environment."
-          '';
-          env = {
-            LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath [ pkgs.stdenv.cc.cc ];
-            POETRY_VIRTUALENVS_IN_PROJECT = "true";
-            POETRY_VIRTUALENVS_PATH = "{project-dir}/.venv";
-            POETRY_VIRTUALENVS_PREFER_ACTIVE_PYTHON = "true";
-          };
+      makePythonShell = shellName: pythonVersion:
+        devenv.lib.mkShell {
+          inherit inputs pkgs;
+          modules = [
+            ({ pkgs, config, ... }: {
+              languages.python = {
+                version = pythonVersion;
+                enable = true;
+                venv.enable = true;
+                poetry = {
+                  enable = true;
+                  activate.enable = true;
+                  package = pkgs.poetry;
+                  install = {
+                    enable = true;
+                    # compile = true;
+                    installRootPackage = true;
+                  };
+                };
+              };
+              env = {
+                LD_LIBRARY_PATH =
+                  pkgs.lib.makeLibraryPath [ pkgs.stdenv.cc.cc ];
+              };
+
+            })
+          ];
         };
     in {
       # mapAttrs runs the given function (makePythonShell) against every value

The LD_LIBRARY_PATH environment variable is still set and works. When I try and bring up python it does use the 3.10 version I set to be default. However, now I can’t import numpy:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.10 from "/home/ipreston/nixpy/.venv/bin/python"
  * The NumPy version is: "1.26.4"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libz.so.1: cannot open shared object file: No such file or directory

So we’re back to this?

Figure out where to go next

With devenv I get a lot of convenience functions and proper support for specifying a python version. On the other hand I can’t import numpy, and my environment activation is sloooooow. Do I hack my older config until it has the nice parts of devenv but not the bad parts? Do I hack devenv to fix my library linking issues? Do I try another tool altogether?

Try fixing devenv

There’s a lot of good stuff in devenv, I feel like my least disruptive path is if I can get it working, so let’s try that first. This issue sounds basically like the one I’m having so let’s see if I can get that working. I’ll comment out the LD_LIBRARY_PATH stuff and add libraries = with pkgs; [zlib]; to the flake and try again.

That doesn’t initially work, but the issue does mention needing to blow things away and retry so let’s delete my flake.lock, poetry.lock, .venv and .devenv, give the system a reboot and give nix develop --impure another shot.

It works! Let’s see if I can work with other versions of python now. Leaving that shell I run nix develop --impure .#python311 and… get the same import error on numpy. Ok, how about going back to my default shell? Now it doesn’t work there either. What happened?

Try and figure out switching python versions

I can’t really blow away my whole environment and rebuild every time I want to test a different python, let’s see if I can narrow down the issue.

Doing just a restart is not sufficient, so I guess it’s not some environment variable that’s being reset. Let’s try deleting .devenv and see what that does? Still nothing. I notice that I’m actually calling nix develop --impure .#python310, which should be the same as my default, but for completeness let’s see if that makes a difference? Nope.

Ok, one thing at a time is going to be slow, let’s attack this from the other direction and make sure I can reproduce making things work again.

I came back to these notes after a break where I was busy on other things. On the initial run just to make sure I knew where I was at I could import numpy but not pandas. Pandas gave an error about ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

When I switched to python 3.11 I was back to my numpy import error from above, and reverting back to the default I could no longer import numpy. So something in the activation of the 3.11 environment broke my 3.10 one. Even if I fix that I’ll still have to fix pandas.

I do think fixing this is worth looking into though, if for no other reason than it might help me understand nix better.

Now I’m jumping back and forth between environments and it’s only consistently giving the pandas error. On the one hand, hurray, problem solved. On the other, what?

And now I came back to it and found myself with the numpy error again.

I don’t think this is the way to go. There’s too much jankiness.

Detour into devcontainers

At this point I was getting really frustrated. It seemed like the mutability of what I wanted to do with python just wasn’t a good fit for nix. So I took a long detour back into devcontainers.

I did manage to get devcontainers working with DevPod. But between fighting to get my ssh keys passed in, to random failures at build, plus super long wait times I (not so) quickly remembered all the reasons I was trying to get away from devcontainers. Even though I found a nix devcontainer feature that allowed me to pass in my home-manager install for dotfile configs into the devcontainer, which was sweet, at the end I decided that I was fighting just as hard to make the devcontainer work as I had been nix, so maybe I should just go back. On a quick sidebar, I did switch out my shell from fish to zsh. I even tried bash again with ble.sh installed on top but it was too slow if I wanted the syntax highlighting. Zsh seems more common than fish, is posix compliant, and I was able to fairly easily get my autocomplete and syntax highlighting preferences configured.

Back to poetry with poetry2nix

I’ve resisted learning poetry2nix. It seemed like a lot of complication, and I was worried about its ability to handle packages that might not be in the nix library. But after fighting with c bindings a bunch in the previous efforts I think I might have to just suck it up and learn. After copying over the default flake I was basically immediately able to access a working python environment and import numpy as well as my dev dependencies. I now have to figure out how to convert this to handle multiple python versions. There are docs on that, it’s just a matter of figuring out enough nix syntax to know how to do it.

Let’s start with the template:


{
  description = "Application packaged using poetry2nix";

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable-small";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, flake-utils, poetry2nix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        # see https://github.com/nix-community/poetry2nix/tree/master#api for more functions and examples.
        pkgs = nixpkgs.legacyPackages.${system};
        inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; })
          mkPoetryApplication;
      in {
        packages = {
          myapp = mkPoetryApplication { projectDir = self; };
          default = self.packages.${system}.myapp;
        };

        # Shell for app dependencies.
        #
        #     nix develop
        #
        # Use this shell for developing your app.
        devShells.default =
          pkgs.mkShell { inputsFrom = [ self.packages.${system}.myapp ]; };

        # Shell for poetry.
        #
        #     nix develop .#poetry
        #
        # Use this shell for changes to pyproject.toml and poetry.lock.
        devShells.poetry = pkgs.mkShell { packages = [ pkgs.poetry ]; };
      });
}

This works, I get a nix shell with all my libraries available and I’m able to import numpy. But I don’t really understand how it works and I need to be able to extend it to work with multiple versions of python, so let’s figure that stuff out.

The first thing I haven’t really used yet is this flake-utils.lib.echDefaultSystem thing from the docs it’s just a way to ensure you get environments built for mac, linux etc. I don’t really need it for my case but no harm keeping it. It’s wrapping well around everything I’d care about so I’ll just leave it on the outside.

Next we have this big long inherit statement, which is just bringing the mkPoetryApplication function into scope for me. That makes sense, there’s a few other functions there like mkPoetryEnv that I might want to add but for now it’s ok.

At the top of the in block we have the packages attribute, which defines what will get built if I run nix build. I’m not really trying to use nix as my package builder right now, although maybe I will be by the end of this. This is where my poetry application requirements get defined though, so I’ll have to modify this part later to make things work. projectDir = self is just saying the poetry project is in the same directory as the flake, which is fine.

The devShells.default is pretty straightforward, it’s making a development shell with the argument to take the build requirements from the package defined earlier in the flake. I don’t think this is actually the way I want to package things in my case, since the mkPoetryEnv function has options to specify the python version and some other things. I’m also confused how this knows to bring in my development dependency for ruff. According to the docs the default for groups is empty. Maybe it’s installing it because it’s in dev and that’s included in checkGroups? Is running tests considered part of the build by default? That kind of makes sense. I guess it must be.

Finally the poetry devshell is where I run poetry commands to add, remove, or otherwise manage dependencies. That’s ok for now. I’ll get some more practice working with that if I can get these other pieces working.

As a start I’m just going to try replacing devShells.default with a call to mkPoetryEnv. Not really trying to change any outcomes, just seeing if I can work off that:

devShells.default = mkPoetryEnv {
  projectDir = self;
  python = pkgs.python3;
  groups = [ "dev" ];
};

If I can get this working then I can use some of the function wrapping I did above to make multiple python development environments pretty easily (I think at least).

A quick run of nix develop -c zsh ran almost instantly and took me into an environment, but I couldn’t actually run python or do anything else. That’s weird, what did I actually accomplish?

Well, nothing. When I look into this a bit more I need to pass the output of mkPoetryEnv as an input to pkgs.mkShell under buildInputs. Let’s try this again as follows:

let
  # see https://github.com/nix-community/poetry2nix/tree/master#api for more functions and examples.
  pkgs = nixpkgs.legacyPackages.${system};
  inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; })
    mkPoetryApplication mkPoetryEnv;
  myappEnv = mkPoetryEnv {
    projectDir = self;
    python = pkgs.python3;
    groups = [ "dev" ];
  };
in {
  # Use this shell for developing your app.
  devShells.default = pkgs.mkShell { buildInputs = [ myappEnv ]; };

Ok, this worked. We haven’t accomplished anything yet, but I have a better foundation for implementing the changes I want. Let’s make some additional changes and see if it still works. The first thing is just going to be adding the argument preferWheels = true; to my env. Building from source worked fine but it was sloooow. It only has to happen once so it’s not the end of the world if I need to do this, but it would be cool to be able to skip it. That seemed to work and my build was quicker so that’s solid. Next let’s try a different python version. I’m still not parameterizing it, which is my long term goal, but let’s just see if we can make it work at all. All I have to change at this point is setting python = pkgs.python310; and re-running. Again it builds quickly (thanks wheels!) and I find myself in a python 3.10 interpreter and able to import all my dependencies. We’re looking pretty good! Now let’s get cocky and try and reproduce the multiple python versions thing I had going on before.

I got it working but I had to do a few sneaky things to make it work so let’s look at the new code and then discuss:

      let
        # see https://github.com/nix-community/poetry2nix/tree/master#api for more functions and examples.
        pkgs = nixpkgs.legacyPackages.${system};
        inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; })
          mkPoetryApplication mkPoetryEnv;
        pythonVersions = {
          python310 = pkgs.python310;
          python311 = pkgs.python311;
          default = pkgs.python310;
        };
        makePoetryEnvPyVer = pythonPackage:
          mkPoetryEnv {
            projectDir = self;
            python = pythonPackage;
            preferWheels = true;
            groups = [ "dev" ];
          };
        makePythonShell = shellName: pythonPackage:
          pkgs.mkShell {
            buildInputs = [ (makePoetryEnvPyVer pythonPackage) ];
          };
        mappedDevShells = builtins.mapAttrs makePythonShell pythonVersions;
        moreDevShells = {
          poetry = pkgs.mkShell { packages = [ pkgs.poetry ]; };
        };
      in { devShells = mappedDevShells // moreDevShells; });
}

Ok, so the pythonVersions set is the same as pre poetry2nix, just a list of python versions mapped to the name of the devShell I want to use them in. python 3.10 is listed twice since it’s the default and can be called explicitly.

I couldn’t figure out how to just dump the mkPoetryEnv call inside the makePythonShell function. When I tried I got errors about extra ; or error: Dependency is not of a valid type: element 1 of buildInputs for nix-shell if I didn’t put a ; at the end of my mkPoetryEnv{}. After consulting chat GPT it suggested the issue might relate to the context of self within this function. So instead I make a function that creates a poetry env with an argument for the python version to use, and then call taht from my makePythonShell function. This works fine.

The next issue I had was that I couldn’t just call the builtins.mapAttrs method to make most of my devShells and then inject devShells.poetry for the poetry environment after. Instead I define maps for all the mapped devShells and the poetry shell in my let statement and then combine them to make the full devShells map in the in statement. Another shout out to ChatGPT for helping me figure that out.

Now that this is all done I can quickly and painlessly enter a development environment running the version of python I want, or enter a poetry environment to update or otherwise manage my dependencies. Hurray!

Test this on a real project

This all worked well on a toy project, but how about the actual library I want to maintain? I copied the flake over to my stats_can project and tried developing. The dependencies all loaded, but I couldn’t actually import my library. Switching back to the toy I realized I couldn’t do it there either, it just hadn’t really come up so I hadn’t thought to test it. Adding the following line to my mkPoetryEnv call fixed it: editablePackageSources = { my-app = ./src;.

After that I tried to run my tests. I got an error about pytest-vcr conflicting with pytest-recording. I realized I did have both specified in my project and that pytest-vcr should have been removed. I’m not sure why my old setup worked with that, but it was easy enough to correct.

I also had some hassles with my cassettes from testing, but that was probably a one off from converting. After that the tests ran ok.

How about CI/CD?

At this point it looks like my project is working, but now I’ve got one approach to testing locally and a different one when I run CI/CD. I happen to know from experience that that’s a nightmare scenario and that I should keep my CI/CD as close as possible to my local dev environment.

There was a nice blog I remember reading on this subject, let’s try it out.

This actually worked great. No notes. I have a Makefile in my project that for tests spins up each python version’s nix shell and runs pytest. This works locally, and after adding the nix setup actions to my CI/CD pipeline I can call it just the same from GitHub actions. This is a surprise bonus for this approach, much more consistent runs between local and CI/CD.

Conclusion

Nix has a pretty solid learning curve. Do not expect to pick it up in an hour. Especially if your development environment is complex/involves python. With that said, I’m super happy with my environment now and think the time I spent was well worth it. It will probably be a couple years at least before I officially break even on time saved with this approach from not having to install packages or rebuild containers, but I’m still glad I did it.