I recently introduced a new function in Nixpkgs named buildGoModule. The new function allows you to package Go application (or modules) with ease, provided the modules have added support for Go modules upstream.

Go, starting at version 1.11, has shipped with an internal solution for vendoring dependencies named: Go Modules. This feature provides a pseudo-lockfile-based implementation for vendoring dependencies, with one guarentee: Reproducible build. This means that all the dependencies are guarenteed to be the same.

This new infrastructure takes advantage of this reproducibility approach to fetch all the dependencies in a special fetcher, validate them against a fixed known hash, and finally use them to build the package.

This happens in two phases:

1. An intermediate fetcher derivation. This derivation will be used to fetch all the dependencies of the Go module. The checksum of the dependencies will be compared against the modSha256 attribute that is passed in.
2. A final derivation will use the output of the intermediate derivation to build the binaries and produce the final output.

The following derivation demonstrate how to package pet:

{ buildGoModule, fetchFromGitHub, lib }:

buildGoModule rec {
  pname = "pet";
  version = "0.3.4";

  src = fetchFromGitHub {
    owner = "knqyf263";
    repo = "pet";
    rev = "v${version}";
    sha256 = "0m2fzpqxk7hrbxsgqplkg7h2p7gv6s1miymv3gvw0cz039skag0s";
  };

  modSha256 = "1879j77k96684wi554rkjxydrj8g3hpp0kvxz03sd8dmwr3lh83j";

  meta = with lib; {
    description = "Simple command-line snippet manager, written in Go";
    homepage = https://github.com/knqyf263/pet;
    license = licenses.mit;
    maintainers = with maintainers; [ kalbasit ];
    platforms = platforms.linux ++ platforms.darwin;
  };
}

Notice that we did not have to specify deps nor goPackagePath for this to work, as with the legacy buildGoPackage.

How does it work

The first phase is referred to as go-modules and the second is referred to as package. The former prepares the dependencies white the latter uses the dependencies in order to compile the final output.

go-modules

The go-modules derivation starts by setting up the environment necessary to allow Go to have network access (even from within the sandbox). It does so by setting the attributes outputHashAlgo and outputHash as is the case with any other fetcher. At the end of the derivation, the hash computed according to outputHashAlgo will be expected to match the hash found in outputHash with is coming from the modSha256 attribute.

The derivation would also set up GOPATH and the GOCACHE in a temporary writable location.

During the build phase, the src will be unpacked and patched with patches, if any. The derivation then executes the command go mod download from within the root directory of the module. This command will download all the dependencies recursively and store them in $GOPATH/pkg/mod.

During the install phase, it takes the entire $GOPATH/pkg/mod/cache/download as the output $out. The hash verification will now be performed before returning $out to the caller.

One might wonder why does the derivation only use $GOPATH/pkg/mod/cache/download instead of the entire $GOPATH/pkg/mod directory. This is because the directory $GOPATH/pkg/mod/cache/vcs is impure as it reflects the repositories upstream as they were seen at the time of the cloning. Checkout the thread over on golang-nuts for more information.

package

This derivation will start similarly to the above one, by setting up the environment in order to have Go find all the necessary modules without network access. The lack of network access is very important here, and it allows us to build the module within an off-line sandbox. This step relies on setting GOPROXY environment variable to the output of the go-modules derivation we’ve built earlier. Go will use this information to locate the repositories of all the dependencies and construct the entire $GOPATH/pkg/mod.

export GOPROXY=file://${go-modules}

When to use buildGoModule?

You should use buildGoModule to build any package containing the files go.mod and go.sum at the root of src. Otherwise, you must use the legacy buildGoPackage infrastructure as the dependencies cannot be reproduced.