How do go modules work with installable commands?

Question

I\'ve recently started with Go 1.11 and love the modules. Apart from runtime dependencies I need to work with go modules during the build, e.g. during go generate

Answer 1

https://github.com/golang/go/issues/25922 proved helpful for me, especially

when using build-only dependencies with modules the main point is version selection (not installing these!)

To avoid installing you can modify your //go:generate directive to something like:

//go:generate go run golang.org/x/tools/cmd/stringer ARGS

There is also the best practices repo: https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md

Answer 2

If you get an error

I was not seeing the dependency that I wanted added to the go.mod and I was getting this error:

internal/tools/tools.go:6:5: import "github.com/UnnoTed/fileb0x" is a program, not an importable package

(fileb0x is the thing I'm trying to add)

I'm not 100% clear on the sequence of events that fixed it, but I did all of these things:

Using a "tools" package

I made a tools directory:

mkdir -p internal/tools

I put the tools package inside of it (as mentioned above):

internal/tools/tools.go:

// +build tools

package tools

import (
    _ "github.com/UnnoTed/fileb0x"
)

Note that the tag is mostly not important. You could use foo:

// +build foo

However, you cannot use ignore. That's a special predefined tag.

// +build ignore

// NO NO NO NO NO
// `ignore` is a special keyword which (surprise) will cause
// the file to be ignore, even for dependencies

Updating go.mod

The best way is probably to run go mod tidy:

go mod tidy

However, before I did that I ran a number of commands trying to figure out which one would cause it to go into go.mod:

go install github.com/UnnoTed/fileb0x # didn't seem to do the trick
go get
go generate ./...
go build ./...
go install ./...
go mod vendor

Later I did a git reset and rm -rf ~/go/pkg/mod; mkdir ~/go/pkg/mod and found that go mod tidy did well enough on its own.

vendoring

In order to actually take advantage of the modules cache in a project you need to copy-in the source code

go mod vendor

That will grab all dependencies from go.mod

You also need to change nearly all of your go commands to use -mod=vendor in any Makefiles, Dockerfiles or other scripts.

go fmt -mod=vendor ./... # has a bug slated to be fixed in go1.15
go generate -mod=vendor ./...
go build -mod=vendor ./...

That includes go build, go get, go install, and any go run called by go generate (and even the go generate itself)

//go:generate go run -mod=vendor github.com/UnnoTed/fileb0x b0x.toml
package main

// ...

Answer 3

tools.go is a great solution if you're building an app or service. But if you're building a library, tools.go still leaks dependencies to things consuming your library (your tools are still there as indirect dependencies, and go mod tidy will pull them in since it considers every possible target). That's not the end of the world since those modules never end up in the actual built binaries of the consumer, but it's still messy.

https://github.com/myitcv/gobin/issues/44 is probably the most promising approach to fixing this long term, but short term I've used a combination of the "internal module" approach explained there along with https://github.com/izumin5210/gex.

First, I install gex globally:

GO111MODULE=off go get github.com/izumin5210/gex/cmd/gex

Then before actually using gex I create a structure like this:

myproject/
\
  - go.mod: module github.com/ysamlan/myproject
  \
    internal/
    \
      tools/
       - go.mod: module github.com/ysamlan/myproject/tools

To install a build-only tool I just cd internal/tools and run gex --add (sometool), which puts that tool in internal/tools/bin. CI scripts and other folks that want to build my stuff locally just need to run cd internal/tools && gex --build to reliably and reproducibly populate the tool binaries, but the top-level go.mod is unchanged.

The key piece there is creating that internal/tools/go.mod file with a different module path than the one the root project uses, and then only running gex from that directory.

Answer 4

The convention is to add a file named "tools.go" that is guarded by a build constraint and imports all required tools:

// +build tools

package tools

import (
    _ "github.com/aprice/embed/cmd/embed"
)

https://github.com/golang/go/issues/25922#issuecomment-412992431

The tools are then installed as usual in one of

• $GOBIN
• $GOPATH/bin
• $HOME/go/bin

You may also want to follow https://github.com/golang/go/issues/27653, which discusses future explicit support for tools.