Packaging Hugo for Debian
I would like to package Hugo for Debian. However, Hugo depends on many Go libraries, about 27 of them in total, all of which have to be available as Debian packages in the form of golang-libraryname-dev. Fortunately, 5 of them have been packaged, but that still leaves 22 that I would have to build. As such, I would really like the whole process to be automated as much as possible, both to save time and to ensure consistency.
Notes/Requirements/Nice-to-Haves/TODOs
- All exist in Git or Hg repositories, i.e. need to study
git-buildpackageandmercurial-buildpackagewell. - Some/Most(?) do not have release tags.
- I need to keep them up-to-date at least semi-automatically.
- I will also need to ask the maintainers of the 5 existing Debian Go library packages to have them updated.
- I will likely use the same license as the upstream source for
debian/*files, e.g. Apache-2.0 rather than GPL-2. - Note dependencies of each library, and of Hugo.
- (Is there a way for
go getcommand or something similar to generate the dependency tree?)
- (Is there a way for
- Add bash-completion for Hugo
I will use /work/debian/hugo as my working directory.
github.com/spf13/hugo (download)
bitbucket.org/kardianos/osext (download)
bitbucket.org/pkg/inflect (download)
github.com/BurntSushi/toml (download)
github.com/PuerkitoBio/purell (download)
github.com/dchest/cssmin (download)
github.com/eknkc/amber (download)
github.com/gorilla/websocket (download)
github.com/howeyc/fsnotify (download)
github.com/russross/blackfriday (download)
github.com/shurcooL/go (download)
github.com/spf13/afero (download)
github.com/spf13/cast (download)
github.com/spf13/jwalterweatherman (download)
github.com/spf13/cobra (download)
github.com/spf13/pflag (download)
github.com/spf13/fsync (download)
github.com/spf13/viper (download)
github.com/kr/pretty (download)
github.com/kr/text (download)
github.com/mitchellh/mapstructure (download)
github.com/xordataexchange/crypt (download)
github.com/armon/consul-api (download)
github.com/coreos/go-etcd (download)
Fetching https://golang.org/x/crypto/openpgp?go-get=1
Parsing meta tags from https://golang.org/x/crypto/openpgp?go-get=1 (status code 200)
get "golang.org/x/crypto/openpgp": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/openpgp?go-get=1
get "golang.org/x/crypto/openpgp": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
golang.org/x/crypto (download)
Fetching https://gopkg.in/yaml.v1?go-get=1
Parsing meta tags from https://gopkg.in/yaml.v1?go-get=1 (status code 200)
get "gopkg.in/yaml.v1": found meta tag main.metaImport{Prefix:"gopkg.in/yaml.v1", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v1"} at https://gopkg.in/yaml.v1?go-get=1
gopkg.in/yaml.v1 (download)
github.com/yosssi/ace (download)
github.com/spf13/nitro (download)
- ☐ bitbucket.org/kardianos/osext (hg) → golang-osext-dev
- ☐ github.com/spf13/afero → golang-afero-dev
- ☐ github.com/spf13/jwalterweatherman → golang-jwalterweatherman-dev
- ☐ github.com/spf13/pflag → golang-pflag-dev
- ☐ github.com/PuerkitoBio/purell → golang-purell-dev
- ☐ github.com/spf13/cast → golang-cast-dev
- ☐ github.com/spf13/cobra → golang-cobra-dev
- ☐ github.com/shurcooL/go → golang-shurcool-go-dev
- ☐ github.com/spf13/fsync → golang-fsync-dev
- ☑ github.com/russross/blackfriday → golang-blackfriday-dev
- ☐ github.com/BurntSushi/toml → golang-toml-dev
- ☑ github.com/kr/text → golang-text-dev
- ☑ github.com/kr/pretty → golang-pretty-dev
- ☐ github.com/armon/consul-api → golang-consul-api-dev
- ☐ github.com/coreos/go-etcd → golang-go-etcd-dev
- ☑(O) golang.org/x/crypto → golang-go.crypto-dev
- ☐ github.com/xordataexchange/crypt → golang-crypt-dev or golang-xordataexchange-crypt-dev?
- ☐ gopkg.in/yaml.v1 → golang-yaml.v1-dev
- ☐ bitbucket.org/pkg/inflect (hg) → golang-inflect-dev
- ☐ github.com/dchest/cssmin → golang-cssmin-dev
- ☐ github.com/eknkc/amber → golang-amber-dev
- ☐ github.com/yosssi/ace → golang-ace-dev
- ☐ github.com/spf13/nitro → golang-nitro-dev
- ☑ github.com/gorilla/websocket → golang-websocket-dev
- ☐ github.com/mitchellh/mapstructure → golang-mapstructure-dev
- ☐ github.com/howeyc/fsnotify → golang-fsnotify-dev
- ☐ github.com/spf13/viper → golang-viper-dev
- ☐ github.com/spf13/hugo → hugo
Actually doing it…
DH_GOLANG_EXCLUDES to the rescue!
- https://bugs.debian.org/721496 Do not recurse into packages when building
* In debian/rules, export DH_GOLANG_EXCLUDES (containing a space-separated
list of perl regular expressions) to filter targets, e.g.:
export DH_GOLANG_EXCLUDES := github.com/coreos/etcd/third_party
ratt and sbuild
https://wiki.debian.org/sbuild
sbuild hangs
So when I have "UNRELEASED" as target distribution and call "sbuild", I
get this hang. The simple solutions is to indicate the desired
distribution: sbuild -d unstable
FAILED?
foka@debian-Aspire-M5-481G:/work/debian/go/stretchr$ ratt golang-testify_1.0-1_amd64.changes
2015/09/25 17:49:46 Loading changes file "golang-testify_1.0-1_amd64.changes"
2015/09/25 17:49:46 - 1 binary packages: golang-testify-dev
2015/09/25 17:49:46 - corresponding .debs (will be injected when building):
2015/09/25 17:49:46 golang-testify-dev_1.0-1_all.deb
2015/09/25 17:49:46 Loading sources index "/var/lib/apt/lists/ftp.us.debian.org_debian_dists_sid_contrib_source_Sources"
2015/09/25 17:49:46 Loading sources index "/var/lib/apt/lists/ftp.us.debian.org_debian_dists_sid_main_source_Sources"
2015/09/25 17:49:49 Loading sources index "/var/lib/apt/lists/ftp.us.debian.org_debian_dists_sid_non-free_source_Sources"
2015/09/25 17:49:49 Building etcd_2.2.0+dfsg-2 (commandline: [sbuild --arch-all --dist=unstable --nolog etcd_2.2.0+dfsg-2 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 17:58:25 Building golang-github-kimor79-gollectd_0.0~git20150616-1 (commandline: [sbuild --arch-all --dist=unstable --nolog golang-github-kimor79-gollectd_0.0~git20150616-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 17:59:46 building golang-github-kimor79-gollectd_0.0~git20150616-1 failed: exit status 3
2015/09/25 17:59:46 Building golang-github-mesos-mesos-go_0.0~git20150903.0.d98afa6-1 (commandline: [sbuild --arch-all --dist=unstable --nolog golang-github-mesos-mesos-go_0.0~git20150903.0.d98afa6-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:03:24 Building golang-github-samalba-dockerclient_0.0~git20150905.0.77b723e-1 (commandline: [sbuild --arch-all --dist=unstable --nolog golang-github-samalba-dockerclient_0.0~git20150905.0.77b723e-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:04:36 building golang-github-samalba-dockerclient_0.0~git20150905.0.77b723e-1 failed: exit status 3
2015/09/25 18:04:36 Building golang-github-spf13-cast_0.0~git20150803.0.ee815aa-1 (commandline: [sbuild --arch-all --dist=unstable --nolog golang-github-spf13-cast_0.0~git20150803.0.ee815aa-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:05:46 building golang-github-spf13-cast_0.0~git20150803.0.ee815aa-1 failed: exit status 3
2015/09/25 18:05:46 Building golang-github-spf13-jwalterweatherman_0.0~git20141219.0.3d60171-1 (commandline: [sbuild --arch-all --dist=unstable --nolog golang-github-spf13-jwalterweatherman_0.0~git20141219.0.3d60171-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:08:26 Building golang-logrus_0.7.3-1 (commandline: [sbuild --arch-all --dist=unstable --nolog golang-logrus_0.7.3-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:11:04 building golang-logrus_0.7.3-1 failed: exit status 2
2015/09/25 18:11:04 Building docker-libkv_0.0~git20150829.0.2954a88-1 (commandline: [sbuild --arch-all --dist=unstable --nolog docker-libkv_0.0~git20150829.0.2954a88-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:13:42 Build results:
2015/09/25 18:13:42 PASSED: golang-github-mesos-mesos-go_0.0~git20150903.0.d98afa6-1
2015/09/25 18:13:42 PASSED: golang-github-spf13-jwalterweatherman_0.0~git20141219.0.3d60171-1
2015/09/25 18:13:42 PASSED: docker-libkv_0.0~git20150829.0.2954a88-1
2015/09/25 18:13:42 PASSED: etcd_2.2.0+dfsg-2
2015/09/25 18:13:42 FAILED: golang-github-samalba-dockerclient_0.0~git20150905.0.77b723e-1 (see buildlogs/golang-github-samalba-dockerclient_0.0~git20150905.0.77b723e-1)
2015/09/25 18:13:42 FAILED: golang-github-spf13-cast_0.0~git20150803.0.ee815aa-1 (see buildlogs/golang-github-spf13-cast_0.0~git20150803.0.ee815aa-1)
2015/09/25 18:13:42 FAILED: golang-logrus_0.7.3-1 (see buildlogs/golang-logrus_0.7.3-1)
2015/09/25 18:13:42 FAILED: golang-github-kimor79-gollectd_0.0~git20150616-1 (see buildlogs/golang-github-kimor79-gollectd_0.0~git20150616-1)
foka@debian-Aspire-M5-481G:/work/debian/go/stretchr$
Well, let’s try again!
foka@debian-Aspire-M5-481G:/work/debian/go/stretchr$ ratt golang-testify_1.0-1_amd64.changes
2015/09/25 18:16:52 Loading changes file "golang-testify_1.0-1_amd64.changes"
2015/09/25 18:16:52 - 1 binary packages: golang-testify-dev
2015/09/25 18:16:52 - corresponding .debs (will be injected when building):
2015/09/25 18:16:52 golang-testify-dev_1.0-1_all.deb
2015/09/25 18:16:52 Loading sources index "/var/lib/apt/lists/ftp.us.debian.org_debian_dists_sid_contrib_source_Sources"
2015/09/25 18:16:52 Loading sources index "/var/lib/apt/lists/ftp.us.debian.org_debian_dists_sid_main_source_Sources"
2015/09/25 18:16:55 Loading sources index "/var/lib/apt/lists/ftp.us.debian.org_debian_dists_sid_non-free_source_Sources"
2015/09/25 18:16:55 Building golang-github-spf13-jwalterweatherman_0.0~git20141219.0.3d60171-1 (commandline: [sbuild --arch-all --dist=unstable --nolog golang-github-spf13-jwalterweatherman_0.0~git20141219.0.3d60171-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:19:18 Building golang-logrus_0.7.3-1 (commandline: [sbuild --arch-all --dist=unstable --nolog golang-logrus_0.7.3-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:21:40 building golang-logrus_0.7.3-1 failed: exit status 2
2015/09/25 18:21:40 Building docker-libkv_0.0~git20150829.0.2954a88-1 (commandline: [sbuild --arch-all --dist=unstable --nolog docker-libkv_0.0~git20150829.0.2954a88-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:24:14 Building etcd_2.2.0+dfsg-2 (commandline: [sbuild --arch-all --dist=unstable --nolog etcd_2.2.0+dfsg-2 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:32:13 Building golang-github-kimor79-gollectd_0.0~git20150616-1 (commandline: [sbuild --arch-all --dist=unstable --nolog golang-github-kimor79-gollectd_0.0~git20150616-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:33:21 building golang-github-kimor79-gollectd_0.0~git20150616-1 failed: exit status 3
2015/09/25 18:33:21 Building golang-github-mesos-mesos-go_0.0~git20150903.0.d98afa6-1 (commandline: [sbuild --arch-all --dist=unstable --nolog golang-github-mesos-mesos-go_0.0~git20150903.0.d98afa6-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:36:24 Building golang-github-samalba-dockerclient_0.0~git20150905.0.77b723e-1 (commandline: [sbuild --arch-all --dist=unstable --nolog golang-github-samalba-dockerclient_0.0~git20150905.0.77b723e-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:37:34 building golang-github-samalba-dockerclient_0.0~git20150905.0.77b723e-1 failed: exit status 3
2015/09/25 18:37:34 Building golang-github-spf13-cast_0.0~git20150803.0.ee815aa-1 (commandline: [sbuild --arch-all --dist=unstable --nolog golang-github-spf13-cast_0.0~git20150803.0.ee815aa-1 --extra-package=golang-testify-dev_1.0-1_all.deb])
2015/09/25 18:38:40 building golang-github-spf13-cast_0.0~git20150803.0.ee815aa-1 failed: exit status 3
2015/09/25 18:38:40 Build results:
2015/09/25 18:38:40 PASSED: docker-libkv_0.0~git20150829.0.2954a88-1
2015/09/25 18:38:40 PASSED: etcd_2.2.0+dfsg-2
2015/09/25 18:38:40 PASSED: golang-github-mesos-mesos-go_0.0~git20150903.0.d98afa6-1
2015/09/25 18:38:40 PASSED: golang-github-spf13-jwalterweatherman_0.0~git20141219.0.3d60171-1
2015/09/25 18:38:40 FAILED: golang-github-samalba-dockerclient_0.0~git20150905.0.77b723e-1 (see buildlogs/golang-github-samalba-dockerclient_0.0~git20150905.0.77b723e-1)
2015/09/25 18:38:40 FAILED: golang-github-spf13-cast_0.0~git20150803.0.ee815aa-1 (see buildlogs/golang-github-spf13-cast_0.0~git20150803.0.ee815aa-1)
2015/09/25 18:38:40 FAILED: golang-logrus_0.7.3-1 (see buildlogs/golang-logrus_0.7.3-1)
2015/09/25 18:38:40 FAILED: golang-github-kimor79-gollectd_0.0~git20150616-1 (see buildlogs/golang-github-kimor79-gollectd_0.0~git20150616-1)
foka@debian-Aspire-M5-481G:/work/debian/go/stretchr$
Post-mortem analysis
So, for the 8 packages that depends on golang-testify-dev, 4 passed and 4 failed.
Why did the four packages fail?
golang-github-spf13-casthasBuild-Depends: golang-github-stretchr-testify-dev | golang-testify-devgolang-github-kimor79-gollectd: Ditto.golang-github-samalba-dockerclient: Depends: golang-github-docker-docker-dev but it is not installable Depends: golang-github-stretchr-testify-dev but it is not installable
golang-github-spf13-jwalterweatherman passed because it has golang-testify-dev listed first, i.e.
Build-Depends: ..., golang-testify-dev | golang-github-stretchr-testify-dev
Why?
and the sbuild(1) man page has this to say:
- --build-dep-resolver=resolver
Use the specified resolver to handle selecting the build dependencies. Supported resolvers are apt (the default), aptitude and xapt. The apt resolver is the most appropriate resolver for most users, for building for unstable, stable and other distributions. If alternative build dependencies are used x (excluding architecture restrictions), only the first alternative will be used; the others will be ignored. The aptitude resolver is very similar, but smarter and slower, and it will consider all alternatives by default; it is suited to more complex situations, such as building packages for the experimental distribution, where packages need installing from multiple suites (unstable and experimental). Due to performance and other issues (bug #139615), aptitude is not recommended for use by default. The xapt resolver is intended only for cross-building, and is a temporary transitional feature which will be removed following the complete introduction of multi-arch support.
- --resolve-alternatives
Allow the use of alternatives in Build-Depends, Build-Depends-Arch and Build-Depends-Indep. This is the default for the aptitude dependency resolver.
- --no-resolve-alternatives
Do not allow the use of alternatives in Build-Depends, Build-Depends-Arch and Build-Depends-Indep. Note that alternatives for the same package (e.g. different versions) are still allowed. This is the default for the apt and xapt dependency resolvers.
So, there you have it! Let’s add --resolve-alternatives to ratt’s source
code and see what happens!
…
In order to get…
diff --git a/ratt.go b/ratt.go
index f270d3f..427b52e 100644
--- a/ratt.go
+++ b/ratt.go
@@ -154,11 +154,19 @@ func main() {
sort.Sort(sort.Reverse(version.Slice(versions)))
newest := versions[0]
target := fmt.Sprintf("%s_%s", src, newest)
+ // aptGetInstall := fmt.Sprintf("--chroot-setup-commands=apt-get -y --no-install-recommends install %s", strings.Join(changes.Binaries, " "))
+ aptGetInstall := fmt.Sprintf("--starting-build-commands=sudo apt-get -y --no-install-recommends install %s", strings.Join(changes.Binaries, " "))
// TODO: discard resulting package immediately?
args := []string{
"--arch-all",
- "--dist=sid",
+ "--dist=unstable",
"--nolog",
+ //"--build-dep-resolver=apt",
+ "--resolve-alternatives",
+ //"--chroot-setup-commands=apt-get update",
+ "--chroot-setup-commands=apt-get update",
+ "--chroot-setup-commands=apt-get install sudo",
+ aptGetInstall,
target,
}
for _, filename := range debs {
And this:
--- /usr/share/perl5/Sbuild/AptResolver.pm.orig 2015-03-24 07:40:45.000000000 -0600
+++ /usr/share/perl5/Sbuild/AptResolver.pm 2015-09-26 09:13:57.182058363 -0600
@@ -105,7 +105,8 @@
}
}
- $self->cleanup_apt_archive();
+ # Commented out by Anthony Fok for github.com/Debian/ratt on 2015-09-26
+ #$self->cleanup_apt_archive();
return $status;
}
And…
- Built-Using is wrong… It cannot handle alternatives, leaving `golang-testify-dev (= )… Not ratt or sbuild’s fault, but some dpkg tools (substvars?) To do.
Accepted into Debian on… January 11, 2016, see https://tracker.debian.org/pkg/hugo