JBake is a static site/blog generator. I have been using jekyll for this blog but I would like to try something new. My main operating system is NixOS which does not have a package to install JBake. It will need to be added. I have already updated visualvm and notion for NixOS but adding a new package is something I haven’t tried yet.
nixpkgs workflow
Contributing to nixpkgs is a great workflow with the goal of having an easy to understand log. It is not a typical workflow and some basic understanding is required to contribute to it. nixpkgs expects only one commit when adding a new package. Normally I would add a commit message for any small change. With the nixpkgs workflow a local branch can be used to make small commits which can then be squashed on the master branch. There is also two upstream repositories that are used for development. One is used to develop from and the other is used for pull requests. Soon this will make more sense but for now lets setup the fork.
creating a fork, adding channels, nixpkgs to fork
Creating a fork is easy with GitHub. Click fork in the top-right corner of the page. More information can be found here.
Once the fork under your github account clone it to your local machine.
git clone https://github.com/moaxcp/nixpkgs.git
Next, add two remotes.
git remote add channels git://github.com/NixOS/nixpkgs-channels.git git remote add nixpkgs https://github.com/NixOS/nixpkgs.git
channels
are collections of packages for the nix package manager. It is useful when you want to reuse what is already built on your system. channels
will be used for the local branch to develop the package.
nixpkgs
is the remote the fork was made from. It is the upstream of this fork. It is used to update master of this fork and to submit pull requests.
syncing master with upstream when outdated
From time to time the master will be outdated and need to be synced with nixpkgs.
git fetch nixpkgs git checkout master git reset --hard nixpkgs/master git push origin master --force
This will help prevent conflicts when a pull request is made.
creating local branch for current commit of NixOS
The version of NixOS being used contains the git commit from channels
.
nix-version 17.09.2476.53e6d671a96 (Hummingbird)
53e6d671a96
should be used to create the local branch. This is a good practice. If you use a different commit you risk building all of NixOS just to test a few changes.
If your system is ever updated the local branch will need to be rebased with that commit.
git checkout local git rebase <NixOS commit>
It is also ok to delete the local branch if the pull request is already.
git branch -d local
adding JBake to local branch of nixpkgs
Within nixpkgs there is a large hierarchy of files. For JBake it makes sense to add it under development/tools/jbake/default.nix
. all-packages.nix
also needs to be changed to include JBake so it can be installed by users. JBake should be added to all-packages.nix
first so it can be found by nix-build
during the development process.
all-packages.nix
First, edit pkgs/top-level/all-packages.nix
and add the jbake
attribute.
jbake = callPackage ../development/tools/jbake { };
default.nix
Starting out we know a few things about JBake. Nix needs to download the package. And the name to use is jbake-2.5.1
. This is enought to start writing the expression.
Edit pkgs/development/tools/jbake/default.nix
and add the following:
{stdenv, fetchurl}: stdenv.mkDerivation { name = "jbake-2.5.1"; src = fetchurl { url = ""; sha256=""; }; }
The the attribute added to all-packages.nix
and default.nix
added we can attempt to build jbake
. Lets build this file and see what happens. Run nix-build from the root of repository.
nix-build -A jbake error: assertion failed at /etc/user/john/projects/nixpkgs/pkgs/build-support/fetchurl/default.nix:90:1 (use ‘--show-trace’ to show detailed location information)
This is probably because the file is missing a url. Set the url in default.nix
.
url = "http://jbake.org/files/jbake-2.5.1-bin.zip";
Notice that how the url also contains the name of the derivation. It would be nice to reuse the name value.
{stdenv, fetchurl}: stdenv.mkDerivation { name = "jbake-2.5.1"; src = fetchurl { url = "http://jbake.org/files/${name}-bin.zip"; sha256=""; }; }
Save the changes and run nix-build
again.
nix-build -A jbake error: undefined variable ‘name’ at /etc/user/john/projects/nixpkgs/pkgs/development/tools/jbake/default.nix:6:41
We get a different error…
In a nix expression attributes cannot be reused in a set unless the set is recursive. Add rec to the set:
{stdenv, fetchurl}: stdenv.mkDerivation rec { name = "jbake-2.5.1"; src = fetchurl { url = "http://jbake.org/files/${name}-bin.zip"; sha256=""; }; }
The name can now be reused within the expression. Run nix-build
again.
nix-build -A jbake error: Specify hash for fetchurl fixed-output derivation: http://jbake.org/files/jbake-2.5.1-bin.zip (use ‘--show-trace’ to show detailed location information)
This is a result of the missing hash. I don’t know the hash but using an invalid hash will give us the real hash to use. Add an invalid hash to the file and run nix-build
again.
sha256="1gkdkxssh51jczhgv680i42jjrlia1vbpcqhxvf45xcq9xj95bm5";
nix-build
will download the file but give the following error:
output path ‘/nix/store/4lj8xga9nxwc2my60z9pjh7r37arhxmx-jbake-2.5.1-bin.zip’ has sha256 hash ‘1r46y84q5x915055hx2vxydaqng3cz0clwz0yhwapgmi4sliygjd’ when ‘1gkdkxssh51jczhgv680i42jjrlia1vbpcqhxvf45xcq9xj95bm5’ was expected cannot build derivation ‘/nix/store/wnmw6yq876ga9anf59lxwjrc6nddwskx-jbake-2.5.1.drv’: 1 dependencies couldn't be built error: build of ‘/nix/store/wnmw6yq876ga9anf59lxwjrc6nddwskx-jbake-2.5.1.drv’ failed
This gives the hash to use. Set the hash in default.nix
.
sha256="1r46y84q5x915055hx2vxydaqng3cz0clwz0yhwapgmi4sliygjd";
Running nix-build again results in
nix-build -A jbake these derivations will be built: /nix/store/r0zvmmjc36wdssyhj1q1ja9lsqwqqlav-jbake-2.5.1.drv building path(s) ‘/nix/store/a5492adfv7sx9qrilgqf591s3pdqly0k-jbake-2.5.1’ unpacking sources unpacking source archive /nix/store/4xwch46y38rdca1zh3zx6z0kk2janiyr-jbake-2.5.1-bin.zip do not know how to unpack source archive /nix/store/4xwch46y38rdca1zh3zx6z0kk2janiyr-jbake-2.5.1-bin.zip builder for ‘/nix/store/r0zvmmjc36wdssyhj1q1ja9lsqwqqlav-jbake-2.5.1.drv’ failed with exit code 1 error: build of ‘/nix/store/r0zvmmjc36wdssyhj1q1ja9lsqwqqlav-jbake-2.5.1.drv’ failed
Which means stdenv
does not know how to unpack the downloaded file because it is a zip
rather than a tar.gz
. Looking back at JBake files there is no tar.gz
download. There are some examples of this in nixpkgs where a zip
file is downloaded and unzipped in the nix expression using unzip
. In these cases, unpackPhase
is changed, adding the unzip
command. Add the code to unzip
the file:
{stdenv, fetchurl, unzip}: stdenv.mkDerivation rec { name = "jbake-2.5.1"; src = fetchurl { url = "http://jbake.org/files/jbake-2.5.1-bin.zip"; sha256="1r46y84q5x915055hx2vxydaqng3cz0clwz0yhwapgmi4sliygjd"; }; buildInputs = [unzip]; unpackPhase = "unzip ${src}"; }
When this is run with nix-build
we get a new error.
... inflating: jbake-2.5.1/lib/thymeleaf-3.0.3.RELEASE.jar patching sources configuring no configure script, doing nothing building no Makefile, doing nothing installing install flags: install make: *** No rule to make target 'install'. Stop. builder for ‘/nix/store/wjhxbm803yagw32kz1hyfys0k8n3kdnq-jbake-2.5.1.drv’ failed with exit code 2 error: build of ‘/nix/store/wjhxbm803yagw32kz1hyfys0k8n3kdnq-jbake-2.5.1.drv’ failed
This means stdenv
was not able to build the package using make
. JBake is already built and we only need to install it. This can be done by changing the installPhase
.
{stdenv, fetchurl, unzip}: stdenv.mkDerivation rec { name = "jbake-2.5.1"; src = fetchurl { url = "http://jbake.org/files/${name}-bin.zip"; sha256="1r46y84q5x915055hx2vxydaqng3cz0clwz0yhwapgmi4sliygjd"; }; buildInputs = [unzip]; unpackPhase = "unzip ${src}"; installPhase = '' mkdir -p $out cp -r $name/* $out ''; }
stdenv
provides $out
as the directory to install the package into. It needs to be created and the files copied over. The zip
contains one directory called jbake-2.5.1
and all of the files in it. cp -r $name/* $out
will copy all of those files into $out
Now when we run nix-build we get
... inflating: jbake-2.5.1/lib/thymeleaf-3.0.3.RELEASE.jar patching sources configuring no configure script, doing nothing building no Makefile, doing nothing installing post-installation fixup shrinking RPATHs of ELF executables and libraries in /nix/store/1cqhcxd4d64r6bgr4ssvk1yi6h5jyx37-jbake-2.5.1 stripping (with flags -S) in /nix/store/1cqhcxd4d64r6bgr4ssvk1yi6h5jyx37-jbake-2.5.1/lib /nix/store/1cqhcxd4d64r6bgr4ssvk1yi6h5jyx37-jbake-2.5.1/bin patching script interpreter paths in /nix/store/1cqhcxd4d64r6bgr4ssvk1yi6h5jyx37-jbake-2.5.1 /nix/store/1cqhcxd4d64r6bgr4ssvk1yi6h5jyx37-jbake-2.5.1/bin/jbake: interpreter directive changed from "/bin/bash" to "/nix/store/hqi64wjn83nw4mnf9a5z9r4vmpl72j3r-bash-4.4-p12/bin/bash" checking for references to /tmp/nix-build-jbake-2.5.1.drv-0 in /nix/store/1cqhcxd4d64r6bgr4ssvk1yi6h5jyx37-jbake-2.5.1... /nix/store/1cqhcxd4d64r6bgr4ssvk1yi6h5jyx37-jbake-2.5.1
nix-build creates a link to $out called result
in the current directory for us.
ls result bin example_project_freemarker.zip example_project_groovy-mte.zip example_project_groovy.zip example_project_jade.zip example_project_thymeleaf.zip jbake-core.jar lib LICENSE
This looks correct. stdenv
will automatically add the bin
directory to PATH
when a user installs JBake. Executables in bin
need to be checked for anything that may not work. Viewing bin/jbake
shows the application uses java
. This means java
is a dependency. It needs to be installed by adding jre
to the expression.
{stdenv, fetchurl, unzip, jre}:
There is still a problem though. The executable may not be using the verison of java
installed by nix
. To fix this the executable needs to use the real path to the java
executable installed by the jre
dependency. stdenv
provides the function substituteInPlace
which will fix the jbake
script during the installPhase.
{stdenv, fetchurl, unzip, jre}: stdenv.mkDerivation rec { name = "jbake-2.5.1"; src = fetchurl { url = "http://jbake.org/files/${name}-bin.zip"; sha256="1r46y84q5x915055hx2vxydaqng3cz0clwz0yhwapgmi4sliygjd"; }; buildInputs = [unzip]; unpackPhase = "unzip ${src}"; installPhase = '' substituteInPlace $name/bin/jbake --replace "java" "${jre}/bin/java" mkdir -p $out cp -r $name/* $out ''; }
The executable now uses the java
dependency installed in the nix-store.
/nix/store/jgpc401wdfg1gibvnlp60ddav4fi68ys-openjdk-8u152b16-jre/bin/java -jar "${EXEC_PARENT}/jbake-core.jar" $@
The results look good and now meta
needs to be added to the file. meta
contains details about the package for users to view. It is also a way to add yourself as a maintainer.
First make sure you are in maintainers.nix
.
moaxcp = "John Mercier <moaxcp@gmail.com>";
The file is in alphabetical order by attribute name.
Then, add meta
to default.nix
.
This is the final version of default.nix
.
{stdenv, fetchurl, unzip, jre}: stdenv.mkDerivation rec { name = "jbake-2.5.1"; src = fetchurl { url = "http://jbake.org/files/${name}-bin.zip"; sha256="1r46y84q5x915055hx2vxydaqng3cz0clwz0yhwapgmi4sliygjd"; }; buildInputs = [unzip]; unpackPhase = "unzip ${src}"; installPhase = '' substituteInPlace $name/bin/jbake --replace "java" "${jre}/bin/java" mkdir -p $out cp -r $name/* $out ''; meta = with stdenv.lib; { description = "JBake is a Java based, open source, static site/blog generator for developers & designers"; homepage = "http://jbake.org/"; license = licenses.mit; maintainers = with maintainers; [ moaxcp ]; }; }
Make sure the file works with nix-build
and also install it with nix-env
.
From nixpkgs run.
nix-build nix-env -f . -iA jbake
squashing commits in local branch
Once the expression is ready for a pull request a single commit needs to be added to the master branch. If there were many commits during development they can be squashed. Squashing commits can be done following this guide. Make sure the commit message follows the guidelines for nixpkgs. The commit message for this change is.
jbake: init at 2.5.1
cherry-pick commits to master branch
View git log
and find the commit hash.
commit dc521d4ad609743008bfa31c0d681c8b4141e815 (HEAD -> local, origin/local) Author: John Mercier <moaxcp@gmail.com> Date: Tue Dec 26 20:06:19 2017 -0500 jbake: init at 2.5.1
Switch to master and cherry-pick the commit.
git cherry-pick dc521d4ad609743008bfa31c0d681c8b4141e815
create pull request to add jbake
Conclusion
Adding packages to nixpkgs can be fun. There is a lot of packages especially for java that should be added/maintained. I hope this helps in some way.