docspell/website/site/content/docs/dev/development.md

+++ title = "Tips & Setup" weight = 20 +++

Setup / Tips

Starting Servers with reStart

When developing, it's very convenient to use the revolver sbt plugin. Start the sbt console and then run:

sbt:docspell-root> restserver/reStart

This starts a REST server. Once this started up, type:

sbt:docspell-root> joex/reStart

if also a joex component is required. Prefixing the commads with ~, results in recompile+restart once a source file is modified.

It is possible to start both in the root project:

sbt:docspell-root> reStart

Custom config file

The sbt build is setup such that a file dev.conf in the directory local (at root of the source tree) is picked up as config file, if it exists. So you can create a custom config file for development. For example, a custom database for development may be setup this way:

#jdbcurl = "jdbc:h2:///home/dev/workspace/projects/docspell/local/docspell-demo.db;MODE=PostgreSQL;DATABASE_TO_LOWER=TRUE;AUTO_SERVER=TRUE"
jdbcurl = "jdbc:postgresql://localhost:5432/docspelldev"
#jdbcurl = "jdbc:mariadb://localhost:3306/docspelldev"

docspell.server {
  backend {
    jdbc {
      url = ${jdbcurl}
      user = "dev"
      password = "dev"
    }
  }
}

docspell.joex {
  jdbc {
    url = ${jdbcurl}
    user = "dev"
    password = "dev"
  }
  scheduler {
    pool-size = 1
  }
}

Developing Frontend

The frontend is a SPA written in Elm. The UI framework in use is tailwind.

The frontend code is in the sub-project webapp. Running sbt's compile task, compiles elm sources and creates the final CSS file. Whenever the restserver module is build by sbt, the webapp sub-project is built as well and the final files to deliver are updated. So, when in sbt shell, "watch-compile" the project restserver, (via ~ restserver/compile), re-compiles elm-code on change. However, it also re-creates the final css, which is a rather long task.

To speed things up when only developing the frontend, a bash script is provided in project/dev-ui-build.sh. Start the restserver once, using restserver/reStart task as described above. Then run this script in the source root. It will watch elm files and the css file and re-compiles only on change writing the resulting files in the correct locations so they get picked up by the restserver.

Now you can edit elm files and the index.css and then only refresh the page. Elm compilation is very fast, it's difficult to reach the refresh button before it is done compiling :). When editing the CSS, it takes a little longer, but this is hardly necessary, thanks to tailwind.

There is still a problem: the browser caches the js and css files by default, so a page refresh is not enough, you need to clear the cache, too. To avoid this annoyance, set a env variable DOCSPELL_ENV to the value dev. Docspell then adds a response header, preventing the browser to cache these files. This must be done, obviously, before starting the restserver:

$ export DOCSPELL_ENV=dev
$ sbt "restserver/reStart"

Developing Backend

OpenAPI

The http API is specified in the corresponding -openapi.yml file. The component section is being used to generate code for the client and the server, so that both are always in sync. However, the route definitions are not checked against the server implementation.

Changes to the openapi files can be checked by running a sbt task:

restapi/openapiLint //and/or
joexapi/openapiLint

These tasks must not show any errors (it is checked by the CI). The warnings should also be fixed.

Nix Expressions

The directory /nix contains Nix Flake to install docspell via the nix package manager and to integrate it into NixOS.

Flake implements checks output which can be run with nix flake check and it defines a development VM which can be used to interactively work with docspell.

To run the VM, issue:

cd $PROJECT_ROOT/nix
nix run '.#nixosConfigurations.dev-vm.config.system.build.vm

To open docspell, wait for docspell-restserver service to report that http listener is up and connect to localhost:64080.

To ssh into the machine, run:

ssh -o StrictHostKeyChecking=no \
    -o UserKnownHostsFile=/dev/null \
    -p 64022 root@localhost

Once connected to the machine, you can see the docspell config file via

systemd-show docspell-joex.service | grep ExecStart | cut -d'=' -f2 | xargs cat | tail -n1 | awk '{print $NF}'| sed 's/.$//' | xargs cat | jq

Release

The CI and making a release is done via github actions. The workflow is roughly like this:

• each PR is only merged if the sbt ci task returns successfully. This is ensured by the ci.yml workflow that triggers on each pull request
• each commit to the master branch is also going through sbt ci and then a prerelease is created. The tag nightly is used to point to the latest commit in master. Note, that this is discouraged by git, but github doesn't allow to create a release without a tag. So this tag moves (and is not really a tag then…). After the prerelease is created, the docker images are built and pushed to docker hub into the docspell organization. The docker images are also tagged with nightly at docker hub. This is all done via the realease-nightly.yml workflow.
• A stable release is started by pushing a tag with pattern v* to github. This triggers the release.yml workflow which builds the packages and creates a release in draft mode. The sbt ci task is not run, because it is meant to only release commits already in the master branch. After this completes, the release notes need to be added manually and then the release must be published at github. This then triggers the docker-images.yml workflow, which builds the corresponding docker images and pushes them to docker hub. The docker images are tagged with the exact version and the latest tag is moved to the new images. Another manual step is to set the branch current-docs to its new state and push it to github. This will trigger a build+publish of the website.
• Publishing the website happens automatically on each push to the branch current-docs. Changes to the current website must be based on this branch.

Some notes: I wanted a 2/3 step process when doing a stable release, to be able to add release notes manually (I don't want this to be automated right now) and to do some testing with the packages before publishing the release. However, for the nightly releases, this doesn't matter - everything must be automated here obviously. I also wanted the docker images to be built from the exact same artifacts that have been released at github (in contrast to being built again).