Initial website

website/site/content/docs/tools/_index.md

@@ -0,0 +1,11 @@
++++
+title = "Tools"
+description = "There are several tools distributed with docspell, like a program to watch a folder and import files to docspell."
+weight = 60
+insert_anchor_links = "right"
+template = "pages.html"
+redirect_to = "docs/tools/ds"
+sort_by = "weight"
+[extra]
+mktoc = false
++++

website/site/content/docs/tools/browserext.md

@@ -0,0 +1,80 @@
++++
+title = "Browser Extension (Firefox)"
+description = "An extension for firefox to upload files from your browser via right-click → upload to docspell."
+weight = 30
++++
+
+The idea is to click on a file in firefox and send it to docspell. It
+is downloaded in the context of your current page. Then handed to an
+application that pushes it to docspell. There is a browser add-on
+implementing this in `tools/webextension`. This add-on only works with
+firefox.
+
+Installation is a bit complicated, since you need to install external
+tools and the web extension. Both work together.
+
+# Install `ds.sh`
+
+First copy the `ds.sh` tool somewhere in your `PATH`, maybe
+`/usr/local/bin` as described above.
+
+
+# Install the native part
+
+Then install the "native" part of the web extension:
+
+Copy or symlink the `native.py` script into some known location. For
+example:
+
+``` bash
+ln -s ~/docspell-checkout/tools/webextension/native/native.py /usr/local/share/docspell/native.py
+```
+
+Then copy the `app_manifest.json` to
+`$HOME/.mozilla/native-messaging-hosts/docspell.json`. For example:
+
+``` bash
+cp ~/docspell-checkout/tools/webextension/native/app_manifest.json  ~/.mozilla/native-messaging-hosts/docspell.json
+```
+
+See
+[here](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Native_manifests#Manifest_location)
+for details.
+
+And you might want to modify this json file, so the path to the
+`native.py` script is correct (it must be absolute).
+
+If the `ds.sh` script is in your `$PATH`, then this should
+work. Otherwise, edit the `native.py` script and change the path to
+the tool. Or create a file `$HOME/.config/docspell/ds.cmd` whose
+content is the path to the `ds.sh` script.
+
+
+# Install the extension
+
+An extension file can be build using the `make-xpi.sh` script. But
+installing it in "standard" firefox won't work, because [Mozilla
+requires extensions to be signed by
+them](https://wiki.mozilla.org/Add-ons/Extension_Signing). This means
+creating an account and going through some process…. So here are two
+alternatives:
+
+1. Open firefox and type `about:debugging` in the addressbar. Then
+   click on *'Load Temporary Add-on...'* and select the
+   `manifest.json` file. The extension is now installed. The downside
+   is, that the extension will be removed once firefox is closed.
+2. Use Firefox ESR, which allows to install Add-ons not signed by
+   Mozilla. But it has to be configured: Open firefox and type
+   `about:config` in the address bar. Search for key
+   `xpinstall.signatures.required` and set it to `false`. This is
+   described on the last paragraph on [this
+   page](https://support.mozilla.org/en-US/kb/add-on-signing-in-firefox).
+
+When you right click on a file link, there should be a context menu
+entry *'Docspell Upload Helper'*. The add-on will download this file
+using the browser and then send the file path to the `native.py`
+script. This script will in turn call `ds.sh` which finally uploads it
+to your configured URLs.
+
+Open the Add-ons page (`Ctrl`+`Shift`+`A`), the new add-on should be
+there.

website/site/content/docs/tools/consumedir.md

@@ -0,0 +1,140 @@
++++
+title = "Consume Directory"
+description = "A script to watch a directory for new files and upload them to docspell."
+weight = 20
++++
+
+The `consumerdir.sh` is a bash script that works in two modes:
+
+- Go through all files in given directories (recursively, if `-r` is
+  specified) and sent each to docspell.
+- Watch one or more directories for new files and upload them to
+  docspell.
+
+It can watch or go through one or more directories. Files can be
+uploaded to multiple urls.
+
+Run the script with the `-h` or `--help` option, to see a short help
+text. The help text will also show the values for any given option.
+
+The script requires `curl` for uploading. It requires the
+`inotifywait` command if directories should be watched for new
+files.
+
+Example for watching two directories:
+
+``` bash
+./tools/consumedir.sh --path ~/Downloads --path ~/pdfs -m -dv http://localhost:7880/api/v1/open/upload/item/5DxhjkvWf9S-CkWqF3Kr892-WgoCspFWDo7-XBykwCyAUxQ
+```
+
+The script by default watches the given directories. If the `-o` or
+`--once` option is used, it will instead go through these directories
+and upload all files in there.
+
+Example for uploading all immediatly (the same as above only with `-o`
+added):
+
+``` bash
+$ consumedir.sh -o --path ~/Downloads --path ~/pdfs/ -m -dv http://localhost:7880/api/v1/open/upload/item/5DxhjkvWf9S-CkWqF3Kr892-WgoCspFWDo7-XBykwCyAUxQ
+```
+
+
+The URL can be any docspell url that accepts uploads without
+authentication. This is usually a [source
+url](@/docs/webapp/uploading.md#anonymous-upload). It is also possible
+to use the script with the [integration
+endpoint](@/docs/webapp/uploading.md#integration-endpoint).
+
+
+## Integration Endpoint
+
+When given the `-i` or `--integration` option, the script changes its
+behaviour slightly to work with the [integration
+endpoint](@/docs/webapp/uploading.md#integration-endpoint).
+
+First, if `-i` is given, it implies `-r` – so the directories are
+watched or traversed recursively. The script then assumes that there
+is a subfolder with the collective name. Files must not be placed
+directly into a folder given by `-p`, but below a sub-directory that
+matches a collective name. In order to know for which collective the
+file is, the script uses the first subfolder.
+
+If the endpoint is protected, these credentials can be specified as
+arguments `--iuser` and `--iheader`, respectively. The format is for
+both `<name>:<value>`, so the username cannot contain a colon
+character (but the password can).
+
+Example:
+``` bash
+$ consumedir.sh -i -iheader 'Docspell-Integration:test123' -m -p ~/Downloads/ http://localhost:7880/api/v1/open/integration/item
+```
+
+The url is the integration endpoint url without the collective, as
+this is amended by the script.
+
+This watches the folder `~/Downloads`. If a file is placed in this
+folder directly, say `~/Downloads/test.pdf` the upload will fail,
+because the collective cannot be determined. Create a subfolder below
+`~/Downloads` with the name of a collective, for example
+`~/Downloads/family` and place files somewhere below this `family`
+subfolder, like `~/Downloads/family/test.pdf`.
+
+
+## Duplicates
+
+With the `-m` option, the script will not upload files that already
+exist at docspell. For this the `sha256sum` command is required.
+
+So you can move and rename files in those folders without worring
+about duplicates. This allows to keep your files organized using the
+file-system and have them mirrored into docspell as well.
+
+
+## Systemd
+
+The script can be used with systemd to run as a service. This is an
+example unit file:
+
+``` systemd
+[Unit]
+After=networking.target
+Description=Docspell Consumedir
+
+[Service]
+Environment="PATH=/set/a/path"
+
+ExecStart=/bin/su -s /bin/bash someuser -c "consumedir.sh --path '/a/path/' -m 'http://localhost:7880/api/v1/open/upload/item/5DxhjkvWf9S-CkWqF3Kr892-WgoCspFWDo7-XBykwCyAUxQ'"
+```
+
+This unit file is just an example, it needs some fiddling. It assumes
+an existing user `someuser` that is used to run this service. The url
+`http://localhost:7880/api/v1/open/upload/...` is an anonymous upload
+url as described [here](@/docs/webapp/uploading.md#anonymous-upload).
+
+
+## Docker
+
+The provided docker image runs this script to watch a single
+directory, `./docs` in current directory, for new files. If a new file
+is detected, it is pushed to docspell.
+
+This utilizes the [integration
+endpoint](@/docs/webapp/uploading.md#integration-endpoint), which is
+enabled in the config file, to allow uploading documents for all
+collectives. A subfolder must be created for each registered
+collective. The docker containers are configured to use http-header
+protection for the integration endpoint. This requires you to provide
+a secret, that is shared between the rest-server and the
+`consumedir.sh` script. This can be done by defining an environment
+variable which gets picked up by the containers defined in
+`docker-compose.yml`:
+
+``` bash
+export DOCSPELL_HEADER_VALUE="my-secret"
+docker-compose up
+```
+
+
+Now you can create a folder `./docs/<collective-name>` and place all
+files in there that you want to import. Once dropped in this folder
+the `consumedir` container will push it to docspell.

website/site/content/docs/tools/ds.md

@@ -0,0 +1,44 @@
++++
+title = "Upload CLI"
+description = "A script to quickly upload files from the command line."
+weight = 10
++++
+
+A bash script to quickly upload files from the command line. It reads
+a configuration file containing the URLs to upload to. Then each file
+given to the script will be uploaded to al URLs in the config.
+
+The config file is expected in
+`$XDG_CONFIG_HOME/docspell/ds.conf`. `$XDG_CONFIG_HOME` defaults to
+`~/.config`.
+
+The config file contains lines with key-value pairs, separated by a
+`=` sign. Lines starting with `#` are ignored. Example:
+
+```
+# Config file
+url.1 = http://localhost:7880/api/v1/open/upload/item/5DxhjkvWf9S-CkWqF3Kr892-WgoCspFWDo7-XBykwCyAUxQ
+url.2 = http://localhost:7880/api/v1/open/upload/item/6DxhjkvWf9S-CkWqF3Kr892-WgoCspFWDo7-XBykwCyAUxQ
+```
+
+The key must start with `url`. The urls should be [anonymous upload
+urls](@/docs/webapp/uploading.md#anonymous-upload).
+
+
+## Usage
+
+- The `-c` option allows to specifiy a different config file.
+- The `-h` option shows a help overview.
+- The `-d` option deletes files after upload was successful
+- The `-e` option can be used to check for file existence in docspell.
+  Instead of uploading, the script only checks whether the file is in
+  docspell or not.
+
+The script takes a list of files as arguments.
+
+
+Example:
+
+``` bash
+./ds.sh ~/Downloads/*.pdf
+```

website/site/content/docs/tools/smtpgateway.md

@@ -0,0 +1,190 @@
++++
+title = "SMTP Gateway with Exim"
+description = "Start a SMTP server that forwards all mails to docspell."
+weight = 40
++++
+
+One possible use case for the [integration
+endpoint](@/docs/webapp/uploading.md#integration-endpoint) is a SMTP
+server that forwards all local mail to docspell. This way there is no
+periodic polling involved and documents (e-mails) get into docspell
+without delay.
+
+The `tools/exim` folder contains a docker file and a sample
+`exim.conf` to help start with this setup. Note that these files
+provide a minimal setup, you might want to add tls and spam protection
+when opening it to the public.
+
+
+## What you need
+
+You need to own a domain and add the appropriate MX records to point
+to your server. In this document, the domain `test.org` is used.
+
+You need to enable the [integration
+endpoint](@/docs/webapp/uploading.md#integration-endpoint) in the
+docspell configuration.
+
+## Exim
+
+[Exim](http://exim.org/) is a popular smtp server (message transfer
+agent). It is used here only because of previous knowledge, but same
+can be achieved with other MTAs.
+
+
+## The Config File
+
+Here is the example config file for exim:
+
+{{ incl_conf(path="templates/shortcodes/sample-exim.conf") }}
+
+
+Exim has good [documentation](https://www.exim.org/docs.html), look
+there for more info. The following is only a quick summary of the file
+above.
+
+The `domainlist local_domains` should list your domain. Only mails to
+this domain are allowed, as specified in the first rule in
+`acl_check_rcpt`. So mails to `name@test.org` are ok, but
+`name@someother.org` not.
+
+Another rule in `acl_check_rcpt` executes a `GET` request against the
+integration endpoint. If that fails, the recipient is wrong (or the
+endpoint disabled) and the mail is rejected right away.
+
+Then the `routers` define how a mail is handled. There is only one
+router that accepts all mails (that have not been rejected by a rule
+in acls) and uses the `docspell` transport to deliver it. The
+transport specifies a command via the `pipe` driver that is run with
+the mail. The mail itself is provided via stdin. So a simple `curl`
+command can upload it to the integration endpoint. Here are some quick
+notes about the used options (see `man curl`):
+
+- `--silent` and `--out /dev/null` don't print upload progress
+  information and no output to stdout
+- `--fail` return non-zero if http status code is not success
+- `-F` use a multipart/form-data request (defaults to a POST request)
+- `"file=@-;filename=\"$_subject:\""` add one part with name `file`
+  and take the data from stdin (`@-`). Since there is no filename, we
+  use the subject of the mail. This is [supported by
+  exim](http://exim.org/exim-html-current/doc/html/spec_html/ch-string_expansions.html)
+  by expanding the subject mail header via `$h_subject:` (the colon is
+  required).
+- `$local_part` this is expanded by exim to the recipient address,
+  only the part until the `@` sign.
+- `${env{DS_HEADER}{$value} fail}` looks up an environment variable by
+  key `DS_HEADER`. This is usually defined in `docker-compose.yml`.
+  The value must be the "secret" header value as defined in docspell's
+  configuration file.
+- `${env{DS_URL}{$value} fail}` the url to docspell. It is looked up
+  from the environment with key `DS_URL`, which is usually defined in
+  `docker-compose.yml`. Adding the `$local_part` at the end means that
+  mails to `somename@test.org` are uploaded to the collective
+  `somename`.
+
+
+## Install with Docker
+
+Go into the `tools/exim` directory and build the docker image:
+
+``` bash
+docker build -t ds-exim:latest -f exim.dockerfile .
+```
+
+Then start docspell somewhere and configure the integration endpoint
+to use http-header protection; i.e. set this in the config file:
+
+``` conf
+docspell.server {
+  integration-endpoint {
+    enabled = true
+    http-header = {
+      enabled = true
+      header-value = "test123"
+    }
+  }
+}
+```
+
+Then edit the `docker-compose.yml` and change the environment
+variables as needed.
+
+Finally start the container:
+
+``` bash
+docker-compose up
+```
+
+
+## Test Run
+
+Now it is possible to send mails to this MTA which will be immediatly
+uploaded to docspell for the collective corresponding to the
+`$local_part` of the recipients address. Here is a quick telnet
+session (the collective is named `family`):
+
+```
+~> telnet localhost 25
+Trying ::1...
+Connected to localhost.
+Escape character is '^]'.
+220 test.org ESMTP Exim 4.93 Sun, 14 Jun 2020 19:03:51 +0000
+ehlo localhost
+250-test.org Hello localhost [::1]
+250-SIZE 31457280
+250-8BITMIME
+250-PIPELINING
+250-CHUNKING
+250 HELP
+mail from:<me@test.org>
+250 OK
+rcpt to:<family@test.org>
+250 Accepted
+data
+354 Enter message, ending with "." on a line by itself
+From: me@test.org
+To: family@test.org
+Subject: This is a test
+
+Test,
+
+this is just a test mail.
+.
+250 OK id=1jkXwf-000007-0d
+quit
+221 test.org closing connection
+Connection closed by foreign host.
+~>
+```
+
+The mail is processed and results in an item:
+
+{{ figure(file="exim-mail.png") }}
+
+However, if a mail is to an unknown collective or not to the
+configured local domain, the server rejects it immediately:
+
+``` bash
+~> telnet localhost 25
+Trying ::1...
+Connected to localhost.
+Escape character is '^]'.
+220 test.org ESMTP Exim 4.93 Sun, 14 Jun 2020 19:07:04 +0000
+ehlo localhost
+250-test.org Hello localhost [::1]
+250-SIZE 31457280
+250-8BITMIME
+250-PIPELINING
+250-CHUNKING
+250 HELP
+mail from:<me@test.org>
+250 OK
+rcpt to:<family22@test.org>
+550 Recipient unknown
+rcpt to:<family@gmail.com>
+550 Administrative prohibition
+quit
+221 test.org closing connection
+Connection closed by foreign host.
+~>
+```