TheAnachronism/docspell
1
0
Fork 0
mirror of https://github.com/TheAnachronism/docspell.git synced 2025-06-21 18:08:25 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add documentation for the cli tool

Browse Source
This commit is contained in:
eikek
2021-07-19 01:39:00 +02:00
parent 16ade6934a
commit b01cb7715e
18 changed files with 383 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
website/site/content/docs/api/upload.md
View File

@ -94,8 +94,8 @@ specified via a JSON structure in a part with name `meta`:
# Endpoints
Docspell needs to the collective that owns the files. There are the
following ways for this.
Docspell needs to know the collective that owns the files. There are
the following ways for this.
## Authenticated User
@ -160,5 +160,6 @@ checksum with:
/api/v1/open/integration/checkfile/[collective-name]/[sha256-checksum]
```
See the [SMTP gateway](@/docs/tools/smtpgateway.md) or the [consumedir
script](@/docs/tools/consumedir.md) for examples to use this endpoint.
See the [SMTP gateway](@/docs/tools/smtpgateway.md) or the [dsc
watch/upload](@/docs/tools/cli.md#docker) command for example can use
this endpoint.

1
website/site/content/docs/features/_index.md
View File

@ -87,3 +87,4 @@ considering docspell at the moment.
- Documents cannot be modified.
- You can remove and add documents but there is no versioning.
- There are no user/groups nor permission management

50
website/site/content/docs/install/docker.md
View File

@ -18,15 +18,16 @@ There are images for all components that are available from the github
release page. The images contain all the necessary
[prerequisites](@/docs/install/prereq.md).
- `docspell-restserver` this images contains the http server
- `docspell-joex` this image contains the job executor and all
- `docspell/restserver` this images contains the http server
- `docspell/joex` this image contains the job executor and all
required software (ocrmypdf, unoconv etc) mentioned in
[prerequisites](@/docs/install/prereq.md).
- `docspell-tools` this image simply contains all the scripts from the
`tools/` folder. They are installed into a location in `$PATH`. It
doesn't specify a `CMD` or `ENTRYPOINT`, so you must choose which
script to run. The scripts are all prefixed by `ds-`. So to run the
`consumedir.sh` script, execute `ds-consumedir`.
- `docspell/dsc` this is an image containing a
[cli](@/docs/tools/cli.md) for docspell that can be used to watch
directories for new files. It doesn't specify a `CMD` or
`ENTRYPOINT`, so you must specify the exact command to run. Here, it
is used to watch a directory for uploading files. This runs the `dsc
watch` command.
### Examples
@ -90,20 +91,20 @@ After this `docker ps` should show these two containers. Go to
When signing up, use the same name for collective and user and then
login with this name.
For the last part, we use the `docspell/tools` image to create another
For the last part, we use the `docspell/dsc` image to create another
container that watches a directory and pushes files to docspell.
``` bash
$ docker run -d --name ds-consume \
--network dsnet --ip 10.4.3.4 \
-v /tmp/inbox:/var/inbox \
docspell/tools:latest ds-consumedir -imdv --iheader "Docspell-Integration:test123" \
--path /var/inbox "http://10.4.3.3:7880/api/v1/open/integration/item"
docspell/dsc:latest dsc -v -d http://10.4.3.3:7880 watch -r --delete -i \
--header "Docspell-Integration:test123" /var/inbox
```
This starts the [consumedir](@/docs/tools/consumedir.md) script that
watches a directory and uploads arriving files to docspell server.
This requires the value from the `integration-endpoint` setting to be
This starts the [dsc](@/docs/tools/cli.md) tool that watches a
directory and uploads arriving files to the docspell server. This
requires the value from the `integration-endpoint` setting to be
allowed to upload files. It also requires you to explicitely enable
this: go to *Collective Profile → Settings* and enable the
*Integration Endpoint*. Then create a subdirectory in `/tmp/inbox`
@ -111,11 +112,14 @@ with the name of the *collective* that you registered and place a file
into the `/tmp/inbox/[collective]` directory. The file is pushed to
docspell and processed shortly after.
To see all available options, run the script with the `--help` option:
To see all available options, run `dsc` with the `--help` option:
``` bash
$ docker run docspell/tools:latest ds-consumedir --help
$ docker run docspell/dsc:latest dsc --help
```
Or just [download the
binary](https://github.com/docspell/dsc/releases/latest), no docker
required.
Note that this is just an example and is only to demonstrate how to
use the docker images. For instance, this setup does not provide
@ -127,9 +131,9 @@ below.
There is a [docker-compose](https://docs.docker.com/compose/) setup
available in the `/docker/docker-compose` folder. This setup is
similiar to the example above, adding fulltext search and a PostgreSQL
database by using just one command. It's only a few steps to get
started.
similiar to the example above, but adding fulltext search and a
PostgreSQL database by using just one command. It's only a few steps
to get started.
### Start Docspell
#### 1. Get the docker-compose files
@ -172,10 +176,12 @@ $ export DOCSPELL_HEADER_VALUE="my-secret-123"
$ docker-compose up
```
The environment variable defines a secret that is shared between some
containers. You can define whatever you like. Please see the
[consumedir.sh](@/docs/tools/consumedir.md#docker) docs for additional
info.
The environment variable defines a secret that is shared between the
container watching a directory and the server. It is the header
defined for the [integration
endpoint](@/docs/api/upload.md#integration-endpoint) containers. You
can use whatever you like. Please see the help to the [dsc
tool](@/docs/tools/cli.md) docs for additional info.
Goto `http://localhost:7880`, signup and login. When signing up, you
choose the same name for collective and user. Then login with this

6
website/site/content/docs/install/download_run.md
View File

@ -116,6 +116,6 @@ page](@/docs/configure/_index.md#full-text-search-solr).
### Watching a directory
The [consumedir](@/docs/tools/consumedir.md) script can be used for
this. Using systemd or something similar, it is possible to create a
system service that runs the script in "watch mode".
The [dsc](@/docs/tools/cli.md) tool with the `watch` subcommand can be
used for this. Using systemd or something similar, it is possible to
create a system service that runs the script in "watch mode".

6
website/site/content/docs/install/nix.md
View File

@ -84,8 +84,8 @@ There are the following modules provided:
- joex
- consumedir
The `consumedir` module defines a systemd unit that starts the
`consumedir.sh` script to watch one or more directories for new files.
The `consumedir` module defines a systemd unit that starts the `dsc
watch` command to watch one or more directories for new files.
You need to import the `release.nix` file as described above in your
`configuration.nix` and then append the docspell module to your list of
@ -125,7 +125,7 @@ in
services.docspell-consumedir = {
enable = true;
watchDirs = ["/tmp/test"];
urls = ["http://localhost:7880/api/v1/open/upload/item/the-source-id"];
source-id = "the-source-id";
};
...

4
website/site/content/docs/joex/file-processing.md
View File

@ -323,8 +323,8 @@ docspell.joex {
When this is changed, you must re-generate all preview images. Check
the api for this, there is an endpoint to regenerate all preview
images for a collective. There is also a bash script provided in the
`tools/` directory that can be used to call this endpoint.
images for a collective. The [cli tool](../../tools/cli/) can be
used for this.
{% end %}

2
website/site/content/docs/tools/_index.md
View File

@ -4,7 +4,7 @@ description = "There are several tools distributed with docspell, like a program
weight = 60
insert_anchor_links = "right"
template = "pages.html"
redirect_to = "docs/tools/ds"
redirect_to = "docs/tools/cli/"
sort_by = "weight"
[extra]
mktoc = false

20
website/site/content/docs/tools/browserext.md
View File

@ -13,10 +13,10 @@ firefox.
Installation is a bit complicated, since you need to install external
tools and the web extension. Both work together.
# Install `ds.sh`
# Install `dsc`
First copy the `ds.sh` tool somewhere in your `PATH`, maybe
`/usr/local/bin` as described above.
First copy the [dsc](@/docs/tools/cli.md) tool somewhere in your
`PATH`, maybe `/usr/local/bin`.
# Install the native part
@ -44,10 +44,14 @@ for details.
And you might want to modify this json file, so the path to the
`native.py` script is correct (it must be absolute).
If the `ds.sh` script is in your `$PATH`, then this should
work. Otherwise, edit the `native.py` script and change the path to
the tool. Or create a file `$HOME/.config/docspell/ds.cmd` whose
content is the path to the `ds.sh` script.
If the `dsc` tool is in your `$PATH`, then this should work. You need
to provide a default source id in your `~/.config/dsc/config.toml` so
that the upload command can be used without further arguments.
Otherwise, edit the `native.py` script and change the path to the tool
and/or the arguments. Or create a file
`$HOME/.config/docspell/dsc.cmd` whose content is the path to the
`dsc` tool.
# Install the extension
@ -73,7 +77,7 @@ alternatives:
When you right click on a file link, there should be a context menu
entry *'Docspell Upload Helper'*. The add-on will download this file
using the browser and then send the file path to the `native.py`
script. This script will in turn call `ds.sh` which finally uploads it
script. This script will in turn call `dsc` which finally uploads it
to your configured URLs.
Open the Add-ons page (`Ctrl`+`Shift`+`A`), the new add-on should be

266
website/site/content/docs/tools/cli.md Normal file
View File

@ -0,0 +1,266 @@
+++
title = "CLI"
description = "A command line interface to."
weight = 5
+++
# Introduction
The **d**oc**s**pell **c**lient, short
[dsc](https://github.com/docspell/dsc), is a tool to use
docspell through the command line. It is also aims to be useful for
your own scripts and programs.
It is supposed to replace most of the shell scripts from the `tools/`
directory.
It is a work in progress; eventually most of the
[api](@/docs/api/_index.md) will be covered.
# Usage
Download the binary for your architecture from the [release
page](https://github.com/docspell/dsc/releases/latest) and rename it
to `dsc`. Then run `dsc help` to see an overview of all commands. The
help of each command is available via `dsc help [command]` or `dsc
[command] --help`.
There are docker images at
[dockerhub](https://hub.docker.com/repository/docker/docspell/dsc),
but it's usually easier to just download the binary. They should work
on most systems without additional setups.
Below are some quick infos to get started, please see [the project
page](https://github.com/docspell/dsc) for more info.
## Configuration
A configuration file can be used to have some predefined settings, for
example the docspell url, the admin secret etc. They can be overriden
by specifying them as options.
The config looks like this:
``` toml
docspell_url = "http://localhost:7880"
default_format = "Tabular"
admin_secret = "admin123"
default_account = "demo"
pdf_viewer = ["zathura", "{}"]
#pass_entry = "my/entry"
```
For linuxes, the default location is `~/.config/dsc/config.toml`. You
can give a config file explicitly via an option or the environment
variable `DSC_CONFIG`.
If you use the [pass](https://passwordstore.org) password manager, you
can add your password entry to the config file as well.
## Output format
The "output format" defines how the information is printed on screen.
The default output format is `Tabular` which prints a simple table.
This table can also be formatted as CSV using `csv` as output format.
These two modes are intended for humans and they may not present all
information available.
Alternatively, there is `json` and `lisp` as output format. These are
intended for machine consumption. They always contain all information.
If you look for some detail, use for example `json` to get all data in
a structured form. On the shell you can use the awesome tool
[jq](https://stedolan.github.io/jq/) to get exactly what you want.
## Login
Many tasks require to be logged in. This can be done via the `login`
subcommand. You can specify account and password or fallback to the
values in the config file.
Once logged in, the session token will be saved to the filesystem
(next to the config file) and is used for subsequent commands. It is
renewed if expiry is near. If you don't issue any commands for a while
you need to `login` again.
# Use Cases / Examples
## Uploads files
The `upload` subcommand can uploads files to docspell. This is the
replacement for the `ds.sh` shell script.
You can specify a list of files that are all being uploaded. This
command doesn't require to be logged in, it can also upload via a
[source id](@/docs/webapp/uploading.md#anonymous-upload) or via the
[integration endpoint](@/docs/api/upload.md#integration-endpoint).
A source id can be given in the config file, then there are no
additional options required. The simplest form is this:
``` shell
❯ dsc upload *.pdf
File already in Docspell: article-velo.pdf
Adding to request: test-ocr.pdf
Sending request …
┌─────────┬──────────────────┐
│ success │ message │
├─────────┼──────────────────┤
│ true │ Files submitted. │
└─────────┴──────────────────┘
```
By default, duplicate files are detected and not uploaded. This
uploads all files in one single request. By default, each file results
in one item. Using `--single-item` all files can be put into one item.
It is possible to specify certain metadata, like tags or a folder,
that is then attached to the resulting item.
## Upload by traversing a directory
The above use case was about uploading files. Using the `upload`
subcommand with the `--traverse` option, you can traverse directories
and upload all files in them. In this mode, each file will be uploaded
in a separate request, so the `--single-item` option cannot be used.
There are options to exclude/include files based on a [glob
pattern](https://docs.rs/glob/0.3.0/glob/struct.Pattern.html).
``` shell
❯ dsc upload --traverse .
File already in Docspell: article-velo.pdf
File already in Docspell: demo/dirc/scan.21-03-12.15-50-54.pdf
File already in Docspell: demo/dirc/test-stamp.pdf
File already in Docspell: demo/letter-de.pdf
Uploading eike/keywords.pdf
File already in Docspell: eike/large-file.pdf
Uploading eike/letter-en.pdf
File already in Docspell: test-ocr.pdf
┌─────────┬────────────┐
│ success │ message │
├─────────┼────────────┤
│ true │ Uploaded 2 │
└─────────┴────────────┘
```
## Watch a directory
The `watch` subcommand can be used to watch one or more directories
and upload files when they arrive. It uses the `upload` command under
the hood and therefore most options are also available here.
It detects file creations and skips a rename within a watched folder.
The flag `-r` or `--recursive` is required to recursively watch a
directory.
``` shell
❯ dsc watch -r .
Watching directory (Recursive): .
Press Ctrl-C to quit.
------------------------------------------------------------------------------
Got: /home/eike/workspace/projects/dsc/local/files/./demo/letter-de.pdf
Adding to request: /home/eike/workspace/projects/dsc/local/files/./demo/letter-de.pdf
Sending request …
Server: Files submitted.
```
The `--matches` option allows to define a pattern for files to include.
If watching a directory is not possible due to system constraints
(e.g. when using NFS or SAMBA shares), a less efficient option is to
use the `upload` subcommand with `--poll` option which periodically
traverses a directory.
## Admin commands
These are a set of commands that simply call a route at the server to
submit a maintenance task or to reset the password of some user. These
commands require the [admin
secret](@/docs/configure/_index.md#admin-endpoint) either in the
config file or as an argument.
Reset user password:
``` shell
❯ dsc admin reset-password --account demo
┌─────────┬──────────────┬──────────────────┐
│ success │ new password │ message │
├─────────┼──────────────┼──────────────────┤
│ true │ 2q2UeCVvMYg │ Password updated │
└─────────┴──────────────┴──────────────────┘
```
Recreate fulltext index:
``` shell
❯ dsc admin --admin-secret admin123 recreate-index
┌─────────┬─────────────────────────────────────┐
│ success │ message │
├─────────┼─────────────────────────────────────┤
│ true │ Full-text index will be re-created. │
└─────────┴─────────────────────────────────────┘
```
## Search for items
The `search` command takes a [query](@/docs/query/_index.md) and
prints the results.
``` shell
❯ dsc search 'corr:*'
┌──────────┬────────────────────────────┬───────────┬────────────┬─────┬───────────────────────────┬───────────────┬────────┬─────────────┬────────────┬───────┐
│ id │ name │ state │ date │ due │ correspondent │ concerning │ folder │ tags │ fields │ files │
├──────────┼────────────────────────────┼───────────┼────────────┼─────┼───────────────────────────┼───────────────┼────────┼─────────────┼────────────┼───────┤
│ HVK7JuCF │ test-ocr.pdf │ created │ 2021-07-18 │ │ Pancake Company │ │ │ Certificate │ │ 1 │
│ 3odNawKE │ letter-en.pdf │ confirmed │ 2021-07-18 │ │ Pancake Company │ │ │ invoice │ │ 1 │
│ 3MA5NdhS │ large-file.pdf │ confirmed │ 2021-07-18 │ │ Axa │ │ │ Certificate │ │ 1 │
│ HDumXkRm │ keywords.pdf │ confirmed │ 2021-07-18 │ │ Pancake Company │ │ │ invoice │ │ 1 │
│ 733gM656 │ test-stamp.pdf │ created │ 2021-07-18 │ │ Pancake Company │ │ │ Contract │ │ 1 │
│ 8LiciitB │ scan.21-03-12.15-50-54.pdf │ confirmed │ 2021-07-18 │ │ Supermarket │ │ │ Receipt │ CHF 89.44 │ 1 │
│ 8nFt2z7T │ article-velo.pdf │ confirmed │ 2021-07-18 │ │ Supermarket/Rudolf Müller │ Rudolf Müller │ │ invoice │ CHF 123.11 │ 1 │
│ kfugGdXU │ letter-de.pdf │ created │ 2021-07-18 │ │ Axa │ Rudolf Müller │ │ invoice │ │ 1 │
└──────────┴────────────────────────────┴───────────┴────────────┴─────┴───────────────────────────┴───────────────┴────────┴─────────────┴────────────┴───────┘
```
The same can be formatted as json and, for example, only print the ids:
``` shell
❯ dsc -f json search 'corr:*' | jq '.groups[].items[].id'
"HVK7JuCFt4W-qxkcwq1cWCV-dvpGo4DpZzU-Q16Xoujojas"
"3odNawKE1Ek-YJrWfPzekAq-47cjt14sexd-GK35JAEAanx"
"3MA5NdhSrbx-3JkjEpqHiyU-XyVNb15tioh-SUVjMLi1aoV"
"HDumXkRmDea-dNryjtRjk3V-ysdJmJNQGQS-UFb4DWNZJ3F"
"733gM656S4T-d4HmEgdAV6Z-9zuHAd3biKM-mBwNriZpqMB"
"8LiciitBVTi-DTmgiEUdqAJ-xXPckMvFHMc-JSiJMYvLaWh"
"8nFt2z7T9go-1qaCTTgodub-592n6gpmdNR-VRcyYAyT7qj"
"kfugGdXUGUc-mReaUnJxyUL-R44Lf7yH6RK-2JbZ1bv7dw"
```
# Docker
The provided docker-compose setup runs this script to watch a single
directory, `./docs` in current directory, for new files. If a new file
is detected, it is pushed to docspell.
This utilizes the [integration
endpoint](@/docs/api/upload.md#integration-endpoint), which is enabled
in the config file, to allow uploading documents for all collectives.
A subfolder must be created for each registered collective. The docker
containers are configured to use http-header protection for the
integration endpoint. This requires you to provide a secret, that is
shared between the rest-server and the `dsc` tool. This can be done by
defining an environment variable which gets picked up by the
containers defined in `docker-compose.yml`:
``` bash
export DOCSPELL_HEADER_VALUE="my-secret"
docker-compose up
```
Now you can create a folder `./docs/<collective-name>` and place all
files in there that you want to import. Once dropped in this folder
the `consumedir` container will push it to docspell.

10
website/site/content/docs/tools/consumedir-cleaner.md
View File

@ -1,9 +1,15 @@
+++
title = "Directory Cleaner"
title = "Directory Cleaner (⊗)"
description = "Clean directories from files in docspell"
weight = 32
weight = 150
+++
{% infobubble(mode="info", title="⚠ Please note") %}
This script is now obsolete, you can use the [**CLI tool**](../cli/) instead.
Use the `cleanup` or the `upload` command.
{% end %}
# Introduction
This script is made for cleaning up the consumption directory used for

11
website/site/content/docs/tools/consumedir.md
View File

@ -1,9 +1,16 @@
+++
title = "Consume Directory"
title = "Consume Directory (⊗)"
description = "A script to watch a directory for new files and upload them to docspell."
weight = 30
weight = 110
+++
{% infobubble(mode="info", title="⚠ Please note") %}
This script is now obsolete, you can use the [**CLI tool**](../cli/) instead.
You can use the `watch` command, or the `upload` command with `--poll`.
{% end %}
# Introduction
The `consumerdir.sh` is a bash script that works in two modes:

12
website/site/content/docs/tools/convert-all-pdf.md
View File

@ -1,9 +1,17 @@
+++
title = "Convert All PDFs"
title = "Convert All PDFs (⊗)"
description = "Convert all PDF files using OcrMyPdf."
weight = 60
weight = 160
+++
{% infobubble(mode="info", title="⚠ Please note") %}
This script is now obsolete, you can use the [**CLI tool**](../cli/) instead.
Use the `convert-all-pdfs` admin command, e.g. `dsc admin
convert-all-pdfs`.
{% end %}
# convert-all-pdf.sh
With version 0.9.0 there was support added for another external tool,

11
website/site/content/docs/tools/ds.md
View File

@ -1,9 +1,16 @@
+++
title = "Upload CLI"
title = "Upload CLI (⊗)"
description = "A script to quickly upload files from the command line."
weight = 10
weight = 100
+++
{% infobubble(mode="info", title="⚠ Please note") %}
This script is now obsolete, you can use the [**CLI tool**](../cli/) instead.
Use the `upload` command (or the `up` alias), like `dsc up *.pdf`.
{% end %}
# Introduction
The `tools/ds.sh` is a bash script to quickly upload files from the

10
website/site/content/docs/tools/regenerate-previews.md
View File

@ -1,9 +1,15 @@
+++
title = "Regenerate Preview Images"
title = "Regenerate Preview Images (⊗)"
description = "Re-generates all preview images."
weight = 80
weight = 130
+++
{% infobubble(mode="info", title="⚠ Please note") %}
This script is now obsolete, you can use the [**CLI tool**](../cli/) instead.
Use the `generate-previews` admin command, e.g. `dsc admin generate-previews`.
{% end %}
# regenerate-previews.sh
This is a simple bash script to trigger the endpoint that submits task

12
website/site/content/docs/tools/reset-password.md
View File

@ -1,9 +1,17 @@
+++
title = "Reset Password"
title = "Reset Password (⊗)"
description = "Resets a user password."
weight = 70
weight = 120
+++
{% infobubble(mode="info", title="⚠ Please note") %}
This script is now obsolete, you can use the [**CLI tool**](../cli/) instead.
Use the `reset-password` admin command, e.g. `dsc admin reset-password
--account "smith/john"`, where `smith` is the collective id and `john`
the username.
{% end %}
This script can be used to reset a user password. This can be done by
admins, who know the `admin-endpoint.secret` value in the

4
website/site/content/docs/webapp/uploading.md
View File

@ -78,8 +78,8 @@ $ curl -XPOST -F file=@test.pdf http://192.168.1.95:7880/api/v1/open/upload/item
{"success":true,"message":"Files submitted."}
```
There is a [script provided](@/docs/tools/ds.md) that uses curl to
upload files from the command line more conveniently.
There is a [cli](@/docs/tools/cli.md) to upload files from the command
line more conveniently.
When files are uploaded to an source endpoint, the items resulting
from this uploads are marked with the name of the source. So you know

2
website/site/templates/shortcodes/infobubble.html
View File

@ -1,4 +1,4 @@
<div class="notification is-{{ mode }} is-light" style="z-index: -1">
<div class="notification is-{{ mode }} is-light" style="z-index: 0">
<div class="content ">
<p class="title is-5">
{{ title }}