Commit Graph

234 Commits

Author SHA1 Message Date
eikek
c21b2cdd29 Update scalafmt to 3.0.8 2021-12-11 22:46:55 +01:00
eikek
4ffc8d1f14 Add support for more generic notification
This is a start to have different kinds of notifications. It is
possible to be notified via e-mail, matrix or gotify. It also extends
the current "periodic query" for due items by allowing notification
over different channels. A "generic periodic query" variant is added
as well.
2021-12-11 18:57:32 +01:00
eikek
501c6f2988 Updating stanford corenlp to 4.3.2; adding more languages
There are models for Spanish, that have been added now. Also the
Hungarian language has been added to the list of supported
languages (for tesseract mainly, no nlp models)
2021-11-20 14:31:39 +01:00
eikek
20fc9955ba Fix reprocessing to lookup the last used language 2021-11-20 01:34:16 +01:00
eikek
0651db9901 Make publish async, replace joexclient in periodic job scheduler 2021-11-16 22:55:37 +01:00
eikek
7a8527f821 Indicate number of running jobs in tob nav
Issue: #1069
2021-11-12 22:01:30 +01:00
eikek
cf933b60a7 Encode ws messages as JSON 2021-11-11 23:54:37 +01:00
eikek
7b8afe8371 Make internal endpoints available to nodes only 2021-11-08 00:13:02 +01:00
eikek
61c01ad79b Use job group instead of parsing the task arguments 2021-11-08 00:07:44 +01:00
eikek
3e58d97f72 Add websockets and notify frontend when an item is processed 2021-11-06 21:32:30 +01:00
eikek
f38d520a1d Hooking the new pubsub impl into the application 2021-11-06 01:44:06 +01:00
eikek
668cd7d974 Refactor config validation 2021-10-25 11:27:06 +02:00
eikek
118d23c3a2 Add list of env variables to documentation
Issue: #1121
2021-10-25 00:23:20 +02:00
eikek
4e5924d796 Refactor config loading and add config from environment
Issue: #1121
2021-10-24 23:02:39 +02:00
eikek
48d2bec9c4 Update http4s to 0.23.6 2021-10-24 18:24:03 +02:00
eikek
aa8f3b82fc Use passwords when reading PDFs 2021-09-30 11:48:59 +02:00
eikek
3c93b63c8a Add option to decrypt PDFs during conversion
Refs: #1074
2021-09-29 23:04:26 +02:00
eikek
9013f2de5b Update scalafmt settings 2021-09-22 17:23:24 +02:00
eikek
20a829cf7a Refactoring for migrating to binny library 2021-09-22 14:18:43 +02:00
eikek
9785db0683 Change license header of all files 2021-09-21 22:35:38 +02:00
eikek
193b81bf7d Fix version check
Refs: #1068
2021-09-21 22:07:19 +02:00
eikek
751fa3da5a Add attachments-only filter to uploads
When uploading a file which is an e-mail, this option allows to skip
the mail body when the file is being processed.
2021-08-21 13:49:12 +02:00
eikek
5d33b3841a Add a task to check for updates periodically
It must be enabled and configured by the admin.

Refs: #990
2021-08-20 00:25:27 +02:00
eikek
90421599ea Fix storing empty-trash task
It was wrongly stored using RPeriodicTask directly, but the higher
level `UserTask` must be used instead, because this ensures a
correctly scoped periodic task using the `updateOneTask` method. Since
this is a system task, it can be given a fixed ID which makes it now
safe even if stored using RPeriodicTask directly.

The bug resulted in multiple empty-trash tasks to be inserted (on each
restart).

Refs: #347
2021-08-20 00:25:25 +02:00
Scala Steward
e4fecefaea
Reformat with scalafmt 3.0.0 2021-08-19 08:50:30 +02:00
eikek
14e4a8f792 Fixup for deleting items
First, when checking for existence of a file, deleted items are not
conisdered.

The working with fulltext search has been changed: deleted items are
removed from fulltext index and are re-added when they are restored.
The fulltext index currently doesn't hold the item state and it would
mean much more work to introduce it into the index (or, worse, to
reprocess the results from the index). Thus, deleted items can only be
searched using database queries. It is probably a very rare use case
when fulltext search should be applied to deleted items. They have
been deleted for a reason and the most likely case is that they are
simply removed.

Refs: #347
2021-08-15 16:00:30 +02:00
eikek
f4a2b86ea8 Use a minimum age of items to remove
In order to keep deleted items for a while, the periodic task can now
use a duration to only remove items with a certain age. This can be
used to ensure that a deleted item stays at least X days before it
will be removed from the database.

Refs: #347
2021-08-15 12:32:50 +02:00
eikek
31d885ed79 Refactor user tasks to support collective and user scopes
Before, there were periodic tasks run per collective and not user by
making sure that submitter + group are the same value. This is now
encoded in `UserTaskScope` so it is now obvious and errors can be
reduced when using this.
2021-08-14 22:07:56 +02:00
eikek
27fd7a5867 Make sure the empty-trash task is started for all collectives 2021-08-14 20:40:04 +02:00
eikek
50706c3d6d Add a task implementation to delete items 2021-08-14 19:33:18 +02:00
eikek
1901fe1a8c Adopt deprecated APIs from fs2; use fs2.Path 2021-08-07 17:51:56 +02:00
eikek
1c0d87527b Log error when setting folder doesn't work 2021-07-17 15:10:00 +02:00
eikek
8e5c88fd32 Add copyright header to source files 2021-07-04 10:57:53 +02:00
eikek
bd791b4593 Upgrade code base to CE3 2021-06-22 22:53:34 +02:00
eikek
ac7d00c28f Refactor re-index task 2021-06-07 21:17:29 +02:00
eikek
3ee0846e19 Remove fts_migration table
It is now stored it SOLR instead.
2021-06-07 17:53:47 +02:00
eikek
5205ee0623 Store solr migration state in a solr document 2021-06-07 17:53:37 +02:00
eikek
bdc7822f50 Add documentation about docker setup 2021-05-31 22:19:49 +02:00
Scala Steward
144ea852bf Update fs2-core, fs2-io to 2.5.4 2021-03-31 21:10:42 +02:00
Eike Kettner
c36073b852 Allow to give human readable summary to user tasks 2021-03-27 22:13:13 +01:00
Eike Kettner
cc38b850a6 Remove deprecated search routes and some refactoring 2021-03-27 22:13:13 +01:00
Eike Kettner
f8bd42e5bd Redo pdf conversion and text extraction on reprocess
When processing a new file conversion and text extraction is skipped
if detected to be already done. This prevents running expensive tasks
again after restarting/retrying. When explicitely reprocessing a file,
these tasks should run again and replace the existing results.
2021-03-12 00:45:28 +01:00
Eike Kettner
a7ee0aa08b Add a flag to processing task to distinguish re-/processing 2021-03-12 00:45:23 +01:00
Eike Kettner
058c31e1f6 Reprocessing now sets metadata to an item if not in state confirmed
When reprocessing an item, the metadat of all *files* are replaced.
This change now also sets some metadat to an item, but only if the
item is not in state "confirmed". Confirmed items are not touched, but
the metadata of the files is updated.
2021-03-12 00:16:19 +01:00
Eike Kettner
0229a867af Add a use colum to metadata entities 2021-03-10 23:55:18 +01:00
Eike Kettner
9013d9264e Add more convenient date parsers and some basic macros 2021-03-01 00:51:01 +01:00
Eike Kettner
e9ed998e3a Basic poc to search via custom query 2021-03-01 00:51:01 +01:00
Eike Kettner
186014a1c6 Refactor search to separate between a base query and user query
The `findBase` is adding only strictly required conditions. Everything
else comes from the user.
2021-03-01 00:51:01 +01:00
Eike Kettner
e6d9ce2c37 Remove obsolete type capabilities
These are now detected by the new scala compiler and lead to compile
errors.
2021-03-01 00:16:30 +01:00
Eike Kettner
d7bc963450 Cleanup nodes that are not reachable anymore 2021-02-18 00:37:18 +01:00