The upload request can now contain a boolean for importing only
attachments when e-mails are uploaded. This option is now also added
to a source url.
Refs: #983
It was wrongly stored using RPeriodicTask directly, but the higher
level `UserTask` must be used instead, because this ensures a
correctly scoped periodic task using the `updateOneTask` method. Since
this is a system task, it can be given a fixed ID which makes it now
safe even if stored using RPeriodicTask directly.
The bug resulted in multiple empty-trash tasks to be inserted (on each
restart).
Refs: #347
Also hard delete the remaining items. They are empty (have no
attachments), because data is moved if possible. Doesn't make much
sense to keep them, because restoring them isn't much useful.
First, when checking for existence of a file, deleted items are not
conisdered.
The working with fulltext search has been changed: deleted items are
removed from fulltext index and are re-added when they are restored.
The fulltext index currently doesn't hold the item state and it would
mean much more work to introduce it into the index (or, worse, to
reprocess the results from the index). Thus, deleted items can only be
searched using database queries. It is probably a very rare use case
when fulltext search should be applied to deleted items. They have
been deleted for a reason and the most likely case is that they are
simply removed.
Refs: #347
In order to keep deleted items for a while, the periodic task can now
use a duration to only remove items with a certain age. This can be
used to ensure that a deleted item stays at least X days before it
will be removed from the database.
Refs: #347
Before, there were periodic tasks run per collective and not user by
making sure that submitter + group are the same value. This is now
encoded in `UserTaskScope` so it is now obvious and errors can be
reduced when using this.
A new query/request parameter can be used to apply a search to only
soft-deleted items.
The query expression `Trashed` has been introduced which selects only
items with state `Deleted`. This is another option an analog to
`ValidItemStates` (both cannot be used together as they would select
no items). This new query node is not added to the parser, because
users may not use it in their own queries - it must be part of the
"fixed" query so the application can control in which subset to search
(it would otherwise be possible to select any items).
When deleting items via the http api, they are not deleted anymore but
a new status "Deleted" is set. The collective insights contains now a
count separately for deleted items.
When using fulltext only search, then only the index must be searched.
This wasn't working anymore, because the routes added a query to
always select valid items (those not being processed). But this lead
to the downstream code to always consult the database, too. Since the
routes are using a "simple-search" interface, this is now adding the
valid-state condition if applicable. There are still more low-level
interfaces that can be used when searching should be done differently.
Closes: #823
If multiple tags of the same category are applied to the same item,
just summing tag counts will produce the wrong results as now items
are counted multiple times.
- put all used strings in one place to have it easier to track
- don't use `$` for shortcuts, it's a detail not interesting to a
user; now names must not clash (which is a good idea anyways)
- Added two more shortcuts `conc` and `corr`
The h2 jdbc driver could not translate the union query in QCollective
when the `kind` was set via a constant value. Using literals works
here. Renamed the corresponding elements in the query builder.
Classifier don't work on each attachment, but on all. So the results
must not be stored at an attachment. This reverts some previous
changes to put the classifier results for item entities into its own
table.
Improves and reorganizes how nlp pipelines are setup. Now users can
choose from many options, depending on their hardware and usage
scenario.
This is the base to use more languages without depending on what
stanford-nlp supports. Support then is involves to text extraction and
simple regex-ner processing.
