The join to retrieve the attachment count per item turned out to be
very expensive. And it is not used anymore for the result, only to
support fulltext search. It is now removed from the query. The
DISTINCT keyword is also removed, because it is not necessary and it
is expensive. With the DISTINCT removed, a new index (provided in the
previous commit) can now be used to avoid sorting items.
The query now searches in more fields. For example, when getting a
list of tags, the query is applied to the tag name *and* category.
When listing persons, the query now also looks in the associated
organization name.
This has been used to make some headers in the meta data tables
clickable to sort the list accordingly.
Refs: #965, #538
The upload request can now contain a boolean for importing only
attachments when e-mails are uploaded. This option is now also added
to a source url.
Refs: #983
Also hard delete the remaining items. They are empty (have no
attachments), because data is moved if possible. Doesn't make much
sense to keep them, because restoring them isn't much useful.
First, when checking for existence of a file, deleted items are not
conisdered.
The working with fulltext search has been changed: deleted items are
removed from fulltext index and are re-added when they are restored.
The fulltext index currently doesn't hold the item state and it would
mean much more work to introduce it into the index (or, worse, to
reprocess the results from the index). Thus, deleted items can only be
searched using database queries. It is probably a very rare use case
when fulltext search should be applied to deleted items. They have
been deleted for a reason and the most likely case is that they are
simply removed.
Refs: #347
In order to keep deleted items for a while, the periodic task can now
use a duration to only remove items with a certain age. This can be
used to ensure that a deleted item stays at least X days before it
will be removed from the database.
Refs: #347
Before, there were periodic tasks run per collective and not user by
making sure that submitter + group are the same value. This is now
encoded in `UserTaskScope` so it is now obvious and errors can be
reduced when using this.
It's not very likely to have more modes of search besides normal and
trashed, but got surprised in that way quite often and it's nicer this
way anyways.
A new query/request parameter can be used to apply a search to only
soft-deleted items.
The query expression `Trashed` has been introduced which selects only
items with state `Deleted`. This is another option an analog to
`ValidItemStates` (both cannot be used together as they would select
no items). This new query node is not added to the parser, because
users may not use it in their own queries - it must be part of the
"fixed" query so the application can control in which subset to search
(it would otherwise be possible to select any items).
When deleting items via the http api, they are not deleted anymore but
a new status "Deleted" is set. The collective insights contains now a
count separately for deleted items.
It doesn't make much sense to have this per collective, because this
is triggered by an admin after changing the server config file. So it
is now implemented as an admin endpoint that affects all files.
When using fulltext only search, then only the index must be searched.
This wasn't working anymore, because the routes added a query to
always select valid items (those not being processed). But this lead
to the downstream code to always consult the database, too. Since the
routes are using a "simple-search" interface, this is now adding the
valid-state condition if applicable. There are still more low-level
interfaces that can be used when searching should be done differently.
Closes: #823
If multiple tags of the same category are applied to the same item,
just summing tag counts will produce the wrong results as now items
are counted multiple times.
