eikek
14e4a8f792
Fixup for deleting items
...
First, when checking for existence of a file, deleted items are not
conisdered.
The working with fulltext search has been changed: deleted items are
removed from fulltext index and are re-added when they are restored.
The fulltext index currently doesn't hold the item state and it would
mean much more work to introduce it into the index (or, worse, to
reprocess the results from the index). Thus, deleted items can only be
searched using database queries. It is probably a very rare use case
when fulltext search should be applied to deleted items. They have
been deleted for a reason and the most likely case is that they are
simply removed.
Refs: #347
2021-08-15 16:00:30 +02:00
eikek
f4a2b86ea8
Use a minimum age of items to remove
...
In order to keep deleted items for a while, the periodic task can now
use a duration to only remove items with a certain age. This can be
used to ensure that a deleted item stays at least X days before it
will be removed from the database.
Refs: #347
2021-08-15 12:32:50 +02:00
eikek
ef31c90e34
Fix submitter column of a collective-scoped periodic job
2021-08-14 22:07:56 +02:00
eikek
4562c77152
Fix migration changeset name for h2
2021-08-14 22:07:56 +02:00
eikek
31d885ed79
Refactor user tasks to support collective and user scopes
...
Before, there were periodic tasks run per collective and not user by
making sure that submitter + group are the same value. This is now
encoded in `UserTaskScope` so it is now obvious and errors can be
reduced when using this.
2021-08-14 22:07:56 +02:00
eikek
27fd7a5867
Make sure the empty-trash task is started for all collectives
2021-08-14 20:40:04 +02:00
eikek
50706c3d6d
Add a task implementation to delete items
2021-08-14 19:33:18 +02:00
eikek
4901276c66
Change "empty trash" settings for a collective and submit the job
2021-08-14 19:33:15 +02:00
eikek
f999662905
Add routes to restore deleted items
2021-08-14 16:48:03 +02:00
eikek
a7b74bd5ae
Allow to search in soft-deleted items
...
A new query/request parameter can be used to apply a search to only
soft-deleted items.
The query expression `Trashed` has been introduced which selects only
items with state `Deleted`. This is another option an analog to
`ValidItemStates` (both cannot be used together as they would select
no items). This new query node is not added to the parser, because
users may not use it in their own queries - it must be part of the
"fixed" query so the application can control in which subset to search
(it would otherwise be possible to select any items).
2021-08-14 14:53:05 +02:00
eikek
cb777e30c0
Delete items by introducing a deleted state
...
When deleting items via the http api, they are not deleted anymore but
a new status "Deleted" is set. The collective insights contains now a
count separately for deleted items.
2021-08-14 14:18:03 +02:00
eikek
48d13a35fc
Fix search summary to restrict on valid items
2021-08-14 14:09:07 +02:00
eikek
1256a6e8cf
Allo wildcards when searching by attachment ids
2021-07-31 01:23:37 +02:00
eikek
15b15666bf
Allow to use the created timestamp in item queries
...
Refs: 925
2021-07-27 17:33:13 +02:00
eikek
d95a78f495
Allow to set folder by id or name
2021-07-17 15:10:19 +02:00
eikek
8e5c88fd32
Add copyright header to source files
2021-07-04 10:57:53 +02:00
eikek
ce6f53cc29
Fix next-job query to do round-robin through job groups
2021-06-27 00:04:56 +02:00
eikek
988367a281
Extend query builder to compare results from subselects
2021-06-26 21:12:02 +02:00
eikek
bd791b4593
Upgrade code base to CE3
2021-06-22 22:53:34 +02:00
eikek
3ee0846e19
Remove fts_migration table
...
It is now stored it SOLR instead.
2021-06-07 17:53:47 +02:00
eikek
c2170d2daf
Fix tag category search being case insensitive
2021-06-03 21:44:55 +02:00
Eike Kettner
25788a0b23
Add routes for storing/retrieving client settings
2021-05-27 21:34:05 +02:00
Eike Kettner
a1a93e5ca6
Fixes searching items with fulltext
...
When using fulltext only search, then only the index must be searched.
This wasn't working anymore, because the routes added a query to
always select valid items (those not being processed). But this lead
to the downstream code to always consult the database, too. Since the
routes are using a "simple-search" interface, this is now adding the
valid-state condition if applicable. There are still more low-level
interfaces that can be used when searching should be done differently.
Closes : #823
2021-05-23 14:14:25 +02:00
Stefan Scheidewig
6149a2ab89
Restored unused imports to make it compile again
2021-04-15 18:34:54 +02:00
Stefan Scheidewig
fa34312020
Implemented endpoint to delete multiple attachments
2021-04-15 18:05:01 +02:00
Eike Kettner
994e3df597
Fix query for getting tag categoy summary
2021-04-12 13:40:22 +02:00
Eike Kettner
3e0914ece7
Correctly count tag categories
...
If multiple tags of the same category are applied to the same item,
just summing tag counts will produce the wrong results as now items
are counted multiple times.
2021-04-11 14:34:44 +02:00
Eike Kettner
4041018c47
Reduce not expressions
...
Fixes queries containing macros inside a "not".
2021-04-11 12:57:42 +02:00
Eike Kettner
e1bbc2edf5
Apply autoformat
2021-04-10 16:31:58 +02:00
Scala Steward
144ea852bf
Update fs2-core, fs2-io to 2.5.4
2021-03-31 21:10:42 +02:00
Eike Kettner
c36073b852
Allow to give human readable summary to user tasks
2021-03-27 22:13:13 +01:00
Eike Kettner
cc38b850a6
Remove deprecated search routes and some refactoring
2021-03-27 22:13:13 +01:00
Eike Kettner
177488817d
Fix h2 migration
...
Using java source code obviously requires `javac` during migration.
2021-03-13 16:38:48 +01:00
Eike Kettner
2e443bc9b9
Fix mariadb migration
2021-03-13 15:52:38 +01:00
Eike Kettner
0229a867af
Add a use colum to metadata entities
2021-03-10 23:55:18 +01:00
Eike Kettner
6a63694a3e
Convert unit tests to munit
2021-03-10 19:48:56 +01:00
Eike Kettner
77a87782b7
Refactoring parser
...
- put all used strings in one place to have it easier to track
- don't use `$` for shortcuts, it's a detail not interesting to a
user; now names must not clash (which is a good idea anyways)
- Added two more shortcuts `conc` and `corr`
2021-03-08 22:51:14 +01:00
Eike Kettner
e681ffa96f
Extend query builder allowing more conditions
...
Before only a column or a dbfunction could be used in a condition. It
is now allowed for all `SelectExpr`.
2021-03-08 22:51:08 +01:00
Eike Kettner
30c901ddf1
Add more ways to query for attachments
...
- find items with a specified attachment count
- find items by attachment id
2021-03-08 09:49:38 +01:00
Eike Kettner
2b2f913e85
Add checksum query expr
2021-03-08 01:53:21 +01:00
Eike Kettner
1c834cbb77
Correctly compare numeric field values
2021-03-03 22:54:55 +01:00
Eike Kettner
71985244f1
Use a better representation for macros
2021-03-03 00:44:49 +01:00
Eike Kettner
a48504debb
Specificly search for field id vs name
2021-03-02 21:09:31 +01:00
Eike Kettner
168f5a1a98
Fix like search for custom fields
2021-03-01 20:56:23 +01:00
Eike Kettner
f8307f77c6
Search by field id or name
2021-03-01 20:56:23 +01:00
Eike Kettner
698ff58aa3
Provide a more convenient interface to search
2021-03-01 11:50:07 +01:00
Eike Kettner
d737da768e
Move to munit in query module
2021-03-01 00:51:01 +01:00
Eike Kettner
9013d9264e
Add more convenient date parsers and some basic macros
2021-03-01 00:51:01 +01:00
Eike Kettner
af73b59ec2
Parser improvements
...
- default expressions into a and node
- fix parsing string lists that end in whitespace
- fix package names of internal classes
2021-03-01 00:51:01 +01:00
Eike Kettner
a80d73d5d2
Optimize imports
2021-03-01 00:51:01 +01:00
Eike Kettner
e9ed998e3a
Basic poc to search via custom query
2021-03-01 00:51:01 +01:00
Eike Kettner
186014a1c6
Refactor search to separate between a base query and user query
...
The `findBase` is adding only strictly required conditions. Everything
else comes from the user.
2021-03-01 00:51:01 +01:00
Eike Kettner
c3cdec416c
Sketching some basic tests
2021-03-01 00:50:52 +01:00
Eike Kettner
be5c7ffb88
First draft of ast and parser
2021-03-01 00:46:57 +01:00
Eike Kettner
e6d9ce2c37
Remove obsolete type capabilities
...
These are now detected by the new scala compiler and lead to compile
errors.
2021-03-01 00:16:30 +01:00
Eike Kettner
7ef3185659
Add language to a source
...
Allows to define upload urls for different languages.
2021-02-18 23:34:42 +01:00
Eike Kettner
d7bc963450
Cleanup nodes that are not reachable anymore
2021-02-18 00:37:18 +01:00
Eike Kettner
5181283b1b
Add a short-name to organizations
2021-02-17 22:55:35 +01:00
Eike Kettner
20ccdda609
Add a notes field to equipments
2021-02-17 22:39:07 +01:00
Eike Kettner
48eee00c0b
Allow person to be correspondent, concerning or both
2021-02-16 22:49:55 +01:00
Eike Kettner
394aeeccb6
Introduce a sql literal and constants in query builder
...
The h2 jdbc driver could not translate the union query in QCollective
when the `kind` was set via a constant value. Using literals works
here. Renamed the corresponding elements in the query builder.
2021-01-25 00:18:24 +01:00
Eike Kettner
1b66e2af5c
Fix classifier_settings table
2021-01-23 21:30:26 +01:00
Eike Kettner
c7e850116f
Make the text length limit optional
2021-01-22 23:06:50 +01:00
mergify[bot]
38e0a50942
Merge pull request #582 from eikek/delete-org-fix
...
Fix deleting organization
2021-01-21 22:56:56 +00:00
Eike Kettner
f4a03e7c69
Fix deleting organization
...
The foreign key in person must be resetted.
2021-01-21 21:27:02 +01:00
Eike Kettner
4cba96f390
Always return classifier results as suggestion
...
The classifier results are spliced into the suggestion list at second
place. When linking they are only used if nlp didn't find anything.
2021-01-21 21:05:28 +01:00
Eike Kettner
9957c3267e
Add constraints from config to classifier training
...
For large and/or many documents, training the classifier can lead to
OOM errors. Some limits have been set by default.
2021-01-21 17:46:39 +01:00
Eike Kettner
363cf5aef0
Quote names in sql changesets
2021-01-21 00:22:58 +01:00
Eike Kettner
38387e00a0
Fix mariadb migration
2021-01-21 00:22:53 +01:00
Eike Kettner
27c24c128d
Store tags guessed with classifier in database
2021-01-20 00:30:40 +01:00
Eike Kettner
9d83cb7fe4
Store item based proposals in separate table
...
Classifier don't work on each attachment, but on all. So the results
must not be stored at an attachment. This reverts some previous
changes to put the classifier results for item entities into its own
table.
2021-01-19 23:48:09 +01:00
Eike Kettner
3ff9284a64
Return classifier results as suggestions
2021-01-19 23:13:51 +01:00
Eike Kettner
1cd3441462
Run classifier for item entities (concerned, correspondent)
...
Store the results separately from nlp results in attachment metadata.
2021-01-19 22:08:29 +01:00
Eike Kettner
d124f0c1a9
Rename db changeset
...
It's not just a fix, but adds new things
2021-01-19 22:08:29 +01:00
Eike Kettner
99dcaae66b
Learn classifiers for item entities
...
Learns classifiers for concerned and correspondent entities. This can
be used as an alternative to or after nlp.
2021-01-19 20:54:47 +01:00
Eike Kettner
a6f29153c4
Control what tag categories to use for auto-tagging
2021-01-19 01:20:13 +01:00
Eike Kettner
cce8878898
Exclude tags w/o category from classifying; remove obsolete models
2021-01-18 21:51:49 +01:00
Eike Kettner
3e28ce1254
Add the sql concat function to query builder
2021-01-18 21:51:45 +01:00
Eike Kettner
249f9e6e2a
Extend guessing tags to all tag categories
2021-01-18 21:51:45 +01:00
Eike Kettner
f01646aeb5
Reorganize nlp pipeline and add nlp-unsupported language italian
...
Improves and reorganizes how nlp pipelines are setup. Now users can
choose from many options, depending on their hardware and usage
scenario.
This is the base to use more languages without depending on what
stanford-nlp supports. Support then is involves to text extraction and
simple regex-ner processing.
2021-01-18 17:41:40 +01:00
Eike Kettner
a70e9ab614
Store used language for processing on attachmentmeta
...
Issue: #570
2021-01-17 22:56:33 +01:00
Eike Kettner
f0f0e6e0d4
Search for categories case-insensitive
...
The string was already lowercased, but the comparison was not.
Fixes #568
2021-01-17 20:10:24 +01:00
Eike Kettner
623a61dbb6
Introduce a lowerEq operator to the query builder
2021-01-17 20:10:00 +01:00
Eike Kettner
3fccc3df39
Return all tags in search stats result
...
Before only tags with a count > 0 were included. Now those that have
not attached to any item are returned as well.
2021-01-11 12:13:13 +01:00
Eike Kettner
0cfd8974d3
Add a flag to imap settings to enable/disable oauth2 scheme
2021-01-04 11:03:04 +01:00
Eike Kettner
95fd386c14
Fixing find-by-checksum with exclusions
...
The NOT-IN query must check for null separately, as everything with
null evaluates to false in sql resulting in not finding existing
duplicates.
2021-01-03 12:29:03 +01:00
Eike Kettner
97dfcece97
Fix duplicate check on restarts
...
Issue: #530
2021-01-02 21:18:05 +01:00
Eike Kettner
a9ed0364d2
Fix linking guessed tags
...
Since tag names are lower-cased the search must happen lower-cased, too.
2021-01-02 01:30:31 +01:00
Eike Kettner
36858da624
Fix search condition for empty items set
2020-12-17 23:07:04 +01:00
Eike Kettner
8fba637ebe
Add folder counts to search summary
2020-12-16 01:14:27 +01:00
Eike Kettner
77627534bc
Improve on basic search summary
2020-12-15 23:37:02 +01:00
Eike Kettner
f3855628d5
Extend query builder with more functions
2020-12-15 23:34:12 +01:00
Eike Kettner
4ca6dfccae
Get basic search summary
2020-12-15 23:10:13 +01:00
Eike Kettner
56d6d2e2ac
Allow changing more parts of a select
2020-12-15 22:12:44 +01:00
Eike Kettner
f1c4b4adb0
Extract find-item query condition
2020-12-15 21:03:47 +01:00
Eike Kettner
2dff686fa0
Introduce unit condition
2020-12-15 21:03:47 +01:00
Eike Kettner
80406cabc2
Refactoring some code into separate files
2020-12-15 21:03:47 +01:00
Eike Kettner
278b1c22c9
Remove old code
2020-12-15 21:03:46 +01:00
Eike Kettner
2cecd01837
Convert rest of QItem
2020-12-15 21:03:46 +01:00
Eike Kettner
d1606d6f16
Remove old commented code
2020-12-15 21:03:46 +01:00
