eikek
3ee0846e19
Remove fts_migration table
...
It is now stored it SOLR instead.
2021-06-07 17:53:47 +02:00
eikek
c2170d2daf
Fix tag category search being case insensitive
2021-06-03 21:44:55 +02:00
Eike Kettner
25788a0b23
Add routes for storing/retrieving client settings
2021-05-27 21:34:05 +02:00
Eike Kettner
a1a93e5ca6
Fixes searching items with fulltext
...
When using fulltext only search, then only the index must be searched.
This wasn't working anymore, because the routes added a query to
always select valid items (those not being processed). But this lead
to the downstream code to always consult the database, too. Since the
routes are using a "simple-search" interface, this is now adding the
valid-state condition if applicable. There are still more low-level
interfaces that can be used when searching should be done differently.
Closes : #823
2021-05-23 14:14:25 +02:00
Stefan Scheidewig
6149a2ab89
Restored unused imports to make it compile again
2021-04-15 18:34:54 +02:00
Stefan Scheidewig
fa34312020
Implemented endpoint to delete multiple attachments
2021-04-15 18:05:01 +02:00
Eike Kettner
994e3df597
Fix query for getting tag categoy summary
2021-04-12 13:40:22 +02:00
Eike Kettner
3e0914ece7
Correctly count tag categories
...
If multiple tags of the same category are applied to the same item,
just summing tag counts will produce the wrong results as now items
are counted multiple times.
2021-04-11 14:34:44 +02:00
Eike Kettner
4041018c47
Reduce not expressions
...
Fixes queries containing macros inside a "not".
2021-04-11 12:57:42 +02:00
Eike Kettner
e1bbc2edf5
Apply autoformat
2021-04-10 16:31:58 +02:00
Scala Steward
144ea852bf
Update fs2-core, fs2-io to 2.5.4
2021-03-31 21:10:42 +02:00
Eike Kettner
c36073b852
Allow to give human readable summary to user tasks
2021-03-27 22:13:13 +01:00
Eike Kettner
cc38b850a6
Remove deprecated search routes and some refactoring
2021-03-27 22:13:13 +01:00
Eike Kettner
177488817d
Fix h2 migration
...
Using java source code obviously requires `javac` during migration.
2021-03-13 16:38:48 +01:00
Eike Kettner
2e443bc9b9
Fix mariadb migration
2021-03-13 15:52:38 +01:00
Eike Kettner
0229a867af
Add a use colum to metadata entities
2021-03-10 23:55:18 +01:00
Eike Kettner
6a63694a3e
Convert unit tests to munit
2021-03-10 19:48:56 +01:00
Eike Kettner
77a87782b7
Refactoring parser
...
- put all used strings in one place to have it easier to track
- don't use `$` for shortcuts, it's a detail not interesting to a
user; now names must not clash (which is a good idea anyways)
- Added two more shortcuts `conc` and `corr`
2021-03-08 22:51:14 +01:00
Eike Kettner
e681ffa96f
Extend query builder allowing more conditions
...
Before only a column or a dbfunction could be used in a condition. It
is now allowed for all `SelectExpr`.
2021-03-08 22:51:08 +01:00
Eike Kettner
30c901ddf1
Add more ways to query for attachments
...
- find items with a specified attachment count
- find items by attachment id
2021-03-08 09:49:38 +01:00
Eike Kettner
2b2f913e85
Add checksum query expr
2021-03-08 01:53:21 +01:00
Eike Kettner
1c834cbb77
Correctly compare numeric field values
2021-03-03 22:54:55 +01:00
Eike Kettner
71985244f1
Use a better representation for macros
2021-03-03 00:44:49 +01:00
Eike Kettner
a48504debb
Specificly search for field id vs name
2021-03-02 21:09:31 +01:00
Eike Kettner
168f5a1a98
Fix like search for custom fields
2021-03-01 20:56:23 +01:00
Eike Kettner
f8307f77c6
Search by field id or name
2021-03-01 20:56:23 +01:00
Eike Kettner
698ff58aa3
Provide a more convenient interface to search
2021-03-01 11:50:07 +01:00
Eike Kettner
d737da768e
Move to munit in query module
2021-03-01 00:51:01 +01:00
Eike Kettner
9013d9264e
Add more convenient date parsers and some basic macros
2021-03-01 00:51:01 +01:00
Eike Kettner
af73b59ec2
Parser improvements
...
- default expressions into a and node
- fix parsing string lists that end in whitespace
- fix package names of internal classes
2021-03-01 00:51:01 +01:00
Eike Kettner
a80d73d5d2
Optimize imports
2021-03-01 00:51:01 +01:00
Eike Kettner
e9ed998e3a
Basic poc to search via custom query
2021-03-01 00:51:01 +01:00
Eike Kettner
186014a1c6
Refactor search to separate between a base query and user query
...
The `findBase` is adding only strictly required conditions. Everything
else comes from the user.
2021-03-01 00:51:01 +01:00
Eike Kettner
c3cdec416c
Sketching some basic tests
2021-03-01 00:50:52 +01:00
Eike Kettner
be5c7ffb88
First draft of ast and parser
2021-03-01 00:46:57 +01:00
Eike Kettner
e6d9ce2c37
Remove obsolete type capabilities
...
These are now detected by the new scala compiler and lead to compile
errors.
2021-03-01 00:16:30 +01:00
Eike Kettner
7ef3185659
Add language to a source
...
Allows to define upload urls for different languages.
2021-02-18 23:34:42 +01:00
Eike Kettner
d7bc963450
Cleanup nodes that are not reachable anymore
2021-02-18 00:37:18 +01:00
Eike Kettner
5181283b1b
Add a short-name to organizations
2021-02-17 22:55:35 +01:00
Eike Kettner
20ccdda609
Add a notes field to equipments
2021-02-17 22:39:07 +01:00
Eike Kettner
48eee00c0b
Allow person to be correspondent, concerning or both
2021-02-16 22:49:55 +01:00
Eike Kettner
394aeeccb6
Introduce a sql literal and constants in query builder
...
The h2 jdbc driver could not translate the union query in QCollective
when the `kind` was set via a constant value. Using literals works
here. Renamed the corresponding elements in the query builder.
2021-01-25 00:18:24 +01:00
Eike Kettner
1b66e2af5c
Fix classifier_settings table
2021-01-23 21:30:26 +01:00
Eike Kettner
c7e850116f
Make the text length limit optional
2021-01-22 23:06:50 +01:00
mergify[bot]
38e0a50942
Merge pull request #582 from eikek/delete-org-fix
...
Fix deleting organization
2021-01-21 22:56:56 +00:00
Eike Kettner
f4a03e7c69
Fix deleting organization
...
The foreign key in person must be resetted.
2021-01-21 21:27:02 +01:00
Eike Kettner
4cba96f390
Always return classifier results as suggestion
...
The classifier results are spliced into the suggestion list at second
place. When linking they are only used if nlp didn't find anything.
2021-01-21 21:05:28 +01:00
Eike Kettner
9957c3267e
Add constraints from config to classifier training
...
For large and/or many documents, training the classifier can lead to
OOM errors. Some limits have been set by default.
2021-01-21 17:46:39 +01:00
Eike Kettner
363cf5aef0
Quote names in sql changesets
2021-01-21 00:22:58 +01:00
Eike Kettner
38387e00a0
Fix mariadb migration
2021-01-21 00:22:53 +01:00
Eike Kettner
27c24c128d
Store tags guessed with classifier in database
2021-01-20 00:30:40 +01:00
Eike Kettner
9d83cb7fe4
Store item based proposals in separate table
...
Classifier don't work on each attachment, but on all. So the results
must not be stored at an attachment. This reverts some previous
changes to put the classifier results for item entities into its own
table.
2021-01-19 23:48:09 +01:00
Eike Kettner
3ff9284a64
Return classifier results as suggestions
2021-01-19 23:13:51 +01:00
Eike Kettner
1cd3441462
Run classifier for item entities (concerned, correspondent)
...
Store the results separately from nlp results in attachment metadata.
2021-01-19 22:08:29 +01:00
Eike Kettner
d124f0c1a9
Rename db changeset
...
It's not just a fix, but adds new things
2021-01-19 22:08:29 +01:00
Eike Kettner
99dcaae66b
Learn classifiers for item entities
...
Learns classifiers for concerned and correspondent entities. This can
be used as an alternative to or after nlp.
2021-01-19 20:54:47 +01:00
Eike Kettner
a6f29153c4
Control what tag categories to use for auto-tagging
2021-01-19 01:20:13 +01:00
Eike Kettner
cce8878898
Exclude tags w/o category from classifying; remove obsolete models
2021-01-18 21:51:49 +01:00
Eike Kettner
3e28ce1254
Add the sql concat function to query builder
2021-01-18 21:51:45 +01:00
Eike Kettner
249f9e6e2a
Extend guessing tags to all tag categories
2021-01-18 21:51:45 +01:00
Eike Kettner
f01646aeb5
Reorganize nlp pipeline and add nlp-unsupported language italian
...
Improves and reorganizes how nlp pipelines are setup. Now users can
choose from many options, depending on their hardware and usage
scenario.
This is the base to use more languages without depending on what
stanford-nlp supports. Support then is involves to text extraction and
simple regex-ner processing.
2021-01-18 17:41:40 +01:00
Eike Kettner
a70e9ab614
Store used language for processing on attachmentmeta
...
Issue: #570
2021-01-17 22:56:33 +01:00
Eike Kettner
f0f0e6e0d4
Search for categories case-insensitive
...
The string was already lowercased, but the comparison was not.
Fixes #568
2021-01-17 20:10:24 +01:00
Eike Kettner
623a61dbb6
Introduce a lowerEq operator to the query builder
2021-01-17 20:10:00 +01:00
Eike Kettner
3fccc3df39
Return all tags in search stats result
...
Before only tags with a count > 0 were included. Now those that have
not attached to any item are returned as well.
2021-01-11 12:13:13 +01:00
Eike Kettner
0cfd8974d3
Add a flag to imap settings to enable/disable oauth2 scheme
2021-01-04 11:03:04 +01:00
Eike Kettner
95fd386c14
Fixing find-by-checksum with exclusions
...
The NOT-IN query must check for null separately, as everything with
null evaluates to false in sql resulting in not finding existing
duplicates.
2021-01-03 12:29:03 +01:00
Eike Kettner
97dfcece97
Fix duplicate check on restarts
...
Issue: #530
2021-01-02 21:18:05 +01:00
Eike Kettner
a9ed0364d2
Fix linking guessed tags
...
Since tag names are lower-cased the search must happen lower-cased, too.
2021-01-02 01:30:31 +01:00
Eike Kettner
36858da624
Fix search condition for empty items set
2020-12-17 23:07:04 +01:00
Eike Kettner
8fba637ebe
Add folder counts to search summary
2020-12-16 01:14:27 +01:00
Eike Kettner
77627534bc
Improve on basic search summary
2020-12-15 23:37:02 +01:00
Eike Kettner
f3855628d5
Extend query builder with more functions
2020-12-15 23:34:12 +01:00
Eike Kettner
4ca6dfccae
Get basic search summary
2020-12-15 23:10:13 +01:00
Eike Kettner
56d6d2e2ac
Allow changing more parts of a select
2020-12-15 22:12:44 +01:00
Eike Kettner
f1c4b4adb0
Extract find-item query condition
2020-12-15 21:03:47 +01:00
Eike Kettner
2dff686fa0
Introduce unit condition
2020-12-15 21:03:47 +01:00
Eike Kettner
80406cabc2
Refactoring some code into separate files
2020-12-15 21:03:47 +01:00
Eike Kettner
278b1c22c9
Remove old code
2020-12-15 21:03:46 +01:00
Eike Kettner
2cecd01837
Convert rest of QItem
2020-12-15 21:03:46 +01:00
Eike Kettner
d1606d6f16
Remove old commented code
2020-12-15 21:03:46 +01:00
Eike Kettner
266fec9eb5
Convert find items query
2020-12-15 21:03:46 +01:00
Eike Kettner
5e2c5d2a50
Extends query builder
2020-12-15 21:03:46 +01:00
Eike Kettner
35c62049f5
Start converting QItem
2020-12-15 21:03:46 +01:00
Eike Kettner
a355767fdb
Convert all query libs besides QItem
2020-12-15 21:03:46 +01:00
Eike Kettner
fd6d09587d
Convert more records
2020-12-15 21:03:46 +01:00
Eike Kettner
613696539f
Minor refactorings
2020-12-15 21:03:46 +01:00
Eike Kettner
d6f28d3eca
Convert folder
2020-12-15 21:03:46 +01:00
Eike Kettner
87eb8c7f55
Convert more records
2020-12-15 21:03:46 +01:00
Eike Kettner
e3f6892abd
Convert job record
2020-12-15 21:03:46 +01:00
Eike Kettner
1aa1f4367e
Convert periodic tasks
2020-12-15 21:03:46 +01:00
Eike Kettner
3cef932ccd
Convert more records
2020-12-15 21:03:46 +01:00
Eike Kettner
fe4815c737
Convert RSentMail
2020-12-15 21:03:46 +01:00
Eike Kettner
5cbf0d5602
Convert more records
2020-12-15 21:03:46 +01:00
Eike Kettner
10b49fccf8
Converting user and userimap records
2020-12-15 21:03:46 +01:00
Eike Kettner
c5c7f7ed3b
Convert equipment record
2020-12-15 21:03:46 +01:00
Eike Kettner
adee496b77
Convert source record
2020-12-15 21:03:46 +01:00
Eike Kettner
2dbb1db2fd
Initial outline for a simple query builder
2020-12-15 21:03:46 +01:00
Eike Kettner
27d087b14c
Fix foreign key constraints
2020-12-14 14:34:22 +01:00
Eike Kettner
a0642905db
Use remember-me cookie if present
2020-12-04 17:59:25 +01:00
