mirror
/
tile38
mirror of https://github.com/tidwall/tile38.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,291 Commits 14 Branches 101 Tags 54 MiB
Commit Graph

136 Commits

Author SHA1 Message Date
tidwall f9e6f74753 Use struct for server options 2021-09-29 07:19:00 -07:00
Benjamin Ramser c87fd4836e test: add more tests 2021-09-05 12:56:45 +02:00
Benjamin Ramser 6eb1cca78c fix: point -> circle fallthrough 2021-09-05 12:05:33 +02:00
Benjamin Ramser bc62edb692 feat: add sector 2021-09-05 11:48:34 +02:00
tidwall 694ad1fb62 Update btree/buntdb 2021-07-31 07:42:58 -07:00
tidwall 401670e621 Fix NEARBY with SPARSE returning too many results 
fixes #618
 2021-07-22 08:39:57 -07:00
tidwall 9e68703841 Update expiration logic 
This commit changes the logic for managing the expiration of
objects in the database.

Before: There was a server-wide hashmap that stored the
collection key, id, and expiration timestamp for all objects
that had a TTL. The hashmap was occasionally probed at 20
random positions, looking for objects that have expired. Those
expired objects were immediately deleted, and if there was 5
or more objects deleted, then the probe happened again, with
no delay. If the number of objects was less than 5 then the
there was a 1/10th of a second delay before the next probe.

Now: Rather than a server-wide hashmap, each collection has
its own ordered priority queue that stores objects with TTLs.
Rather than probing, there is a background routine that
executes every 1/10th of a second, which pops the expired
objects from the collection queues, and deletes them.

The collection/queue method is a more stable approach than
the hashmap/probing method. With probing, we can run into
major cache misses for some cases where there is wide
TTL duration, such as in the hours or days. This may cause
the system to occasionally fall behind, leaving should-be
expired objects in memory. Using a queue, there is no
cache misses, all objects that should be expired will be
right away, regardless of the TTL durations.

Fixes #616
 2021-07-12 13:37:50 -07:00
tidwall 3c699183e0 Additional KNN test 2021-07-11 14:49:23 -07:00
tidwall 579a41abae Merge branch 'housecanary-fix-knn' 2021-07-11 10:02:59 -07:00
tidwall 0d83b1ca53 Merge branch 'rshura-optimize-field-match' 2021-07-10 17:59:22 -07:00
Josh 1467cba769
Merge pull request #552 from rshura/clip-by 
Add CLIPBY subcommand to INTERSECTS/WITHIN
 2021-07-10 09:24:57 -07:00
Oliver 56a0ad70c6 fix tests 2021-05-13 21:14:06 -04:00
Oliver de055384f9 add tests 2021-05-13 21:14:06 -04:00
Oliver 6b0282d960 add more metrics, add metrics index page 2021-05-13 21:14:05 -04:00
Oliver 7ec4d58392 cleanup / fix tests 2021-05-13 21:14:05 -04:00
Oliver c78415e26e go fmt 2021-05-13 21:12:49 -04:00
Benjamin Ramser 2a8b98778b fix: distance if point and object have the same coordinates 2021-03-19 09:42:38 +01:00
tidwall 4f4954020f enable fence tests 2020-11-03 14:56:31 -07:00
tidwall 4504ea75ac Updated Github Action files 
Disabled some fence tests due to them hanging and
timing out in the Github Actions workflow, which is
something that did not happen with Travis CI.

I suspect that there's something up with Go routines
and or num of CPUs in the GH virtual environment.

Tests pass locally though
 2020-11-03 14:36:07 -07:00
tidwall 100be7be3c Added tests 2020-10-23 09:50:51 -07:00
tidwall b1dc463c41 Updated dependencies 2020-09-22 16:43:58 -07:00
Alex Roitman 25579a052c Fix a bug in WHEREIN -- 0 values would always match, incorrectly. 2020-04-12 16:06:10 -07:00
Mike Poindexter fe6e3863ba Add bench tests for knn 2020-04-08 11:36:37 -07:00
Alex Roitman 34cb2affdc Add clipby subcommand to INTERSECTS/WITHIN 2020-04-03 16:49:06 -07:00
tidwall ff48054d3d Fixed a missing faraway event for roaming geofences 
This commit fixes a case where a roaming geofence will not fire
a "faraway" event when it's supposed to.

The fix required rewriting the nearby/faraway detection logic. It
is now much more accurate and takes overall less memory, but it's
also a little slower per operation because each object proximity
is checked twice per update. Once to compare the old object's
surrounding, and once to evaulated the new object. The two lists
are then used to generate accurate "nearby" and "faraway" results.
 2020-03-22 11:54:56 -07:00
Josh Baker 90c2474e3d
Merge pull request #501 from JordanArmstrong/strict-jset-numbers 
Strictly check if values passed to JSET are numbers
 2019-10-29 15:28:40 -07:00
Josh Baker df477bf3f4
Merge pull request #464 from rshura/area-expression 
Add area expressions.
 2019-10-28 13:45:10 -07:00
Jordan Armstrong 41fb410e2e Strictly check if values to JSET are numbers 
Fixes #493
 2019-10-26 23:37:48 -03:00
tidwall f7888c1edf Fixed malformed json for chans command 
Mentioned by ds2xor on Slack
 2019-08-03 10:10:28 -07:00
Alex Roitman dd09ffbe13 Fix parser for multiple negations. 2019-06-14 10:02:26 -07:00
Alex Roitman 2d83e18934 Add expression errors test. Make parser stricter. 2019-06-13 13:10:47 -07:00
Alex Roitman 0c3a5d02ca Fixes 2019-06-13 12:04:04 -07:00
Alex Roitman 496ace25d3 Refactoring/cleanup/fixes. 2019-06-11 17:13:33 -07:00
Alex Roitman d0ca579f5c Add area expressions. 
Add parser.
Hook up to test command.
Add tests for expressions in test command.
 2019-06-10 14:47:42 -07:00
Melissa Baker bedddcbe13 Added RESP output fields test for NEARBY/WITHIN 2019-06-04 10:43:32 -07:00
Melissa Baker 4ba237b10e Added test for correct json fields output 2019-05-28 12:28:56 -07:00
tidwall b86d2287f4 Removed elapsed member for json testing 2019-05-28 09:22:15 -07:00
Alex Roitman cbf10b7472 Rework timeouts to allow prepending any command with the TIMEOUT seconds 2019-04-25 14:15:53 -07:00
Alex Roitman 2020f537dc Add tests for script timeouts. 2019-04-24 17:17:49 -07:00
Alex Roitman 31525487c3 Add timeout tests. Fix a bug. 2019-04-24 13:20:57 -07:00
tidwall fc3e8b4359 Fix nearby with match query invalid results 
closes #421
 2019-03-01 06:55:26 -07:00
tidwall 30f903bd51 Require properties member for geojson features 2019-02-12 06:49:13 -07:00
tidwall fb7259b10b Changed clip errors and json result type 2019-02-12 05:33:20 -07:00
Alex Roitman 1cffdcc31f Add tests for the TEST comand 2019-02-08 13:58:04 -08:00
Steven Wolfe ffd8ca915c Testing for valid INFO and CLIENT Json output 2019-01-15 11:08:19 -07:00
Steven Wolfe 9c8639497f Adding test for 'not' operator 2019-01-14 11:37:11 -07:00
Steven Wolfe 4392fb44bd Fixing KEYS command and tests 2019-01-14 11:06:12 -07:00
Steven Wolfe 087975abd1 Roaming geofence tests 2019-01-10 11:28:29 -07:00
Steven Wolfe a5fbcf5f15 Ensuring channel publish order 2019-01-09 00:23:53 -07:00
Alex Roitman 5e215c247d Add distance_to function to the tile38 namespace in lua. 
distance_to(lat_a, lon_a, lat_b, lon_b) returns distance in meters between points A and B.
 2019-01-04 14:57:00 -08:00
Alex Roitman e1c82e17f7 Refactor to be more consistent with redis responses. 2018-12-28 14:46:54 -08:00
Alex Roitman 01a7dda2a1 Add RENAME and RENAMENX commands. 2018-12-27 17:15:53 -08:00
Alex Roitman 0933c541f4 Refactor cursor/paging. 2018-10-31 22:01:37 -07:00
tidwall 60b054d58c Fixed test 2018-10-29 08:16:04 -07:00
tidwall e577f60481 Updated redigo imports 2018-10-29 05:00:54 -07:00
tidwall 555e47036c Replaced net package with evio 
- Added threads startup flag
- Replaced net package with evio
- Refactored controller into server
 2018-10-28 15:51:47 -07:00
Alex Roitman 7c803f355b Make NEARBY always use knn 2018-10-25 16:37:06 -07:00
tidwall cc75cf22a8 Fix #369 poly in hole query 2018-10-18 06:28:31 -07:00
Alex Roitman 1517a2a770 Add unit tests for WITHIN and INTERSECTS. 2018-10-17 13:34:11 -07:00
tidwall 6257ddba78 Faster point in polygon / GeoJSON updates 
The big change is that the GeoJSON package has been completely
rewritten to fix a few of geometry calculation bugs, increase
performance, and to better follow the GeoJSON spec RFC 7946.

GeoJSON updates

- A LineString now requires at least two points.
- All json members, even foreign, now persist with the object.
- The bbox member persists too but is no longer used for geometry
  calculations. This is change in behavior. Previously Tile38 would
  treat the bbox as the object's physical rectangle.
- Corrections to geometry intersects and within calculations.

Faster spatial queries

- The performance of Point-in-polygon and object intersect operations
  are greatly improved for complex polygons and line strings. It went
  from O(n) to roughly O(log n).
- The same for all collection types with many children, including
  FeatureCollection, GeometryCollection, MultiPoint, MultiLineString,
  and MultiPolygon.

Codebase changes

- The pkg directory has been renamed to internal
- The GeoJSON internal package has been moved to a seperate repo at
  https://github.com/tidwall/geojson. It's now vendored.

Please look out for higher memory usage for datasets using complex
shapes. A complex shape is one that has 64 or more points. For these
shapes it's expected that there will be increase of least 54 bytes per
point.
 2018-10-13 04:30:48 -07:00
Alex Roitman 656f9df17c Typo 2018-05-04 13:45:51 -07:00
Alex Roitman a01281a526 Add tests for the new CIRCLE subcommand. 2018-05-04 11:42:35 -07:00
Josh Baker 7e9871bb69 Refactor project layout 
Move internal and support packages to pkg directory
 2018-04-19 08:43:32 -07:00
Alex Roitman 9ffca9aa10 Tests and commands.json for WHEREEVAL. 2018-02-15 16:42:07 -08:00
Alex Roitman a3bf8b6572 Tweak multi-field fset to return the count of updated fields 2017-11-10 14:31:38 -08:00
Alex Roitman 79acc0efe5 Allow setting multiple fields in a single fset command. Add xx flag to fset. 2017-11-03 11:50:03 -07:00
Alex Roitman b55300b729 Lua scripting feature. (#224) 
* Start on lua scripting

* Implement evalsha, script load, script exists, and script flush

* Type conversions from lua to resp/json.
Refactor to make luastate and luascripts persistent in the controller.

* Change controller.command and all underlying commands to return resp.Value.
Serialize only during the ouput.

* First stab at tile38 call from lua

* Change tile38 into tile38.call in Lua

* Property return errors from scripts

* Minor refactoring.  No locking on script run

* Cleanup/refactoring

* Create a pool of 5 lua states, allow for more as needed. Refactor.

* Use safe map for scripts.  Add a limit for max number of lua states.  Refactor.

* Refactor

* Refactor script commands into atomic, read-only, and non-atomic classes.
Proper locking for all three classes.
Add tests for scripts

* More tests for scripts

* Properly escape newlines in lua-produced errors

* Better test for readonly failure

* Correctly convert ok/err messages between lua and resp.
Add pcall, sha1hex, error_reply, status_reply functions to tile38 namespace in lua.

* Add pcall test. Change writeErr to work with string argument

* Make sure eval/evalsha never attempt to write AOF

* Add eval-set and eval-get to benchmarks

* Fix eval benchmark tests, add more

* Improve benchmarks

* Optimizations and refactoring.

* Add lua memtest

* Typo

* Add dependency

* golint fixes

* gofmt fixes

* Add scripting commands to the core/commands.json

* Use ARGV for args inside lua
 2017-10-05 08:20:40 -07:00
Josh Baker d3329f07aa discard quiet logs 2017-10-03 08:53:09 -07:00
Alex Roitman 1e4b635e19 Add tests for WHEREIN. Add WHEREIN to the commands.json 2017-08-24 10:57:07 -07:00
Josh Baker cd057088d9 index optimizations 2017-08-10 17:32:40 -07:00
Josh Baker f9fa48db21 use redis-style expires 
Updated the Tile38 expires to match the Redis implmentation at
https://redis.io/commands/expire#how-redis-expires-keys.

It now supports passive and active expires with sub-millisecond
accuracy.

This addresses issue #156
 2017-03-29 12:50:04 -07:00
Josh Baker 374d199d07 detect inside on enter, fixes #150 2017-02-24 06:03:11 -07:00
Josh Baker 7f1b311b3d added fence tests 2017-02-12 07:58:03 -07:00
Josh Baker 04290ec535 KNN results for NEARBY command 
This commit includes the ability to search for k nearest neighbors using
a NEARBY command. When the LIMIT keyword is included and the 'meters'
param is excluded, the knn algorithm will be used instead of the
standard overlap+haversine algorithm.

   NEARBY fleet LIMIT 10 POINT 33.5 -115.8

This will find the 10 closest points to 33.5,-115.8.

closes #136, #130, and #138.
ping @tomquas, @joernroeder, and @m1ome
 2017-01-30 16:41:12 -07:00
Josh Baker 627555cdc8 Merge branch 'memoptz' 2017-01-13 10:03:02 -07:00
Pavel Makarenko a4e5d38461 Disable HTTP & WebSocket transports on demand. (#128) 
* Added .idea folder into .gitignore

* Added --http-transport flag

* Fixed test & Removed some dumps

* Added support of yes/no
 2017-01-13 08:45:28 -07:00
Josh Baker d6ca25d14b updated collection ReplaceOrInsert 10% bump 2016-12-30 18:12:18 -07:00
Josh Baker 63268dad3f added PDEL test 2016-12-29 10:43:22 -07:00
Josh Baker 3e3d364911 test app for #107 2016-12-28 11:16:28 -07:00
Josh Baker 44cf149325 added JSET, JGET, JDEL commands 
JSET key id path value [RAW]
JGET key id path [RAW]
JDEL key id path

Allows for working with JSON strings, for example:

  JSET user 901 name Tom
  JGET user 901
  > '{"name":"Tom"}'
  JSET user 901 name.first Tom
  JSET user 901 name.last Anderson
  > '{"name":{"first":"Tom","last":"Anderson"}'
  JDEL user 901 name.last
  > '{"name":{"first":"Tom"}'

All commands use the GJSON path syntax, for more information:

  Setting JSON: https://github.com/tidwall/sjson
  Getting JSON: https://github.com/tidwall/gjson
 2016-12-12 10:33:28 -07:00
Josh Baker ae0eb64526 updated stats test 2016-12-04 12:35:04 -07:00
Josh Baker 2e41d49c4b wip tests 2016-12-04 08:52:38 -07:00
Josh Baker a664bade48 added black-box testing 2016-12-02 09:14:34 -07:00
Josh Baker cc560646a7 correct log import 2016-09-08 21:37:00 -07:00
Josh Baker a49d1ddf15 test with very large keys 2016-09-08 16:27:28 -07:00
Josh Baker 6d944ada32 fixed #49. fragmented pipeline requests. 2016-09-08 16:11:53 -07:00