This commit changes the logic for managing the expiration of
objects in the database.
Before: There was a server-wide hashmap that stored the
collection key, id, and expiration timestamp for all objects
that had a TTL. The hashmap was occasionally probed at 20
random positions, looking for objects that have expired. Those
expired objects were immediately deleted, and if there was 5
or more objects deleted, then the probe happened again, with
no delay. If the number of objects was less than 5 then the
there was a 1/10th of a second delay before the next probe.
Now: Rather than a server-wide hashmap, each collection has
its own ordered priority queue that stores objects with TTLs.
Rather than probing, there is a background routine that
executes every 1/10th of a second, which pops the expired
objects from the collection queues, and deletes them.
The collection/queue method is a more stable approach than
the hashmap/probing method. With probing, we can run into
major cache misses for some cases where there is wide
TTL duration, such as in the hours or days. This may cause
the system to occasionally fall behind, leaving should-be
expired objects in memory. Using a queue, there is no
cache misses, all objects that should be expired will be
right away, regardless of the TTL durations.
Fixes#616
Disabled some fence tests due to them hanging and
timing out in the Github Actions workflow, which is
something that did not happen with Travis CI.
I suspect that there's something up with Go routines
and or num of CPUs in the GH virtual environment.
Tests pass locally though
This commit fixes a case where a roaming geofence will not fire
a "faraway" event when it's supposed to.
The fix required rewriting the nearby/faraway detection logic. It
is now much more accurate and takes overall less memory, but it's
also a little slower per operation because each object proximity
is checked twice per update. Once to compare the old object's
surrounding, and once to evaulated the new object. The two lists
are then used to generate accurate "nearby" and "faraway" results.
The big change is that the GeoJSON package has been completely
rewritten to fix a few of geometry calculation bugs, increase
performance, and to better follow the GeoJSON spec RFC 7946.
GeoJSON updates
- A LineString now requires at least two points.
- All json members, even foreign, now persist with the object.
- The bbox member persists too but is no longer used for geometry
calculations. This is change in behavior. Previously Tile38 would
treat the bbox as the object's physical rectangle.
- Corrections to geometry intersects and within calculations.
Faster spatial queries
- The performance of Point-in-polygon and object intersect operations
are greatly improved for complex polygons and line strings. It went
from O(n) to roughly O(log n).
- The same for all collection types with many children, including
FeatureCollection, GeometryCollection, MultiPoint, MultiLineString,
and MultiPolygon.
Codebase changes
- The pkg directory has been renamed to internal
- The GeoJSON internal package has been moved to a seperate repo at
https://github.com/tidwall/geojson. It's now vendored.
Please look out for higher memory usage for datasets using complex
shapes. A complex shape is one that has 64 or more points. For these
shapes it's expected that there will be increase of least 54 bytes per
point.
* Start on lua scripting
* Implement evalsha, script load, script exists, and script flush
* Type conversions from lua to resp/json.
Refactor to make luastate and luascripts persistent in the controller.
* Change controller.command and all underlying commands to return resp.Value.
Serialize only during the ouput.
* First stab at tile38 call from lua
* Change tile38 into tile38.call in Lua
* Property return errors from scripts
* Minor refactoring. No locking on script run
* Cleanup/refactoring
* Create a pool of 5 lua states, allow for more as needed. Refactor.
* Use safe map for scripts. Add a limit for max number of lua states. Refactor.
* Refactor
* Refactor script commands into atomic, read-only, and non-atomic classes.
Proper locking for all three classes.
Add tests for scripts
* More tests for scripts
* Properly escape newlines in lua-produced errors
* Better test for readonly failure
* Correctly convert ok/err messages between lua and resp.
Add pcall, sha1hex, error_reply, status_reply functions to tile38 namespace in lua.
* Add pcall test. Change writeErr to work with string argument
* Make sure eval/evalsha never attempt to write AOF
* Add eval-set and eval-get to benchmarks
* Fix eval benchmark tests, add more
* Improve benchmarks
* Optimizations and refactoring.
* Add lua memtest
* Typo
* Add dependency
* golint fixes
* gofmt fixes
* Add scripting commands to the core/commands.json
* Use ARGV for args inside lua
This commit includes the ability to search for k nearest neighbors using
a NEARBY command. When the LIMIT keyword is included and the 'meters'
param is excluded, the knn algorithm will be used instead of the
standard overlap+haversine algorithm.
NEARBY fleet LIMIT 10 POINT 33.5 -115.8
This will find the 10 closest points to 33.5,-115.8.
closes#136, #130, and #138.
ping @tomquas, @joernroeder, and @m1ome
JSET key id path value [RAW]
JGET key id path [RAW]
JDEL key id path
Allows for working with JSON strings, for example:
JSET user 901 name Tom
JGET user 901
> '{"name":"Tom"}'
JSET user 901 name.first Tom
JSET user 901 name.last Anderson
> '{"name":{"first":"Tom","last":"Anderson"}'
JDEL user 901 name.last
> '{"name":{"first":"Tom"}'
All commands use the GJSON path syntax, for more information:
Setting JSON: https://github.com/tidwall/sjson
Getting JSON: https://github.com/tidwall/gjson
Benjamin Ramser