The current KNN implementation has two areas that can be improved:
- The current behavior is somewhat incorrect. When performing a kNN
query, the current code fetches k items from the index, and then sorts
these items according to Haversine distance. The problem with this
approach is that since the items fetched from the index are ordered by
a Euclidean metric, there is no guarantee that item k + 1 is not closer
than item k in great circle distance, and hence incorrect results can be
returned when closer items beyond k exist.
- The secondary sort is a performance killer. This requires buffering
all k items (again...they were already run through a priority queue in)
the index, and then a sort. Since the items are mostly sorted, and
Go's sort implementation is a quickSort this is the worst case for the
sort algorithm.
Both of these can be fixed by applying a proper distance metric in
the index nearby operation. In addition, this cleans up the code
considerably, removing a number of special cases that applied only
to NEARBY operations.
This change implements a geodetic distance metric that ensures that
the order from the index is correct, eliminating the need for the
secondary sort and special filtering cases in the ScanWriter code.
The big change is that the GeoJSON package has been completely
rewritten to fix a few of geometry calculation bugs, increase
performance, and to better follow the GeoJSON spec RFC 7946.
GeoJSON updates
- A LineString now requires at least two points.
- All json members, even foreign, now persist with the object.
- The bbox member persists too but is no longer used for geometry
calculations. This is change in behavior. Previously Tile38 would
treat the bbox as the object's physical rectangle.
- Corrections to geometry intersects and within calculations.
Faster spatial queries
- The performance of Point-in-polygon and object intersect operations
are greatly improved for complex polygons and line strings. It went
from O(n) to roughly O(log n).
- The same for all collection types with many children, including
FeatureCollection, GeometryCollection, MultiPoint, MultiLineString,
and MultiPolygon.
Codebase changes
- The pkg directory has been renamed to internal
- The GeoJSON internal package has been moved to a seperate repo at
https://github.com/tidwall/geojson. It's now vendored.
Please look out for higher memory usage for datasets using complex
shapes. A complex shape is one that has 64 or more points. For these
shapes it's expected that there will be increase of least 54 bytes per
point.
Mike Poindexter