tile38/README.md

Tile38 is an open source (MIT licensed), in-memory geolocation data store, spatial index, and realtime geofence. It supports a variety of object types including lat/lon points, bounding boxes, XYZ tiles, Geohashes, and GeoJSON.

Features

• Spatial index with search methods such as Nearby, Within, and Intersects.
• Realtime geofencing.
• Variety of client protocols, including http (curl), websockets, telnet, and a native interface.
• Object types of lat/lon, bbox, Geohash, GeoJSON, QuadKey, and XYZ tile.
• Server responses are in json.
• Full command line interface.
• Leader / follower replication.
• In-memory database that persists on disk.
• Simliar feel and syntax style to the fantastic Redis api.

Components

• tile38-server - The server
• tile38-cli - Command line interface tool

Building Tile38

Tile38 can be compiled and used on Linux, OSX, Windows, FreeBSD, and probably others since the codebase is 100% Go. We support both 32 bit and 64 bit systems. Go must be installed on the build machine.

To build everything simply:

$ make

To test:

$ make test

Running

For command line options invoke:

$ ./tile38-server -h

To run a single server:

$ ./tile38-server

# The tile38 shell connects to localhost:9851
$ ./tile38-cli
> help

Playing with Tile38

Basic operations:

$ ./tile38-cli

# add a couple of points named 'truck1' and 'truck2' to a collection named 'fleet'.
> set fleet truck1 point 33.5123 -112.2693   # on the Loop 101 in Phoenix
> set fleet truck2 point 33.4626 -112.1695   # on the I-10 in Phoenix

# search the 'fleet' collection.
> scan fleet                                 # returns both trucks in 'fleet'
> nearby fleet point 33.462 -112.268 6000    # search 6 kilometers around a point. returns one truck.

# crud operations
> get fleet truck1                           # returns 'truck1'
> del fleet truck2                           # deletes 'truck2'
> drop fleet                                 # removes all 

Fields

Fields are extra data that belongs to an object. A field is always a double precision floating point. There is no limit to the number of fields that an object can have.

To set a field when setting an object:

> set fleet truck1 field speed 90 point 33.5123 -112.2693             
> set fleet truck1 field speed 90 field age 21 point 33.5123 -112.2693

To set a field when an object already exists:

> fset fleet truck1 speed 90

Searching

Tile38 has support to search for objects and points that are within or intersects other objects. All object types can be searched including Polygons, MultiPolygons, GeometryCollections, etc.

#### Within WITHIN searches a collection for objects that are fully contained inside a specified bounding area.
#### Intersects INTERSECTS searches a collection for objects that intersect a specified bounding area.
#### Nearby NEARBY searches a collection for objects that intersect a specified radius.

Search options

SPARSE - This option will distribute the results of a search evenly across the requested area.
This is very helpful for example; when you have many (perhaps millions) of objects and do not want them all clustered together on a map. Sparse will limit the number of objects returned and provide them evenly distributed so that your map looks clean.

You can choose a value between 1 and 8. The value 1 will result in no more than 4 items. The value 8 will result in no more than 65536. 1=4, 2=16, 3=64, 4=256, 5=1024, 6=4098, 7=16384, 8=65536.

No Sparsing

Sparse 1

Sparse 2

Sparse 3

Sparse 4

Sparse 5

Sparse 6

*Please note that the higher the sparse value, the slower the performance. Also, LIMIT and CURSOR are not available when using SPARSE.*

WHERE - This option allows for filtering out results based on field values. For example
nearby fleet where speed 70 +inf point 33.462 -112.268 6000 will return only the objects in the 'fleet' collection that are within the 6 km radius and have a field named speed that is greater than 70.

Multiple WHEREs are concatenated as and clauses. WHERE speed 70 +inf WHERE age -inf 24 would be interpreted as speed is over 70 and age is less than 24.

The default value for a field is always 0. Thus if you do a WHERE on the field speed and an object does not have that field set, the server will pretend that the object does and that the value is Zero.

MATCH - MATCH is similar to WHERE except that it works on the object id instead of fields.
nearby fleet match truck* point 33.462 -112.268 6000 will return only the objects in the 'fleet' collection that are within the 6 km radius and have an object id that starts with truck. There can be multiple MATCH options in a single search. The MATCH value is a simple glob pattern.

CURSOR - CURSOR is used to iterate though many objects from the search results. An iteration begins when the CURSOR is set to Zero or not included with the request, and completes when the cursor returned by the server is Zero.

NOFIELDS - NOFIELDS tells the server that you do not want field values returned with the search results.

LIMIT - LIMIT can be used to limit the number of objects returned for a single search request.

Geofencing

A [geofence](https://en.wikipedia.org/wiki/Geo-fence) is a virtual boundary that can detect when an object enters or exits the area. This boundary can be a radius, bounding box, or a polygon. Tile38 can turn any standard search into a geofence monitor by adding the FENCE keyword to the search.

A simple example:

> nearby fleet fence point 33.462 -112.268 6000

This command opens a geofence that monitors the 'fleet' collection. The server will respond with:

{"ok":true,"live":true}

And the connection will be kept open. If any object enters or exits the 6 km radius around 33.462,-112.268 the server will respond in realtime with a message such as:

{"command":"set","detect":"enter","id":"truck02","object":{"type":"Point","coordinates":[-112.2695,33.4626]}}

The server will notify the client if the command is del | set | drop.

• del notifies the client that an object has been deleted from the collection that is being fenced.
• drop notifies the client that the entire collection is dropped.
• set notifies the client that an object has been added or updated, and when it's position is detected by the fence.

The detect may be enter | exit | cross.

• enter is when an object that was not previously in the fence has entered the area.
• exit is when an object that was previously in the fence has exited the area.
• cross is when an object that was not previously in the fence has entered and exited the area.

Object types

All object types except for XYZ Tiles and QuadKeys can be stored in a collection. XYZ Tiles and QuadKeys are reserved for the SEARCH keyword only.

Lat/lon point

The most basic object type is a point that is composed of a latitude and a longitude. There is an optional z member that may be used for auxilary data such as elevation or a timestamp.

set fleet truck1 point 33.5123 -112.2693     # plain lat/lon
set fleet truck1 point 33.5123 -112.2693 225 # lat/lon with z member

Bounding box

A bounding box consists of two points. The first being the southwestern most point and the second is the northeastern most point.

set fleet truck1 bounds 30 -110 40 -100

Geohash

A geohash is a string respresentation of a point. With the length of the string indicating the precision of the point.

set fleet truck1 hash 9tbnthxzr # this would be equivlent to 'point 33.5123 -112.2693'

GeoJSON

GeoJSON is an industry standard format for representing a variety of object types including a point, multipoint, linestring, multilinestring, polygon, multipolygon, geometrycollection, feature, and featurecollection. Tile38 supports all of the standards with these exceptions.

1. The crs member is not supported and will be ignored. The CRS84/WGS84 projection is assumed.
2. Any member that is not recognized (including crs) will be ignored.
3. All coordinates can be 2 or 3 axes. Less than 2 axes or more than 3 will result in a parsing error.

* All ignored members will not persist.

Important to note that all coordinates are in Longitude, Latitude order.

set city tempe object {"type":"Polygon","coordinates":[[[0,0],[10,10],[10,0],[0,0]]]}

XYZ Tile

An XYZ tile is rectangle bounding area on earth that is represented by an X, Y coordinate and a Z (zoom) level. Check out maptiler.org for an interactive example.

QuadKey

A QuadKey used the same coordinate system as an XYZ tile except that the string representation is a string characters composed of 0, 1, 2, or 3. For a detailed explanation checkout The Bing Maps Tile System.

Network protocols

It's recommended to use the native interface, but there are times when only HTTP is available or when you need to test from a remote terminal. In those cases we provide an HTTP and telnet options.

HTTP

One of the simplest ways to call a tile38 command is to use HTTP. From the command line you can use curl. For example:

# call with request in the body
curl --data "set fleet truck3 point 33.4762 -112.10923" localhost:9851

# call with request in the url path
curl localhost:9851/set+fleet+truck3+point+33.4762+-112.10923

Websockets

Websockets can be used when you need to Geofence and keep the connection alive. It works just like the HTTP example above, with the exception that the connection stays alive and the data is sent from the server as text websocket messages.

Telnet

There is the option to use a plain telnet connection.

telnet localhost 9851
set fleet truck3 point 33.4762 -112.10923
{"ok":true,"elapsed":"18.73µs"}

Native interface

The native interface is very simple. A single message is composed of a '$' + TEXT_DATA_SIZE + SPACE + DATA + CRLF.

So the request message:

get fleet truck1

Should be sent to the server as (without quotes):

"$16 get fleet truck1\r\n"

The server will always respond in JSON, and will include the top level member ok. When ok is false there will also be an accompanied err member describing the problem. In nearly every response there will also be an elapsed member that is the duration of time that it took to process the request on the server. For more information on this string please refer to the time.Duration Go documentation.

So the response message:

{"ok":true,"elapsed":"37.829µs"}

Will be sent to the client as (without quotes):

"$32 {"ok":true,"elapsed":"37.829µs"}\r\n"

Clients

Currently we have only one native client written in Go. Though is should be trivial to write one in your language of choice.

• Go

Commands

This is the full list of commands available to Tile38.

Crud

GET key id [OBJECT|POINT|BOUNDS|(HASH precision)]
summary: Get the object of an id

SET key string [FIELD name value ...] (OBJECT geojson)|(POINT lat lon [z])|(BOUNDS minlat minlon maxlat maxlon)|(HASH geohash)
summary: Sets the value of an id

FSET key id field value
summary: Set the value for a single field of an id

DEL key id
summary: Delete an id from a key

DROP key
summary: Remove a key from the database

KEYS pattern
summary: Finds all keys matching the given pattern

STATS key [key ...]
summary: Show stats for one or more keys

Search

INTERSECTS key [CURSOR start] [LIMIT count] [SPARSE spread] [MATCH pattern] [WHERE field min max ...] [NOFIELDS] [FENCE] [COUNT|IDS|OBJECTS|POINTS|BOUNDS|(HASHES precision)] (GET key id)|(BOUNDS minlat minlon maxlat maxlon)|(OBJECT geojson)|(TILE x y z)|(QUADKEY quadkey)|(HASH precision)
summary: Searches for ids that are nearby a point

NEARBY key [CURSOR start] [LIMIT count] [SPARSE spread] [MATCH pattern] [WHERE field min max ...] [NOFIELDS] [FENCE] [COUNT|IDS|OBJECTS|POINTS|BOUNDS|(HASHES precision)] POINT lat lon meters
summary: Searches for ids that are nearby a point

WITHIN key [CURSOR start] [LIMIT count] [SPARSE spread] [MATCH pattern] [WHERE field min max ...] [NOFIELDS] [FENCE] [COUNT|IDS|OBJECTS|POINTS|BOUNDS|(HASHES precision)] (GET key id)|(BOUNDS minlat minlon maxlat maxlon)|(OBJECT geojson)|(TILE x y z)|(QUADKEY quadkey)|(HASH precision)
summary: Searches for ids that are nearby a point

SCAN key [CURSOR start] [LIMIT count] [MATCH pattern] [WHERE field min max ...] [NOFIELDS] [COUNT|IDS|OBJECTS|POINTS|BOUNDS|(HASHES precision)]
summary: Incrementally iterate though a key

Connection

AUTH password
summary: Authenticate to the server

PING
summary: Ping the server

QUIT
summary: Close the connection

Server

CONFIG GET parameter
summary: Get the value of a configuration parameter

CONFIG REWRITE
summary: Rewrite the configuration file with the in memory configuration

CONFIG SET parameter [value]
summary: Set a configuration parameter to the given value

FLUSHDB
summary: Removes all keys

GC
summary: Forces a garbage collection

READONLY yes|no
summary: Turns on or off readonly mode

SERVER
summary: Show server stats and details

Replication

FOLLOW host port
summary: Follows a leader host

AOFSHRINK
summary: Shrinks the aof in the background

AOF pos
summary: Downloads the AOF start from pos and keeps the connection alive

AOFMD5 pos size
summary: Performs a checksum on a portion of the aof

Contact

Josh Baker @tidwall

License

Tile38 source code is available under the MIT License.