Commit Graph

640 Commits

Author SHA1 Message Date
Gert-Jan Timmer 7fe42b0c27 Update: Travis CI
Added Cross-Compile jobs for additional feature(s)
2018-05-24 12:35:44 +02:00
Gert-Jan Timmer 4d6bb50892 Update: Travis CI
Usage of new build tag names
2018-05-24 12:07:11 +02:00
Gert-Jan Timmer 7319fcc0d8 Renamed build option `trace` 2018-05-24 12:06:54 +02:00
Gert-Jan Timmer 5a602a653e Update: README
Add documentation to feature table for tag `trace`
2018-05-24 12:00:17 +02:00
Gert-Jan Timmer 23bb9ee3cb Update: Travis CI 2018-05-24 11:59:17 +02:00
Gert-Jan Timmer ef74e33f4a Update: README
Fix: New build tag(s) names
2018-05-24 11:55:24 +02:00
Gert-Jan Timmer 765f6e93d2 Update prefix of build tags to `sqlite_`
* Keep current build tags backwards compliant
* Added alias for `sqlite_json1` => `sqlite_json`
2018-05-24 11:55:02 +02:00
Gert-Jan Timmer 6f78e03131 Update build tags with prefix `sqlite_` 2018-05-24 11:53:45 +02:00
Gert-Jan Timmer fdd603e859 Merge branch 'master' into update/options 2018-05-24 11:43:17 +02:00
mattn 29ac65fc30
Merge pull request from GJRTimmer/fix/travis-ci
Update Travis CI
2018-05-24 17:53:35 +09:00
mattn 323f3f4bee
Merge pull request from GJRTimmer/fix/wal
ADD: SQLITE_DEFAULT_WAL_SYNCHRONOUS
2018-05-24 17:43:25 +09:00
Gert-Jan Timmer 6510a04266 Add: EOL 2018-05-24 09:25:16 +02:00
mattn 2614d46676
Merge branch 'master' into fix/wal 2018-05-24 10:35:25 +09:00
mattn 323a32be5a
Merge pull request from GJRTimmer/fix/213
Fix 
2018-05-24 10:34:43 +09:00
mattn 9aa67ff0d6
Merge pull request from GJRTimmer/fix/deprecated
ADD: SQLITE_OMIT_DEPRECATED
2018-05-24 10:32:29 +09:00
Gert-Jan Timmer 347aa38e98 ADD: SQLITE_OMIT_DEPRECATED 2018-05-23 23:10:05 +02:00
Gert-Jan Timmer 46cd598502 ADD: SQLITE_DEFAULT_WAL_SYNCHRONOUS 2018-05-23 23:05:26 +02:00
Gert-Jan Timmer 79bf9ba26b ADD: SQLITE_ENABLE_UPDATE_DELETE_LIMIT
Fixes 
2018-05-23 22:43:42 +02:00
Gert-Jan Timmer a909aeb37f Update: README
Add: Feature Table
Add: Links
Add: TOC
Fix: References for TOC
2018-05-23 22:30:51 +02:00
Gert-Jan Timmer ebb175735b Add: SQLITE_DEFAULT_AUTOVACUUM 2018-05-23 22:21:51 +02:00
Gert-Jan Timmer 6929098dc1 Add: SQLITE_ENABLE_STAT4 2018-05-23 22:21:33 +02:00
Gert-Jan Timmer c5ff80d7a8 Add: SQLITE_SECURE_DELETE 2018-05-23 22:21:22 +02:00
Gert-Jan Timmer bdda6e10e3 Add: SQLITE_INTROSPECTION_PRAGMAS 2018-05-23 22:21:11 +02:00
Gert-Jan Timmer e2fc6d49c6 Add: SQLITE_DEFAULT_FOREIGN_KEYS 2018-05-23 22:21:00 +02:00
Gert-Jan Timmer 507e9be49b Add: SQLITE_ENABLE_API_ARMOR 2018-05-23 22:20:48 +02:00
Gert-Jan Timmer de217e4b87 Add: SQLITE_ALLOW_URI_AUTHORITY 2018-05-23 22:20:35 +02:00
Gert-Jan Timmer ba97a23a7f Rename
Renamed files containing current features.
2018-05-23 22:20:06 +02:00
Gert-Jan Timmer 56ffb7a49b Update Travis CI
Add: OSX
Add: Windows Cross-Compile
Add: Golang: 1.10.x

Exclusions:
- Golang: 1.7 from OSX
- Golang: Master does not cause build failure
- Tests for Windows Cross-Platform
2018-05-23 19:23:25 +02:00
mattn a72efd674f
Merge pull request from mattn/open-mode
add _mutex flag to specify SQLITE_OPEN_NOMUTEX or SQLITE_OPEN_FULLMUTEX
2018-04-19 16:32:57 +09:00
mattn 20a06cb34c
Merge pull request from mattn/disable-pread64-android
disable pread64/pwrite64 on android
2018-04-18 11:10:46 +09:00
Yasuhiro Matsumoto aaf32cc923
disable pread64/pwrite64 on android 2018-04-18 10:36:16 +09:00
mattn 1024cb20b4
Merge pull request from mattn/upgrade-3.23.1
bump sqlite3 3.23.1
2018-04-17 23:31:54 +09:00
Yasuhiro Matsumoto d4680fe41a
bump sqlite3 3.23.1 2018-04-17 22:18:20 +09:00
mattn 397c95cd5e
Merge pull request from mjtrangoni/fix-linter-issues
Fix linter issues
2018-04-17 22:17:04 +09:00
Mario Trangoni b75aefcf46 fix small codespell issue 2018-04-17 12:33:03 +02:00
Mario Trangoni 4bde157d91 fix all goconst issues 2018-04-17 12:33:03 +02:00
Mario Trangoni 28e6f6d675 fix all gosimple issues 2018-04-17 12:21:29 +02:00
Mario Trangoni 847660225d fix all unconvert issues 2018-04-17 10:55:42 +02:00
mattn d896508f87
Merge pull request from mattn/upgrade-3.23
bump sqlite 3.23.0
2018-04-04 23:27:34 +09:00
Yasuhiro Matsumoto 953b2322bf
bump sqlite 3.23.0
closes 
2018-04-04 22:48:19 +09:00
Yasuhiro Matsumoto 57ca83473c
add _mutex flag to specify SQLITE_OPEN_NOMUTEX or SQLITE_OPEN_FULLMUTEX 2018-03-16 21:40:16 +09:00
mattn 9101028e5c
Merge pull request from ifraixedes/if-update-readme
README: Update list issues concurrent read/writes
2018-03-07 16:20:36 +09:00
Ivan Fraixedes 766b269f59
README: Update list issues concurrent read/writes
Update the README FAQ section adding one issues more to the list of the
issues related with the concurrent reads and writes. This new added
issues gives a lot of insights, much more than the ones which where
already present.
2018-02-24 19:53:12 +00:00
mattn 696e2e43cb
Merge pull request from navytux/y/no-go-if-notneeded
Don't spawn interrupt goroutine if we know that context cannot be canceled
2018-02-18 22:15:52 +09:00
mattn 2a20b89318
Merge pull request from basvanbeek/master
only enable pread/pwrite for linux. fixes  and fixes 
2018-02-18 07:50:39 +09:00
Bas van Beek bd9ec0844c only enable pread/pwrite for linux. fixes and fixes 2018-02-17 21:38:35 +01:00
Kirill Smelkov 00a23ba538 Don't spawn interrupt goroutine if we know that context cannot be canceled
For a Go-only project the following code pattern

	go func() {
		select {
		case <-ctx.Done():
			// call some cancel

		case <-done:
			// work finished ok
		}
	}()

	// do some work
	close(done)

works good and fast - without high scheduling overhead because scheduler
usually puts spawned goroutine into run queue on the same OS thread and so
after done is closed control is passed to spawned goroutine without OS context
switch.

However in the presence of Cgo calls in "do some work" the situation can
become different - Cgo calls are treated by go runtime similarly to
system calls with the effect that goroutines spawned on original OS
thread tend to be migrated by scheduler to be executed on another OS
thread.

This in turn can bring high overhead for communicating on "done", which
ultimately can result in full context switch: if the spawned goroutine
had chance to run, already checked done and ctx to be not ready, and went
into sleep via wait on futex - showing as something like below in strace for
one read query (note futex calls):

	27867 00:38:39.782146 stat(".../neo.sqlite-journal", 0x7f83809c4a20) = -1 ENOENT (No such file or directory)
	27867 00:38:39.782165 pread64(3, "\0\0\0\33\0\0\10\235\0\0\10]\0\0\0\27", 16, 24) = 16
	27871 00:38:39.782179 <... pselect6 resumed> ) = 0 (Timeout)
	27868 00:38:39.782187 <... pselect6 resumed> ) = 0 (Timeout)
	27871 00:38:39.782193 futex(0xc4200f8538, FUTEX_WAIT, 0, NULL <unfinished ...>
	27868 00:38:39.782199 futex(0xc420013138, FUTEX_WAIT, 0, NULL <unfinished ...>
	27867 00:38:39.782205 stat(".../neo.sqlite-wal", 0x7f83809c4a20) = -1 ENOENT (No such file or directory)
	27867 00:38:39.782224 fstat(3, {st_mode=S_IFREG|0644, st_size=9031680, ...}) = 0
	27867 00:38:39.782247 futex(0xc420013138, FUTEX_WAKE, 1 <unfinished ...>
	27868 00:38:39.782259 <... futex resumed> ) = 0
	27867 00:38:39.782265 <... futex resumed> ) = 1
	27868 00:38:39.782270 pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=3000}, NULL <unfinished ...>
	27867 00:38:39.782279 fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
	27867 00:38:39.782315 fcntl(3, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1}) = 0
	27868 00:38:39.782336 <... pselect6 resumed> ) = 0 (Timeout)
	27867 00:38:39.782342 fcntl(3, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741826, l_len=510} <unfinished ...>
	27868 00:38:39.782348 futex(0xc4200f8538, FUTEX_WAKE, 1 <unfinished ...>
	27867 00:38:39.782355 <... fcntl resumed> ) = 0
	27871 00:38:39.782360 <... futex resumed> ) = 0
	27868 00:38:39.782367 <... futex resumed> ) = 1
	27871 00:38:39.782372 futex(0xc4200f8138, FUTEX_WAKE, 1 <unfinished ...>
	27868 00:38:39.782377 pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=3000}, NULL <unfinished ...>
	27871 00:38:39.782384 <... futex resumed> ) = 1
	27870 00:38:39.782389 <... futex resumed> ) = 0
	27867 00:38:39.782394 fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1} <unfinished ...>
	27870 00:38:39.782400 pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=3000}, NULL <unfinished ...>
	27867 00:38:39.782408 <... fcntl resumed> ) = 0

Below link shows that go scheduler itself might be significantly improved for
cases when there are several Cgo calls made for a request in a server:

https://github.com/golang/go/issues/21827#issuecomment-329092317

in particular CGo-4 case should be closely related to this sqlite3 go package,
because for one query many CGo calls are made to SQLite.

However until there are proper scheduler fixes, let's make what could
be made to improve time to do queries:

If we know that the context under which a query is executed will never
be canceled - we know we can safely skip spawning the interrupt
goroutine and this was avoid ping-pong on done in between different OS
threads.

This brings the following speedup on my notebook with go1.10:

name               old req/s    new req/s    delta
Exec                 254k ± 1%    379k ± 1%  +48.89%  (p=0.000 n=10+10)
Query               90.6k ± 2%   96.4k ± 1%   +6.37%  (p=0.000 n=10+10)
Params              81.5k ± 1%   87.0k ± 1%   +6.83%  (p=0.000 n=10+10)
Stmt                 122k ± 2%    129k ± 1%   +6.07%  (p=0.000 n=10+9)
Rows                2.98k ± 1%   3.06k ± 1%   +2.77%  (p=0.000 n=9+10)
StmtRows            3.10k ± 1%   3.13k ± 1%   +1.12%  (p=0.000 n=9+10)

name               old time/op  new time/op  delta
CustomFunctions-4  10.6µs ± 1%  10.1µs ± 1%   -5.01%  (p=0.000 n=10+10)
2018-02-17 21:09:05 +03:00
mattn 788c147ad7
Merge pull request from navytux/y/preadwrite
Let SQLite use pread/pwrite
2018-02-17 23:37:09 +09:00
Kirill Smelkov f675967c54 Let SQLite use pread/pwrite
With current settings SQLite was using lseek/read syscalls to read data, e.g.:

	20:43:17.640660 fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
	20:43:17.640683 fcntl(3, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1}) = 0
	20:43:17.640705 fcntl(3, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741826, l_len=510}) = 0
	20:43:17.640725 fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1}) = 0
	20:43:17.640744 stat(".../neo.sqlite-journal", 0x7ffef2c91080) = -1 ENOENT (No such file or directory)
	20:43:17.640764 lseek(3, 24, SEEK_SET)  = 24
	20:43:17.640779 read(3, "\0\0\0\33\0\0\10\235\0\0\10]\0\0\0\27", 16) = 16
	20:43:17.640795 stat(".../neo.sqlite-wal", 0x7ffef2c91080) = -1 ENOENT (No such file or directory)

but if we allow it to use pread it will be only 1 system call instead of 2 and
reading this way can also be done in parallel because there is no global to
file seeking:

	20:48:42.668466 fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
	20:48:42.668501 fcntl(3, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1}) = 0
	20:48:42.668522 fcntl(3, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741826, l_len=510}) = 0
	20:48:42.668542 fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1}) = 0
	20:48:42.668561 stat(".../neo.sqlite-journal", 0x7ffdbc1f22c0) = -1 ENOENT (No such file or directory)
	20:48:42.668580 pread64(3, "\0\0\0\33\0\0\10\235\0\0\10]\0\0\0\27", 16, 24) = 16
	20:48:42.668597 stat(".../neo.sqlite-wal", 0x7ffdbc1f22c0) = -1 ENOENT (No such file or directory)

(if needed this enablement can be done per OS)
2018-02-16 20:44:19 +03:00
mattn 3c6eef45ff
Merge pull request from mattn/sqlite-amalgamation-3220000
bump sqlite 3.22.0
2018-02-07 21:14:15 +09:00