00a23ba538
For a Go-only project the following code pattern
go func() {
select {
case <-ctx.Done():
// call some cancel
case <-done:
// work finished ok
}
}()
// do some work
close(done)
works good and fast - without high scheduling overhead because scheduler
usually puts spawned goroutine into run queue on the same OS thread and so
after done is closed control is passed to spawned goroutine without OS context
switch.
However in the presence of Cgo calls in "do some work" the situation can
become different - Cgo calls are treated by go runtime similarly to
system calls with the effect that goroutines spawned on original OS
thread tend to be migrated by scheduler to be executed on another OS
thread.
This in turn can bring high overhead for communicating on "done", which
ultimately can result in full context switch: if the spawned goroutine
had chance to run, already checked done and ctx to be not ready, and went
into sleep via wait on futex - showing as something like below in strace for
one read query (note futex calls):
27867 00:38:39.782146 stat(".../neo.sqlite-journal", 0x7f83809c4a20) = -1 ENOENT (No such file or directory)
27867 00:38:39.782165 pread64(3, "\0\0\0\33\0\0\10\235\0\0\10]\0\0\0\27", 16, 24) = 16
27871 00:38:39.782179 <... pselect6 resumed> ) = 0 (Timeout)
27868 00:38:39.782187 <... pselect6 resumed> ) = 0 (Timeout)
27871 00:38:39.782193 futex(0xc4200f8538, FUTEX_WAIT, 0, NULL <unfinished ...>
27868 00:38:39.782199 futex(0xc420013138, FUTEX_WAIT, 0, NULL <unfinished ...>
27867 00:38:39.782205 stat(".../neo.sqlite-wal", 0x7f83809c4a20) = -1 ENOENT (No such file or directory)
27867 00:38:39.782224 fstat(3, {st_mode=S_IFREG|0644, st_size=9031680, ...}) = 0
27867 00:38:39.782247 futex(0xc420013138, FUTEX_WAKE, 1 <unfinished ...>
27868 00:38:39.782259 <... futex resumed> ) = 0
27867 00:38:39.782265 <... futex resumed> ) = 1
27868 00:38:39.782270 pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=3000}, NULL <unfinished ...>
27867 00:38:39.782279 fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
27867 00:38:39.782315 fcntl(3, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1}) = 0
27868 00:38:39.782336 <... pselect6 resumed> ) = 0 (Timeout)
27867 00:38:39.782342 fcntl(3, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741826, l_len=510} <unfinished ...>
27868 00:38:39.782348 futex(0xc4200f8538, FUTEX_WAKE, 1 <unfinished ...>
27867 00:38:39.782355 <... fcntl resumed> ) = 0
27871 00:38:39.782360 <... futex resumed> ) = 0
27868 00:38:39.782367 <... futex resumed> ) = 1
27871 00:38:39.782372 futex(0xc4200f8138, FUTEX_WAKE, 1 <unfinished ...>
27868 00:38:39.782377 pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=3000}, NULL <unfinished ...>
27871 00:38:39.782384 <... futex resumed> ) = 1
27870 00:38:39.782389 <... futex resumed> ) = 0
27867 00:38:39.782394 fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1} <unfinished ...>
27870 00:38:39.782400 pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=3000}, NULL <unfinished ...>
27867 00:38:39.782408 <... fcntl resumed> ) = 0
Below link shows that go scheduler itself might be significantly improved for
cases when there are several Cgo calls made for a request in a server:
https://github.com/golang/go/issues/21827#issuecomment-329092317
in particular CGo-4 case should be closely related to this sqlite3 go package,
because for one query many CGo calls are made to SQLite.
However until there are proper scheduler fixes, let's make what could
be made to improve time to do queries:
If we know that the context under which a query is executed will never
be canceled - we know we can safely skip spawning the interrupt
goroutine and this was avoid ping-pong on done in between different OS
threads.
This brings the following speedup on my notebook with go1.10:
name old req/s new req/s delta
Exec 254k ± 1% 379k ± 1% +48.89% (p=0.000 n=10+10)
Query 90.6k ± 2% 96.4k ± 1% +6.37% (p=0.000 n=10+10)
Params 81.5k ± 1% 87.0k ± 1% +6.83% (p=0.000 n=10+10)
Stmt 122k ± 2% 129k ± 1% +6.07% (p=0.000 n=10+9)
Rows 2.98k ± 1% 3.06k ± 1% +2.77% (p=0.000 n=9+10)
StmtRows 3.10k ± 1% 3.13k ± 1% +1.12% (p=0.000 n=9+10)
name old time/op new time/op delta
CustomFunctions-4 10.6µs ± 1% 10.1µs ± 1% -5.01% (p=0.000 n=10+10)
|
||
|---|---|---|
| _example | ||
| tool | ||
| .gitignore | ||
| .travis.yml | ||
| LICENSE | ||
| README.md | ||
| backup.go | ||
| backup_test.go | ||
| callback.go | ||
| callback_test.go | ||
| doc.go | ||
| error.go | ||
| error_test.go | ||
| sqlite3-binding.c | ||
| sqlite3-binding.h | ||
| sqlite3.go | ||
| sqlite3_context.go | ||
| sqlite3_fts3_test.go | ||
| sqlite3_fts5.go | ||
| sqlite3_go18.go | ||
| sqlite3_go18_test.go | ||
| sqlite3_icu.go | ||
| sqlite3_json1.go | ||
| sqlite3_libsqlite3.go | ||
| sqlite3_load_extension.go | ||
| sqlite3_omit_load_extension.go | ||
| sqlite3_other.go | ||
| sqlite3_solaris.go | ||
| sqlite3_test.go | ||
| sqlite3_trace.go | ||
| sqlite3_type.go | ||
| sqlite3_vtable.go | ||
| sqlite3_vtable_test.go | ||
| sqlite3_windows.go | ||
| sqlite3ext.h | ||
| static_mock.go | ||
README.md
go-sqlite3
Description
sqlite3 driver conforming to the built-in database/sql interface
Installation
This package can be installed with the go get command:
go get github.com/mattn/go-sqlite3
go-sqlite3 is cgo package.
If you want to build your app using go-sqlite3, you need gcc.
However, after you have built and installed go-sqlite3 with go install github.com/mattn/go-sqlite3 (which requires gcc), you can build your app without relying on gcc in future.
Documentation
API documentation can be found here: http://godoc.org/github.com/mattn/go-sqlite3
Examples can be found under the ./_example directory
FAQ
-
Want to build go-sqlite3 with libsqlite3 on my linux.
Use
go build --tags "libsqlite3 linux" -
Want to build go-sqlite3 with libsqlite3 on OS X.
Install sqlite3 from homebrew:
brew install sqlite3Use
go build --tags "libsqlite3 darwin" -
Want to build go-sqlite3 with icu extension.
Use
go build --tags "icu"Available extensions:
json1,fts5,icu -
Can't build go-sqlite3 on windows 64bit.
Probably, you are using go 1.0, go1.0 has a problem when it comes to compiling/linking on windows 64bit. See: #27
-
Getting insert error while query is opened.
You can pass some arguments into the connection string, for example, a URI. See: #39
-
Do you want to cross compile? mingw on Linux or Mac?
See: #106 See also: http://www.limitlessfx.com/cross-compile-golang-app-for-windows-from-linux.html
-
Want to get time.Time with current locale
Use
_loc=autoin SQLite3 filename schema likefile:foo.db?_loc=auto. -
Can I use this in multiple routines concurrently?
-
Why is it racy if I use a
sql.Open("sqlite3", ":memory:")database?Each connection to :memory: opens a brand new in-memory sql database, so if the stdlib's sql engine happens to open another connection and you've only specified ":memory:", that connection will see a brand new database. A workaround is to use "file::memory:?mode=memory&cache=shared". Every connection to this string will point to the same in-memory database. See #204 for more info.
License
MIT: http://mattn.mit-license.org/2012
sqlite3-binding.c, sqlite3-binding.h, sqlite3ext.h
The -binding suffix was added to avoid build failures under gccgo.
In this repository, those files are an amalgamation of code that was copied from SQLite3. The license of that code is the same as the license of SQLite3.
Author
Yasuhiro Matsumoto (a.k.a mattn)