The busy_timeout pragma was added in sqlite 3.7.15 as an alternative
to calling sqlite3_busy_timeout directly:
https://sqlite.org/pragma.html#pragma_busy_timeout
While there's no functional change here, using the pragma does align
setting busy_timeout with other settings and removes the special case
for calling sqlite3_busy_timeout directly.
This can be used like in the test; I wrote a little wrapper around
sql.DB which uses this, and allows concurrent reads but just one single
write. This is perhaps a better generic "table locked"-solution than
setting the connections to 1 and/or cache=shared (although even better
would be to design your app in such a way that this doesn't happpen in
the first place, but even then a little seat belt isn't a bad thing).
The parsing adds about 0.1ms to 0.2ms of overhead in the wrapper, which
isn't too bad (and it caches the results, so only needs to do this
once).
At any rate, I can't really access functions from sqlite3-binding.c from
my application, so expose it via SQLiteStmt.
Add a shortcut for PRAGMA cache_size; this is a pretty useful setting:
the default of -2000 (2M) is not especially high, and a lot of people
will probably want to increase this.
For example, while running a bunch of fairy expensive queries in
parallel:
With SetMaxOpenConns(1):
-2000: 5762ms
-20000: 4714ms
With SetMaxOpenConns(20):
-2000: 3067ms
-20000: 2532ms
Which isn't a bad performance boost for changing a single number.
This option was enabled by default in sqlite3 on 2014-07-03.
This setting does nothing. It can now be disabled with
SQLITE_DISABLE_FTS3_UNICODE. See the upstream commit:
https://sqlite.org/src/info/0cc0230ae9cfc976
I think this change was imported into this project with commit
ee9da4840d on 2015-06-12.
This "disables the use of compiler-specific built-in functions such
as __builtin_bswap32()" (from the SQLite docs) so this change might
produce slightly better code. My primary motivation, however, is that
the "default" configuration for SQLite, which is widely tested, does
not set this preprocessor macro.
From looking at Github issues, it appears this was added to avoid a
build error on Mac OS X 10.11, in 2017:
https://github.com/mattn/go-sqlite3/issues/386
There have been a number of changes to sqlite3 since we tried this
last. I think it would be worth trying to remove this setting again.
I found a machine running Mac OS X 10.11.6. It was able to build and
run the tests in this package with this change.
Mac OS X 10.11 is has not been supported by Apple since 2018
(currently Apple is releasing updates for Mac OS 10.13 and newer; 11
is the current release). However, Go 1.14 is supported, and it
requires Mac OS X 10.11 or newer: https://golang.org/doc/go1.14
Go 1.15 only supports Mac OS 10.12 and newer:
https://golang.org/doc/go1.15
This test fails fairly often. On my system, I can trigger it with:
go test . -run=TestExecContextCancel -count=10 -failfast -v
This makes the test less flaky by timing out the context after a
consistent 50 millisecond delay. This was enough time for the query
to start, then get cancelled with sqlite3_interrupt() in my tests.
This now passes the above check.
This is a modified version of the change suggested in:
https://github.com/mattn/go-sqlite3/pull/865
Commit 4f7abea96e added a test that uses Conn.Raw, which was added in
Go >= 1.13. The go-sqlite3 project runs tests with Go >= 1.11. Remove
the test from sqlite3_test.go, so it only runs with the correct
versions of Go.
Instead of adding a new test, modify the existing test that already
uses Conn.Raw() to check the type of driverConn.
* Fix "cannot start a transaction within a transaction" issue
[why]
If db.BeginTx(ctx, nil) context is cancelled too fast, "BEGIN" statement can be
completed inside DB, but we still try to cancel it with sqlite3_interrupt.
In such case we get context.Cancelled or context.DeadlineExceeded from exec(),
but operation really completed. Connection returned into pool, and returns "cannot
start a transaction within a transaction" error for next db.BeginTx() call.
[how]
Handle status code returned from cancelled operation.
[testing]
Added unit-test which reproduces issue.
* Reduce TestQueryRowContextCancelParallel concurrency
[why]
Tests times out in travis-ci when run with -race option.
Set go_import_path to tell Travis-CI where to checkout the code, so the
Travis build can also work on forks. This is important when building
without Go modules support.
Changes in Go versions for the Travis-CI builds:
- reverse order of Go versions so the latest ones are checked first
- add Go 1.14.x (latest stable) at the first position
- use 'tip' instead of 'master' (like https://tip.golang.org) and move
it just after 1.14.x
My app can use PostgreSQL and – optionally – SQLite. I would like to be
able to compile the app without cgo when SQLite isn't used, as this
removes the need for a C compiler which makes builds easier and faster,
especially for end-users.
In the simple case, this is not a problem go-sqlite3 already provides a
simple non-cgo mock so it compiles and gives a runtime error if you try
to use it anyway.
However, now I'd like to register a function for my SQLite connection to
match a PostgreSQL function like so:
sql.Register("sqlite3_custom", &sqlite3.SQLiteDriver{
ConnectHook: func(conn *sqlite3.SQLiteConn) error {
return conn.RegisterFunc("pow", pow, true); err != nil {
},
})
But this makes it quite hard to keep the same logic since it refers to
types that don't exist with CGO_ENABLED=0. I will need to create a db.go
with `+build !cgo` and db_cgo.go with `+buid cgo` which duplicates all
the logic but with the sqlite hooks. In my case, this actually affects
quite a lot; for example I have a helper function which connects and
runs migrations and whatnot which looks like:
type ConnectOptions struct {
Connect string // Connect string.
Schema []byte // Database schema to create on startup.
Migrate *Migrate
SQLiteHook func(*sqlite3.SQLiteConn) error
}
And I'd have to have two versions of that, too. You could perhaps do
something with interfaces, but because the sql.Register() call above
references the `sqlite3.SQLiteDriver.ConnectHook` struct field that's
not so straightforward (and wrapping stuff in interfaces probably won't
do much for the general clarity either).
This simplifies all of that by providing some common types that may be
used when setting up a SQLite connectin. I renamed the
`SQLiteDriverMock` to `&SQLiteDriver` for this reason. As far as I can
tell in my testing, this has no real downsides (but perhaps I missed
something?)
---
Note: it might also be worth doing something similar for error.go, as I
already have two variants of the below function (one with cgo as below,
and one without cgo which checks just PostgreSQL):
// ErrUnique reports if this error reports a UNIQUE constraint violation.
//
// This is the cgo version which works for PostgreSQL and SQLite.
func ErrUnique(err error) bool {
var sqlErr *sqlite3.Error
if errors.As(err, &sqlErr) && sqlErr.ExtendedCode == sqlite3.ErrConstraintUnique {
return true
}
var pqErr *pq.Error
if errors.As(err, &pqErr) && pqErr.Code == "23505" {
return true
}
return false
}
This is a lot more manageable than the ConnectHook case, but it would be
nicer if it would work without the need for build tags.
* Allow unused named parameters
Try to bind all named parameters and ignore those not used.
* Allow "@" and "$" for named parameters
* Add tests for named parameters
Co-authored-by: Guido Berhoerster <guido+go-sqlite3@berhoerster.name>
Dan Peterson