Connections to a database

A Connection encapsulates access to a database. You then use cursors to issue queries against the database.

You can have multiple Connections open against the same database in the same process, across threads and in other processes.

Connection class

class Connection(filename: str, flags: int = SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, vfs: Optional[str] = None, statementcachesize: int = 100)

This object wraps a sqlite3 pointer.

Opens the named database. You can use :memory: to get a private temporary in-memory database that is not shared with any other connections.

Parameters

  • flags – One or more of the open flags orred together

  • vfs – The name of the vfs to use. If None then the default vfs will be used.

  • statementcachesize – Use zero to disable the statement cache, or a number larger than the total distinct SQL statements you execute frequently.

See also

Calls: sqlite3_open_v2

Connection.__enter__() → Connection

You can use the database as a context manager as defined in PEP 0343. When you use with a transaction is started. If the block finishes with an exception then the transaction is rolled back, otherwise it is committed. For example:

with connection:
    connection.execute("....")
    with connection:
        # nested is supported
        call_function(connection)
        connection.execute("...")
        with connection as db:
            # You can also use 'as'
            call_function2(db)
            db.execute("...")

Behind the scenes the savepoint functionality introduced in SQLite 3.6.8 is used to provide nested transactions.

Connection.__exit__(etype: Optional[type[BaseException]], evalue: Optional[BaseException], etraceback: Optional[types.TracebackType]) → Optional[bool]

Implements context manager in conjunction with __enter__(). Any exception that happened in the with block is raised after committing or rolling back the savepoint.

Connection.authorizer: Optional[Authorizer]

While preparing statements, SQLite will call any defined authorizer to see if a particular action is ok to be part of the statement.

Typical usage would be if you are running user supplied SQL and want to prevent harmful operations. You should also set the statementcachesize to zero.

The authorizer callback has 5 parameters:

  • An operation code

  • A string (or None) dependent on the operation (listed as 3rd)

  • A string (or None) dependent on the operation (listed as 4th)

  • A string name of the database (or None)

  • Name of the innermost trigger or view doing the access (or None)

The authorizer callback should return one of SQLITE_OK, SQLITE_DENY or SQLITE_IGNORE. (SQLITE_DENY is returned if there is an error in your Python code).

See also

Calls: sqlite3_set_authorizer

Connection.autovacuum_pages(callable: Optional[Callable[[str, int, int, int], int]]) → None

Calls callable to find out how many pages to autovacuum. The callback has 4 parameters:

  • Database name: str (eg “main”)

  • Database pages: int (how many pages make up the database now)

  • Free pages: int (how many pages could be freed)

  • Page size: int (page size in bytes)

Return how many pages should be freed. Values less than zero or more than the free pages are treated as zero or free page count. On error zero is returned.

READ THE NOTE IN THE SQLITE DOCUMENTATION. Calling into SQLite can result in crashes, corrupt databases or worse.

Calls: sqlite3_autovacuum_pages

Connection.backup(databasename: str, sourceconnection: Connection, sourcedatabasename: str) → Backup

Opens a backup object. All data will be copied from source database to this database.

Parameters

  • databasename – Name of the database. This will be main for the main connection and the name you specified for attached databases.

  • sourceconnection – The Connection to copy a database from.

  • sourcedatabasename – Name of the database in the source (eg main).

Return type

backup

See also

Calls: sqlite3_backup_init

Connection.blobopen(database: str, table: str, column: str, rowid: int, writeable: bool) → Blob

Opens a blob for incremental I/O.

Parameters

  • database –

    Name of the database. This will be main for the main connection and the name you specified for attached databases.

  • table – The name of the table

  • column – The name of the column

  • rowid – The id that uniquely identifies the row.

  • writeable – If True then you can read and write the blob. If False then you can only read it.

Return type

Blob

See also

Calls: sqlite3_blob_open

Connection.cache_stats(include_entries: bool = False) → Dict[str, int]

Returns information about the statement cache as dict.

Note

Calling execute with “select a; select b; insert into c …” will result in 3 cache entries corresponding to each of the 3 queries present.

The returned dictionary has the following information.

Key

Explanation

size

Maximum number of entries in the cache

evictions

How many entries were removed (expired) to make space for a newer entry

no_cache

Queries that had can_cache parameter set to False

hits

A match was found in the cache

misses

No match was found in the cache, or the cache couldn’t be used

no_vdbe

The statement was empty (eg a comment) or SQLite took action during parsing (eg some pragmas). These are not cached and also included in the misses count

too_big

UTF8 query size was larger than considered for caching. These are also included in the misses count.

max_cacheable_bytes

Maximum size of query (in bytes of utf8) that will be considered for caching

entries

(Only present if include_entries is True) A list of the cache entries

If entries is present, then each list entry is a dict with the following information.

Key

Explanation

query

Text of the query itself (first statement only)

prepare_flags

Flags passed to sqlite3_prepare_v3 for this query

uses

How many times this entry has been (re)used

has_more

Boolean indicating if there was more query text than the first statement
Connection.changes() → int

Returns the number of database rows that were changed (or inserted or deleted) by the most recently completed INSERT, UPDATE, or DELETE statement.

Calls: sqlite3_changes64

Connection.close(force: bool = False) → None

Closes the database. If there are any outstanding cursors, blobs or backups then they are closed too. It is normally not necessary to call this method as the database is automatically closed when there are no more references. It is ok to call the method multiple times.

If your user defined functions or collations have direct or indirect references to the Connection then it won’t be automatically garbage collected because of circular referencing that can’t be automatically broken. Calling close will free all those objects and what they reference.

SQLite is designed to survive power failures at even the most awkward moments. Consequently it doesn’t matter if it is closed when the process is exited, or even if the exit is graceful or abrupt. In the worst case of having a transaction in progress, that transaction will be rolled back by the next program to open the database, reverting the database to a know good state.

If force is True then any exceptions are ignored.

Calls: sqlite3_close

Connection.collationneeded(callable: Optional[Callable[[Connection, str], None]]) → None

callable will be called if a statement requires a collation that hasn’t been registered. Your callable will be passed two parameters. The first is the connection object. The second is the name of the collation. If you have the collation code available then call Connection.createcollation().

This is useful for creating collations on demand. For example you may include the locale in the collation name, but since there are thousands of locales in popular use it would not be useful to prereigster them all. Using collationneeded() tells you when you need to register them.

See also

Calls: sqlite3_collation_needed

Connection.config(op: int, *args: int) → int
Parameters

Only optiona that take an int and return one are implemented.

Calls: sqlite3_db_config

Connection.createaggregatefunction(name: str, factory: Optional[AggregateFactory], numargs: int = -1) → None

Registers an aggregate function. Aggregate functions operate on all the relevant rows such as counting how many there are.

Parameters

  • name – The string name of the function. It should be less than 255 characters

  • factory – The function that will be called. Use None to delete the function.

  • numargs – How many arguments the function takes, with -1 meaning any number

When a query starts, the factory will be called and must return a tuple of 3 items:

a context object

This can be of any type

a step function

This function is called once for each row. The first parameter will be the context object and the remaining parameters will be from the SQL statement. Any value returned will be ignored.

a final function

This function is called at the very end with the context object as a parameter. The value returned is set as the return for the function. The final function is always called even if an exception was raised by the step function. This allows you to ensure any resources are cleaned up.

Note

You can register the same named function but with different callables and numargs. See createscalarfunction() for an example.

See also

Calls: sqlite3_create_function_v2

Connection.createcollation(name: str, callback: Optional[Callable[[str, str], int]]) → None

You can control how SQLite sorts (termed collation) when giving the COLLATE term to a SELECT. For example your collation could take into account locale or do numeric sorting.

The callback will be called with two items. It should return -1 if the first is less then the second, 0 if they are equal, and 1 if first is greater:

def mycollation(one, two):
    if one < two:
        return -1
    if one == two:
        return 0
    if one > two:
        return 1

Passing None as the callback will unregister the collation.

See also

Calls: sqlite3_create_collation_v2

Connection.createmodule(name: str, datasource: Any) → None

Registers a virtual table. See Virtual Tables for details.

See also

Calls: sqlite3_create_module_v2

Connection.createscalarfunction(name: str, callable: Optional[ScalarProtocol], numargs: int = -1, deterministic: bool = False) → None

Registers a scalar function. Scalar functions operate on one set of parameters once.

Parameters

  • name – The string name of the function. It should be less than 255 characters

  • callable – The function that will be called. Use None to unregister.

  • numargs – How many arguments the function takes, with -1 meaning any number

  • deterministic – When True this means the function always returns the same result for the same input arguments. SQLite’s query planner can perform additional optimisations for deterministic functions. For example a random() function is not deterministic while one that returns the length of a string is.

Note

You can register the same named function but with different callable and numargs. For example:

connection.createscalarfunction("toip", ipv4convert, 4)
connection.createscalarfunction("toip", ipv6convert, 16)
connection.createscalarfunction("toip", strconvert, -1)

The one with the correct numargs will be called and only if that doesn’t exist then the one with negative numargs will be called.

See also

Calls: sqlite3_create_function_v2

Connection.cursor() → Cursor

Creates a new Cursor object on this database.

Return type

Cursor

Connection.cursor_factory: Callable[[Connection], Any]

Defaults to Cursor

Called with a Connection as the only parameter when a cursor is needed such as by the cursor() method, or Connection.execute().

Note that whatever is returned doesn’t have to be an actual Cursor instance, and just needs to have the methods present that are actually called. These are likely to be execute, executemany, close etc.

Connection.db_filename(name: str) → str

Returns the full filename of the named (attached) database. The main database is named “main”.

Calls: sqlite3_db_filename

Connection.db_names() → List[str]

Returns the list of database names. For example the first database is named ‘main’, the next ‘temp’, and the rest with the name provided in ATTACH

Calls: sqlite3_db_name

Connection.deserialize(name: str, contents: bytes) → None

Replaces the named database with an in-memory copy of contents. name is “main” for the main database, “temp” for the temporary database etc.

The resulting database is in-memory, read-write, and the memory is owned, resized, and freed by SQLite.

See also

Calls: sqlite3_deserialize

Connection.enableloadextension(enable: bool) → None

Enables/disables extension loading which is disabled by default.

Parameters

enable – If True then extension loading is enabled, else it is disabled.

See also

Calls: sqlite3_enable_load_extension

Connection.exectrace: Optional[ExecTracer]

Called with the cursor, statement and bindings for each execute() or executemany() on this Connection, unless the Cursor installed its own tracer. Your execution tracer can also abort execution of a statement.

If callable is None then any existing execution tracer is removed.

See also

Connection.execute(statements: str, bindings: Optional[Bindings] = None, *, can_cache: bool = True, prepare_flags: int = 0) → Cursor

Executes the statements using the supplied bindings. Execution returns when the first row is available or all statements have completed. (A cursor is automatically obtained).

See Cursor.execute() for more details.

Connection.executemany(statements: str, sequenceofbindings: Sequence[Bindings], *, can_cache: bool = True, prepare_flags: int = 0) → Cursor

This method is for when you want to execute the same statements over a sequence of bindings, such as inserting into a database. (A cursor is automatically obtained).

See Cursor.executemany() for more details.

Connection.filecontrol(dbname: str, op: int, pointer: int) → bool

Calls the xFileControl() method on the Virtual File System (VFS) implementing file access for the database.

Parameters

  • dbname – The name of the database to affect (eg “main”, “temp”, attached name)

  • op – A numeric code with values less than 100 reserved for SQLite internal use.

  • pointer – A number which is treated as a void pointer at the C level.

Returns

True or False indicating if the VFS understood the op.

If you want data returned back then the pointer needs to point to something mutable. Here is an example using ctypes of passing a Python dictionary to xFileControl() which can then modify the dictionary to set return values:

obj={"foo": 1, 2: 3}                 # object we want to pass
objwrap=ctypes.py_object(obj)        # objwrap must live before and after the call else
                                     # it gets garbage collected
connection.filecontrol(
         "main",                     # which db
         123,                        # our op code
         ctypes.addressof(objwrap))  # get pointer

The xFileControl() method then looks like this:

def xFileControl(self, op, pointer):
    if op==123:                      # our op code
        obj=ctypes.py_object.from_address(pointer).value
        # play with obj - you can use id() to verify it is the same
        print(obj["foo"])
        obj["result"]="it worked"
        return True
    else:
        # pass to parent/superclass
        return super(MyFile, self).xFileControl(op, pointer)

This is how you set the chunk size by which the database grows. Do not combine it into one line as the c_int would be garbage collected before the filecontrol call is made:

chunksize=ctypes.c_int(32768)
connection.filecontrol("main", apsw.SQLITE_FCNTL_CHUNK_SIZE, ctypes.addressof(chunksize))

Calls: sqlite3_file_control

Connection.filename: str

The filename of the database.

Calls: sqlite3_db_filename

Connection.getautocommit() → bool

Returns if the Connection is in auto commit mode (ie not in a transaction).

Calls: sqlite3_get_autocommit

Connection.getexectrace() → Optional[ExecTracer]

Returns the currently installed execution tracer

Connection.getrowtrace() → Optional[RowTracer]

Returns the currently installed row tracer

Connection.in_transaction: bool

True if currently in a transaction, else False

Calls: sqlite3_get_autocommit

Connection.interrupt() → None

Causes any pending operations on the database to abort at the earliest opportunity. You can call this from any thread. For example you may have a long running query when the user presses the stop button in your user interface. InterruptError will be raised in the query that got interrupted.

Calls: sqlite3_interrupt

Connection.last_insert_rowid() → int

Returns the integer key of the most recent insert in the database.

Calls: sqlite3_last_insert_rowid

Connection.limit(id: int, newval: int = -1) → int

If called with one parameter then the current limit for that id is returned. If called with two then the limit is set to newval.

Parameters

  • id – One of the runtime limit ids

  • newval – The new limit. This is a 32 bit signed integer even on 64 bit platforms.

Returns

The limit in place on entry to the call.

See also

Calls: sqlite3_limit

Connection.loadextension(filename: str, entrypoint: Optional[str] = None) → None

Loads filename as an extension

Parameters

  • filename – The file to load. This must be Unicode or Unicode compatible

  • entrypoint – The initialization method to call. If this parameter is not supplied then the SQLite default of sqlite3_extension_init is used.

Raises

ExtensionLoadingError – If the extension could not be loaded. The exception string includes more details.

See also

Calls: sqlite3_load_extension

Connection.open_flags: int

The integer flags used to open the database.

Connection.open_vfs: str

The string name of the vfs used to open the database.

Connection.overloadfunction(name: str, nargs: int) → None

Registers a placeholder function so that a virtual table can provide an implementation via VTTable.FindFunction().

Parameters

  • name – Function name

  • nargs – How many arguments the function takes

Due to cvstrac 3507 underlying errors will not be returned.

Calls: sqlite3_overload_function

Connection.readonly(name: str) → bool

True or False if the named (attached) database was opened readonly or file permissions don’t allow writing. The main database is named “main”.

An exception is raised if the database doesn’t exist.

Calls: sqlite3_db_readonly

Connection.rowtrace: Optional[RowTracer]

Called with the cursor and row being returned for cursors associated with this Connection, unless the Cursor installed its own tracer. You can change the data that is returned or cause the row to be skipped altogether.

If callable is None then any existing row tracer is removed.

See also

Connection.serialize(name: str) → bytes

Returns a memory copy of the database. name is “main” for the main database, “temp” for the temporary database etc.

The memory copy is the same as if the database was backed up to disk.

If the database name doesn’t exist or is empty, then None is returned, not an exception (this is SQLite’s behaviour).

Calls: sqlite3_serialize

Connection.set_last_insert_rowid(rowid: int) → None

Sets the value calls to last_insert_rowid() will return.

Calls: sqlite3_set_last_insert_rowid

Connection.setauthorizer(callable: Optional[Authorizer]) → None

Sets the authorizer

Connection.setbusyhandler(callable: Optional[Callable[[int], bool]]) → None

Sets the busy handler to callable. callable will be called with one integer argument which is the number of prior calls to the busy callback for the same lock. If the busy callback returns False, then SQLite returns SQLITE_BUSY to the calling code. If the callback returns True, then SQLite tries to open the table again and the cycle repeats.

If you previously called setbusytimeout() then calling this overrides that.

Passing None unregisters the existing handler.

See also

Calls: sqlite3_busy_handler

Connection.setbusytimeout(milliseconds: int) → None

If the database is locked such as when another connection is making changes, SQLite will keep retrying. This sets the maximum amount of time SQLite will keep retrying before giving up. If the database is still busy then apsw.BusyError will be returned.

Parameters

milliseconds – Maximum thousandths of a second to wait.

If you previously called setbusyhandler() then calling this overrides that.

See also

Calls: sqlite3_busy_timeout

Connection.setcommithook(callable: Optional[CommitHook]) → None

callable will be called just before a commit. It should return False for the commit to go ahead and True for it to be turned into a rollback. In the case of an exception in your callable, a True (ie rollback) value is returned. Pass None to unregister the existing hook.

See also

Calls: sqlite3_commit_hook

Connection.setexectrace(callable: Optional[ExecTracer]) → None

Method to set Connection.exectrace

Connection.setprofile(callable: Optional[Callable[[str, int], None]]) → None

Sets a callable which is invoked at the end of execution of each statement and passed the statement string and how long it took to execute. (The execution time is in nanoseconds.) Note that it is called only on completion. If for example you do a SELECT and only read the first result, then you won’t reach the end of the statement.

Calls: sqlite3_profile

Connection.setprogresshandler(callable: Optional[Callable[[], bool]], nsteps: int = 20) → None

Sets a callable which is invoked every nsteps SQLite inststructions. The callable should return True to abort or False to continue. (If there is an error in your Python callable then True/abort will be returned).

See also

Calls: sqlite3_progress_handler

Connection.setrollbackhook(callable: Optional[Callable[[], None]]) → None

Sets a callable which is invoked during a rollback. If callable is None then any existing rollback hook is unregistered.

The callable is called with no parameters and the return value is ignored.

Calls: sqlite3_rollback_hook

Connection.setrowtrace(callable: Optional[RowTracer]) → None

Method to set Connection.rowtrace

Connection.setupdatehook(callable: Optional[Callable[[int, str, str, int], None]]) → None

Calls callable whenever a row is updated, deleted or inserted. If callable is None then any existing update hook is unregistered. The update hook cannot make changes to the database while the query is still executing, but can record them for later use or apply them in a different connection.

The update hook is called with 4 parameters:

type (int)

SQLITE_INSERT, SQLITE_DELETE or SQLITE_UPDATE

database name (string)

This is main for the database or the name specified in ATTACH

table name (string)

The table on which the update happened

rowid (64 bit integer)

The affected row

See also

Calls: sqlite3_update_hook

Connection.setwalhook(callable: Optional[Callable[[Connection, str, int], int]]) → None

callable will be called just after data is committed in Write Ahead Logging mode. It should return SQLITE_OK or an error code. The callback is called with 3 parameters:

  • The Connection

  • The database name (eg “main” or the name of an attached database)

  • The number of pages in the wal log

You can pass in None in order to unregister an existing hook.

Calls: sqlite3_wal_hook

Connection.sqlite3pointer() → int

Returns the underlying sqlite3 * for the connection. This method is useful if there are other C level libraries in the same process and you want them to use the APSW connection handle. The value is returned as a number using PyLong_FromVoidPtr under the hood. You should also ensure that you increment the reference count on the Connection for as long as the other libraries are using the pointer. It is also a very good idea to call sqlitelibversion() and ensure it is the same as the other libraries.

Connection.status(op: int, reset: bool = False) → Tuple[int, int]

Returns current and highwater measurements for the database.

Parameters

  • op – A status parameter

  • reset – If True then the highwater is set to the current value

Returns

A tuple of current value and highwater value

See also

The status() example which works in exactly the same way.

Calls: sqlite3_db_status

Connection.totalchanges() → int

Returns the total number of database rows that have be modified, inserted, or deleted since the database connection was opened.

Calls: sqlite3_total_changes64

Connection.txn_state(schema: Optional[str] = None) → int

Returns the current transaction state of the database, or a specific schema if provided. ValueError is raised if schema is not None or a valid schema name. apsw.mapping_txn_state contains the names and values returned.

Calls: sqlite3_txn_state

Connection.wal_autocheckpoint(n: int) → None

Sets how often the Write Ahead Logging checkpointing is run.

Parameters

n – A number representing the checkpointing interval or zero/negative to disable auto checkpointing.

Calls: sqlite3_wal_autocheckpoint

Connection.wal_checkpoint(dbname: Optional[str] = None, mode: int = apsw.SQLITE_CHECKPOINT_PASSIVE) → Tuple[int, int]

Does a WAL checkpoint. Has no effect if the database(s) are not in WAL mode.

Parameters

  • dbname – The name of the database or all databases if None

  • mode – One of the checkpoint modes.

Returns

A tuple of the size of the WAL log in frames and the number of frames checkpointed as described in the documentation.

Calls: sqlite3_wal_checkpoint_v2