Virtual File System (VFS)¶

SQLite 3.6 added VFS functionality which defines the interface between the SQLite core and the underlying operating system. The majority of the functionality deals with files. APSW exposes this functionality letting you provide your own routines. You can also inherit from an existing vfs making it easy to augment or override specific routines. For example you could obfuscate your database by XORing the data implemented by augmenting the read and write methods. The method names are exactly the same as SQLite uses making it easier to read the SQLite documentation, trouble tickets, web searches or mailing lists. The SQLite convention results in names like xAccess, xCurrentTime and xWrite.

You specify which VFS to use as a parameter to the Connection constructor.

db=apsw.Connection("file", vfs="myvfs")

The easiest way to get started is to make a VFS derived class that inherits from the default vfs. Then override methods you want to change behaviour of. If you want to just change how file operations are done then you have to override VFS.xOpen() to return a file instance that has your overridden VFSFile methods. The example demonstrates obfuscating the database file contents.

Note

All strings supplied and returned to VFS/VFSFile routines are treated as Unicode.

Exceptions and errors¶

To return an error from any routine you should raise an exception. The exception will be translated into the appropriate SQLite error code for SQLite. To return a specific SQLite error code use exceptionfor(). If the exception does not map to any specific error code then SQLITE_ERROR which corresponds to SQLError is returned to SQLite.

The SQLite code that deals with VFS errors behaves in varying ways. Some routines have no way to return an error (eg xDlOpen just returns zero/NULL on being unable to load a library, xSleep has no error return parameter), others are unified (eg almost any error in xWrite will be returned to the user as disk full error). Sometimes errors are ignored as they are harmless such as when a journal can’t be deleted after a commit (the journal is marked as obsolete before being deleted). Simple operations such as opening a database can result in many different VFS function calls such as hot journals being detected, locking, and read/writes for playback/rollback.

To avoid confusion with exceptions being raised in the VFS and exceptions from normal code to open Connections or execute SQL queries, VFS exceptions are not raised in the normal way. (If they were, only one could be raised and it would obscure whatever exceptions the Connection open or SQL query execute wanted to raise.) Instead the VFS.excepthook() or VFSFile.excepthook() method is called with a tuple of exception type, exception value and exception traceback. The default implementation of excepthook calls sys.excepthook() under Python 3 merely prints the exception value. (If sys.excepthook fails then PyErr_Display() is called.)

In normal VFS usage there will be no exceptions raised, or specific expected ones which APSW clears after noting them and returning the appropriate value back to SQLite. The exception hooking behaviour helps you find issues in your code or unexpected behaviour of the external environment. Remember that augmented stack traces are available which significantly increase detail about the exceptions.

As an example, lets say you have a divide by zero error in your xWrite routine. The table below shows what happens with time going down and across.

Python Query Code	SQLite and APSW C code	Python VFS code
`cursor.execute("update table set foo=3")`
	SQLite starts executing query
		Your VFS routines are called
		Your xWrite divides by zero
		`VFSFile.excepthook()` is called with ZeroDivision exception
	SQLITE_ERROR (closest matching SQLite error code) is returned to SQLite by APSW
	SQLite error handling and recovery operates which calls more VFS routines.	More VFS routines are called. Any exceptions in these routines will result in `VFSFile.excepthook()` being called with them.
	SQLite returns SQLITE_FULL to APSW
APSW returns `apsw.FullError`

VFS class¶

class VFS(name: str, base: Optional[str] = None, makedefault: bool = False, maxpathname: int = 1024)¶

Provides operating system access. You can get an overview in the SQLite documentation. To create a VFS your Python class must inherit from VFS.

Parameters

name – The name to register this vfs under. If the name already exists then this vfs will replace the prior one of the same name. Use apsw.vfsnames() to get a list of registered vfs names.
base – If you would like to inherit behaviour from an already registered vfs then give their name. To inherit from the default vfs, use a zero length string "" as the name.
makedefault – If true then this vfs will be registered as the default, and will be used by any opens that don’t specify a vfs.
maxpathname – The maximum length of database name in bytes when represented in UTF-8. If a pathname is passed in longer than this value then SQLite will not be able to open it.

Raises

ValueError – If base is not None and the named vfs is not currently registered.

Calls:

VFS.excepthook(etype: type[BaseException], evalue: BaseException, etraceback: Optional[types.TracebackType]) → Any¶: Called when there has been an exception in a VFS routine. The default implementation passes args to sys.excepthook and if that fails then PyErr_Display. The three arguments correspond to what sys.exc_info() would return.

VFS.unregister() → None¶

Unregisters the VFS making it unavailable to future database opens. You do not need to call this as the VFS is automatically unregistered by when the VFS has no more references or open databases using it. It is however useful to call if you have made your VFS be the default and wish to immediately make it be unavailable. It is safe to call this routine multiple times.

Calls: sqlite3_vfs_unregister

VFS.xAccess(pathname: str, flags: int) → bool¶

SQLite wants to check access permissions. Return True or False accordingly.

Parameters

pathname – File or directory to check
flags – One of the access flags

VFS.xCurrentTime() → float¶: Return the Julian Day Number as a floating point number where the integer portion is the day and the fractional part is the time. Do not adjust for timezone (ie use UTC).

VFS.xDelete(filename: str, syncdir: bool) → None¶

Delete the named file. If the file is missing then raise an IOError exception with extendedresult SQLITE_IOERR_DELETE_NOENT

Parameters

filename – File to delete
syncdir – If True then the directory should be synced ensuring that the file deletion has been recorded on the disk platters. ie if there was an immediate power failure after this call returns, on a reboot the file would still be deleted.

VFS.xDlClose(handle: int) → None¶

Close and unload the library corresponding to the handle you returned from xDlOpen(). You can use ctypes to do this:

def xDlClose(handle):
   # Note leading underscore in _ctypes
   _ctypes.dlclose(handle)       # Linux/Mac/Unix
   _ctypes.FreeLibrary(handle)   # Windows

VFS.xDlError() → str¶: Return an error string describing the last error of xDlOpen() or xDlSym() (ie they returned zero/NULL). If you do not supply this routine then SQLite provides a generic message. To implement this method, catch exceptions in xDlOpen() or xDlSym(), turn them into strings, save them, and return them in this routine. If you have an error in this routine or return None then SQLite’s generic message will be used.

VFS.xDlOpen(filename: str) → int¶

Load the shared library. You should return a number which will be treated as a void pointer at the C level. On error you should return 0 (NULL). The number is passed as is to xDlSym()/xDlClose() so it can represent anything that is convenient for you (eg an index into an array). You can use ctypes to load a library:

def xDlOpen(name):
   return ctypes.cdll.LoadLibrary(name)._handle

VFS.xDlSym(handle: int, symbol: str) → int¶

Returns the address of the named symbol which will be called by SQLite. On error you should return 0 (NULL). You can use ctypes:

def xDlSym(ptr, name):
   return _ctypes.dlsym (ptr, name)  # Linux/Unix/Mac etc (note leading underscore)
   return ctypes.win32.kernel32.GetProcAddress (ptr, name)  # Windows

Parameters

handle – The value returned from an earlier xDlOpen() call
symbol – A string

VFS.xFullPathname(name: str) → str¶: Return the absolute pathname for name. You can use os.path.abspath to do this.

VFS.xGetLastError() → Tuple[int, str]¶: This method is to return an integer error code and (optional) text describing the last error that happened in this thread.

Note

SQLite 3.12 changed the semantics in an incompatible way from earlier versions. You will need to rewrite earlier implementations.

VFS.xGetSystemCall(name: str) → Optional[int]¶: Returns a pointer for the current method implementing the named system call. Return None if the call does not exist.

VFS.xNextSystemCall(name: Optional[str]) → Optional[str]¶: This method is repeatedly called to iterate over all of the system calls in the vfs. When called with None you should return the name of the first system call. In subsequent calls return the name after the one passed in. If name is the last system call then return None.

Note

Because of internal SQLite implementation semantics memory will be leaked on each call to this function. Consequently you should build up the list of call names once rather than repeatedly doing it.

VFS.xOpen(name: Optional[Union[str, URIFilename]], flags: List[int, int]) → VFSFile¶

This method should return a new file object based on name. You can return a VFSFile from a completely different VFS.

Parameters

name – File to open. Note that name may be None in which case you should open a temporary file with a name of your choosing. May be an instance of URIFilename.
flags – A list of two integers [inputflags, outputflags]. Each integer is one or more of the open flags binary orred together. The inputflags tells you what SQLite wants. For example SQLITE_OPEN_DELETEONCLOSE means the file should be automatically deleted when closed. The outputflags describes how you actually did open the file. For example if you opened it read only then SQLITE_OPEN_READONLY should be set.

VFS.xRandomness(numbytes: int) → bytes¶: This method is called once when SQLite needs to seed the random number generator. It is called on the default VFS only. It is not called again, even across apsw.shutdown() calls. You can return less than the number of bytes requested including None. If you return more then the surplus is ignored.

VFS.xSetSystemCall(name: Optional[str], pointer: int) → bool¶

Change a system call used by the VFS. This is useful for testing and some other scenarios such as sandboxing.

Parameters

name – The string name of the system call
pointer – A pointer provided as an int. There is no reference counting or other memory tracking of the pointer. If you provide one you need to ensure it is around for the lifetime of this and any other related VFS.

Raise an exception to return an error. If the system call does not exist then raise NotFoundError.

If name is None, then all systemcalls are reset to their defaults. This behaviour is not documented.

Returns: True if the system call was set. False if the system call is not known.

VFS.xSleep(microseconds: int) → int¶

Pause execution of the thread for at least the specified number of microseconds (millionths of a second). This routine is typically called from the busy handler.

Returns: How many microseconds you actually requested the operating system to sleep for. For example if your operating system sleep call only takes seconds then you would have to have rounded the microseconds number up to the nearest second and should return that rounded up value.

VFSFile class¶

class VFSFile(vfs: str, filename: Union[str, URIFilename], flags: List[int])¶

Wraps access to a file. You only need to derive from this class if you want the file object returned from VFS.xOpen() to inherit from an existing VFS implementation.

Note

All file sizes and offsets are 64 bit quantities even on 32 bit operating systems.

Parameters

vfs – The vfs you want to inherit behaviour from. You can use an empty string "" to inherit from the default vfs.
name – The name of the file being opened. May be an instance of URIFilename.
flags – A two item list [inflags, outflags] as detailed in VFS.xOpen().

Raises

ValueError – If the named VFS is not registered.

Note

If the VFS that you inherit from supports write ahead logging then your VFSFile will also support the xShm methods necessary to implement wal.

See also

VFS.xOpen()

VFSFile.excepthook(etype: type[BaseException], evalue: BaseException, etraceback: Optional[types.TracebackType]) → None¶

Called when there has been an exception in a VFSFile routine. The default implementation calls sys.excepthook and if that fails then PyErr_Display. The three arguments correspond to what sys.exc_info() would return.

Parameters

etype – The exception type
evalue – The exception value
etraceback – The exception traceback. Note this includes all frames all the way up to the thread being started.

VFSFile.xCheckReservedLock() → bool¶: Returns True if any database connection (in this or another process) has a lock other than SQLITE_LOCK_NONE or SQLITE_LOCK_SHARED.

VFSFile.xClose() → None¶: Close the database. Note that even if you return an error you should still close the file. It is safe to call this method multiple times.

VFSFile.xDeviceCharacteristics() → int¶: Return I/O capabilities (bitwise or of appropriate values). If you do not implement the function or have an error then 0 (the SQLite default) is returned.

VFSFile.xFileControl(op: int, ptr: int) → bool¶

Receives file control request typically issued by Connection.filecontrol(). See Connection.filecontrol() for an example of how to pass a Python object to this routine.

Parameters

op – A numeric code. Codes below 100 are reserved for SQLite internal use.
ptr – An integer corresponding to a pointer at the C level.

Returns

A boolean indicating if the op was understood

As of SQLite 3.6.10, this method is called by SQLite if you have inherited from an underlying VFSFile. Consequently ensure you pass any unrecognised codes through to your super class. For example:

        def xFileControl(self, op, ptr):
            if op==1027:
                process_quick(ptr)
            elif op==1028:
                obj=ctypes.py_object.from_address(ptr).value
            else:
                # this ensures superclass implementation is called
                return super(MyFile, self).xFileControl(op, ptr)
# we understood the op
      return True

VFSFile.xFileSize() → int¶: Return the size of the file in bytes. Remember that file sizes are 64 bit quantities even on 32 bit operating systems.

VFSFile.xLock(level: int) → None¶: Increase the lock to the level specified which is one of the SQLITE_LOCK family of constants. If you can’t increase the lock level because someone else has locked it, then raise BusyError.

VFSFile.xRead(amount: int, offset: int) → bytes¶

Read the specified amount of data starting at offset. You should make every effort to read all the data requested, or return an error. If you have the file open for non-blocking I/O or if signals happen then it is possible for the underlying operating system to do a partial read. You will need to request the remaining data. Except for empty files SQLite considers short reads to be a fatal error.

Parameters

amount – Number of bytes to read
offset – Where to start reading. This number may be 64 bit once the database is larger than 2GB.

VFSFile.xSectorSize() → int¶: Return the native underlying sector size. SQLite uses the value returned in determining the default database page size. If you do not implement the function or have an error then 4096 (the SQLite default) is returned.

VFSFile.xSync(flags: int) → None¶: Ensure data is on the disk platters (ie could survive a power failure immediately after the call returns) with the sync flags detailing what needs to be synced. You can sync more than what is requested.

VFSFile.xTruncate(newsize: int) → None¶: Set the file length to newsize (which may be more or less than the current length).

VFSFile.xUnlock(level: int) → None¶: Decrease the lock to the level specified which is one of the SQLITE_LOCK family of constants.

VFSFile.xWrite(data: bytes, offset: int) → None¶

Write the data starting at absolute offset. You must write all the data requested, or return an error. If you have the file open for non-blocking I/O or if signals happen then it is possible for the underlying operating system to do a partial write. You will need to write the remaining data.

Parameters: offset – Where to start writing. This number may be 64 bit once the database is larger than 2GB.

URIFilename class¶

class URIFilename¶

SQLite uses a convoluted method of storing uri parameters after the filename binding the C filename representation and parameters together. This class encapsulates that binding. The example shows usage of this class.

Your VFS.xOpen() method will generally be passed one of these instead of a string as the filename if the URI flag was used or the main database flag is set.

You can safely pass it on to the VFSFile constructor which knows how to get the name back out.

URIFilename.filename() → str¶: Returns the filename.

URIFilename.uri_boolean(name: str, default: bool) → bool¶

Returns the boolean value for parameter name or default if not present.

Calls: sqlite3_uri_boolean

URIFilename.uri_int(name: str, default: int) → int¶

Returns the integer value for parameter name or default if not present.

Calls: sqlite3_uri_int64

URIFilename.uri_parameter(name: str) → Optional[str]¶

Returns the value of parameter name or None.

Calls: sqlite3_uri_parameter

Virtual File System (VFS)¶

Exceptions and errors¶

VFS class¶

VFSFile class¶

URIFilename class¶

Table of Contents

Previous topic

Next topic

This Page