Building

See version info to understand the relationship between Python, APSW, and SQLite versions.

setup.py

Short story: You run setup.py but you should ideally follow the recommended way which will also fetch needed components for you.

Command

Result
python setup.py install test

Compiles APSW with default Python compiler, installs it into Python site library directory and then runs the test suite.
python setup.py install –user

Compiles APSW with default Python compiler and installs it into a subdirectory of your home directory. See PEP 370 for more details.
python setup.py build_ext –force –inplace test

Compiles the extension but doesn’t install it. The test suite is then run.
python setup.py build –debug install

Compiles APSW with debug information. This also turns on assertions in APSW that double check the code assumptions. If you are using the SQLite amalgamation then assertions are turned on in that too. Note that this will considerably slow down APSW and SQLite.

Additional setup.py flags

There are a number of APSW specific flags to commands you can specify.

fetch

setup.py can automatically fetch SQLite and other optional components. You can set the environment variable http_proxy to control proxy usage for the download.

If any files are downloaded then the build step will automatically use them. This still applies when you do later builds without re-fetching.

python setup.py fetch options

fetch flag

Result
–version=VERSION

By default the SQLite version corresponding to the APSW release is retrieved, You can also ask for specific versions, or for latest which uses the SQLite download page to work out the most recent version.
–missing-checksum-ok

Allows setup to continue if the checksum is missing.
–all

Gets all components listed below.
–sqlite

Automatically downloads the SQLite amalgamation. The amalgamation is the preferred way to use SQLite as you have total control over what components are included or excluded (see below) and have no dependencies on any existing libraries on your developer or deployment machines. The amalgamation includes the fts3/4/5, rtree, json1 and icu extensions. Nny existing sqlite3/ directory will be erased and the downloaded code placed in a newly created sqlite3/ directory.

Note

The SQLite downloads are not digitally signed which means you have no way of verifying they were produced by the SQLite team or were not modified between the SQLite servers and your computer.

Consequently APSW ships with a checksums file that includes checksums for the various SQLite downloads. If the download does not match the checksum then it is rejected and an error occurs.

The SQLite download page is not checksummed, so in theory a bad guy could modify it to point at a malicious download version instead. (setup only uses the page to determine the current version number - the SQLite download site URL is hard coded.)

If the URL is not listed in the checksums file then setup aborts. You can use –missing-checksum-ok to continue. You are recommended instead to update the checksums file with the correct information.

Note

(This note only applies to non-Windows platforms.) By default the amalgamation will work on your platform. It detects the operating system (and compiler if relevant) and uses the appropriate APIs. However it then only uses the oldest known working APIs. For example it will use the sleep system call. More recent APIs may exist but the amalgamation needs to be told they exist. As an example sleep can only sleep in increments of one second while the usleep system call can sleep in increments of one microsecond. The default SQLite busy handler does small sleeps (eg 1/50th of a second) backing off as needed. If sleep is used then those will all be a minimum of a second. A second example is that the traditional APIs for getting time information are not re-entrant and cannot be used concurrently from multiple threads. Consequently SQLite has mutexes to ensure that concurrent calls do not happen. However you can tell it you have more recent re-entrant versions of the calls and it won’t need to bother with the mutexes.

After fetching the amalgamation, setup automatically determines what new APIs you have by running the configure script that comes with SQLite and noting the output. The information is placed in sqlite3/sqlite3config.h. The build stage will automatically take note of this as needed.

build/build_ext

You can enable or omit certain functionality by specifying flags to the build and/or build_ext commands of setup.py:

python setup.py build *options*

Note that the options do not accumulate. If you want to specify multiple enables or omits then you need to give the flag once and giving a comma separated list. For example:

python setup.py build --enable=fts3,fts3_parenthesis,rtree,icu

SQLite includes many options defined to the C compiler. If you want to change compiled in default values, or provide defines like SQLITE_CUSTOM_INCLUDE then you can use –definevalues using = and comma separating. For example:

python setup.py build_ext --definevalues SQLITE_DEFAULT_FILE_FORMAT=1,SQLITE_CUSTOM_INCLUDE=config.h

build/build_ext flag

Result
–enable-all-extensions

Enables the STAT4, FTS3/4/5, RTree, JSON1, RBU, and ICU extensions if icu-config is on your path
–enable=fts3
–enable=fts4
–enable=fts5

Enables the full text search extension. This flag only helps when using the amalgamation. If not using the amalgamation then you need to separately ensure fts3/4/5 is enabled in the SQLite install. You are likely to want the parenthesis option on unless you have legacy code (–enable-all-extensions turns it on).
–enable=rtree

Enables the spatial table extension. This flag only helps when using the amalgamation. If not using the amalgamation then you need to separately ensure rtree is enabled in the SQLite install.
–enable=rbu

Enables the reumable bulk update extension. This flag only helps when using the amalgamation. If not using the amalgamation then you need to separately ensure rbu is enabled in the SQLite install.
–enable=icu

Enables the International Components for Unicode extension. Note that you must have the ICU libraries on your machine which setup will automatically try to find using icu-config. This flag only helps when using the amalgamation. If not using the amalgamation then you need to separately ensure ICU is enabled in the SQLite install.
–omit=ITEM

Causes various functionality to be omitted. For example –omit=load_extension will omit code to do with loading extensions. If using the amalgamation then this will omit the functionality from APSW and SQLite, otherwise the functionality will only be omitted from APSW (ie the code will still be in SQLite, APSW just won’t call it). In almost all cases you will need to regenerate the SQLite source because the omits also alter the generated SQL parser. See the relevant SQLite documentation.

Matching APSW and SQLite options

APSW needs to see the same options as SQLite to correctly match it. For example if SQLite is compiled without loadable extensions, then APSW also needs to know that at compile time because the APIs won’t be present. Another example is Cursor.description_full needs to know if SQLITE_ENABLE_COLUMN_METADATA was defined when building SQLite for the same reason.

If you use the amalgamation (recommended configuration) then APSW and SQLite will see the same options and will be correctly in sync.

If you are using the system provided SQLite then specify –use-system-sqlite-config to build_ext, and the configuration will be automatically obtained (using ctypes find_library)

You can use the amalgamation and –use-system-sqlite-config simultaneously in which case the amalgamation will have an identical configuration to the system one. This is useful if you are using a newer SQLite version in the amalgamation, but still want to match the system.

Finding SQLite 3

SQLite 3 is needed during the build process. If you specify fetch –sqlite to the setup.py command line then it will automatically fetch the current version of the SQLite amalgamation. (The current version is determined by parsing the SQLite download page). You can manually specify the version, for example fetch –sqlite –version=3.39.4.

These methods are tried in order:

Amalgamation

The file sqlite3.c and then sqlite3/sqlite3.c is looked for. The SQLite code is then statically compiled into the APSW extension and is invisible to the rest of the process. There are no runtime library dependencies on SQLite as a result. When you use fetch this is where it places the downloaded amalgamation.

Local build

The header sqlite3/sqlite3.h and library sqlite3/libsqlite3.a,so,dll is looked for.

User directories

If specifying –user then your user directory is searched first. See PEP 370 for more details.

System directories

The default compiler include path (eg /usr/include) and library path (eg /usr/lib) are used.

Note

If you compiled SQLite with any OMIT flags (eg SQLITE_OMIT_LOAD_EXTENSION) then you must include them in the setup.py command or file. For this example you could use setup.py build –omit=load_extension to add the same flags.

Source distribution (advanced)

If you want to make a source distribution or a binary distribution that creates an intermediate source distribution such as bdist_rpm then you can have the SQLite amalgamation automatically included as part of it. If you specify the fetch command as part of the same command line then everything fetched is included in the source distribution. For example this will fetch all components, include them in the source distribution and build a rpm using those components:

$ python setup.py fetch --all bdist_rpm

Testing

SQLite itself is extensively tested. It has considerably more code dedicated to testing than makes up the actual database functionality.

APSW includes tests which use the standard Python testing modules to verify correct operation. New code is developed alongside the tests. Reported issues also have test cases to ensure the issue doesn’t happen or doesn’t happen again.:

$ python3 -m apsw.tests
                Python  /usr/bin/python3 sys.version_info(major=3, minor=10, micro=4, releaselevel='final', serial=0)
Testing with APSW file  /space/apsw/apsw/__init__.cpython-310-x86_64-linux-gnu.so
          APSW version  3.39.2.0
    SQLite lib version  3.39.2
SQLite headers version  3039002
    Using amalgamation  True
...............................................................................................
----------------------------------------------------------------------
Ran 95 tests in 25.990s

OK

The tests also ensure that as much APSW code as possible is executed including alternate paths through the code. 95.5% of the APSW code is executed by the tests. If you checkout the APSW source then there is a script tools/coverage.sh that enables extra code that deliberately induces extra conditions such as memory allocation failures, SQLite returning undocumented error codes etc. That brings coverage up to 99.6% of the code.

A memory checker Valgrind is used while running the test suite. The test suite is run multiple times to make any memory leaks or similar issues stand out. A checking version of Python is also used. See tools/valgrind.sh in the source. The same testing is also done with the compiler’s sanitizer option.

To ensure compatibility with the various Python versions, a script downloads and compiles all supported Python versions in both debug and release configurations (and 32 and 64 bit) against the APSW and SQLite supported versions running the tests. See tools/megatest.py in the source.

In short both SQLite and APSW have a lot of testing!