Bundle URIs

The bundle URI standard aims to be flexible enough to satisfy multiple workloads. The bundle provider and the Git client have several choices in how they create and consume bundle URIs.

Each repository is different and every Git server has different needs. Hopefully the bundle URI feature is flexible enough to satisfy all needs. If not, then the feature can be extended through its versioning mechanism.

To provide a server-side implementation of bundle servers, no other parts of the Git protocol are required. This allows server maintainers to use static content solutions such as CDNs in order to serve the bundle files.

At the current scope of the bundle URI feature, all URIs are expected to be HTTP(S) URLs where content is downloaded to a local file using a GET request to that URL. The server could include authentication requirements to those requests with the aim of triggering the configured credential helper for secure access. (Future extensions could use "file://" URIs or SSH URIs.)

Assuming a 200 OK response from the server, the content at the URL is inspected. First, Git attempts to parse the file as a bundle file of version 2 or higher. If the file is not a bundle, then the file is parsed as a plain-text file using Git’s config parser. The key-value pairs in that config file are expected to describe a list of bundle URIs. If neither of these parse attempts succeed, then Git will report an error to the user that the bundle URI provided erroneous data.

Any other data provided by the server is considered erroneous.

The Git server can advertise bundle URIs using a set of key=value pairs. A bundle URI can also serve a plain-text file in the Git config format containing these same key=value pairs. In both cases, we consider this to be a bundle list. The pairs specify information about the bundles that the client can use to make decisions for which bundles to download and which to ignore.

A few keys focus on properties of the list itself.

The remaining keys include an <id> segment which is a server-designated name for each available bundle. The <id> must contain only alphanumeric and - characters.

Here is an example bundle list using the Git config format:

This example uses bundle.mode=all as well as the bundle.<id>.creationToken heuristic. It also uses the bundle.<id>.filter options to present two parallel sets of bundles: one for full clones and another for blobless partial clones.

Suppose that this bundle list was found at the URI https://bundles.example.com/git/git/ and so the two blobless bundles have the following fully-expanded URIs:

If a user knows a bundle URI for the repository they are cloning, then they can specify that URI manually through a command-line option. However, a Git host may want to advertise bundle URIs during the clone operation, helping users unaware of the feature.

The only thing required for this feature is that the server can advertise one or more bundle URIs. This advertisement takes the form of a new protocol v2 capability specifically for discovering bundle URIs.

The client could choose an arbitrary bundle URI as an option or select the URI with best performance by some exploratory checks. It is up to the bundle provider to decide if having multiple URIs is preferable to a single URI that is geodistributed through server-side infrastructure.

The primary need for bundle URIs is to speed up clones. The Git client will interact with bundle URIs according to the following flow:

Note that during a clone we expect that all bundles will be required, and heuristics such as bundle.<uri>.creationToken can be used to download bundles in chronological order or in parallel.

If a given bundle URI is a bundle list with a bundle.heuristic value, then the client can choose to store that URI as its chosen bundle URI. The client can then navigate directly to that URI during later git fetch calls.

When downloading bundle URIs, the client can choose to inspect the initial content before committing to downloading the entire content. This may provide enough information to determine if the URI is a bundle list or a bundle. In the case of a bundle, the client may inspect the bundle header to determine that all advertised tips are already in the client repository and cancel the remaining download.

When the client fetches new data, it can decide to fetch from bundle servers before fetching from the origin remote. This could be done via a command-line option, but it is more likely useful to use a config value such as the one specified during the clone.

The fetch operation follows the same procedure to download bundles from a bundle list (although we do not want to use parallel downloads here). We expect that the process will end when all prerequisite commit OIDs in a thin bundle are already in the object database.

When using the creationToken heuristic, the client can avoid downloading any bundles if their creation tokens are not larger than the stored creation token. After fetching new bundles, Git updates this local creation token.

If the bundle provider does not provide a heuristic, then the client should attempt to inspect the bundle headers before downloading the full bundle data in case the bundle tips already exist in the client repository.

If the Git client discovers something unexpected while downloading information according to a bundle URI or the bundle list found at that location, then Git can ignore that data and continue as if it was not given a bundle URI. The remote Git server is the ultimate source of truth, not the bundle URI.

Here are a few example error conditions:

There are also situations that could be seen as wasteful, but are not error conditions:

The bundle URI feature is intentionally designed to be flexible to different ways a bundle provider wants to organize the object data. However, it can be helpful to have a complete organization model described here so providers can start from that base.

This example organization is a simplified model of what is used by the GVFS Cache Servers (see section near the end of this document) which have been beneficial in speeding up clones and fetches for very large repositories, although using extra software outside of Git.

The bundle provider deploys servers across multiple geographies. Each server manages its own bundle set. The server can track a number of Git repositories, but provides a bundle list for each based on a pattern. For example, when mirroring a repository at https://<domain>/<org>/<repo> the bundle server could have its bundle list available at https://<server-url>/<domain>/<org>/<repo>. The origin Git server can list all of these servers under the "any" mode:

This "list of lists" is static and only changes if a bundle server is added or removed.

Each bundle server manages its own set of bundles. The initial bundle list contains only a single bundle, containing all of the objects received from cloning the repository from the origin server. The list uses the creationToken heuristic and a creationToken is made for the bundle based on the server’s timestamp.

The bundle server runs regularly-scheduled updates for the bundle list, such as once a day. During this task, the server fetches the latest contents from the origin server and generates a bundle containing the objects reachable from the latest origin refs, but not contained in a previously-computed bundle. This bundle is added to the list, with care that the creationToken is strictly greater than the previous maximum creationToken.

When the bundle list grows too large, say more than 30 bundles, then the oldest "N minus 30" bundles are combined into a single bundle. This bundle’s creationToken is equal to the maximum creationToken among the merged bundles.

An example bundle list is provided here, although it only has two daily bundles and not a full list of 30:

To avoid storing and serving object data in perpetuity despite becoming unreachable in the origin server, this bundle merge can be more careful. Instead of taking an absolute union of the old bundles, instead the bundle can be created by looking at the newer bundles and ensuring that their necessary commits are all available in this merged bundle (or in another one of the newer bundles). This allows "expiring" object data that is not being used by new commits in this window of time. That data could be reintroduced by a later push.

The intention of this data organization has two main goals. First, initial clones of the repository become faster by downloading precomputed object data from a closer source. Second, git fetch commands can be faster, especially if the client has not fetched for a few days. However, if a client does not fetch for 30 days, then the bundle list organization would cause redownloading a large amount of object data.

One way to make this organization more useful to users who fetch frequently is to have more frequent bundle creation. For example, bundles could be created every hour, and then once a day those "hourly" bundles could be merged into a "daily" bundle. The daily bundles are merged into the oldest bundle after 30 days.

It is recommended that this bundle strategy is repeated with the blob:none filter if clients of this repository are expecting to use blobless partial clones. This list of blobless bundles stays in the same list as the full bundles, but uses the bundle.<id>.filter key to separate the two groups. For very large repositories, the bundle provider may want to only provide blobless bundles.

This design document is being submitted on its own as an aspirational document, with the goal of implementing all of the mentioned client features over the course of several patch series. Here is a potential outline for submitting these features:

As these features are reviewed, this plan might be updated. We also expect that new designs will be discovered and implemented as this feature matures and becomes used in real-world scenarios.

The Git protocol already has a capability where the Git server can list a set of URLs along with the packfile response when serving a client request. The client is then expected to download the packfiles at those locations in order to have a complete understanding of the response.

This mechanism is used by the Gerrit server (implemented with JGit) and has been effective at reducing CPU load and improving user performance for clones.

A major downside to this mechanism is that the origin server needs to know exactly what is in those packfiles, and the packfiles need to be available to the user for some time after the server has responded. This coupling between the origin and the packfile data is difficult to manage.

Further, this implementation is extremely hard to make work with fetches.

The GVFS Protocol [2] is a set of HTTP endpoints designed independently of the Git project before Git’s partial clone was created. One feature of this protocol is the idea of a "cache server" which can be colocated with build machines or developer offices to transfer Git data without overloading the central server.

The endpoint that VFS for Git is famous for is the GET /gvfs/objects/{oid} endpoint, which allows downloading an object on-demand. This is a critical piece of the filesystem virtualization of that product.

However, a more subtle need is the GET /gvfs/prefetch?lastPackTimestamp=<t> endpoint. Given an optional timestamp, the cache server responds with a list of precomputed packfiles containing the commits and trees that were introduced in those time intervals.

The cache server computes these "prefetch" packfiles using the following strategy:

When a user runs gvfs clone or scalar clone against a repo with cache servers, the client requests all prefetch packfiles, which is at most 24 + 30 + 1 packfiles downloading only commits and trees. The client then follows with a request to the origin server for the references, and attempts to checkout that tip reference. (There is an extra endpoint that helps get all reachable trees from a given commit, in case that commit was not already in a prefetch packfile.)

During a git fetch, a hook requests the prefetch endpoint using the most-recent timestamp from a previously-downloaded prefetch packfile. Only the list of packfiles with later timestamps are downloaded. Most users fetch hourly, so they get at most one hourly prefetch pack. Users whose machines have been off or otherwise have not fetched in over 30 days might redownload all prefetch packfiles. This is rare.

It is important to note that the clients always contact the origin server for the refs advertisement, so the refs are frequently "ahead" of the prefetched pack data. The missing objects are downloaded on-demand using the GET gvfs/objects/{oid} requests, when needed by a command such as git checkout or git log. Some Git optimizations disable checks that would cause these on-demand downloads to be too aggressive.

[1] https://lore.kernel.org/git/RFC-cover-00.13-0000000000-20210805T150534Z-avarab@gmail.com/ An earlier RFC for a bundle URI feature.

[2] https://github.com/microsoft/VFSForGit/blob/master/Protocol.md The GVFS Protocol