Virtual memory management. More...

#include <stddef.h>
#include "cpl_port.h"
#include "cpl_vsi.h"

Typedefs
typedef struct CPLVirtualMem	CPLVirtualMem
	Opaque type that represents a virtual memory mapping. More...

typedef void(*	CPLVirtualMemCachePageCbk) (CPLVirtualMem ctxt, size_t nOffset, void pPageToFill, size_t nToFill, void *pUserData)
	Callback triggered when a still unmapped page of virtual memory is accessed. More...

typedef void(*	CPLVirtualMemUnCachePageCbk) (CPLVirtualMem ctxt, size_t nOffset, const void pPageToBeEvicted, size_t nToBeEvicted, void *pUserData)
	Callback triggered when a dirty mapped page is going to be freed. More...

typedef void(*	CPLVirtualMemFreeUserData) (void *pUserData)
	Callback triggered when a virtual memory mapping is destroyed. More...

Enumerations
enum	CPLVirtualMemAccessMode { VIRTUALMEM_READONLY, VIRTUALMEM_READONLY_ENFORCED, VIRTUALMEM_READWRITE }
	Access mode of a virtual memory mapping. More...

Functions
size_t	CPLGetPageSize (void)
	Return the size of a page of virtual memory. More...

CPLVirtualMem *	CPLVirtualMemNew (size_t nSize, size_t nCacheSize, size_t nPageSizeHint, int bSingleThreadUsage, CPLVirtualMemAccessMode eAccessMode, CPLVirtualMemCachePageCbk pfnCachePage, CPLVirtualMemUnCachePageCbk pfnUnCachePage, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
	Create a new virtual memory mapping. More...

int	CPLIsVirtualMemFileMapAvailable (void)
	Return if virtual memory mapping of a file is available. More...

CPLVirtualMem *	CPLVirtualMemFileMapNew (VSILFILE fp, vsi_l_offset nOffset, vsi_l_offset nLength, CPLVirtualMemAccessMode eAccessMode, CPLVirtualMemFreeUserData pfnFreeUserData, void pCbkUserData)
	Create a new virtual memory mapping from a file. More...

CPLVirtualMem *	CPLVirtualMemDerivedNew (CPLVirtualMem pVMemBase, vsi_l_offset nOffset, vsi_l_offset nSize, CPLVirtualMemFreeUserData pfnFreeUserData, void pCbkUserData)
	Create a new virtual memory mapping derived from an other virtual memory mapping. More...

void	CPLVirtualMemFree (CPLVirtualMem *ctxt)
	Free a virtual memory mapping. More...

void *	CPLVirtualMemGetAddr (CPLVirtualMem *ctxt)
	Return the pointer to the start of a virtual memory mapping. More...

size_t	CPLVirtualMemGetSize (CPLVirtualMem *ctxt)
	Return the size of the virtual memory mapping. More...

int	CPLVirtualMemIsFileMapping (CPLVirtualMem *ctxt)
	Return if the virtual memory mapping is a direct file mapping. More...

CPLVirtualMemAccessMode	CPLVirtualMemGetAccessMode (CPLVirtualMem *ctxt)
	Return the access mode of the virtual memory mapping. More...

size_t	CPLVirtualMemGetPageSize (CPLVirtualMem *ctxt)
	Return the page size associated to a virtual memory mapping. More...

int	CPLVirtualMemIsAccessThreadSafe (CPLVirtualMem *ctxt)
	Return TRUE if this memory mapping can be accessed safely from concurrent threads. More...

void	CPLVirtualMemDeclareThread (CPLVirtualMem *ctxt)
	Declare that a thread will access a virtual memory mapping. More...

void	CPLVirtualMemUnDeclareThread (CPLVirtualMem *ctxt)
	Declare that a thread will stop accessing a virtual memory mapping. More...

void	CPLVirtualMemPin (CPLVirtualMem ctxt, void pAddr, size_t nSize, int bWriteOp)
	Make sure that a region of virtual memory will be realized. More...

void	CPLVirtualMemManagerTerminate (void)
	Cleanup any resource and handlers related to virtual memory. More...

Detailed Description

Virtual memory management.

This file provides mechanism to define virtual memory mappings, whose content is allocated transparently and filled on-the-fly. Those virtual memory mappings can be much larger than the available RAM, but only parts of the virtual memory mapping, in the limit of the allowed the cache size, will actually be physically allocated.

This exploits low-level mechanisms of the operating system (virtual memory allocation, page protection and handler of virtual memory exceptions).

It is also possible to create a virtual memory mapping from a file or part of a file.

The current implementation is Linux only.

Typedef Documentation

◆ CPLVirtualMem

typedef struct CPLVirtualMem CPLVirtualMem

Opaque type that represents a virtual memory mapping.

◆ CPLVirtualMemCachePageCbk

typedef void(* CPLVirtualMemCachePageCbk) (CPLVirtualMem *ctxt, size_t nOffset, void *pPageToFill, size_t nToFill, void *pUserData)

Callback triggered when a still unmapped page of virtual memory is accessed.

The callback has the responsibility of filling the page with relevant values

Parameters

ctxt	virtual memory handle.
nOffset	offset of the page in the memory mapping.
pPageToFill	address of the page to fill. Note that the address might be a temporary location, and not at CPLVirtualMemGetAddr() + nOffset.
nToFill	number of bytes of the page.
pUserData	user data that was passed to CPLVirtualMemNew().

◆ CPLVirtualMemFreeUserData

typedef void(* CPLVirtualMemFreeUserData) (void *pUserData)

Callback triggered when a virtual memory mapping is destroyed.

Parameters

pUserData user data that was passed to CPLVirtualMemNew().

◆ CPLVirtualMemUnCachePageCbk

typedef void(* CPLVirtualMemUnCachePageCbk) (CPLVirtualMem *ctxt, size_t nOffset, const void *pPageToBeEvicted, size_t nToBeEvicted, void *pUserData)

Callback triggered when a dirty mapped page is going to be freed.

(saturation of cache, or termination of the virtual memory mapping).

Parameters

ctxt	virtual memory handle.
nOffset	offset of the page in the memory mapping.
pPageToBeEvicted	address of the page that will be flushed. Note that the address might be a temporary location, and not at CPLVirtualMemGetAddr() + nOffset.
nToBeEvicted	number of bytes of the page.
pUserData	user data that was passed to CPLVirtualMemNew().

Enumeration Type Documentation

◆ CPLVirtualMemAccessMode

enum CPLVirtualMemAccessMode

Access mode of a virtual memory mapping.

Enumerator
VIRTUALMEM_READONLY	The mapping is meant at being read-only, but writes will not be prevented. Note that any content written will be lost.
VIRTUALMEM_READONLY_ENFORCED	The mapping is meant at being read-only, and this will be enforced through the operating system page protection mechanism.
VIRTUALMEM_READWRITE	The mapping is meant at being read-write, and modified pages can be saved thanks to the pfnUnCachePage callback

Function Documentation

◆ CPLGetPageSize()

size_t CPLGetPageSize ( void )

Return the size of a page of virtual memory.

Returns: the page size.

Since: GDAL 1.11

◆ CPLIsVirtualMemFileMapAvailable()

int CPLIsVirtualMemFileMapAvailable ( void )

Return if virtual memory mapping of a file is available.

Returns: TRUE if virtual memory mapping of a file is available.

Since: GDAL 1.11

◆ CPLVirtualMemDeclareThread()

void CPLVirtualMemDeclareThread ( CPLVirtualMem * ctxt )

Declare that a thread will access a virtual memory mapping.

This function must be called by a thread that wants to access the content of a virtual memory mapping, except if the virtual memory mapping has been created with bSingleThreadUsage = TRUE.

This function must be paired with CPLVirtualMemUnDeclareThread().

Parameters

ctxt	context returned by CPLVirtualMemNew().

Since: GDAL 1.11

◆ CPLVirtualMemDerivedNew()

CPLVirtualMem* CPLVirtualMemDerivedNew	(	CPLVirtualMem *	pVMemBase,
		vsi_l_offset	nOffset,
		vsi_l_offset	nSize,
		CPLVirtualMemFreeUserData	pfnFreeUserData,
		void *	pCbkUserData
	)

Create a new virtual memory mapping derived from an other virtual memory mapping.

This may be useful in case of creating mapping for pixel interleaved data.

The new mapping takes a reference on the base mapping.

Parameters

pVMemBase	Base virtual memory mapping
nOffset	Offset in the base virtual memory mapping from which to start the new mapping.
nSize	Size of the base virtual memory mapping to expose in the the new mapping.
pfnFreeUserData	callback that is called when the object is destroyed.
pCbkUserData	user data passed to pfnFreeUserData.

Returns: a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.

Since: GDAL 1.11

◆ CPLVirtualMemFileMapNew()

CPLVirtualMem* CPLVirtualMemFileMapNew	(	VSILFILE *	fp,
		vsi_l_offset	nOffset,
		vsi_l_offset	nLength,
		CPLVirtualMemAccessMode	eAccessMode,
		CPLVirtualMemFreeUserData	pfnFreeUserData,
		void *	pCbkUserData
	)

Create a new virtual memory mapping from a file.

The file must be a "real" file recognized by the operating system, and not a VSI extended virtual file.

In VIRTUALMEM_READWRITE mode, updates to the memory mapping will be written in the file.

On Linux AMD64 platforms, the maximum value for nLength is 128 TB. On Linux x86 platforms, the maximum value for nLength is 2 GB.

Supported on Linux only in GDAL <= 2.0, and all POSIX systems supporting mmap() in GDAL >= 2.1

Parameters

fp	Virtual file handle.
nOffset	Offset in the file to start the mapping from.
nLength	Length of the portion of the file to map into memory.
eAccessMode	Permission to use for the virtual memory mapping. This must be consistent with how the file has been opened.
pfnFreeUserData	callback that is called when the object is destroyed.
pCbkUserData	user data passed to pfnFreeUserData.

Returns: a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.

Since: GDAL 1.11

◆ CPLVirtualMemFree()

void CPLVirtualMemFree ( CPLVirtualMem * ctxt )

Free a virtual memory mapping.

The pointer returned by CPLVirtualMemGetAddr() will no longer be valid. If the virtual memory mapping was created with read/write permissions and that they are dirty (i.e. modified) pages, they will be flushed through the pfnUnCachePage callback before being freed.

Parameters

ctxt	context returned by CPLVirtualMemNew().

Since: GDAL 1.11

◆ CPLVirtualMemGetAccessMode()

CPLVirtualMemAccessMode CPLVirtualMemGetAccessMode ( CPLVirtualMem * ctxt )

Return the access mode of the virtual memory mapping.

Parameters

ctxt	context returned by CPLVirtualMemNew().

Returns: the access mode of the virtual memory mapping.

Since: GDAL 1.11

◆ CPLVirtualMemGetAddr()

void* CPLVirtualMemGetAddr ( CPLVirtualMem * ctxt )

Return the pointer to the start of a virtual memory mapping.

The bytes in the range [p:p+CPLVirtualMemGetSize()-1] where p is the pointer returned by this function will be valid, until CPLVirtualMemFree() is called.

Note that if a range of bytes used as an argument of a system call (such as read() or write()) contains pages that have not been "realized", the system call will fail with EFAULT. CPLVirtualMemPin() can be used to work around this issue.

Parameters

ctxt	context returned by CPLVirtualMemNew().

Returns: the pointer to the start of a virtual memory mapping.

Since: GDAL 1.11

◆ CPLVirtualMemGetPageSize()

size_t CPLVirtualMemGetPageSize ( CPLVirtualMem * ctxt )

Return the page size associated to a virtual memory mapping.

The value returned will be at least CPLGetPageSize(), but potentially larger.

Parameters

ctxt	context returned by CPLVirtualMemNew().

Returns: the page size

Since: GDAL 1.11

◆ CPLVirtualMemGetSize()

size_t CPLVirtualMemGetSize ( CPLVirtualMem * ctxt )

Return the size of the virtual memory mapping.

Parameters

ctxt	context returned by CPLVirtualMemNew().

Returns: the size of the virtual memory mapping.

Since: GDAL 1.11

◆ CPLVirtualMemIsAccessThreadSafe()

int CPLVirtualMemIsAccessThreadSafe ( CPLVirtualMem * ctxt )

Return TRUE if this memory mapping can be accessed safely from concurrent threads.

The situation that can cause problems is when several threads try to access a page of the mapping that is not yet mapped.

The return value of this function depends on whether bSingleThreadUsage has been set of not in CPLVirtualMemNew() and/or the implementation.

On Linux, this will always return TRUE if bSingleThreadUsage = FALSE.

Parameters

ctxt	context returned by CPLVirtualMemNew().

Returns: TRUE if this memory mapping can be accessed safely from concurrent threads.

Since: GDAL 1.11

◆ CPLVirtualMemIsFileMapping()

int CPLVirtualMemIsFileMapping ( CPLVirtualMem * ctxt )

Return if the virtual memory mapping is a direct file mapping.

Parameters

ctxt	context returned by CPLVirtualMemNew().

Returns: TRUE if the virtual memory mapping is a direct file mapping.

Since: GDAL 1.11

◆ CPLVirtualMemManagerTerminate()

void CPLVirtualMemManagerTerminate ( void )

Cleanup any resource and handlers related to virtual memory.

This function must be called after the last CPLVirtualMem object has been freed.

Since: GDAL 2.0

◆ CPLVirtualMemNew()

CPLVirtualMem* CPLVirtualMemNew	(	size_t	nSize,
		size_t	nCacheSize,
		size_t	nPageSizeHint,
		int	bSingleThreadUsage,
		CPLVirtualMemAccessMode	eAccessMode,
		CPLVirtualMemCachePageCbk	pfnCachePage,
		CPLVirtualMemUnCachePageCbk	pfnUnCachePage,
		CPLVirtualMemFreeUserData	pfnFreeUserData,
		void *	pCbkUserData
	)

Create a new virtual memory mapping.

This will reserve an area of virtual memory of size nSize, whose size might be potentially much larger than the physical memory available. Initially, no physical memory will be allocated. As soon as memory pages will be accessed, they will be allocated transparently and filled with the pfnCachePage callback. When the allowed cache size is reached, the least recently used pages will be unallocated.

On Linux AMD64 platforms, the maximum value for nSize is 128 TB. On Linux x86 platforms, the maximum value for nSize is 2 GB.

Only supported on Linux for now.

Note that on Linux, this function will install a SIGSEGV handler. The original handler will be restored by CPLVirtualMemManagerTerminate().

Parameters

nSize	size in bytes of the virtual memory mapping.
nCacheSize	size in bytes of the maximum memory that will be really allocated (must ideally fit into RAM).
nPageSizeHint	hint for the page size. Must be a multiple of the system page size, returned by CPLGetPageSize(). Minimum value is generally 4096. Might be set to 0 to let the function determine a default page size.
bSingleThreadUsage	set to TRUE if there will be no concurrent threads that will access the virtual memory mapping. This can optimize performance a bit.
eAccessMode	permission to use for the virtual memory mapping.
pfnCachePage	callback triggered when a still unmapped page of virtual memory is accessed. The callback has the responsibility of filling the page with relevant values.
pfnUnCachePage	callback triggered when a dirty mapped page is going to be freed (saturation of cache, or termination of the virtual memory mapping). Might be NULL.
pfnFreeUserData	callback that can be used to free pCbkUserData. Might be NULL
pCbkUserData	user data passed to pfnCachePage and pfnUnCachePage.

Returns: a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.

Since: GDAL 1.11

◆ CPLVirtualMemPin()

void CPLVirtualMemPin	(	CPLVirtualMem *	ctxt,
		void *	pAddr,
		size_t	nSize,
		int	bWriteOp
	)

Make sure that a region of virtual memory will be realized.

Calling this function is not required, but might be useful when debugging a process with tools like gdb or valgrind that do not naturally like segmentation fault signals.

It is also needed when wanting to provide part of virtual memory mapping to a system call such as read() or write(). If read() or write() is called on a memory region not yet realized, the call will fail with EFAULT.

Parameters

ctxt	context returned by CPLVirtualMemNew().
pAddr	the memory region to pin.
nSize	the size of the memory region.
bWriteOp	set to TRUE if the memory are will be accessed in write mode.

Since: GDAL 1.11

◆ CPLVirtualMemUnDeclareThread()

void CPLVirtualMemUnDeclareThread ( CPLVirtualMem * ctxt )

Declare that a thread will stop accessing a virtual memory mapping.

This function must be called by a thread that will no longer access the content of a virtual memory mapping, except if the virtual memory mapping has been created with bSingleThreadUsage = TRUE.

This function must be paired with CPLVirtualMemDeclareThread().

Parameters

ctxt	context returned by CPLVirtualMemNew().

Since: GDAL 1.11

Typedefs

Enumerations

Functions

Detailed Description

Typedef Documentation

◆ CPLVirtualMem

◆ CPLVirtualMemCachePageCbk

◆ CPLVirtualMemFreeUserData

◆ CPLVirtualMemUnCachePageCbk

Enumeration Type Documentation

◆ CPLVirtualMemAccessMode

Function Documentation

◆ CPLGetPageSize()

◆ CPLIsVirtualMemFileMapAvailable()

◆ CPLVirtualMemDeclareThread()

◆ CPLVirtualMemDerivedNew()

◆ CPLVirtualMemFileMapNew()

◆ CPLVirtualMemFree()

◆ CPLVirtualMemGetAccessMode()

◆ CPLVirtualMemGetAddr()

◆ CPLVirtualMemGetPageSize()

◆ CPLVirtualMemGetSize()

◆ CPLVirtualMemIsAccessThreadSafe()

◆ CPLVirtualMemIsFileMapping()

◆ CPLVirtualMemManagerTerminate()

◆ CPLVirtualMemNew()

◆ CPLVirtualMemPin()

◆ CPLVirtualMemUnDeclareThread()