casacore
Public Member Functions | Private Member Functions | Private Attributes | List of all members
casacore::BucketCache Class Reference

Cache for buckets in a part of a file. More...

#include <BucketCache.h>

Public Member Functions

 BucketCache (BucketFile *file, Int64 startOffset, uInt bucketSize, uInt nrOfBuckets, uInt cacheSize, void *ownerObject, BucketCacheToLocal readCallBack, BucketCacheFromLocal writeCallBack, BucketCacheAddBuffer addCallBack, BucketCacheDeleteBuffer deleteCallBack)
 Create the cache for (a part of) a file. More...
 
 ~BucketCache ()
 
Bool flush (uInt fromSlot=0)
 Flush the cache from the given slot on. More...
 
void clear (uInt fromSlot=0, Bool doFlush=True)
 Clear the cache from the given slot on. More...
 
void resize (uInt cacheSize)
 Resize the cache. More...
 
void resync (uInt nrBucket, uInt nrOfFreeBucket, Int firstFreeBucket)
 Resynchronize the object (after another process updated the file). More...
 
uInt nBucket () const
 Get the current nr of buckets in the file. More...
 
uInt cacheSize () const
 Get the current cache size (in buckets). More...
 
void setDirty ()
 Set the dirty bit for the current bucket. More...
 
char * getBucket (uInt bucketNr)
 Make another bucket current. More...
 
void extend (uInt nrBucket)
 Extend the file with the given number of buckets. More...
 
uInt addBucket (char *data)
 Add a bucket to the file and make it the current one. More...
 
void removeBucket ()
 Remove the current bucket; i.e. More...
 
void get (char *buf, uInt length, Int64 offset)
 Get a part from the file outside the cached area. More...
 
void put (const char *buf, uInt length, Int64 offset)
 Put a part from the file outside the cached area. More...
 
Int firstFreeBucket () const
 Get the bucket number of the first free bucket. More...
 
uInt nFreeBucket () const
 Get the number of free buckets. More...
 
void initStatistics ()
 (Re)initialize the cache statistics. More...
 
void showStatistics (ostream &os) const
 Show the statistics. More...
 

Private Member Functions

 BucketCache (const BucketCache &)
 Copy constructor is not possible. More...
 
BucketCacheoperator= (const BucketCache &)
 Assignment is not possible. More...
 
void setLRU ()
 Set the LRU information for the current slot. More...
 
void getSlot (uInt bucketNr)
 Get a cache slot for the bucket. More...
 
void writeBucket (uInt slotNr)
 Write a bucket. More...
 
void readBucket (uInt slotNr)
 Read a bucket. More...
 
void initializeBuckets (uInt bucketNr)
 Initialize the bucket buffer. More...
 
void checkOffset (uInt length, Int64 offset) const
 Check if the offset of a non-cached part is correct. More...
 

Private Attributes

BucketFileits_file
 The file used. More...
 
void * its_Owner
 The owner object. More...
 
BucketCacheToLocal its_ReadCallBack
 The read callback function. More...
 
BucketCacheFromLocal its_WriteCallBack
 The write callback function. More...
 
BucketCacheAddBuffer its_InitCallBack
 The add bucket callback function. More...
 
BucketCacheDeleteBuffer its_DeleteCallBack
 The delete callback function. More...
 
Int64 its_StartOffset
 The starting offsets of the buckets in the file. More...
 
uInt its_BucketSize
 The bucket size. More...
 
uInt its_CurNrOfBuckets
 The current nr of buckets in the file. More...
 
uInt its_NewNrOfBuckets
 The new nr of buckets in the file (after extension). More...
 
uInt its_CacheSize
 The size of the cache (i.e. More...
 
uInt its_CacheSizeUsed
 The nr of slots used in the cache. More...
 
PtrBlock< char * > its_Cache
 The cache itself. More...
 
uInt its_ActualSlot
 The cache slot actually used. More...
 
Block< Intits_SlotNr
 The slot numbers of the buckets in the cache (-1 = not in cache). More...
 
Block< uIntits_BucketNr
 The buckets in the cache. More...
 
Block< uIntits_Dirty
 Determine if a block is dirty (i.e. More...
 
Block< uIntits_LRU
 Determine when a block is used for the last time. More...
 
uInt its_LRUCounter
 The Least Recently Used counter. More...
 
char * its_Buffer
 The internal buffer. More...
 
uInt its_NrOfFree
 The number of free buckets. More...
 
Int its_FirstFree
 The first free bucket (-1 = no free buckets). More...
 
uInt naccess_p
 The statistics. More...
 
uInt nread_p
 
uInt ninit_p
 
uInt nwrite_p
 

Detailed Description

Cache for buckets in a part of a file.

Intended use:

Public interface

Review Status

Reviewed By:
UNKNOWN
Date Reviewed:
before2004/08/25

Prerequisite

Etymology

BucketCache implements a cache for buckets in (a part of) a file.

Synopsis

A cache may allow more efficient quasi-random IO. It can, for instance, be used when a limited number of blocks in a file have to be accessed again and again.

The class BucketCache provides such a cache. It can be used on a consecutive part of a file as long as that part is not simultaneously accessed in another way (including another BucketCache object).

BucketCache stores the data as given. It uses (see (group=BucketCache_CallBack))callback functions to allocate/delete buffers and to convert the data to/from local format.

When a new bucket is needed and all slots in the cache are used, BucketCache will remove the least recently used bucket from the cache. When the dirty flag is set, it will first be written.

BucketCache maintains a list of free buckets. Initially this list is empty. When a bucket is removed, it is added to the free list. AddBucket will take buckets from the free list before extending the file.

Since it is possible to handle only a part of a file by a BucketCache object, it is also possible to have multiple BucketCache objects on the same file (as long as they access disjoint parts of the file). Each BucketCache object can have its own bucket size. This can, for example, be used to have tiled arrays with different tile shapes in the same file.

Statistics are kept to know how efficient the cache is working. It is possible to initialize and show the statistics.

Motivation

A cache may reduce IO traffix considerably. Furthermore it is more efficient to keep a cache in local format. In that way conversion to/from local only have to be done when data gets read/written. It also allows for precalculations.

Example

// Define the callback function for reading a bucket.
char* bToLocal (void*, const char* data)
{
char* ptr = new char[32768];
memcpy (ptr, data, 32768);
return ptr;
}
// Define the callback function for writing a bucket.
void bFromLocal (void*, char* data, const char* local)
{
memcpy (data, local, 32768);
}
// Define the callback function for initializing a new bucket.
char* bAddBuffer (void*)
{
char* ptr = new char[32768];
for (uInt i=0; i++; i<32768) {
ptr[i] = 0;
}
return ptr;
}
// Define the callback function for deleting a bucket.
void bDeleteBuffer (void*, char* buffer)
{
delete [] buffer;
}
void someFunc()
{
// Open the filebuf.
BucketFile file(...);
file.open();
uInt i;
// Create a cache for the part of the file starting at offset 512
// consisting of 1000 buckets. The cache consists of 10 buckets.
// Each bucket is 32768 bytes.
BucketCache cache (&file, 512, 32768, 1000, 10, 0,
bToLocal, bFromLocal, bAddBuffer, bDeleteBuffer);
// Write all buckets into the file.
for (i=0; i<100; i++) {
char* buf = new char[32768];
cache.addBucket (buf);
}
Flush the cache to write all buckets in it.
cache.flush();
// Read all buckets from the file.
for (i=0; i<1000; i++) {
char* buf = cache.getBucket(i);
..\.
}
cout << cache.nBucket() << endl;
}
BucketCache(BucketFile *file, Int64 startOffset, uInt bucketSize, uInt nrOfBuckets, uInt cacheSize, void *ownerObject, BucketCacheToLocal readCallBack, BucketCacheFromLocal writeCallBack, BucketCacheAddBuffer addCallBack, BucketCacheDeleteBuffer deleteCallBack)
Create the cache for (a part of) a file.
unsigned int uInt
Definition: aipstype.h:51
LatticeExprNode all(const LatticeExprNode &expr)

To Do

Definition at line 217 of file BucketCache.h.

Constructor & Destructor Documentation

◆ BucketCache() [1/2]

casacore::BucketCache::BucketCache ( BucketFile file,
Int64  startOffset,
uInt  bucketSize,
uInt  nrOfBuckets,
uInt  cacheSize,
void *  ownerObject,
BucketCacheToLocal  readCallBack,
BucketCacheFromLocal  writeCallBack,
BucketCacheAddBuffer  addCallBack,
BucketCacheDeleteBuffer  deleteCallBack 
)

Create the cache for (a part of) a file.

The file part used starts at startOffset. Its length is bucketSize*nrOfBuckets bytes. When the file is smaller, the remainder is indicated as an extension similarly to the behaviour of function extend.

◆ ~BucketCache()

casacore::BucketCache::~BucketCache ( )

◆ BucketCache() [2/2]

casacore::BucketCache::BucketCache ( const BucketCache )
private

Copy constructor is not possible.

Member Function Documentation

◆ addBucket()

uInt casacore::BucketCache::addBucket ( char *  data)

Add a bucket to the file and make it the current one.

When no more cache slots are available, the one least recently used is flushed.
When no free buckets are available, the file will be extended with one bucket. It returns the new bucket number. The buffer must have been allocated on the heap. It will get part of the cache; its contents are not copied. Thus the buffer should hereafter NOT be used for other purposes. It will be deleted later via the DeleteBuffer callback function. The data is copied into the bucket. A pointer to the data in local format is returned.

◆ cacheSize()

uInt casacore::BucketCache::cacheSize ( ) const
inline

Get the current cache size (in buckets).

Definition at line 403 of file BucketCache.h.

References its_CacheSize.

◆ checkOffset()

void casacore::BucketCache::checkOffset ( uInt  length,
Int64  offset 
) const
private

Check if the offset of a non-cached part is correct.

◆ clear()

void casacore::BucketCache::clear ( uInt  fromSlot = 0,
Bool  doFlush = True 
)

Clear the cache from the given slot on.

By default the entire cache is cleared. It will remove the buckets in the cleared part. If wanted and needed, the buckets are flushed to the file before removing them. It can be used to enforce rereading buckets from the file.

◆ extend()

void casacore::BucketCache::extend ( uInt  nrBucket)

Extend the file with the given number of buckets.

The buckets get initialized when they are acquired (using getBucket) for the first time.

◆ firstFreeBucket()

Int casacore::BucketCache::firstFreeBucket ( ) const
inline

Get the bucket number of the first free bucket.

-1 = no free buckets.

Definition at line 406 of file BucketCache.h.

References its_FirstFree.

◆ flush()

Bool casacore::BucketCache::flush ( uInt  fromSlot = 0)

Flush the cache from the given slot on.

By default the entire cache is flushed. When the entire cache is flushed, possible remaining uninitialized buckets will be initialized first. A True status is returned when buckets had to be written.

◆ get()

void casacore::BucketCache::get ( char *  buf,
uInt  length,
Int64  offset 
)

Get a part from the file outside the cached area.

It is checked if that part is indeed outside the cached file area.

◆ getBucket()

char* casacore::BucketCache::getBucket ( uInt  bucketNr)

Make another bucket current.

When no more cache slots are available, the one least recently used is flushed. The data in the bucket is converted using the ToLocal callback function. When the bucket does not exist yet in the file, it gets added and initialized using the AddBuffer callback function. A pointer to the data in converted format is returned.

◆ getSlot()

void casacore::BucketCache::getSlot ( uInt  bucketNr)
private

Get a cache slot for the bucket.

◆ initializeBuckets()

void casacore::BucketCache::initializeBuckets ( uInt  bucketNr)
private

Initialize the bucket buffer.

The uninitialized buckets before this bucket are also initialized. It returns a pointer to the buffer.

◆ initStatistics()

void casacore::BucketCache::initStatistics ( )

(Re)initialize the cache statistics.

◆ nBucket()

uInt casacore::BucketCache::nBucket ( ) const

Get the current nr of buckets in the file.

◆ nFreeBucket()

uInt casacore::BucketCache::nFreeBucket ( ) const
inline

Get the number of free buckets.

Definition at line 409 of file BucketCache.h.

References its_NrOfFree.

◆ operator=()

BucketCache& casacore::BucketCache::operator= ( const BucketCache )
private

Assignment is not possible.

◆ put()

void casacore::BucketCache::put ( const char *  buf,
uInt  length,
Int64  offset 
)

Put a part from the file outside the cached area.

It is checked if that part is indeed outside the cached file area.

◆ readBucket()

void casacore::BucketCache::readBucket ( uInt  slotNr)
private

Read a bucket.

◆ removeBucket()

void casacore::BucketCache::removeBucket ( )

Remove the current bucket; i.e.

add it to the beginning of the free bucket list.

◆ resize()

void casacore::BucketCache::resize ( uInt  cacheSize)

Resize the cache.

When the cache gets smaller, the latter buckets are cached out. It does not take "least recently used" into account.

◆ resync()

void casacore::BucketCache::resync ( uInt  nrBucket,
uInt  nrOfFreeBucket,
Int  firstFreeBucket 
)

Resynchronize the object (after another process updated the file).

It clears the cache (so all data will be reread) and sets the new sizes.

◆ setDirty()

void casacore::BucketCache::setDirty ( )

Set the dirty bit for the current bucket.

◆ setLRU()

void casacore::BucketCache::setLRU ( )
private

Set the LRU information for the current slot.

◆ showStatistics()

void casacore::BucketCache::showStatistics ( ostream &  os) const

Show the statistics.

◆ writeBucket()

void casacore::BucketCache::writeBucket ( uInt  slotNr)
private

Write a bucket.

Member Data Documentation

◆ its_ActualSlot

uInt casacore::BucketCache::its_ActualSlot
private

The cache slot actually used.

Definition at line 350 of file BucketCache.h.

◆ its_BucketNr

Block<uInt> casacore::BucketCache::its_BucketNr
private

The buckets in the cache.

Definition at line 354 of file BucketCache.h.

◆ its_BucketSize

uInt casacore::BucketCache::its_BucketSize
private

The bucket size.

Definition at line 338 of file BucketCache.h.

◆ its_Buffer

char* casacore::BucketCache::its_Buffer
private

The internal buffer.

Definition at line 362 of file BucketCache.h.

◆ its_Cache

PtrBlock<char*> casacore::BucketCache::its_Cache
private

The cache itself.

Definition at line 348 of file BucketCache.h.

◆ its_CacheSize

uInt casacore::BucketCache::its_CacheSize
private

The size of the cache (i.e.

#buckets fitting in it).

Definition at line 344 of file BucketCache.h.

Referenced by cacheSize().

◆ its_CacheSizeUsed

uInt casacore::BucketCache::its_CacheSizeUsed
private

The nr of slots used in the cache.

Definition at line 346 of file BucketCache.h.

◆ its_CurNrOfBuckets

uInt casacore::BucketCache::its_CurNrOfBuckets
private

The current nr of buckets in the file.

Definition at line 340 of file BucketCache.h.

◆ its_DeleteCallBack

BucketCacheDeleteBuffer casacore::BucketCache::its_DeleteCallBack
private

The delete callback function.

Definition at line 334 of file BucketCache.h.

◆ its_Dirty

Block<uInt> casacore::BucketCache::its_Dirty
private

Determine if a block is dirty (i.e.

changed) (1=dirty).

Definition at line 356 of file BucketCache.h.

◆ its_file

BucketFile* casacore::BucketCache::its_file
private

The file used.

Definition at line 324 of file BucketCache.h.

◆ its_FirstFree

Int casacore::BucketCache::its_FirstFree
private

The first free bucket (-1 = no free buckets).

Definition at line 366 of file BucketCache.h.

Referenced by firstFreeBucket().

◆ its_InitCallBack

BucketCacheAddBuffer casacore::BucketCache::its_InitCallBack
private

The add bucket callback function.

Definition at line 332 of file BucketCache.h.

◆ its_LRU

Block<uInt> casacore::BucketCache::its_LRU
private

Determine when a block is used for the last time.

Definition at line 358 of file BucketCache.h.

◆ its_LRUCounter

uInt casacore::BucketCache::its_LRUCounter
private

The Least Recently Used counter.

Definition at line 360 of file BucketCache.h.

◆ its_NewNrOfBuckets

uInt casacore::BucketCache::its_NewNrOfBuckets
private

The new nr of buckets in the file (after extension).

Definition at line 342 of file BucketCache.h.

◆ its_NrOfFree

uInt casacore::BucketCache::its_NrOfFree
private

The number of free buckets.

Definition at line 364 of file BucketCache.h.

Referenced by nFreeBucket().

◆ its_Owner

void* casacore::BucketCache::its_Owner
private

The owner object.

Definition at line 326 of file BucketCache.h.

◆ its_ReadCallBack

BucketCacheToLocal casacore::BucketCache::its_ReadCallBack
private

The read callback function.

Definition at line 328 of file BucketCache.h.

◆ its_SlotNr

Block<Int> casacore::BucketCache::its_SlotNr
private

The slot numbers of the buckets in the cache (-1 = not in cache).

Definition at line 352 of file BucketCache.h.

◆ its_StartOffset

Int64 casacore::BucketCache::its_StartOffset
private

The starting offsets of the buckets in the file.

Definition at line 336 of file BucketCache.h.

◆ its_WriteCallBack

BucketCacheFromLocal casacore::BucketCache::its_WriteCallBack
private

The write callback function.

Definition at line 330 of file BucketCache.h.

◆ naccess_p

uInt casacore::BucketCache::naccess_p
private

The statistics.

Definition at line 368 of file BucketCache.h.

◆ ninit_p

uInt casacore::BucketCache::ninit_p
private

Definition at line 370 of file BucketCache.h.

◆ nread_p

uInt casacore::BucketCache::nread_p
private

Definition at line 369 of file BucketCache.h.

◆ nwrite_p

uInt casacore::BucketCache::nwrite_p
private

Definition at line 371 of file BucketCache.h.


The documentation for this class was generated from the following file: