casacore
Public Member Functions | Private Member Functions | Private Attributes | List of all members
casacore::MultiFile Class Reference

More...

#include <MultiFile.h>

Public Member Functions

 MultiFile (const String &name, ByteIO::OpenOption, Int blockSize=0, Bool useODirect=False)
 Open or create a MultiFile with the given name. More...
 
virtual ~MultiFile ()
 The destructor flushes and closes the file. More...
 
virtual void reopenRW ()
 Reopen the underlying file for read/write access. More...
 
virtual void fsync ()
 Fsync the file (i.e., force the data to be physically written). More...
 
- Public Member Functions inherited from casacore::MultiFileBase
 MultiFileBase (const String &name, Int blockSize, Bool useODirect)
 Open or create a MultiFileBase with the given name. More...
 
virtual ~MultiFileBase ()
 The destructor flushes and closes the file. More...
 
Int fileId (const String &name, Bool throwExcp=True) const
 Return the file id of a file in the MultiFileBase object. More...
 
Int addFile (const String &name)
 Add a file to the MultiFileBase object. More...
 
void deleteFile (Int fileId)
 Delete a file. More...
 
Int64 read (Int fileId, void *buffer, Int64 size, Int64 offset)
 Read a block at the given offset. More...
 
Int64 write (Int fileId, const void *buffer, Int64 size, Int64 offset)
 Write a block at the given offset. More...
 
void flush ()
 Flush the file by writing all dirty data and all header info. More...
 
void resync ()
 Resync with another process by clearing the buffers and rereading the header. More...
 
String fileName () const
 Get the file name of the MultiFileBase. More...
 
Bool isWritable () const
 Is the file writable? More...
 
Bool useODirect () const
 Will O_DIRECT be used? More...
 
Int64 blockSize () const
 Get the block size used. More...
 
uInt nfile () const
 Get the nr of virtual files. More...
 
Int64 size () const
 Get the total nr of data blocks used. More...
 
const vector< MultiFileInfo > & info () const
 Get the info object (for test purposes mainly). More...
 
const vector< Int64 > & freeBlocks () const
 Get the free blocks (for test purposes mainly). More...
 

Private Member Functions

virtual void doAddFile (MultiFileInfo &)
 Do the class-specific actions on adding a file. More...
 
virtual void doDeleteFile (MultiFileInfo &)
 Do the class-specific actions on deleting a file. More...
 
virtual void flushFile ()
 Flush the file itself. More...
 
virtual void close ()
 Flush and close the file. More...
 
virtual void writeHeader ()
 Write the header info. More...
 
virtual void readHeader (Bool always=True)
 Read the header info. More...
 
virtual void extend (MultiFileInfo &info, Int64 lastblk)
 Extend the virtual file to fit lastblk. More...
 
virtual void writeBlock (MultiFileInfo &info, Int64 blknr, const void *buffer)
 Write a data block. More...
 
virtual void readBlock (MultiFileInfo &info, Int64 blknr, void *buffer)
 Read a data block. More...
 

Private Attributes

FiledesIO itsIO
 
int itsFD
 

Additional Inherited Members

- Protected Member Functions inherited from casacore::MultiFileBase
void setNewFile ()
 Set the flags and blockSize for a new MultiFile/HDF5. More...
 
- Protected Attributes inherited from casacore::MultiFileBase
String itsName
 
Int64 itsBlockSize
 
Int64 itsNrBlock
 
Int64 itsHdrCounter
 
vector< MultiFileInfoitsInfo
 
std::shared_ptr< MultiFileBufferitsBuffer
 
Bool itsUseODirect
 
Bool itsWritable
 
Bool itsChanged
 
vector< Int64itsFreeBlocks
 

Detailed Description


Class to combine multiple files in a single one.

Intended use:

Public interface

Review Status

Test programs:
tMultiFile

Synopsis

This class is a container file holding multiple virtual files. It is primarily meant as a container file for the storage manager files of a table to reduce the number of files used (especially for Lustre) and to reduce the number of open files (especially when concatenating tables).
A secondary goal is offering the ability to use an IO buffer size that matches the file system well (large buffer size for e.g. ZFS).
A third goal is offering the ability to use O_DIRECT (if supported by OS) to tell the OS kernel to bypass its file cache. It makes the I/O behaviour more predictable which a real-time system might need.

The SetupNewTable constructor has a StorageOption argument to define if a MultiFile has to be used and if so, the buffer size to use. It is also possible to specify that through aipsrc variables.

A virtual file is spread over multiple (fixed size) data blocks in the MultiFile. A data block is never shared by multiple files. For each virtual file MultiFile keeps a MultiFileInfo object telling the file size and the blocks numbers used for the file. When flushing the MultiFile, this meta info is written into the header block. If it does not fit in the header block, the rest is written in a separate "-ext" file. if needed, continuation blocks. On open and resync, it is read back.

A virtual file is represented by an MFFileIO object, which is derived from ByteIO and as such part of the casacore IO framework. It makes it possible for applications to access a virtual file in the same way as a regular file.

It is possible to delete a virtual file. Its blocks will be added to the free block list (which is also stored in the meta info).

Example

In principle it is possible to use the MultiFile functions directly. However, in general it is much easier to use an MFFileIO object per virtual file as shown below.

// Create a new MultiFile using a block size of 1 MB.
MultiFile mfile("file.mf', ByteIO::New, 1048576);
// Create a virtual file in it.
MFFileIO mf1(mfile, "mf1", ByteIO::New);
// Use it (for example) as the sink of AipsIO.
AipsIO stream (&mf1);
// Write values.
stream << (Int)10;
stream << True;
// Seek to beginning of file and read data in.
stream.setpos (0);
Int vali;
Bool valb;
stream >> vali >> valb;
MultiFile(const String &name, ByteIO::OpenOption, Int blockSize=0, Bool useODirect=False)
Open or create a MultiFile with the given name.

To Do

Definition at line 110 of file MultiFile.h.

Constructor & Destructor Documentation

◆ MultiFile()

casacore::MultiFile::MultiFile ( const String name,
ByteIO::OpenOption  ,
Int  blockSize = 0,
Bool  useODirect = False 
)

Open or create a MultiFile with the given name.

Upon creation the block size can be given. If 0, it uses the block size of the file system the file is on. If useODirect=True, the O_DIRECT flag in used (if supported). It tells the kernel to bypass its file cache to have more predictable I/O behaviour.

◆ ~MultiFile()

virtual casacore::MultiFile::~MultiFile ( )
virtual

The destructor flushes and closes the file.

Member Function Documentation

◆ close()

virtual void casacore::MultiFile::close ( )
privatevirtual

Flush and close the file.

Implements casacore::MultiFileBase.

◆ doAddFile()

virtual void casacore::MultiFile::doAddFile ( MultiFileInfo )
privatevirtual

Do the class-specific actions on adding a file.

Implements casacore::MultiFileBase.

◆ doDeleteFile()

virtual void casacore::MultiFile::doDeleteFile ( MultiFileInfo )
privatevirtual

Do the class-specific actions on deleting a file.

Implements casacore::MultiFileBase.

◆ extend()

virtual void casacore::MultiFile::extend ( MultiFileInfo info,
Int64  lastblk 
)
privatevirtual

Extend the virtual file to fit lastblk.

Implements casacore::MultiFileBase.

◆ flushFile()

virtual void casacore::MultiFile::flushFile ( )
privatevirtual

Flush the file itself.

Implements casacore::MultiFileBase.

◆ fsync()

virtual void casacore::MultiFile::fsync ( )
virtual

Fsync the file (i.e., force the data to be physically written).

Implements casacore::MultiFileBase.

◆ readBlock()

virtual void casacore::MultiFile::readBlock ( MultiFileInfo info,
Int64  blknr,
void *  buffer 
)
privatevirtual

Read a data block.

Implements casacore::MultiFileBase.

◆ readHeader()

virtual void casacore::MultiFile::readHeader ( Bool  always = True)
privatevirtual

Read the header info.

If always==False, the info is only read if the header counter has changed.

Implements casacore::MultiFileBase.

◆ reopenRW()

virtual void casacore::MultiFile::reopenRW ( )
virtual

Reopen the underlying file for read/write access.

Nothing will be done if the file is writable already. Otherwise it will be reopened and an exception will be thrown if it is not possible to reopen it for read/write access.

Implements casacore::MultiFileBase.

◆ writeBlock()

virtual void casacore::MultiFile::writeBlock ( MultiFileInfo info,
Int64  blknr,
const void *  buffer 
)
privatevirtual

Write a data block.

Implements casacore::MultiFileBase.

◆ writeHeader()

virtual void casacore::MultiFile::writeHeader ( )
privatevirtual

Write the header info.

Implements casacore::MultiFileBase.

Member Data Documentation

◆ itsFD

int casacore::MultiFile::itsFD
private

Definition at line 159 of file MultiFile.h.

◆ itsIO

FiledesIO casacore::MultiFile::itsIO
private

Definition at line 158 of file MultiFile.h.


The documentation for this class was generated from the following file: