\chapter{Data file details}
\label{app-datafile}

\section{Basic native format}
\label{native}

In gretl's basic native data format--for which we use the suffix
\texttt{gdt}---a data set is stored in XML (extensible mark-up
language). Data files correspond to the simple DTD (document type
definition) given in \verb+gretldata.dtd+, which is supplied with the
gretl distribution and is installed in the system data directory
(e.g.\ \url{/usr/share/gretl/data} on Linux.)  Such files may be plain
text (uncompressed) or gzipped.  They contain the actual data values
plus additional information such as the names and descriptions of
variables, the frequency of the data, and so on.

In a \texttt{gdt} file the actual data values are written to 17
significant figures (for generated data such as logs or pseudo-random
numbers) or to a maximum of 15 figures for primary data. The C
\texttt{printf} format ``\verb|%.*g|'' is used (for \texttt{*} = 17 or
15) so that trailing zeros are not printed.

Most users will probably not have need to read or write such files
other than via gretl itself, but if you want to manipulate them
using other software tools you should examine the DTD and also take a
look at a few of the supplied practice data files: \verb+data4-1.gdt+
gives a simple example; \verb+data4-10.gdt+ is an example where
observation labels are included.

\section{Binary data file format}
\label{bindata}

As of gretl 1.9.15, an alternative, binary format is available for
data storage. Files of this sort have suffix \texttt{gdtb}, and they
take the form of a \app{PKZIP} archive containing two files,
\texttt{data.xml} and \texttt{data.bin}, with the following
characteristics.
\begin{itemize}
\item \texttt{data.xml} is an XML file conforming to the gretldata DTD
  mentioned above, holding all the metadata.
\item \texttt{data.bin} starts with a header composed of one of the
  strings

  \texttt{gretl-bin:little-endian} \,\,or \\
  \texttt{gretl-bin:big-endian}

  padded to 24 bytes with nul characters.  This is followed by a
  binary dump of the data series, by variable, as double-precision
  floating-point values.
\end{itemize}

Binary values are saved in the endianness of the machine on which
they're written; the header information enables gretl to convert to
the endianness of the host on which the data are read if need be.

The rationale for introducing the binary \texttt{gdtb} format is that
for very large datasets it is a lot faster to write and read data in
this form rather than as text. For small to moderately sized datasets
(say, up to 10 megabytes or so) there is no appreciable advantage in the
binary format and we recommend use of plain \texttt{gdt}. 

Some illustrative timings are shown in Table~\ref{tab:dataspeed};
these were obtained on a Lenovo ThinkPad X1 Carbon running gretl
1.9.15.  The datasets contained a mixture of random normal series and
random binary (dummy) series. The largest comprised 50000 observations
on 1000 series and the smallest 5000 observations on 250 series.  As
can be seen, there is a big time saving from the binary format when
writing (and to a lesser extent, when reading) a dataset in the
hundreds of megabytes range. At a size of around 10 megabytes,
however, \texttt{gdt} files can be both written and read in under a
second, surely fast enough for most purposes.

\begin{table}[htbp]
  \centering
  \begin{tabular}{lrr@{\hskip 2em}rr@{\hskip 2em}rr@{\hskip 2em}rr}
    & \multicolumn{2}{c}{381\,MB}
    & \multicolumn{2}{c}{38\,MB}
    & \multicolumn{2}{c}{19\,MB}
    & \multicolumn{2}{c}{10\,MB} \\
    format & write & read & write & read & write & read & write & read\\
    \texttt{gdt} & 
    21.76 & 4.91 & 2.20 & 0.48 & 1.19 & 0.32 & 0.59 & 0.16\\
    \texttt{gdtb} & 
    3.82 & 1.10 & 0.39 & 0.11 & 0.25 & 0.06 & 0.13 & 0.03
  \end{tabular}
  \caption{Data write and read timings in seconds for datasets
    of various sizes in megabytes. The MB numbers represent the 
    size of the datasets in memory; files of both formats are
    substantially smaller when compression is applied.}
  \label{tab:dataspeed}
\end{table}


\section{Native database format}
\label{dbdetails}

A gretl database consists of two parts: an ASCII index file (with
filename suffix \texttt{.idx}) containing information on the series,
and a binary file (suffix \texttt{.bin}) containing the actual data.
Two examples of the format for an entry in the \texttt{idx} file are
shown below:

\begin{code}
G0M910  Composite index of 11 leading indicators (1987=100) 
M 1948.01 - 1995.11  n = 575
currbal Balance of Payments: Balance on Current Account; SA 
Q 1960.1 - 1999.4 n = 160
\end{code}

The first field is the series name.  The second is a description of
the series (maximum 128 characters).  On the second line the first
field is a frequency code: \texttt{M} for monthly, \texttt{Q} for
quarterly, \texttt{A} for annual, \texttt{B} for business-daily (daily
with five days per week), \texttt{D} for 7-day daily, \texttt{S} for
6-day daily, \texttt{U} for undated.

No other frequencies are accepted at present.  Then comes the starting
date (N.B. with two digits following the point for monthly data, one
for quarterly data, none for annual), a space, a hyphen, another
space, the ending date, the string ``\verb+n = +'' and the integer
number of observations. In the case of daily data the starting and
ending dates should be given in the form \verb+YYYY-MM-DD+. This
format must be respected exactly.

Optionally, the first line of the index file may contain a short
comment (up to 64 characters) on the source and nature of the data,
following a hash mark.  For example:

\begin{code}
# Federal Reserve Board (interest rates)
\end{code}

The corresponding binary database file holds the data values,
represented as ``floats'', that is, single-precision floating-point
numbers, typically taking four bytes apiece.  The numbers are packed
``by variable'', so that the first \emph{n} numbers are the
observations of variable 1, the next \emph{m} the observations on
variable 2, and so on.

\chapter{Building gretl}
\label{app-build}

Here we give instructions detailed enough to allow a user
with only a basic knowledge of a Unix-type system to build gretl.
These steps were tested on a fresh installation of Debian Etch. For
other Linux distributions (especially Debian-based ones, like Ubuntu
and its derivatives) little should change. Other Unix-like operating
systems such as Mac OS X and BSD would probably require more substantial
adjustments.

In this guided example, we will build gretl complete with
documentation.  This introduces a few more requirements, but gives you
the ability to modify the documentation files as well, like the help
files or the manuals.

\section{Installing the prerequisites}

We assume that the basic GNU utilities are already installed on the
system, together with these other programs:
\begin{itemize}
\item some \TeX/\LaTeX system (\texttt{texlive} will do beautifully)
\item Gnuplot
\item ImageMagick
\end{itemize}
We also assume that the user has administrative privileges and knows
how to install packages.  The examples below are carried out using the
\texttt{apt-get} shell command, but they can be performed with
menu-based utilities like \texttt{aptitude}, \texttt{dselect} or the
GUI-based program \texttt{synaptic}. Users of Linux distributions
which employ rpm packages (e.g.\ Red Hat/Fedora, Mandriva, SuSE) may
want to refer to the
\href{http://gretl.sourceforge.net/depend.html}{dependencies} page on
the gretl website.

The first step is installing the C compiler and related basic
utilities, if these are not already in place. On a Debian (or
derivative) system, these are contained in a bunch of packages that
can be installed via the command
\begin{code}
apt-get install gcc autoconf automake1.9 libtool flex bison gcc-doc \
libc6-dev libc-dev gfortran gettext pkgconfig
\end{code}

Then it is necessary to install the ``development'' (\texttt{dev})
packages for the libraries that gretl uses: 
\begin{center}
  \begin{tabular}{ll}
    \textit{Library} & \textit{command} \\ [4pt]
    GLIB     & \texttt{apt-get install libglib2.0-dev} \\
    GTK 3.0  & \texttt{apt-get install libgtk3.0-dev} \\
    PNG      & \texttt{apt-get install libpng12-dev} \\
    XSLT     & \texttt{apt-get install libxslt1-dev} \\
    LAPACK   & \texttt{apt-get install liblapack-dev} \\
    FFTW     & \texttt{apt-get install libfftw3-dev} \\
    READLINE & \texttt{apt-get install libreadline-dev} \\
    ZLIB     & \texttt{apt-get install zlib1g-dev} \\
    XML      & \texttt{apt-get install libxml2-dev} \\
    GMP      & \texttt{apt-get install libgmp-dev} \\
    CURL     & \texttt{apt-get install libcurl4-gnutls-dev} \\
    MPFR     & \texttt{apt-get install libmpfr-dev}
  \end{tabular}
\end{center}

It is possible to substitute GTK 2.0 for GTK 3.0.  The \texttt{dev}
packages for these libraries are necessary to \emph{compile}
gretl---you'll also need the plain, non-\texttt{dev} library packages
to \emph{run} gretl, but most of these should already be part of a
standard installation.  In order to enable other optional features,
like audio support, you may need to install more libraries.

\tip{The above steps can be much simplified on Linux systems
that provide deb-based package managers, such as Debian and its
derivatives (Ubuntu, Knoppix and other distributions). The command

\texttt{apt-get build-dep gretl}

will download and install all the necessary packages for building the
version of gretl that is currently present in your APT
sources. Technically, this does not guarantee that all the software
necessary to build the git version is included, because the version of
gretl on your repository may be quite old and build requirements
may have changed in the meantime. However, the chances of a mismatch
are rather remote for a reasonably up-to-date system, so in most cases
the above command should take care of everything correctly.}

\section{Getting the source: release or git}

At this point, it is possible to build from the source.  You have two
options here: obtain the latest released source package, or retrieve
the current git version of gretl (git = the version control software
currently in use for gretl).  The usual caveat applies to the git
version, namely, that it may not build correctly and may contain
``experimental'' code; on the other hand, git often contains bug-fixes
relative to the released version.  If you want to help with testing
and to contribute bug reports, we recommend using git gretl.

To work with the released source:
\begin{enumerate}
\item Download the gretl source package from
  \href{http://gretl.sourceforge.net/}{gretl.sourceforge.net}.
\item Unzip and untar the package.  On a system with the GNU utilities
  available, the command would be \cmd{tar xvfJ gretl-N.tar.xz}
  (replace \cmd{N} with the specific version number of the file you
  downloaded at step 1).
\item Change directory to the gretl source directory created at step 2
  (e.g.\ \verb+gretl-2020a+).
\item Proceed to the next section, ``Configure and make''.
\end{enumerate}

To work with git you'll first need to install the \app{git} client
program if it's not already on your system.  Relevant resources you
may wish to consult include the main git website at
\href{https://git-scm.com/}{git-scm.com} and instructions specific to
gretl:
\href{http://gretl.sourceforge.net/gretl-git-basics.html}{gretl
  git basics}.

When grabbing the git sources \textit{for the first time}, you should
first decide where you want to store the code.  For example, you might
create a directory called \texttt{git} under your home directory.
Open a terminal window, \texttt{cd} into this directory, and type
the following commands:
%
\begin{code}
git clone git://git.code.sf.net/p/gretl/git gretl-git
\end{code}
%
At this point \app{git} should create a subdirectory named
\texttt{gretl-git} and fill it with the current sources.

When you want to \textit{update the source}, this is very simple: just
move into the \texttt{gretl-git} directory and type
\begin{code}
git pull
\end{code}

Assuming you're now in the \texttt{gretl-git} directory, you can
proceed in the same manner as with the released source package.


\section{Configure the source}
          
The next command you need is \texttt{./configure}; this is a complex
script that detects which tools you have on your system and sets
things up. The \texttt{configure} command accepts many
options; you may want to run 
\begin{code}
./configure --help
\end{code}
first to see what options are available. One option you way wish to
tweak is \cmd{--prefix}.  By default the installation goes under
\verb+/usr/local+ but you can change this.  For example
\begin{code}
./configure --prefix=/usr
\end{code}
will put everything under the \verb+/usr+ tree.

Note that the recommended location to build gretl is \emph{not} in
the source directory. The way to achieve that is quite simple: the
invocation of the configure script has to take into account the
relative path to the source tree. So if your build directory is
inside (underneath) the source tree it is
\begin{code}
../configure 
\end{code}
while if it is in a parallel tree it would be (something like):
\begin{code}
../gretl-git/configure   
\end{code}

If you have a multi-core machine you may want to activate support
for OpenMP, which permits the parallelization of matrix
multiplication and some other tasks. This requires adding the
\texttt{configure} flag
\begin{code}
--enable-openmp
\end{code}

By default the gretl GUI is built using version 3.0 of the GTK
library, if available, otherwise version 2.0. If you have both
versions installed and prefer to use GTK 2.0, use the flag
\begin{code}
--enable-gtk2
\end{code}

In order to have the documentation built, we need to pass the relevant
option to \texttt{configure}, as in
\begin{code}
--enable-build-doc
\end{code}
But please note that this option will work only if you are using
the git source.


\tip{In order to build the documentation, there is the possibility
  that you will have to install some extra software on top of the
  packages mentioned in the previous section. For example, you may
  need some extra \LaTeX\ packages to compile the manuals. Two of the
  required packages, that not every standard \LaTeX\ installation
  include, are typically \app{pifont.sty} and \app{appendix.sty}. You
  could install the corresponding packages from your distribution or
  you could simply download them from CTAN and install them by hand.}


This, for example, if you want to install under \texttt{/usr}, with
OpenMP support, and also build the documentation, you would do
\begin{code}
./configure --prefix=/usr \
 --enable-openmp \
 --enable-build-doc
\end{code}

You will see a number of checks being run, and if everything goes
according to plan, you should see a summary similar to that displayed
in Listing~\ref{configure-output}.

\begin{script}[htbp]
  \caption{Sample output from \texttt{./configure}}
  \label{configure-output}
\begin{scode}
Configuration:

  Installation path:                      /usr
  Use readline library:                   yes
  Use gnuplot for graphs:                 yes
  Use LaTeX for typesetting output:       yes
  Use libgsf for zip/unzip:               no
  sse2 support for RNG:                   yes
  OpenMP support:                         yes
  MPI support:                            no
  AVX support for arithmetic:             no
  Build with GTK version:                 2.0
  Build gretl documentation:              yes
  Use Lucida fonts:                       no
  Build message catalogs:                 yes
  X-12-ARIMA support:                     yes
  TRAMO/SEATS support:                    yes
  libR support:                           yes
  ODBC support:                           no
  Experimental audio support:             no
  Use xdg-utils in installation:          if DESTDIR not set
  LAPACK libraries:
    -llapack -lblas -lgfortran

Now type 'make' to build gretl.
You can also do 'make pdfdocs' to build the PDF documentation.
\end{scode}
\end{script}

\tip{If you're using git, it's a good idea to re-run the
  \texttt{configure} script after doing an update.  This is not always
  necessary, but sometimes it is, and it never does any harm.  For
  this purpose, you may want to write a little shell script that calls
  \texttt{configure} with any options you want to use.}


\section{Build and install}

We are now ready to undertake the compilation proper: this is done by
running the \texttt{make} command, which takes care of compiling all
the necessary source files in the correct order. All you need to do is
type
\begin{code}
make 
\end{code}

This step will likely take several minutes to complete; a lot of
output will be produced on screen. Once this is done, you can install
your freshly baked copy of gretl on your system via
\begin{code}
make install
\end{code}

On most systems, the \texttt{make install} command requires you to
have administrative privileges.  Hence, either you log in as
\texttt{root} before launching \texttt{make install} or you may want
to use the \texttt{sudo} utility, as in:
\begin{code}
sudo make install
\end{code}

Now try if everything works: go back to your home directory and run gretl
\begin{code}
cd ~
gretl &
\end{code}

If all is well, you ought to see gretl start, at which point just exit
the program in the usual way. On the other hand, there is the
possibility that gretl doesn't start and instead you see a message
like

\begin{code}
   /usr/local/bin/gretl_x11: error while loading shared libraries:
   libgretl-1.0.so.0: cannot open shared object file: No such file or directory
\end{code}

In this case, just run
\begin{code}
  sudo ldconfig
\end{code}
The problem should be fixed once and for all.

\chapter{Numerical accuracy}
\label{app-accuracy}

Gretl uses double-precision arithmetic throughout---except for
the multiple-precision plugin invoked by the menu item ``Model, Other
linear models, High precision OLS'' which represents floating-point
values using a number of bits given by the environment variable
\verb+GRETL_MP_BITS+ (default value 256).  

The normal equations of Least Squares are by default solved via
Cholesky decomposition, which is highly accurate provided the matrix
of cross-products of the regressors, $X'X$, is not very ill
conditioned.  If this problem is detected, gretl automatically
switches to use QR decomposition.

The program has been tested rather thoroughly on the statistical
reference datasets provided by NIST (the U.S.  National Institute of
Standards and Technology) and a full account of the results may be
found on the gretl website (follow the link ``Numerical accuracy'').

To date, two published reviews have discussed gretl's accuracy:
Giovanni Baiocchi and Walter Distaso \citeyearpar{baiocchi03}, and
Talha Yalta and Yasemin Yalta \citeyearpar{yalta07}.  We are grateful
to these authors for their careful examination of the program.  Their
comments have prompted several modifications including the use of
Stephen Moshier's \app{cephes} code for computing p-values and other
quantities relating to probability distributions (see
\href{http://www.netlib.org/cephes/}{netlib.org}), changes to the
formatting of regression output to ensure that the program displays a
consistent number of significant digits, and attention to compiler
issues in producing the MS Windows version of gretl (which at
one time was slighly less accurate than the Linux version).

Gretl now includes a ``plugin'' that runs the NIST linear
regression test suite.  You can find this under the ``Tools'' menu in
the main window.  When you run this test, the introductory text
explains the expected result.  If you run this test and see anything
other than the expected result, please send a bug report to
\verb+cottrell@wfu.edu+.

All regression statistics are printed to 6 significant figures in the
current version of gretl (except when the multiple-precision
plugin is used, in which case results are given to 12 figures).  If
you want to examine a particular value more closely, first save it
(for example, using the \cmd{genr} command) then print it using
\cmd{printf}, to as many digits as you like (see the \GCR).  

\chapter{Related free software}
\label{app-advanced}

Gretl's capabilities are substantial, and are expanding.
Nonetheless you may find there are some things you can't do in
gretl, or you may wish to compare results with other programs.
If you are looking for complementary functionality in the realm of
free, open-source software we recommend the following programs.  The
self-description of each program is taken from its website.

\begin{itemize}

\item \textbf{GNU R} \href{http://www.r-project.org/}{r-project.org}:
  ``R is a system for statistical computation and graphics. It
  consists of a language plus a run-time environment with graphics, a
  debugger, access to certain system functions, and the ability to run
  programs stored in script files\dots\ It compiles and runs on a wide
  variety of UNIX platforms, Windows and MacOS.''  Comment: There are
  numerous add-on packages for R covering most areas of statistical
  work.

\item \textbf{GNU Octave}
  \href{http://www.octave.org/}{www.octave.org}:
  ``GNU Octave is a high-level language, primarily intended for
  numerical computations. It provides a convenient command line
  interface for solving linear and nonlinear problems numerically, and
  for performing other numerical experiments using a language that is
  mostly compatible with Matlab. It may also be used as a
  batch-oriented language.''

\item \textbf{Julia} \href{http://julialang.org/}{julialang.org}:
  ``Julia is a high-level, high-performance dynamic programming
  language for technical computing, with syntax that is familiar to
  users of other technical computing environments. It provides a
  sophisticated compiler, distributed parallel execution, numerical
  accuracy, and an extensive mathematical function library.''

\item \textbf{JMulTi} \href{http://www.jmulti.de/}{www.jmulti.de}:
  ``JMulTi was originally designed as a tool for certain econometric
  procedures in time series analysis that are especially difficult to
  use and that are not available in other packages, like Impulse
  Response Analysis with bootstrapped confidence intervals for VAR/VEC
  modelling. Now many other features have been integrated as well to
  make it possible to convey a comprehensive analysis.''  Comment:
  JMulTi is a java GUI program: you need a java run-time environment to
  make use of it.

\end{itemize}

As mentioned above, gretl offers the facility of exporting
data in the formats of both Octave and R.  In the case of Octave, the
gretl data set is saved as a single matrix, \verb+X+. You can
pull the \verb+X+ matrix apart if you wish, once the data are loaded
in Octave; see the Octave manual for details.  As for R, the exported
data file preserves any time series structure that is apparent to
gretl.  The series are saved as individual structures. The data
should be brought into R using the \cmd{source()} command.
  
In addition, gretl has a convenience function for moving data
quickly into R.  Under gretl's ``Tools'' menu, you will find the
entry ``Start GNU R''.  This writes out an R version of the current
gretl data set (in the user's gretl directory), and sources it
into a new R session.  The particular way R is invoked depends on the
internal gretl variable \verb+Rcommand+, whose value may be set
under the ``Tools, Preferences'' menu.  The default command is
\cmd{RGui.exe} under MS Windows. Under X it is \cmd{xterm -e R}.
Please note that at most three space-separated elements in this
command string will be processed; any extra elements are ignored.

\chapter{Listing of URLs}
\label{app-urls}

Below is a listing of the full URLs of websites mentioned in the text.

\begin{description}

\item[Estima (RATS)] \url{http://www.estima.com/}
\item[FFTW3] \url{http://www.fftw.org/}
\item[Gnome desktop homepage] \url{http://www.gnome.org/}
\item[GNU Multiple Precision (GMP) library]
  \url{http://gmplib.org/}
\item[CURL library]
  \url{http://curl.haxx.se/libcurl/}
\item[GNU Octave homepage] \url{http://www.octave.org/}
\item[GNU R homepage] \url{http://www.r-project.org/}
\item[GNU R manual]
  \url{http://cran.r-project.org/doc/manuals/R-intro.pdf}
\item[Gnuplot homepage] \url{http://www.gnuplot.info/}
\item[Gretl data page]
  \url{http://gretl.sourceforge.net/gretl_data.html}
\item[Gretl homepage] \url{http://gretl.sourceforge.net/}
\item[GTK+ homepage] \url{http://www.gtk.org/}
\item[GTK+ port for win32]
  \url{https://wiki.gnome.org/Projects/GTK/Win32/}
\item[InfoZip homepage]
  \url{http://www.info-zip.org/pub/infozip/zlib/}
\item[JMulTi homepage] \url{http://www.jmulti.de/}
\item[JRSoftware] \url{http://www.jrsoftware.org/}
\item[Julia homepage] \url{http://julialang.org/}
\item[Mingw (gcc for win32) homepage] \url{http://www.mingw.org/}
\item[Minpack] \url{http://www.netlib.org/minpack/}
\item[Penn World Table] \url{http://pwt.econ.upenn.edu/}
\item[Readline homepage]
  \url{http://cnswww.cns.cwru.edu/~chet/readline/rltop.html}
\item[Readline manual]
  \url{http://cnswww.cns.cwru.edu/~chet/readline/readline.html}
\item[Xmlsoft homepage] \url{http://xmlsoft.org/}

\end{description}


%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "gretl-guide"
%%% End: 

