\chapter{Data files}
\label{chap:datafiles}

\section{Data file formats}
\label{sec:data-formats}

Gretl has its own native format for data files.  Most users will
probably not want to read or write such files outside of gretl itself,
but occasionally this may be useful and details on the file formats
are given in Appendix~\ref{app-datafile}. The program can also import
data from a variety of other formats. In the GUI program this can be
done via the ``File, Open Data, User file'' menu---note the drop-down
list of acceptable file types. In script mode, simply use the
\cmd{open} command. The supported import formats are as follows.

\begin{itemize}
\item Plain text files (comma-separated or ``CSV'' being the most
  common type).  For details on what gretl expects of such files, see
  Section~\ref{scratch}.
\item Spreadsheets: MS \app{Excel}, \app{Gnumeric} and Open Document
  (ODS). The requirements for such files are given in
  Section~\ref{scratch}.
\item \app{Stata} data files (\texttt{.dta}).
\item \app{SPSS} data files (\texttt{.sav}).
\item \app{SAS} ``xport'' files (\texttt{.xpt}).
\item \app{Eviews} workfiles (\texttt{.wf1}).\footnote{See
    \url{http://users.wfu.edu/cottrell/eviews_format/}.}
\item \app{JMulTi} data files.
\end{itemize}

When you import data from a plain text format, gretl opens a
``diagnostic'' window, reporting on its progress in reading the data.
If you encounter a problem with ill-formatted data, the messages in
this window should give you a handle on fixing the problem.

Note that gretl has a facility for writing out data in the
native formats of GNU \app{R}, \app{Octave}, \app{JMulTi} and
\app{PcGive} (see Appendix~\ref{app-advanced}).  In the GUI client
this option is found under the ``File, Export data'' menu; in the
command-line client use the \cmd{store} command with the appropriate
option flag.

\section{Databases}
\label{dbase}

For working with large amounts of data gretl is supplied with a
database-handling routine.  A \emph{database}, as opposed to a
\emph{data file}, is not read directly into the program's workspace.
A database can contain series of mixed frequencies and sample ranges.
You open the database and select series to import into the working
dataset.  You can then save those series in a native format data file
if you wish. Databases can be accessed via the menu item ``File,
Databases''.

For details on the format of gretl databases, see
Appendix~\ref{app-datafile}.

\subsection{Online access to databases}
\label{online-data}

Several gretl databases are available from Wake Forest University.
Your computer must be connected to the internet for this option to
work.  Please see the description of the ``data'' command under
the Help menu.

\tip{Visit the gretl
  \href{http://gretl.sourceforge.net/gretl_data.html}{data page} for
  details and updates on available data.}


\subsection{Foreign database formats}
\label{RATS}

Thanks to Thomas Doan of \emph{Estima}, who made available the
specification of the database format used by RATS 4 (Regression
Analysis of Time Series), gretl can handle such databases---or at
least, a subset of same, namely time-series databases containing
monthly and quarterly series.

Gretl can also import data from \app{PcGive} databases.  These
take the form of a pair of files, one containing the actual data (with
suffix \texttt{.bn7}) and one containing supplementary information
(\texttt{.in7}).  

In addition, gretl offers ODBC connectivity. Be warned: this feature
is meant for somewhat advanced users; there is currently no graphical
interface.  Interested readers will find more info in appendix
\ref{chap:odbc}.

\section{Creating a dataset from scratch}
\label{scratch}

There are several ways of doing this:

\begin{enumerate}
\item Find, or create using a text editor, a plain text data file and
  open it via ``Import''.
\item Use your favorite spreadsheet to establish the data file, save
  it in comma-separated format if necessary (this may not be
  necessary if the spreadsheet format is MS Excel, Gnumeric or Open
  Document), then use one of the ``Import'' options.
\item Use gretl's built-in spreadsheet.
\item Select data series from a suitable database.
\item Use your favorite text editor or other software tools to a
  create data file in gretl format independently.
\end{enumerate}

Here are a few comments and details on these methods.

\subsection{Common points on imported data}

Options (1) and (2) involve using gretl's ``import'' mechanism.
For the program to read such data successfully, certain general
conditions must be satisfied:

\begin{itemize}

\item The first row must contain valid variable names.  A valid
  variable name is of 31 characters maximum; starts with a letter; and
  contains nothing but letters, numbers and the underscore character,
  \verb+_+.  (Longer variable names will be truncated to 31
  characters.)  Qualifications to the above: First, in the case of an
  plain text import, if the file contains no row with variable names
  the program will automatically add names, \verb+v1+, \verb+v2+ and
  so on.  Second, by ``the first row'' is meant the first
  \emph{relevant} row.  In the case of plain text imports, blank
  rows and rows beginning with a hash mark, \verb+#+, are ignored.  In
  the case of Excel, Gnumeric and ODS imports, you are presented with a
  dialog box where you can select an offset into the spreadsheet, so
  that gretl will ignore a specified number of rows and/or
  columns.
          
\item Data values: these should constitute a rectangular block, with
  one variable per column (and one observation per row).  The number
  of variables (data columns) must match the number of variable names
  given. See also section~\ref{missing-data}.  Numeric data are
  expected, but in the case of importing from plain text, the program
  offers limited handling of character (string) data: if a given
  column contains character data only, consecutive numeric codes are
  substituted for the strings, and once the import is complete a table
  is printed showing the correspondence between the strings and the
  codes.
          
\item Dates (or observation labels): Optionally, the \emph{first}
  column may contain strings such as dates, or labels for
  cross-sectional observations.  Such strings have a maximum of 15
  characters (as with variable names, longer strings will be
  truncated).  A column of this sort should be headed with the string
  \verb+obs+ or \verb+date+, or the first row entry may be left
  blank.

  For dates to be recognized as such, the date strings should adhere
  to one or other of a set of specific formats, as follows.  For
  \emph{annual} data: 4-digit years.  For \emph{quarterly} data: a
  4-digit year, followed by a separator (either a period, a colon, or
  the letter \verb+Q+), followed by a 1-digit quarter.  Examples:
  \verb+1997.1+, \verb+2002:3+, \verb+1947Q1+.  For \emph{monthly}
  data: a 4-digit year, followed by a period or a colon, followed by a
  two-digit month.  Examples: \verb+1997.01+, \verb+2002:10+.
          
\end{itemize}

Plain text (``CSV'') files can use comma, space, tab or semicolon as
the column separator.  When you open such a file via the GUI you are
given the option of specifying the separator, though in most cases it
should be detected automatically.

If you use a spreadsheet to prepare your data you are able to carry
out various transformations of the ``raw'' data with ease (adding
things up, taking percentages or whatever): note, however, that you
can also do this sort of thing easily---perhaps more easily---within
gretl, by using the tools under the ``Add'' menu.

\subsection{Appending imported data}

You may wish to establish a dataset piece by piece, by incremental
importation of data from other sources.  This is supported via the
``File, Append data'' menu items: gretl will check the new data for
conformability with the existing dataset and, if everything seems OK,
will merge the data.  You can add new variables in this way, provided
the data frequency matches that of the existing dataset.  Or you can
append new observations for data series that are already present; in
this case the variable names must match up correctly.  Note that by
default (that is, if you choose ``Open data'' rather than ``Append
data''), opening a new data file closes the current one.

\subsection{Using the built-in spreadsheet}

Under the ``File, New data set'' menu you can choose the sort of
dataset you want to establish (e.g.\ quarterly time series,
cross-sectional).  You will then be prompted for starting and ending
dates (or observation numbers) and the name of the first variable to
add to the dataset. After supplying this information you will be faced
with a simple spreadsheet into which you can type data values.  In the
spreadsheet window, clicking the right mouse button will invoke a
popup menu which enables you to add a new variable (column), to add an
observation (append a row at the foot of the sheet), or to insert an
observation at the selected point (move the data down and insert a
blank row.)

Once you have entered data into the spreadsheet you import these into
gretl's workspace using the spreadsheet's ``Apply changes''
button.

Please note that gretl's spreadsheet is quite basic and has no
support for functions or formulas.  Data transformations are done via
the ``Add'' or ``Variable'' menus in the main window.

\subsection{Selecting from a database}

Another alternative is to establish your dataset by selecting
variables from a database.  

Begin with the ``File, Databases'' menu item. This has four forks:
``Gretl native'', ``RATS 4'', ``PcGive'' and ``On database server''.
You should be able to find the file \verb+fedstl.bin+ in the file
selector that opens if you choose the ``Gretl native'' option since
this file, which contains a large collection of US macroeconomic time
series, is supplied with the distribution.

You won't find anything under ``RATS 4'' unless you have purchased
RATS data.\footnote{See \href{http://www.estima.com/}{www.estima.com}}
If you do possess RATS data you should go into the ``Tools,
Preferences, General'' dialog, select the Databases tab, and fill in
the correct path to your RATS files.  

If your computer is connected to the internet you should find several
databases (at Wake Forest University) under ``On database server''.
You can browse these remotely; you also have the option of installing
them onto your own computer.  The initial remote databases window has
an item showing, for each file, whether it is already installed
locally (and if so, if the local version is up to date with the
version at Wake Forest).

Assuming you have managed to open a database you can import selected
series into gretl's workspace by using the ``Series, Import''
menu item in the database window, or via the popup menu that appears
if you click the right mouse button, or by dragging the series into
the program's main window.

\subsection{Creating a gretl data file independently}

It is possible to create a data file in one or other of gretl's own
formats using a text editor or software tools such as \app{awk},
\app{sed} or \app{perl}.  This may be a good choice if you have large
amounts of data already in machine readable form. You will, of course,
need to study these data formats (XML-based or ``traditional'') as
described in Appendix~\ref{app-datafile}.

\section{Structuring a dataset}
\label{sec:data-structure}

Once your data are read by gretl, it may be necessary to supply
some information on the nature of the data. We distinguish between
three kinds of datasets:
\begin{enumerate}
\item Cross section
\item Time series
\item Panel data
\end{enumerate}

The primary tool for doing this is the ``Data, Dataset structure''
menu entry in the graphical interface, or the \texttt{setobs} command
for scripts and the command-line interface.

\subsection{Cross sectional data}
\label{sec:cross-section-data}

By a cross section we mean observations on a set of ``units'' (which
may be firms, countries, individuals, or whatever) at a common point
in time.  This is the default interpretation for a data file: if there
is insufficient information to interpret data as time-series or panel
data, they are automatically interpreted as a cross section.  In the
unlikely event that cross-sectional data are wrongly interpreted as
time series, you can correct this by selecting the ``Data, Dataset
structure'' menu item.  Click the ``cross-sectional'' radio button in
the dialog box that appears, then click ``Forward''.  Click ``OK'' to
confirm your selection.

\subsection{Time series data}
\label{sec:timeser-data}

When you import data from a spreadsheet or plain text file,
gretl will make fairly strenuous efforts to glean time-series
information from the first column of the data, if it looks at all
plausible that such information may be present.  If time-series
structure is present but not recognized, again you can use the ``Data,
Dataset structure'' menu item.  Select ``Time series'' and click
``Forward''; select the appropriate data frequency and click
``Forward'' again; then select or enter the starting observation and
click ``Forward'' once more.  Finally, click ``OK'' to confirm the
time-series interpretation if it is correct (or click ``Back'' to make
adjustments if need be).

Besides the basic business of getting a data set interpreted as time
series, further issues may arise relating to the frequency of
time-series data.  In a gretl time-series data set, all the series
must have the same frequency.  Suppose you wish to make a combined
dataset using series that, in their original state, are not all of the
same frequency.  For example, some series are monthly and some are
quarterly.

Your first step is to formulate a strategy: Do you want to end up with
a quarterly or a monthly data set?  A basic point to note here is
that ``compacting'' data from a higher frequency (e.g.\ monthly) to
a lower frequency (e.g.\ quarterly) is usually unproblematic.  You
lose information in doing so, but in general it is perfectly
legitimate to take (say) the average of three monthly observations to
create a quarterly observation.  On the other hand, ``expanding'' data
from a lower to a higher frequency is not, in general, a valid
operation.  

In most cases, then, the best strategy is to start by creating a data
set of the \textit{lower} frequency, and then to compact the higher
frequency data to match.  When you import higher-frequency data from a
database into the current data set, you are given a choice of
compaction method (average, sum, start of period, or end of period).
In most instances ``average'' is likely to be appropriate.  

You \textit{can} also import lower-frequency data into a
high-frequency data set, but this is generally not recommended.  What
gretl does in this case is simply replicate the values of the
lower-frequency series as many times as required. For example, suppose
we have a quarterly series with the value 35.5 in 1990:1, the first
quarter of 1990.  On expansion to monthly, the value 35.5 will be
assigned to the observations for January, February and March of 1990.
The expanded variable is therefore useless for fine-grained
time-series analysis, outside of the special case where you know that
the variable in question does in fact remain constant over the
sub-periods.

When the current data frequency is appropriate, gretl offers
both ``Compact data'' and ``Expand data'' options under the ``Data''
menu.  These options operate on the whole data set, compacting or
exanding all series.  They should be considered ``expert'' options
and should be used with caution. 


\subsection{Panel data}
\label{sec:panel-data}

Panel data are inherently three dimensional---the dimensions being
variable, cross-sectional unit, and time-period.  For example, a
particular number in a panel data set might be identified as the
observation on capital stock for General Motors in 1980.  (A note on
terminology: we use the terms ``cross-sectional unit'', ``unit'' and
``group'' interchangeably below to refer to the entities that compose
the cross-sectional dimension of the panel.  These might, for
instance, be firms, countries or persons.)

For representation in a textual computer file (and also for gretl's
internal calculations) the three dimensions must somehow be flattened
into two.  This ``flattening'' involves taking layers of the data that
would naturally stack in a third dimension, and stacking them in the
vertical dimension.

gretl always expects data to be arranged ``by observation'',
that is, such that each row represents an observation (and each
variable occupies one and only one column).  In this context the
flattening of a panel data set can be done in either of two ways:

\begin{itemize}
\item Stacked time series: the successive vertical blocks each
  comprise a time series for a given unit.
\item Stacked cross sections: the successive vertical blocks each
  comprise a cross-section for a given period.
\end{itemize}

You may input data in whichever arrangement is more convenient.
Internally, however, gretl always stores panel data in
the form of stacked time series.

\section{Panel data specifics}
\label{sec:more-panel}

When you import panel data into gretl from a spreadsheet or
comma separated format, the panel nature of the data will not be
recognized automatically (most likely the data will be treated as
``undated'').  A panel interpretation can be imposed on the data
using the graphical interface or via the \cmd{setobs} command.

In the graphical interface, use the menu item ``Data, Dataset
structure''.  In the first dialog box that appears, select ``Panel''.
In the next dialog you have a three-way choice.  The first two
options, ``Stacked time series'' and ``Stacked cross sections'' are
applicable if the data set is already organized in one of these two
ways.  If you select either of these options, the next step is to
specify the number of cross-sectional units in the data set.  The
third option, ``Use index variables'', is applicable if the data set
contains two variables that index the units and the time periods
respectively; the next step is then to select those variables.  For
example, a data file might contain a country code variable and a
variable representing the year of the observation.  In that case
gretl can reconstruct the panel structure of the data regardless
of how the observation rows are organized.

The \cmd{setobs} command has options that parallel those in the
graphical interface.  If suitable index variables are available
you can do, for example
%
\begin{code}
setobs unitvar timevar --panel-vars
\end{code}
%
where \texttt{unitvar} is a variable that indexes the units and
\texttt{timevar} is a variable indexing the periods.  Alternatively
you can use the form \verb+setobs+ \textsl{freq} \verb+1:1+
\textsl{structure}, where \textsl{freq} is replaced by the ``block
size'' of the data (that is, the number of periods in the case of
stacked time series, or the number of units in the case of stacked
cross-sections) and structure is either \option{stacked-time-series}
or \option{stacked-cross-section}.  Two examples are given below: the
first is suitable for a panel in the form of stacked time series with
observations from 20 periods; the second for stacked cross sections
with 5 units.
%
\begin{code}
setobs 20 1:1 --stacked-time-series
setobs 5 1:1 --stacked-cross-section
\end{code}

\subsection{Panel data arranged by variable}

Publicly available panel data sometimes come arranged ``by variable.''
Suppose we have data on two variables, \varname{x1} and \varname{x2},
for each of 50 states in each of 5 years (giving a total of 250
observations per variable).  One textual representation of such a data
set would start with a block for \varname{x1}, with 50 rows
corresponding to the states and 5 columns corresponding to the years.
This would be followed, vertically, by a block with the same structure
for variable \varname{x2}.  A fragment of such a data file is shown
below, with quinquennial observations 1965--1985.  Imagine the table
continued for 48 more states, followed by another 50 rows for variable
\varname{x2}.

\begin{center}
  \begin{tabular}{rrrrrr}
  \varname{x1} \\
     & 1965 & 1970 & 1975 & 1980 & 1985 \\
  AR & 100.0 & 110.5 & 118.7 & 131.2 & 160.4\\
  AZ & 100.0 & 104.3 & 113.8 & 120.9 & 140.6\\
  \end{tabular}
\end{center}

If a datafile with this sort of structure is read into
gretl,\footnote{Note that you will have to modify such a
  datafile slightly before it can be read at all.  The line containing
  the variable name (in this example \varname{x1}) will have to be
  removed, and so will the initial row containing the years,
  otherwise they will be taken as numerical data.}  the program
will interpret the columns as distinct variables, so the data will not
be usable ``as is.''  But there is a mechanism for correcting the
situation, namely the \cmd{stack} function.

Consider the first data column in the fragment above: the first 50 rows
of this column constitute a cross-section for the variable \varname{x1}
in the year 1965.  If we could create a new series by stacking the
first 50 entries in the second column underneath the first 50 entries
in the first, we would be on the way to making a data set ``by
observation'' (in the first of the two forms mentioned above, stacked
cross-sections).  That is, we'd have a column comprising a
cross-section for \varname{x1} in 1965, followed by a cross-section for
the same variable in 1970.

The following gretl script illustrates how we can accomplish the
stacking, for both \varname{x1} and \varname{x2}.  We assume
that the original data file is called \texttt{panel.txt}, and that in
this file the columns are headed with ``variable names'' \varname{v1},
\varname{v2}, \dots, \varname{v5}.  (The columns are not really
variables, but in the first instance we ``pretend'' that they are.)

\begin{code}
open panel.txt
series x1 = stack(v1..v5, 50)
series x2 = stack(v1..v5, 50, 50)
setobs 50 1:1 --stacked-cross-section
store panel.gdt x1 x2
\end{code}

The second and third lines illustrate the syntax of the \cmd{stack}
function, which has this signature:
\begin{code}
series stack(list L, scalar length, scalar offset)
\end{code}
\begin{itemize}
\item \texttt{L}: a list of series on which to operate.
\item \texttt{length}: an integer giving the number of observations to
  take from each series.
\item \texttt{offset}: an integer giving the offset from the top of
  the dataset at which to start taking values (optional, defaults to
  0).
\end{itemize}

The ``\texttt{..}'' syntax in the example above constructs a list of
the 5 contiguous series to be stacked.  More generally, you can define
a named list of series and pass that as the first argument to
\texttt{stack} (see chapter~\ref{chap:lists-strings}). In this example
we're supposing that the full data set contains 100 rows, and that in
the stacking of variable \varname{x1} we wish to read only the first
50 rows from each column, so we give \texttt{50} as the second
argument.

On line 3 we do the stacking for variable \varname{x2}.  Again we want
a \texttt{length} of 50 for the components of the stacked series, but
this time we want to start reading from the 50th row of the original
data, and so we add a third \texttt{offset} argument of 50.  Line 4
then imposes a panel interpretation on the data. Finally, we save the
stacked data to file, with the panel interpretation.

The illustrative script above is appropriate when the number of
variables to be processed is small.  When then are many variables in
the dataset it will be more convenient to use a loop to accomplish the
stacking, as shown in the following script.  The setup is presumed to
be the same as in the previous case (50 units, 5 periods), but with 20
variables rather than 2.

\begin{code}
open panel.txt
list L = v1..v5 # predefine a list of series
scalar length = 50
loop i=1..20
  scalar offset = (i - 1) * length
  series x$i = stack(L, length, offset)
endloop
setobs 50 1.01 --stacked-cross-section
store panel.gdt x1..x20
\end{code}

\subsection{Side-by-side time series}

There's a second sort of data that you may wish to convert to gretl's
panel format, namely side-by-side time series for a number of
cross-sectional units. For example, a data file might contain separate
GDP series of common length $T$ for each of $N$ countries. To turn
these into a single stacked time series the \texttt{stack} function
can again be used. An example follows, where we suppose the original
data source is a comma-separated file named \texttt{GDP.csv},
containing GDP data for countries from Austria (\texttt{GDP\_AT}) to
Zimbabwe (\texttt{GDP\_ZW}) in consecutive columns.

\begin{code}
open GDP.csv
scalar T = $nobs # the number of periods
list L = GDP_AT..GDP_ZW
series GDP = stack(L, T)
setobs T 1:01 --stacked-time-series
store panel.gdt GDP
\end{code}

The resulting data file, \texttt{panel.gdt}, will contain a single
series of length $NT$ where $N$ is the number of countries and
$T$ is the length of the original dataset. One could insert revised
variants of lines 3 and 4 of the script if the original file contained
additional side-by-side per-country series for investment, consumption
or whatever.

\subsection{Panel data marker strings}

It can be helpful with panel data to have the observations identified
by mnemonic markers.  A special function in the \texttt{genr} command
is available for this purpose.

In the example under the heading ``Panel data arranged by variable''
above, suppose all the states are identified by two-letter codes in
the left-most column of the original datafile.  When the
\texttt{stack} function is invoked as shown, these codes will be
stacked along with the data values.  If the first row is marked
\texttt{AR} for Arkansas, then the marker \texttt{AR} will end up
being shown on each row containing an observation for Arkansas.
That's all very well, but these markers don't tell us anything about
the date of the observation.  To rectify this we could do:

\begin{code}
genr time
series year = 1960 + (5 * time)
genr markers = "%s:%d", marker, year
\end{code}

The first line generates a 1-based index representing the period of
each observation, and the second line uses the \texttt{time} variable
to generate a variable representing the year of the observation.  The
third line contains this special feature: if (and only if) the name of
the new ``variable'' to generate is \texttt{markers}, the portion of
the command following the equals sign is taken as a C-style format
string (which must be wrapped in double quotes), followed by a
comma-separated list of arguments.  The arguments will be printed
according to the given format to create a new set of observation
markers.  Valid arguments are either the names of variables in the
dataset, or the string \texttt{marker} which denotes the pre-existing
observation marker.  The format specifiers which are likely to be
useful in this context are \texttt{\%s} for a string and \texttt{\%d}
for an integer.  Strings can be truncated: for example \texttt{\%.3s}
will use just the first three characters of the string.  To chop
initial characters off an existing observation marker when
constructing a new one, you can use the syntax \texttt{marker + n},
where \texttt{n} is a positive integer: in the case the first
\texttt{n} characters will be skipped.

After the commands above are processed, then, the observation markers
will look like, for example, \texttt{AR:1965}, where the two-letter
state code and the year of the observation are spliced together with a
colon.

\subsection{Panel dummy variables}
\label{panel-dummies}

In a panel study you may wish to construct dummy variables of one or
both of the following sorts: (a) dummies as unique identifiers for the
units or groups, and (b) dummies as unique identifiers for the time
periods.  The former may be used to allow the intercept of the
regression to differ across the units, the latter to allow the
intercept to differ across periods.

Two special functions are available to create such dummies.  These are
found under the ``Add'' menu in the GUI, or under the \cmd{genr}
command in script mode or \app{gretlcli}.

\begin{enumerate}
\item ``unit dummies'' (script command \cmd{genr unitdum}).  This
  command creates a set of dummy variables identifying the
  cross-sectional units.  The variable \verb+du_1+ will have value 1
  in each row corresponding to a unit 1 observation, 0 otherwise;
  \verb+du_2+ will have value 1 in each row corresponding to a unit 2
  observation, 0 otherwise; and so on.
\item ``time dummies'' (script command \cmd{genr timedum}).  This
  command creates a set of dummy variables identifying the periods.
  The variable \verb+dt_1+ will have value 1 in each row
  corresponding to a period 1 observation, 0 otherwise; \verb+dt_2+
  will have value 1 in each row corresponding to a period 2
  observation, 0 otherwise; and so on.
\end{enumerate}

If a panel data set has the \verb+YEAR+ of the observation entered as
one of the variables you can create a periodic dummy to pick out a
particular year, e.g.\ \cmd{genr dum = (YEAR==1960)}.  You can also
create periodic dummy variables using the modulus operator,
\verb+%+.  For instance, to create a dummy with
value 1 for the first observation and every thirtieth observation
thereafter, 0 otherwise, do
%
\begin{code}
genr index 
series dum = ((index-1) % 30) == 0
\end{code}

\subsection{Lags, differences, trends}
\label{panel-lagged}

If the time periods are evenly spaced you may want to use lagged
values of variables in a panel regression (but see also
chapter~\ref{chap:dpanel}); you may also wish to construct first
differences of variables of interest.

Once a dataset is identified as a panel, gretl will handle the
generation of such variables correctly.  For example the command
\verb+genr x1_1 = x1(-1)+ will create a variable that contains the
first lag of \verb+x1+ where available, and the missing value code
where the lag is not available (e.g.\ at the start of the time series
for each group).  When you run a regression using such variables, the
program will automatically skip the missing observations.

When a panel data set has a fairly substantial time dimension, you may
wish to include a trend in the analysis.  The command \cmd{genr time} 
creates a variable named \varname{time} which runs from 1 to $T$ for
each unit, where $T$ is the length of the time-series dimension of the
panel.  If you want to create an index that runs consecutively from 1
to $m\times T$, where $m$ is the number of units in the panel, use
\cmd{genr index}.

\subsection{Basic statistics by unit}
\label{panel-stats}

gretl contains functions which can be used to generate basic
descriptive statistics for a given variable, on a per-unit basis;
these are \texttt{pnobs()} (number of valid cases), \texttt{pmin()}
and \texttt{pmax()} (minimum and maximum) and \texttt{pmean()} and
\texttt{psd()} (mean and standard deviation).

As a brief illustration, suppose we have a panel data set comprising 8
time-series observations on each of $N$ units or groups.  Then the
command
%
\begin{code}
series pmx = pmean(x)
\end{code}
%
creates a series of this form: the first 8 values (corresponding to
unit 1) contain the mean of \varname{x} for unit 1, the next 8 values
contain the mean for unit 2, and so on.  The \texttt{psd()} function
works in a similar manner.  The sample standard deviation for group
$i$ is computed as
\[
s_i = \sqrt{\frac{\sum(x-\bar{x}_i)^2}{T_i-1}}
\]
where $T_i$ denotes the number of valid observations on \varname{x}
for the given unit, $\bar{x}_i$ denotes the group mean, and the
summation is across valid observations for the group.  If $T_i < 2$,
however, the standard deviation is recorded as 0.

One particular use of \texttt{psd()} may be worth noting.  If you want
to form a sub-sample of a panel that contains only those units for
which the variable \varname{x} is time-varying, you can either use 
%
\begin{code}
smpl pmin(x) < pmax(x) --restrict
\end{code}
or
%
\begin{code}
smpl psd(x) > 0 --restrict
\end{code}

\section{Missing data values}
\label{missing-data}

\subsection{Representation and handling}

Missing values are represented internally as \verb+NaN+ (``not a
number''), as defined in the IEEE 754 floating-point standard. In a
native-format data file they should be represented as \verb+NA+. When
importing CSV data gretl accepts several common representations of
missing values including $-$999, the string \verb+NA+ (in upper or
lower case), a single dot, or simply a blank cell.  Blank cells
should, of course, be properly delimited, e.g.\ \verb+120.6,,5.38+, in
which the middle value is presumed missing.

As for handling of missing values in the course of statistical
analysis, gretl does the following:

\begin{itemize}
\item In calculating descriptive statistics (mean, standard deviation,
  etc.) under the \cmd{summary} command, missing values are simply
  skipped and the sample size adjusted appropriately.
\item In running regressions gretl first adjusts the beginning
  and end of the sample range, truncating the sample if need be.
  Missing values at the beginning of the sample are common in time
  series work due to the inclusion of lags, first differences and so
  on; missing values at the end of the range are not uncommon due to
  differential updating of series and possibly the inclusion of leads.
\end{itemize}

If gretl detects any missing values ``inside'' the (possibly
truncated) sample range for a regression, the result depends on the
character of the dataset and the estimator chosen.  In many cases, the
program will automatically skip the missing observations when
calculating the regression results.  In this situation a message is
printed stating how many observations were dropped.  On the other
hand, the skipping of missing observations is not supported for all
procedures: exceptions include all autoregressive estimators, system
estimators such as SUR, and nonlinear least squares.  In the case of
panel data, the skipping of missing observations is supported only if
their omission leaves a balanced panel. If missing observations are
found in cases where they are not supported, gretl gives an
error message and refuses to produce estimates.

\subsection{Manipulating missing values}
\label{sec:genr-missing}

Some special functions are available for the handling of missing
values.  The Boolean function \verb+missing()+ takes the name of a
variable as its single argument; it returns a series with value 1 for
each observation at which the given variable has a missing value, and
value 0 otherwise (that is, if the given variable has a valid value at
that observation).  The function \verb+ok()+ is complementary to
\verb+missing+; it is just a shorthand for \verb+!missing+ (where
\verb+!+ is the Boolean NOT operator).  For example, one can count the
missing values for variable \verb+x+ using

\begin{code}
scalar nmiss_x = sum(missing(x))
\end{code}

The function \verb+zeromiss()+, which again takes a single series as
its argument, returns a series where all zero values are set to the
missing code.  This should be used with caution---one does not want to
confuse missing values and zeros---but it can be useful in some
contexts.  For example, one can determine the first valid observation
for a variable \verb+x+ using

\begin{code}
genr time
scalar x0 = min(zeromiss(time * ok(x)))
\end{code}

The function \verb+misszero()+ does the opposite of \verb+zeromiss+,
that is, it converts all missing values to zero.

If missing values get involved in calculations, they propagate
according to the IEEE rules: notably, if one of the operands to an
arithmetical operation is a \texttt{NaN}, the result will also be
\texttt{NaN}.
    
\section{Maximum size of data sets}
\label{data-limits}

Basically, the size of data sets (both the number of variables and the
number of observations per variable) is limited only by the
characteristics of your computer.  Gretl allocates memory
dynamically, and will ask the operating system for as much memory as
your data require.  Obviously, then, you are ultimately limited by the
size of RAM.

Aside from the multiple-precision OLS option, gretl uses
double-precision floating-point numbers throughout.  The size of such
numbers in bytes depends on the computer platform, but is typically
eight.  To give a rough notion of magnitudes, suppose we have a data
set with 10,000 observations on 500 variables.  That's 5 million
floating-point numbers or 40 million bytes.  If we define the megabyte
(MB) as $1024 \times 1024$ bytes, as is standard in talking about RAM,
it's slightly over 38 MB.  The program needs additional memory for
workspace, but even so, handling a data set of this size should be
quite feasible on a current PC, which at the time of writing is likely
to have at least 256 MB of RAM.  

If RAM is not an issue, there is one further limitation on data size
(though it's very unlikely to be a binding constraint).  That is,
variables and observations are indexed by signed integers, and on a
typical PC these will be 32-bit values, capable of representing
a maximum positive value of $2^{31} - 1 = 2,147,483,647$.

The limits mentioned above apply to gretl's ``native''
functionality.  There are tighter limits with regard to two
third-party programs that are available as add-ons to gretl for
certain sorts of time-series analysis including seasonal adjustment,
namely \app{TRAMO/SEATS} and \app{X-12-ARIMA}.  These programs employ
a fixed-size memory allocation, and can't handle series of more than
600 observations.


\section{Data file collections}
\label{collections}

If you're using gretl in a teaching context you may be
interested in adding a collection of data files and/or scripts that
relate specifically to your course, in such a way that students can
browse and access them easily.

There are three ways to access such collections of files:

\begin{itemize}
\item For data files: select the menu item ``File, Open data, Sample
  file'', or click on the folder icon on the gretl toolbar.
\item For script files: select the menu item ``File, Script
  files, Example scripts''.
\end{itemize}

When a user selects one of the items:

\begin{itemize}
\item The data or script files included in the gretl distribution are
  automatically shown (this includes files relating to Ramanathan's
  \emph{Introductory Econometrics} and Greene's \emph{Econometric
    Analysis}).
\item The program looks for certain known collections of data files
  available as optional extras, for instance the datafiles from
  various econometrics textbooks (Davidson and MacKinnon, Gujarati,
  Stock and Watson, Verbeek, Wooldridge) and the Penn World Table (PWT
  5.6).  (See \href{http://gretl.sourceforge.net/gretl_data.html}{the
    data page} at the gretl website for information on these
  collections.)  If the additional files are found, they are added to
  the selection windows.
\item The program then searches for valid file collections (not
  necessarily known in advance) in these places: the ``system'' data
  directory, the system script directory, the user directory, and all
  first-level subdirectories of these.  For reference, typical values
  for these directories are shown in Table~\ref{tab-colls}.  (Note that
  \texttt{PERSONAL} is a placeholder that is expanded by Windows,
  corresponding to ``My Documents'' on English-language systems.)
\end{itemize}

\begin{table}[htbp]
  \begin{center}
    \begin{tabular}{lll}
      & \multicolumn{1}{c}{\textit{Linux}} & 
      \multicolumn{1}{c}{\textit{MS Windows}} \\
        system data dir & 
        {\small \verb+/usr/share/gretl/data+} &
        {\small \verb+c:\Program Files\gretl\data+} \\
        system script dir & 
        {\small \verb+/usr/share/gretl/scripts+} &
        {\small \verb+c:\Program Files\gretl\scripts+} \\
        user dir & 
        {\small \verb+$HOME/gretl+} &
        {\small \verb+PERSONAL\gretl+}\\
  \end{tabular}
 \end{center}
 \caption{Typical locations for file collections}
 \label{tab-colls}
\end{table}

Any valid collections will be added to the selection windows. So what
constitutes a valid file collection?  This comprises either a set of
data files in gretl XML format (with the \verb+.gdt+ suffix) or
a set of script files containing gretl commands (with \verb+.inp+
suffix), in each case accompanied by a ``master file'' or catalog.
The gretl distribution contains several example catalog files,
for instance the file \verb+descriptions+ in the \verb+misc+
sub-directory of the gretl data directory and
\verb+ps_descriptions+ in the \verb+misc+ sub-directory of the scripts
directory.

If you are adding your own collection, data catalogs should be named
\verb+descriptions+ and script catalogs should be be named
\verb+ps_descriptions+.  In each case the catalog should be placed
(along with the associated data or script files) in its own specific
sub-directory (e.g.\ \url{/usr/share/gretl/data/mydata} or
\verb+c:\userdata\gretl\data\mydata+).

The catalog files are plain text; if they contain non-ASCII characters
they must be encoded as UTF-8. The syntax of such files is
straightforward.  Here, for example, are the first few lines of
gretl's ``misc'' data catalog:

\begin{code}
# Gretl: various illustrative datafiles
"arma","artificial data for ARMA script example"
"ects_nls","Nonlinear least squares example"
"hamilton","Prices and exchange rate, U.S. and Italy"
\end{code}

The first line, which must start with a hash mark, contains a short
name, here ``Gretl'', which will appear as the label for this
collection's tab in the data browser window, followed by a colon,
followed by an optional short description of the collection.

Subsequent lines contain two elements, separated by a comma and
wrapped in double quotation marks.  The first is a datafile name
(leave off the \verb+.gdt+ suffix here) and the second is a short
description of the content of that datafile.  There should be one such
line for each datafile in the collection.

A script catalog file looks very similar, except that there are three
fields in the file lines: a filename (without its \verb+.inp+ suffix),
a brief description of the econometric point illustrated in the
script, and a brief indication of the nature of the data used.  Again,
here are the first few lines of the supplied ``misc'' script catalog:

\begin{code}
# Gretl: various sample scripts
"arma","ARMA modeling","artificial data"
"ects_nls","Nonlinear least squares (Davidson)","artificial data"
"leverage","Influential observations","artificial data"
"longley","Multicollinearity","US employment"
\end{code}

If you want to make your own data collection available to users, these
are the steps:

\begin{enumerate}
\item Assemble the data, in whatever format is convenient.
\item Convert the data to gretl format and save as \verb+gdt+
  files.  It is probably easiest to convert the data by importing them
  into the program from plain text, CSV, or a spreadsheet format (MS
  Excel or Gnumeric) then saving them. You may wish to add
  descriptions of the individual variables (the ``Variable, Edit
  attributes'' menu item), and add information on the source of the
  data (the ``Data, Edit info'' menu item).
\item Write a descriptions file for the collection using a text
  editor.
\item Put the datafiles plus the descriptions file in a subdirectory
  of the gretl data directory (or user directory).
\item If the collection is to be distributed to other people, package
  the data files and catalog in some suitable manner, e.g.\ as a
  zipfile.
\end{enumerate}

If you assemble such a collection, and the data are not proprietary, we
would encourage you to submit the collection for packaging as a
gretl optional extra.

\section{Assembling data from multiple sources}

In many contexts researchers need to bring together data from multiple
source files, and in some cases these sources are not organized such
that the data can simply be ``stuck together'' by appending rows or
columns to a base dataset. In gretl, the \texttt{join} command
can be used for this purpose; this command is discussed in detail in
chapter~\ref{chap:join}.


%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "gretl-guide"
%%% End: 

