---
title: "Merge ILSA files"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Merge ILSA files}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE
)
```

```{r setup, echo=FALSE}
library(ILSAmerge)
```

## Downloaded data

As an example, we can use the data of 'REDS' 2021 provided by the package.

In the case of 'REDS' 2021, we have 12 files and 3 populations:
```{r}
ILSAfile.info(inputdir = system.file("extdata/reds", package = "ILSAmerge"))
```

With `ILSAmerge()` we can combine all the files into the number of populations
we have.

The default
type will be an "rds", but also we can make "zsav" and "sav" files:

```{r, eval=FALSE}
ILSAmerge(inputdir = system.file("extdata/reds", package = "ILSAmerge"), 
          outputdir = tempdir(), 
          filetype = c("rds", "zsav", "sav"))
```


## Adjust merge options

Depending on your goals or the capabilities of your computer, 
you may consider adjusting some options of `ILSAmerge()`:

- `population`: If you need to merge only some of the populations from the
path, you could include them here as a character vector. It should be in the 
same format as the output of `ILSAfile.info()`.
- `MBlimit`: If set to `NULL`, the default, all files will be
merged within R. If set a numeric value that will establish a limit (from the
output of `ILSAfile.info()`) of which files are going to be merged by R, 
the populations that go over this limit will not be merged by R, but instead an
SPSS syntax will be produced. Later, you can use this syntax to merge the files.
- `MBlistlimit`: In `ILSAmerge()` files are merged in two ways: using a list
produced by an lapply function, or an empty matrix filled up by subscripts.
The first method is almost always faster, but it can use a lot of memory.
This argument establishes a limit for merging by list and not by matrix.
We recommend not setting this higher than 200.
- `SPSSlimit`: If SPSS syntaxes are going to be produced it is important to
take into account the SPSS requirements for handling files. This could vary
by SPSS version, but normally it is a total of 50 files per command.

## Progress and time for merging

When `ILSAmerge()` runs, information about the progress of the merging and the
running time will be produced:

```{r, eval=TRUE}
ILSAmerge(inputdir = system.file("extdata/reds", package = "ILSAmerge"), 
          outputdir = tempdir(), 
          filetype = c("rds", "zsav", "sav"))
```


## Merging between respondents

After merging countries it is also possible to merge datasets of the same or
different respondents.

Two functions are available to merge respondents: `combineStudents()`, and 
`addSchools`. 

### Combine students

`combineStudents()` combines multiple student files, e.g., achievement
and background questionnaires, using 'TIMSS' Advanced 1995:
```{r,echo=FALSE, include = FALSE}
unlink(file.path(tempdir(),"combineStudents"),recursive = TRUE)
```
```{r}
# Path were raw 'SPSS' files are
input <- system.file("extdata/timssadv", package = "ILSAmerge")

# Path were merged files will be saved
dir.create(file.path(tempdir(),"combineStudents"))
output <- file.path(tempdir(),"combineStudents")

# Merging 'TIMSS' Advanced 1995, as .rds file
ILSAmerge(inputdir = input, outputdir = output, filetype = "rds", quiet = TRUE)

# Rename files
ILSArename(output)

# Check file names
list.files(output,pattern = ".rds")

# Combine student files
combineStudents(inputdir = output, outputdir = output)

# Check file names
list.files(output,pattern = ".rds")
```

As we can see, a new file was created combining the achievement and the
student questionnaire.


### Combine schools

`addSchools()` adds school data to student and teacher files, 
using 'TIMSS' Advanced 1995:
```{r}
# Check file names
list.files(output,pattern = ".rds")

# Add school data
addSchools(inputdir = output, outputdir = output)

# Check file names
list.files(output,pattern = ".rds")
```
```{r,echo=FALSE, include = FALSE}
unlink(file.path(tempdir(),"combineStudents"),recursive = TRUE)
```

As we can see, a new file was created combining the information of students
and schools.