Files and data files

Loading data from files

In order to process data, it is necessary to load it first. You can load data by dragging data files into the command window or by using the load command. Data files are any files that usually contain data in tabular form:

• Text files: *.dat, *.csv, *.txt, *.labx, *.jdx

• Binary files: *.ndat, *.xls, *.xlsx, *.ods, *.ibw

Note: in general, the differences in the German and English number formats should not be a problem for NumeRe. However, if it still happens that a number format was misinterpreted, replace all commas with periods and try again. If that still doesn't work, see the generic text files below.

We start with an example that you can enter directly into the command window:

<- load "<scriptpath>/examples/amplitude.dat"

-> The data from "[...]/scripts/examples/amplitude.dat" was successfully loaded into memory: the data set consists of 11 row(s)

and 4 column(s).

-> ans = {1, 11, 1, 4}

<- data(#,:)

-> ans = {"omega", "b=3", "b=5", "b=0"}

This command loaded the data from the file "scripts/examples/amplitude.dat" into the table data(). This table is the default target when data is to be loaded. However, the -totable[=TABLE()] option can be used to customize the target table. Here -totable (without explicitly specified table) would generate a new table from the file name; so in our case amplitude().

• The token <scriptpath> is a placeholder containing the scripts folder of NumeRe. There are a few more of these:

• <loadpath> for the default load path (The "Data files" folder in the tree view).

• <savepath> for the default save path (The "Saved Files" folder in the tree view).

• <plotpath> for the default storage path for generated plots (if they are saved as images).

• <procpath> for the procedure library

• <wp> for a fast changeable working path (can be changed with the command workpath PATH)

• <> for the NumeRe root directory (e.g. numere.exe can be found here)

• <this> for the current directory of the currently executed code unit (in the command window identical to <>, for scripts and procedures identical to their directory)

You can now look at the data even further, for example by using the show data() command. This will by default display the table in a new window in a tabular view as you know it from common spreadsheets. Also feel free to try extracting individual rows or columns from the dataset.

Saving data to files

To save data you have two commands available: save and export. The save command will by default save data to NumeRe's own binary data format NDAT to preserve metadata and precision. The export command will by default create text files to make the data available to a wider audience. You can try both commands:

<- save data()

-> The data set was successfully saved to "[...]/save/data_2022-05-31_093748.ndat.

<- export data()

-> The data set was successfully saved to "[...]/save/data_2022-05-31_093755.dat".

As you can see, the files are stored in the <savepath> under an automatically generated file name containing the current date. But you can also specify your own file names:

<- save data() -file="my_amplitude"

-> The data set was successfully saved to "[...]/save/my_amplitude.ndat".

<- export data() -file="my_exported_amplitude"

-> The data set was successfully saved to "[...]/save/my_exported_amplitude.dat".

Of course, you could also specify a different path or file format. The only conditions are that the path exists and NumeRe knows the file format:

<- export data() -file="<loadpath>/my_exported_amplitude.csv"

-> The data set was successfully saved to "[...]/data/my_exported_amplitude.csv.

Finally, you can of course restrict the range of the table to be saved, e.g.

<- export data(:,1:2) -file="my_extraction.txt"

-> The data set was successfully saved to "[...]/save/my_extraction.txt".

Read generic text files

Unfortunately, text files have the unpleasant property that they can appear in any format. So it is quite likely that NumeRe can only interpret text files incorrectly or sometimes not at all. To solve this problem you can use the read command to read text files as unformatted strings. You can save the return value of this command directly into a cluster:

<- htmlFile{} = read "<scriptpath>/examples/htmlfile.html";

<- num(htmlFile{})

-> ans = 1266

<- htmlFile{1}

-> ans = "<area shape="poly" coords="903,90,893,90,892,130,902,130" title=31,3725 />"

Here we have introduced the semicolon to suppress the response. Otherwise, NumeRe would have printed the 1266 lines of HTML (actually, it's just a collection of <area /> tags) on the command line. We used the num() function to read the number of elements in the cluster.

We can now use various string functions to try to translate the contents of the file into something interpretable. For example, we can find the contents of the title attribute using strfnd() and then read the value of the attribute using the following space. However, this is tedious:

<- substr(htmlFile{}, strfnd("title=", htmlFile{})+6, strrfnd(" ", htmlFile{})-strfnd("title=", htmlFile{})-6)

-> ans = {"31,3725", "5,8824", "19,7183", "27,1318", "2,2556", ...}

It's easier with the textparse() function. Here a pattern takes over the search in the string and extracts the desired element. The extractors %s are a string, %f a numeric value and %a an arbitrary value to be ignored. It is only important that the boundaries of the respective extractors are unique.

<- textparse(htmlFile{}, "<area shape=\"poly\" coords=\"%a\" title=%s />")

-> ans = {"31,3725", "5,8824", "19,7183", "27,1318", "2,2556", ...}

You will have noticed that in both cases the operation was automatically vectorized. In fact, htmlFile{} is an abbreviation for htmlFile{:}, so this automatically iterates over all indices in htmlFile{}. Likewise, data() is an abbreviation for data(:, :).

The function textparse() can also convert the read values into numeric values by means of %f (despite the German number format), so that you can continue working with them:

<- textparse(htmlFile{}, "<area shape=\"poly\" coords=\"%a\" title=%f />")

-> ans = {31.3725, 5.8824, 19.7183, 27.1318, 2.2556, ...}

Writing text files

Of course you can also write your strings into a file. To do this, you use the write command, to which you pass the strings and the target file name. You also have to consider a few options:

<- write htmlFile{} -set file="<savepath>/my_text_file.txt" mode=trunc nq

The content of the htmlFile{} cluster is written line by line to the file my_text_file.txt in the default storage path. With mode=trunc you say that you want to overwrite a possibly already existing file. With mode=app you would simply append new content to the end (useful e.g. for logfiles). The option nq is necessary for historical reasons and defines that the enclosing quotes of the strings should not be written into the file.

You can also use write without clusters by passing the individual lines comma-separated or explicitly wrapped:

<- write "First line", "Second line" -set file="<savepath>/my_four_line_file.txt" mode=trunc nq

<- write "Third line\nFourth line" -set file="<savepath>/my_four_line_file.txt" mode=app nq

With the last example we have then already illustrated the functionality of the mode=app option. The content of the file my_four_line_file.txt will then look like this:

First line

Second line

Third line

Fourth line