Files and data files

Loading data from files

In order to process data, it is necessary to load it first. You can load data by dragging data files into the command window or by using the load command. Data files are any files that usually contain data in tabular form:

• Text files: *.dat, *.csv, *.txt, *.labx, *.jdx

• Binary files: *.ndat, *.xls, *.xlsx, *.ods, *.ibw

Note: in general, the differences in the German and English number formats should not be a problem for NumeRe. However, if it still happens that a number format was misinterpreted, replace all commas with periods and try again. If that still doesn't work, see the generic text files below.

We start with an example that you can enter directly into the command window:

<- load "<scriptpath>/examples/amplitude.dat"

-> The data from "[...]/scripts/examples/amplitude.dat" was successfully loaded into memory: the data set consists of 11 row(s)

   and 4 column(s).

-> ans = {1,   11,    1,   4}

<- data(#,:)

-> ans = {"omega",   "b=3",   "b=5",   "b=0"}

This command loaded the data from the file "scripts/examples/amplitude.dat" into the table data(). This table is the default target when data is to be loaded. However, the -totable[=TABLE()] option can be used to customize the target table. Here -totable (without explicitly specified table) would generate a new table from the file name; so in our case amplitude().

• The token <scriptpath> is a placeholder containing the scripts folder of NumeRe. There are a few more of these:

• <loadpath> for the default load path (The "Data files" folder in the tree view).

• <savepath> for the default save path (The "Saved Files" folder in the tree view).

• <plotpath> for the default storage path for generated plots (if they are saved as images).

• <procpath> for the procedure library

• <wp> for a fast changeable working path (can be changed with the command workpath PATH)

• <> for the NumeRe root directory (e.g. numere.exe can be found here)

• <this> for the current directory of the currently executed code unit (in the command window identical to <>, for scripts and procedures identical to their directory)

You can now look at the data even further, for example by using the show data() command. This will by default display the table in a new window in a tabular view as you know it from common spreadsheets. Also feel free to try extracting individual rows or columns from the dataset.

Saving data to files

To save data you have two commands available: save and export. The save command will by default save data to NumeRe's own binary data format NDAT to preserve metadata and precision. The export command will by default create text files to make the data available to a wider audience. You can try both commands:

<- save data()

-> The data set was successfully saved to "[...]/save/data_2022-05-31_093748.ndat.

<- export data()

-> The data set was successfully saved to "[...]/save/data_2022-05-31_093755.dat".

As you can see, the files are stored in the <savepath> under an automatically generated file name containing the current date. But you can also specify your own file names:

<- save data() -file="my_amplitude"

-> The data set was successfully saved to "[...]/save/my_amplitude.ndat".

<- export data() -file="my_exported_amplitude"

-> The data set was successfully saved to "[...]/save/my_exported_amplitude.dat".

Of course, you could also specify a different path or file format. The only conditions are that the path exists and NumeRe knows the file format:

<- export data() -file="<loadpath>/my_exported_amplitude.csv"

-> The data set was successfully saved to "[...]/data/my_exported_amplitude.csv.

Finally, you can of course restrict the range of the table to be saved, e.g.

<- export data(:,1:2) -file="my_extraction.txt"

-> The data set was successfully saved to "[...]/save/my_extraction.txt".

Read generic text files

Unfortunately, text files have the unpleasant property that they can appear in any format. So it is quite likely that NumeRe can only interpret text files incorrectly or sometimes not at all. To solve this problem you can use the read command to read text files as unformatted strings. You can save the return value of this command directly into a string array:

<- sHtmlFile = read "<scriptpath>/examples/htmlfile.html";

<- sHtmlFile.num

-> ans = 1266

<- sHtmlFile[1]

-> ans = "<area shape="poly" coords="903,90,893,90,892,130,902,130" title=31,3725 />"

Here we have introduced the semicolon to suppress the response. Otherwise, NumeRe would have printed the 1266 lines of HTML (actually, it's just a collection of <area /> tags) on the command line. We used the num method to read the number of elements in the array.

We can now use various string functions to try to translate the contents of the file into something interpretable. For example, we can find the contents of the title attribute using strfnd() and then read the value of the attribute using the following space. However, this is tedious:

<- substr(sHtmlFile, strfnd("title=", sHtmlFile)+6, strrfnd(" ", sHtmlFile)-strfnd("title=", sHtmlFile)-6)

-> ans = {"31,3725",   "5,8824",   "19,7183",   "27,1318",   "2,2556",  ...}

It's easier with the textparse() function. Here a pattern takes over the search in the string and extracts the desired element. The extractors %s are a string, %f a numeric value and %a an arbitrary value to be ignored. It is only important that the boundaries of the respective extractors are unique.

<- textparse(sHtmlFile, "<area shape=\"poly\" coords=\"%a\" title=%s />")

-> ans = {"31,3725",   "5,8824",   "19,7183",   "27,1318",   "2,2556",  ...}

You will have noticed that in both cases the operation was automatically vectorized.

The function textparse() can also convert the read values into numeric values by means of %f (despite the German number format), so that you can continue working with them:

<- textparse(sHtmlFile, "<area shape=\"poly\" coords=\"%a\" title=%f />")

-> ans = {31.3725,   5.8824,   19.7183,   27.1318,   2.2556,   ...}

Writing text files

Of course you can also write your strings into a file. To do this, you use the write command, to which you pass the strings and the target file name. You also have to consider a few options:

<- write sHtmlFile -set file="<savepath>/my_text_file.txt" mode=trunc

The content of the sHtmlFile array is written line by line to the file my_text_file.txt in the default storage path. With mode=trunc you say that you want to overwrite a possibly already existing file. With mode=app you would simply append new content to the end (useful e.g. for logfiles).

You can also use write without arrays by passing the individual lines comma-separated or explicitly wrapped:

<- write "First line", "Second line" -set file="<savepath>/my_four_line_file.txt" mode=trunc

<- write "Third line\nFourth line" -set file="<savepath>/my_four_line_file.txt" mode=app

With the last example we have then already illustrated the functionality of the mode=app option. The content of the file my_four_line_file.txt will then look like this:

First line

Second line

Third line

Fourth line

Reading and writing general files

In the previous section, we described the object’s data type. Among these objects is the file object, which allows you to read from and write to any file, even if it requires binary mode.

You can create a file object using the function of the same name and open the stream to the file at the same time. There are various opening modes to choose from. For reading from a file, "r" (text mode) and "rb" (binary mode) are generally sufficient. There are significantly more options for writing, but here we’ll make do with "r+" (read and write in text mode) or "r+b" (read and write in binary mode). In the following example, we write the first 7 Fibonacci numbers in binary as UINT8 to a file and later read them back in binary.

<- oFib = file("<savepath>/fibonacci.bin", "r+b");

<- oFib.write(ui8({1,2,3,5,8,13,21}))

-> ans = true

<- oFib.close

-> ans = true

<- oFib.open("<savepath>/fibonacci.bin", "rb");

<- oFib.read("value.ui8", 7)

-> ans = {  1,   2,   3,   5,   8,  13,  21}