Query the number of lines in a text file conveniently

[zoziha]

Motivation

I have met many novices who have no programming background and are mostly related to scientific computing. Fortran beginners often fail to master the reading of files well and will encounter EOF (End of File) errors.
If we can query the number of lines in the file in advance, we can slightly avoid EOF and guide the allocation of the allocatable array.

So, I see that there is already a function number_of_rows in stdlib_io, I think it can be overloaded:

interface count_lines
    module procedure :: number_of_rows
end interface count_lines

! - - -

counts = count_lines(unit)

Prior Art

Count lines:

• number_of_rows (stdlib_io)
• countlines1 (forlab_io)
• countlines2 (forlab_io)

Read the context of the file as a string (or array)

• toml-f.toml_parse
• fortran-csv-module.read_csv_file
• json-fortran.json_file_load

Additional Information

• counts = count_lines(filename [, iostat] )
• type(string_type) :: string(:)/string = read_file(unit/filename [, iostat] )
• call count_lines( unit/filename, lines [, columns, max_width, len_lines(:), iostat] )
• call count_lines( unit/file, lines [, max_width, iostat] )

[Beliavsky]

I have a function like this and agree it would be convenient. I suggest offering the option of passing a file name as the argument.

[awvwgk]

How about a function to return an array of strings instead? It is easy enough to query the size of the returned array and it saves actually reading the file yourself.

[arjenmarkus]

Hm, that means that you cannot prepare for reading the file - like: determine the number of lines, knowing one value per line, allocate the array and then read the numbers, Or check that enough memory is available. Op wo 10 nov. 2021 om 13:20 schreef Sebastian Ehlert < ***@***.***>:

…

How about a function to return an array of strings instead. It is easy enough to query the size of the returned array and it saves actually reading the file yourself. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#560 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN6YR4FYDLJ5KW36ZFFLBLULJPQLANCNFSM5HXCZKWA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[zoziha]

There are some functions in the Go language standard library, which do well in these aspects:

• ReadFile (returns character array)
• ReadSlice (delim) (returns character array)
• ReadString (delim) (return string type)
• ReadLine (returns character array)

I agree and like to read the content of the file as a string, which can avoid IO occupation, and the program can parse the read string during the later running process. ReadFile will be useful.

[Beliavsky]

How about a function to return an array of strings instead? It is easy enough to query the size of the returned array and it saves actually reading the file yourself.

Out of habit, I don't use character variables with allocatable length. I would declare

character (len=1000), allocatable :: lines(:)

in the caller and pass this as an allocatable, intent(out) argument to a subroutine. The problems are that some lines may be longer than 1000 characters and that lines may be shorter than 1000 characters, wasting space. I know stdlib has string_type, but I have not looked at it carefully. Could a subroutine or function be written to return a 1D array of this type, with each element having length determined by the length of the line read?

[jvdp1]

How about a function to return an array of strings instead? It is easy enough to query the size of the returned array and it saves actually reading the file yourself.

this could be a solution for small datasets. But for large datasets with million/billion of lines containing reals, such a solution will not be applicable/feasible.

Anyway, I am in favour of @zoziha proposition to add a function that returns the number of lines/columns in a file in stdlib.

[milancurcic]

I like it. Here's a naive implementation:

https://github.com/modern-fortran/stock-prices/blob/392fed96a6ccb42c6398485e9ba8fb4c721b10c4/src/mod_io.f90#L14-L26

I don't know how efficient it is but it works.

[Beliavsky]

In addition to the number of line in a text file, the user may also want to know the length of the longest line, so that he can allocate a character variable of the proper len when reading it. In addition to a function which returns the number of lines, a subroutine with two optional, intent(out) arguments, num_lines and max_len could be created. There could also be an optional integer array argument len_lines(:) returning the length of each line.

[ivan-pi]

I like it. Here's a naive implementation:

https://github.com/modern-fortran/stock-prices/blob/392fed96a6ccb42c6398485e9ba8fb4c721b10c4/src/mod_io.f90#L14-L26

I don't know how efficient it is but it works.

I always thought that branch specifiers to labelled statements have fallen out of favor?

An alternative possibility is to query the iostat argument:

do
  read(unit,*,iostat=stat)
  if (stat < 0) exit ! end of file reached
  count = count + 1
end do

Is count_lines allowed to fail if an error occurred? One possibility could be to provide propagate upward the optional iostat flag or, return a negative count value to signify an error occured. I don't think the second is a good idea, because it is opposite to the iostat meaning in read, where:

• a positive integer means an error occured
• a negative integers means end-of-file or end-of-record (non-advancing io)
• zero means no error, not end-of-file, not end-of-record

[Beliavsky]

Before reading a file, in addition to knowing the number of lines, one may also want to know the number of fields (columns) in a file. This can be defined as the number of strings that can be read from the 1st line or a line number specified by the user, since for data files read by R or Python pandas, it is common for the first line to have one fewer fields than the other lines. One could have either

, A, B
2021-01-01, 10.0, 20.0
2021-01-02, 10.1, 20.1

or

Date, A, B
2021-01-01, 10.0, 20.0
2021-01-02, 10.1, 20.1

So to my suggested subroutine above one could add an intent(out) argument num_fields.

[wiremoons]

I agree this would be a handy function to have available. Just about every solution in AoC 2021 required me to write (or cut and paste) this snippet code every day, so being able to call a stdlib function instead to get the number of lines in a file would of been great.

The original suggestion above by @zoziha (#560 (comment)) of using the existing code for number_of_row via an overload (interface) would be easiest? This would also ensure only one function had to be maintained for this coded feature too?

Re @Beliavsky suggestion (#560 (comment)) above - I noticed just above the function number_of_rows there is a function called number_of_columns as well. Adapting this code to get a function to achieve a get_longest_line or perhaps max_line_len would be less work...

The above functions are private to the module by the looks of it - so they would need to be added to public as well, if this approach is used?