-
Notifications
You must be signed in to change notification settings - Fork 164
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add filesystem interaction #874
base: master
Are you sure you want to change the base?
Add filesystem interaction #874
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this work, @minhqdao. Looking forward to the Stdlib starting to integrate filesystem-related routines. This is the beginning of a new category. In addition to the two noted issues, the function interfaces introduced by this PR should be documented in the specifications later.
(see also fpm::filesystem
)
test/io/test_filesystem.f90
Outdated
call test_failed(error, "Creating file 1 in directory '"//temp_list_dir//"' failed."); return | ||
end if | ||
|
||
call run('mkdir '//temp_list_dir//'/'//dir, iostat=stat) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is an error in CI under Windows OS. This is because the shell command mkdir
called may follow the mkdir
command in CMD.exe
, like mkdir \a\b\c\d
. The path connector uses \
instead of /
. It may be possible to implement a routine to determine the current operating system in Stdlib later.
A temporary solution can be to make two judgments. If the /
scheme fails, then try the \
scheme.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A simple solution that I've been using and think would be useful to have in stdlib would be to add in the constants module something like:
#ifdef _WIN32
character(len=1), parameter, public :: os_sep='\'
#else
character(len=1), parameter, public :: os_sep='/'
#endif
Which then would enable the equivalent of python
import os
print(os.sep)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The C preprocessor in Windows-GFortran does not predefine the _WIN32
macro definition. This may require CMake to determine the operating system and define _WIN32
. There are indeed two solutions:
- During compilation, solve this problem through the C preprocessor and CMake macro definitions.
- During runtime, solve this problem by implementing a routine to determine the operating system. (see also fpm::get_os_type)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The C preprocessor approach is already in use in stdlib_system, so it could be one solution. The runtime approach is also very good.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I chose the runtime approach because, as mentioned by @zoziha, Windows-GFortran does not set the _WIN32
macro. We had that problem in fpm
already and I don't think the solution in stdlib_system
works on Windows using gfortran
.
src/stdlib_io_filesystem.f90
Outdated
stat = 0 | ||
|
||
if (.not. exists(temp_dir)) then | ||
call run('mkdir '//temp_dir, stat) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that under Windows OS, mkdir temp
will create a temp
folder under the current path pwd
. Is this appropriate?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is a very pragmatic solution, but I think it should be fine for now
src/stdlib_io_filesystem.f90
Outdated
!> | ||
!> Run a command in the shell. | ||
!> [Specification](../page/specs/stdlib_io.html#run) | ||
subroutine run(command, iostat, iomsg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe an option to redirect the command output (stdout, stderr) to a string variable would be very useful. It is being done in the fpm version of this function already.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it slightly exceeds the scope of this PR but it can easily be done in a follow-up PR
Windows has NO command "rm", I just confirmed that. You need "del" instead
and quite possibly "del /q" to make sure no questions aer asked.
To get a clean listing of all the files in a directory, use "dir /b".
A side note: many commands on Windows do accept the forward slash as well
as the backward slash to indicate directories. Although, you can get
strange surprises. Where "cd" accepts the forward slash, "dir" does not. Oh
well, best stick with the backslash ;).
Op do 19 sep 2024 om 08:22 schreef Federico Perini ***@***.***
…:
***@***.**** commented on this pull request.
------------------------------
In src/stdlib_io_filesystem.f90
<#874 (comment)>:
> +
+ integer :: unit, stat
+ character(len=256) :: line
+
+ stat = 0
+
+ if (.not. exists(temp_dir)) then
+ call run('mkdir '//temp_dir, stat)
+ if (stat /= 0) then
+ if (present(iostat)) iostat = stat
+ if (present(iomsg)) iomsg = "Failed to create temporary directory '"//temp_dir//"'."
+ return
+ end if
+ end if
+
+ call run('ls '//dir//' > '//listed_contents, stat)
On the same line as @zoziha <https://github.com/zoziha>, ls may not work
under MS Windows unless you are in a WSL environment. There, it would be
worth wrapping a call to dir.
------------------------------
In src/stdlib_io_filesystem.f90
<#874 (comment)>:
> + end if
+
+ allocate(files(0))
+ do
+ read(unit, '(A)', iostat=stat) line
+ if (stat /= 0) exit
+ files = [files, string_type(line)]
+ end do
+ close(unit, status="delete")
+ end
+
+ !> Version: experimental
+ !>
+ !> Run a command in the shell.
+ !> [Specification](../page/specs/stdlib_io.html#run)
+ subroutine run(command, iostat, iomsg)
I believe an option to redirect the command output (stdout, stderr) to a
string variable would be very useful. It is being done in the fpm
<https://github.com/fortran-lang/fpm/blob/main/src/fpm_filesystem.F90>
version of this function already.
------------------------------
In test/io/test_filesystem.f90
<#874 (comment)>:
> + subroutine fs_run_valid_command(error)
+ type(error_type), allocatable, intent(out) :: error
+
+ integer :: stat
+
+ call run("whoami", iostat=stat)
+ call check(error, stat, "Running a valid command should not fail.")
+ end
+
+ subroutine fs_list_dir_empty(error)
+ type(error_type), allocatable, intent(out) :: error
+
+ integer :: stat
+ type(string_type), allocatable :: files(:)
+
+ call run('rm -rf '//temp_list_dir, iostat=stat)
Same as my previous comment, I am not sure if rm is available on non-WSL
Windows versions.
—
Reply to this email directly, view it on GitHub
<#874 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAN6YR24RFI2GBR2ABYIYW3ZXJURBAVCNFSM6AAAAABOJXKJV6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDGMJRG42DEMZVGU>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Changes made:
|
!> | ||
!> Whether the operating system is Windows. | ||
!> [Specification](../page/specs/stdlib_io.html#is_windows) | ||
logical function is_windows() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In fpm, the method of using local save
variables seems very advisable to avoid repeatedly accessing environment variables when is_windows
is frequently called.
Hi I went through the PR again and would like to make a few comments. The first one is purely cosmetic: I think that all procedures should have The second point: I would like to come back to the preprocessing... I know that it is a sad state of affairs that This does indeed impose: either relying on CMake, or documenting that one should pass the appropriate definition flags if using plain Makefiles or I've opened a discussion to see if While I find it "tolerable" for fpm having to rely on the runtime implementation as a tool for building, I'm expecting stdlib to be used for more computationally intensive tasks. Thus, it seems to me better to shift the issue to the building documentation and not the library. |
This PR introduces filesystem interactions by pragmatically invoking the system shell. It is part of the broader strategy outlined in #865 (see #865 (review)) to support file zipping and unzipping. These functionalities are essential for handling
.npz
files.