libstdc++ seems to lack AC_SYS_LARGEFILE in configury and thus uses fopen/open in fstream and friends that can fail not only because of large files but files with large inode numbers depending on the underlying filesystem.
This bug also affects std::filesystem when the capacity of modern storage devices are progressing in a fast pace. Without 64bit _FILE_OFFSET_BITS, and consider that there's no trends among the hardware vendors to migrate to a block size larger than 4096 in the foreseeable future, not even the fixed in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=85632 can tell the user of a 32bit system the correct size of a partition that is as large as 20TB. I think we need this as the ultimate fix and should force it no matter whether it is a 64bit build or not.
I was concerned this would be an ABI change for 32-bit targets, but libstdc++ already ensures that std::streamoff is a 64-bit type even when ::off_t isn't. So now I think building it with large file support should not change the type of anything in the libstdc++ API or the exported ABI.
(In reply to Richard Biener from comment #0) > libstdc++ seems to lack AC_SYS_LARGEFILE in configury and thus uses > fopen/open > in fstream and friends that can fail not only because of large files but > files with large inode numbers depending on the underlying filesystem. Richi, are you sure about that? config/io/basic_file_stdio.cc does: #ifdef _GLIBCXX_USE_LFS if ((_M_cfile = fopen64(__name, __c_mode))) #else if ((_M_cfile = fopen(__name, __c_mode))) #endif And that macro should be set during configure, via: dnl dnl Check whether LFS support is available. dnl AC_DEFUN([GLIBCXX_CHECK_LFS], [ AC_LANG_SAVE AC_LANG_CPLUSPLUS ac_save_CXXFLAGS="$CXXFLAGS" CXXFLAGS="$CXXFLAGS -fno-exceptions" AC_MSG_CHECKING([for LFS support]) AC_CACHE_VAL(glibcxx_cv_LFS, [ GCC_TRY_COMPILE_OR_LINK( [#include <unistd.h> #include <stdio.h> #include <sys/stat.h> ], [FILE* fp; fopen64("t", "w"); fseeko64(fp, 0, SEEK_CUR); ftello64(fp); lseek64(1, 0, SEEK_CUR); struct stat64 buf; fstat64(1, &buf);], [glibcxx_cv_LFS=yes], [glibcxx_cv_LFS=no]) ]) if test $glibcxx_cv_LFS = yes; then AC_DEFINE(_GLIBCXX_USE_LFS, 1, [Define if LFS support is available.]) fi AC_MSG_RESULT($glibcxx_cv_LFS) CXXFLAGS="$ac_save_CXXFLAGS" AC_LANG_RESTORE ]) So I think fstream is OK. However, there are some bugs elsewhere due to not checking _GLIBCXX_USE_LFS and so not using the largefile APIs: src/c++11/random.cc: _M_fd = ::open(fname, O_RDONLY); src/c++11/random.cc: _M_file = static_cast<void*>(std::fopen(fname, "rb")); This should be OK in practice, because fname is guaranteed to be either "/dev/random" or "/dev/urandom" (it seems unlikely that those devices will have inode numbers that don't fit in 32 bits). Nothing in src/filesystem and std/c++17/fs_* checks _GLIBCXX_USE_LFS or uses the LARGEFILE64 APIs. I think those files should be built with _FILE_OFFSET_BITS=64 so that stat, lstat, open, truncate etc. handle large files automatically, without having to remember to use the LARGEFILE64 versions explicitly.
(In reply to Jonathan Wakely from comment #3) > (In reply to Richard Biener from comment #0) > > libstdc++ seems to lack AC_SYS_LARGEFILE in configury and thus uses > > fopen/open > > in fstream and friends that can fail not only because of large files but > > files with large inode numbers depending on the underlying filesystem. > > Richi, are you sure about that? I think I looked at all unresolved symbols of libstdc++.so and there were some not using the LFS interfaces. I didn't actually see you are "manually" selecting the API.
Author: redi Date: Fri Oct 4 15:08:23 2019 New Revision: 276585 URL: https://gcc.gnu.org/viewcvs?rev=276585&root=gcc&view=rev Log: Build filesystem library with large file support Enable AC_SYS_LARGEFILE to set the macros needed for large file APIs to be used by default. We do not want to define those macros in the public headers that users include. The values of the macros are copied to a separate file that is only included by the filesystem sources during the build, and then the macros in <bits/c++config.h> are renamed so that they don't have any effect in user code including our headers. Also use larger type for result of filesystem::file_size to avoid truncation of large values on 32-bit systems (PR 91947). PR libstdc++/81091 PR libstdc++/91947 * configure.ac: Use AC_SYS_LARGEFILE to enable 64-bit file APIs. * config.h.in: Regenerate: * configure: Regenerate: * include/Makefile.am (${host_builddir}/largefile-config.h): New target to generate config header for filesystem library. (${host_builddir}/c++config.h): Rename macros for large file support. * include/Makefile.in: Regenerate. * src/c++17/fs_dir.cc: Include new config header. * src/c++17/fs_ops.cc: Likewise. (filesystem::file_size): Use uintmax_t for size. * src/filesystem/dir.cc: Include new config header. * src/filesystem/ops.cc: Likewise. (experimental::filesystem::file_size): Use uintmax_t for size. Modified: trunk/libstdc++-v3/ChangeLog trunk/libstdc++-v3/config.h.in trunk/libstdc++-v3/configure trunk/libstdc++-v3/configure.ac trunk/libstdc++-v3/include/Makefile.am trunk/libstdc++-v3/include/Makefile.in trunk/libstdc++-v3/src/c++17/fs_dir.cc trunk/libstdc++-v3/src/c++17/fs_ops.cc trunk/libstdc++-v3/src/filesystem/dir.cc trunk/libstdc++-v3/src/filesystem/ops.cc
Fixed on trunk so far. This should be OK to backport too, but I'll let it bake on trunk for a while.
The releases/gcc-9 branch has been updated by Jonathan Wakely <redi@gcc.gnu.org>: https://gcc.gnu.org/g:6cb662745d38e680a1a46fa04b108734cbc3df58 commit r9-8130-g6cb662745d38e680a1a46fa04b108734cbc3df58 Author: Jonathan Wakely <jwakely@redhat.com> Date: Thu Jan 9 13:38:43 2020 +0000 Build filesystem library with large file support Enable AC_SYS_LARGEFILE to set the macros needed for large file APIs to be used by default. We do not want to define those macros in the public headers that users include. The values of the macros are copied to a separate file that is only included by the filesystem sources during the build, and then the macros in <bits/c++config.h> are renamed so that they don't have any effect in user code including our headers. Also use larger type for result of filesystem::file_size to avoid truncation of large values on 32-bit systems (PR 91947). Backport from mainlne 2019-10-04 Jonathan Wakely <jwakely@redhat.com> PR libstdc++/81091 PR libstdc++/91947 * configure.ac: Use AC_SYS_LARGEFILE to enable 64-bit file APIs. * config.h.in: Regenerate: * configure: Regenerate: * include/Makefile.am (${host_builddir}/largefile-config.h): New target to generate config header for filesystem library. (${host_builddir}/c++config.h): Rename macros for large file support. * include/Makefile.in: Regenerate. * src/c++17/fs_dir.cc: Include new config header. * src/c++17/fs_ops.cc: Likewise. (filesystem::file_size): Use uintmax_t for size. * src/filesystem/dir.cc: Include new config header. * src/filesystem/ops.cc: Likewise. (experimental::filesystem::file_size): Use uintmax_t for size.
Fixed for GCC 9.3, but I don't plan to fix this on the gcc-8 branch.
The releases/gcc-8 branch has been updated by Jonathan Wakely <redi@gcc.gnu.org>: https://gcc.gnu.org/g:56b6c2ba3a45c768a33f726b130a56fc19cca650 commit r8-10084-g56b6c2ba3a45c768a33f726b130a56fc19cca650 Author: Jonathan Wakely <jwakely@redhat.com> Date: Thu Jan 9 13:38:43 2020 +0000 Build filesystem library with large file support Enable AC_SYS_LARGEFILE to set the macros needed for large file APIs to be used by default. We do not want to define those macros in the public headers that users include. The values of the macros are copied to a separate file that is only included by the filesystem sources during the build, and then the macros in <bits/c++config.h> are renamed so that they don't have any effect in user code including our headers. Also use larger type for result of filesystem::file_size to avoid truncation of large values on 32-bit systems (PR 91947). Backport from mainlne 2019-10-04 Jonathan Wakely <jwakely@redhat.com> PR libstdc++/81091 PR libstdc++/91947 * configure.ac: Use AC_SYS_LARGEFILE to enable 64-bit file APIs. * config.h.in: Regenerate: * configure: Regenerate: * include/Makefile.am (${host_builddir}/largefile-config.h): New target to generate config header for filesystem library. (${host_builddir}/c++config.h): Rename macros for large file support. * include/Makefile.in: Regenerate. * src/c++17/fs_dir.cc: Include new config header. * src/c++17/fs_ops.cc: Likewise. (filesystem::file_size): Use uintmax_t for size. * src/filesystem/dir.cc: Include new config header. * src/filesystem/ops.cc: Likewise. (experimental::filesystem::file_size): Use uintmax_t for size.