C++ Logo

std-proposals

Advanced search

Re: [std-proposals] Add an interface to std::fstream::open to open unnamed file wb+

From: Thiago Macieira <thiago_at_[hidden]>
Date: Tue, 25 Apr 2023 15:47:28 -0700
On Tuesday, 25 April 2023 13:40:47 PDT Louis Langholtz via Std-Proposals
wrote:
> My primary goal is providing an fstream opened for an exclusively created
> file not completely named by the user and that as much as possible goes
> away automatically

Again, from experience not all randomly-named files are temporary. Some of
those get renamed to permanent names and remain for long periods. For others,
ownership and the responsibility to delete are transferred to other processes,
after closing the file. And in some OSes for some purposes, closing the file is
required -- even on Linux you can get a ETXTBUSY if you try to execve() a file
that is still open for writing.

> - which is like what std::tmpfile provides.
> https://en.cppreference.com/w/cpp/io/c/tmpfile notes how the implementation
> of std::tmpfile has the liberty to vary as per the concerns you are
> raising. This seems like a reasonable lee-way to carry-over to an fstream
> interface to opening a temporary file like I’m wanting and seems to speak
> partly to this concern. But std::tmpfile doesn’t have to be used/required.

In my informed[*] opinion, tmpfile() is too limited. I've just given two
reasonable use-cases where it wouldn't suffice. I believe it would be best to go
beyond it.

[*] informed because I am the maintainer of QFile, QTemporaryFile and
QSaveFile.

> I realize C++23 std::ios_base::noreplace gives us exclusively opened
> writable files which provides part of my goal. That by itself doesn’t
> provide an interface to underlying facilities for creating those
> exclusively opened writable file that only exists temporarily however.

Sure it does, on Unix systems, which is how tmpfile() was first implemented and
designed for. The way you do it on Linux is that you delete the file after
you've created-opened it. Aside from having to have a good random generator
and attempting multiple times, it's no different than O_TMPFILE. See glibc's
implementation and note how it unlink()s the file before returning.
https://codebrowser.dev/glibc/glibc/stdio-common/tmpfile.c.html

It's different on Windows with _O_TEMPORARY, which causes CreateFile to get the
FILE_FLAG_DELETE_ON_CLOSE flag. That does exactly what the C standard says:
"will automatically be removed when it is closed". So for Windows, you'd need
to pass an extra flag to indicate that you want it to be ephemeral.

> It occurs to me now that a new open mode - that effectively plumbed
> O_TMPFILE like support back to the user - could be added as an alternative
> strategy for supporting what I want. This avoids the “horrible”-ness of
> adding an open function overload that took no arguments. ;) And it even
> works in conjunction with noreplace (that does O_EXCL like support).

Yes and no. That's would allow using the Windows flag.

The problem is that it doesn't work for the survives-after-close scenarios
from above. If the temporary file is meant to do that, one still needs the
ability to generate a random and unique name.

> > QTemporaryFile supports both. It will try to create an unnamed file in the
> > directory you asked of it and fall back to randomly-generated names
> > otherwise. And if you ask it for the name, it'll materialise the file by
> > using the linkat() trick.
>
> Thank you for recognizing this prior art. It seems reasonable to consider it
> as such in regards to this proposal. I’m looking at its documentation
> <https://doc.qt.io/qt-6/qtemporaryfile.html> for an eye on how it differs
> functionality wise from std::tmpfile.

See also QSaveFile. It's a QTemporaryFile that gets atomically renamed to the
final name, to any other observer seeing the file in intermediate stage. It also
does fsync() to improve the chances of surviving power failures.

There's also QLockFile, which is an ephemeral file but with a non-random name
(and other locking behaviours).

> Seems a difference is the ability to pick the directory in which to attempt
> to create the file by way of its templateName parameter. That parameter
> looks suggestive that it uses mkstemp
> <https://pubs.opengroup.org/onlinepubs/9699919799/functions/mkdtemp.html>
> or was patterned from similar functions taking a filename template with six
> X characters.

Originally it did. I've long since replaced mktemp / mkstemp() with our own
generator. See https://code.qt.io/cgit/qt/qtbase.git/commit/?
id=5483b30868e44bc0799d7a1998f1907775f50fec
and https://code.qt.io/cgit/qt/qtbase.git/commit/?
id=9a3ce25f983b7ef842bbc055081474f044efb738

> Having an optional parameter to specify the directory path in
> which the temporary should be made seems like a useful option in any case.

More than "nice to have". This is one of the "important to have" features,
even if not "must have". You can't rename a file across filesystems or mount
points. Even renaming across directories inside the same filesystem/mount point
can fail if the filesystem is full and the OS must grow the directory entry
table, but renaming inside the same directory usually doesn't fail -- this is
how system upgrade tools work.

> I don’t like the idea
> of a filename template for which having X’s in it is possibly replaced. I
> do however like that this parameter allows control over the temporary
> file’s prefix and suffix characters (allowing configurability of before and
> after components for a file path).

The XXXXXX template replacement is a well-known technique. For many uses where
the name of the file is important, the prefix or the suffix of the name might need
to be fixed so the file is interpreted correctly by whatever is consuming it.
Think for example of creating a new executable on Windows: it MUST end in .exe
(or one of a handful of other suffixes).

-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
   Software Architect - Intel DCAI Cloud Engineering

Received on 2023-04-25 22:47:30