A write() system call can return successfully and even the following
close() syscall may not return an error, even if there was an error
like EFBIG (File too large).

if ((fd = creat(name, 0666)) < 0) {
perror(name);
exit(1);
}
while (have_more_data) {
if (write(fd, buffer, count) < 0) {
perror("write");
exit(1);
}
}
if (close(fd) < 0) {
perror("close");
exit(1);
}

This may complete successfully, e.g. when EFBIG occurs in the last
write() syscall before the loop ends.

So what is the best way to check for errors with write()?

Steve

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, 16 Dec 2021 18:58:28 +0100, Steve Keller wrote:

A write() system call can return successfully and even the following
close() syscall may not return an error, even if there was an error like EFBIG (File too large).

if ((fd = creat(name, 0666)) < 0) {
perror(name);
exit(1);
}
while (have_more_data) {
if (write(fd, buffer, count) < 0) {
perror("write");
exit(1);
}
}
if (close(fd) < 0) {
perror("close");
exit(1);
}

This may complete successfully, e.g. when EFBIG occurs in the last
write() syscall before the loop ends.

So what is the best way to check for errors with write()?

According to the Linux write(2) manpage,
A successful return from write() does not make any guarantee that data
has been committed to disk. In fact, on some buggy implementations, it
does not even guarantee that space has successfully been reserved for
the data. The only way to be sure is to call fsync(2) after you are
done writing all your data.

While this may not be true for other Unix and Unix-like systems, an appropriately-placed sync(2)/fsync(2) call wouldn't hurt.


--
Lew Pitcher
"In Skills, We Trust"

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Steve Keller <keller.steve@gmx.de> writes:

A write() system call can return successfully and even the following
close() syscall may not return an error, even if there was an error
like EFBIG (File too large).

if ((fd = creat(name, 0666)) < 0) {
perror(name);
exit(1);
}
while (have_more_data) {
if (write(fd, buffer, count) < 0) {
perror("write");
exit(1);
}
}
if (close(fd) < 0) {
perror("close");
exit(1);
}

This may complete successfully, e.g. when EFBIG occurs in the last
write() syscall before the loop ends.

So what is the best way to check for errors with write()?

Steve

ssize_t diag;
bool done = false;
while (!done) {
ssize_t diag = write(fd, buffer, count);
if (diag == -1) {
fprintf(stderr, "%s: Unable to write '%s': %s\n", argv[0], filename,
strerror(errno));
done = true;
continue;
} else if (diag < count) {
fprintf(stderr, "%s: Short write, %zd of %zu bytes written\n", argv[0], diag,
count);
done = true;
continue;
}
/* load buffer with next 'count' bytes of data */
}
int rval = fdatasync(fd);
if (rval == -1) {
fprintf(stderr, "%s: Unable to sync data to disk: %s\n", argv[0], strerror(errno));
}
rval = close(fd);
if (rval == -1) {
fprintf(stderr, "%s: Close of '%s' failed: %s\n", argv[0], filename, strerror(errno));
}
fd = -1;

One may also catch the SIGXFSZ signal to ensure one detects file size resource limit violations
(see setrlimit(2), signal(7))

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Lew Pitcher <lew.pitcher@digitalfreehold.ca> writes:

On Thu, 16 Dec 2021 18:58:28 +0100, Steve Keller wrote:

A write() system call can return successfully and even the following
close() syscall may not return an error, even if there was an error like
EFBIG (File too large).

if ((fd = creat(name, 0666)) < 0) {
perror(name);
exit(1);
}
while (have_more_data) {
if (write(fd, buffer, count) < 0) {
perror("write");
exit(1);
}
}
if (close(fd) < 0) {
perror("close");
exit(1);
}

This may complete successfully, e.g. when EFBIG occurs in the last
write() syscall before the loop ends.

So what is the best way to check for errors with write()?

According to the Linux write(2) manpage,
A successful return from write() does not make any guarantee that data
has been committed to disk. In fact, on some buggy implementations, it
does not even guarantee that space has successfully been reserved for
the data. The only way to be sure is to call fsync(2) after you are
done writing all your data.

While this may not be true for other Unix and Unix-like systems, an appropriately-placed sync(2)/fsync(2) call wouldn't hurt.

Usually, it will hurt. The buffered I/O subsystem in the kernel ("page
cache") is there for a reason and it shouldn't be circumvented just
because it seems somehow suspicious. The default assumption should be
that the system will not suddenly stop and hence, that leaving it to the
kernel to decide when to write what will be ok.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Steve Keller <keller.steve@gmx.de> writes:

A write() system call can return successfully and even the following
close() syscall may not return an error, even if there was an error
like EFBIG (File too large).

if ((fd = creat(name, 0666)) < 0) {
perror(name);
exit(1);
}
while (have_more_data) {
if (write(fd, buffer, count) < 0) {
perror("write");
exit(1);
}
}
if (close(fd) < 0) {
perror("close");
exit(1);
}

This may complete successfully, e.g. when EFBIG occurs in the last
write() syscall before the loop ends.

So what is the best way to check for errors with write()?

Avoid checking the return value of close. The way to ensure that a
"physical write" took place is to use fsync before close (and checking
the return value of that) and this should usually be avoided.

The only thing checking the return value of close is actually good for
is to tell a user that "my program lost your data because I didn't use
fsync" --- not terribly useful.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, 2021-12-16, Steve Keller wrote:

A write() system call can return successfully and even the following
close() syscall may not return an error, even if there was an error
like EFBIG (File too large).

The Linux write(2) man page says:

RETURN VALUE
On success, the number of bytes written is returned. On error, -1 is re-
turned, and errno is set to indicate the cause of the error.

...

EFBIG An attempt was made to write a file that exceeds the implementation-
defined maximum file size or the process's file size limit, or to
write at a position past the maximum allowed offset.

The OpenBSD one is similar. Are you talking about other Unix-like
systems, or are you saying the man page is wrong? Because it seems to
say that in your scenario, the fatal write() will return -1 and set
EFBIG.

if ((fd = creat(name, 0666)) < 0) {
perror(name);
exit(1);
}
while (have_more_data) {
if (write(fd, buffer, count) < 0) {
perror("write");
exit(1);
}
}
if (close(fd) < 0) {
perror("close");
exit(1);
}

This may complete successfully, e.g. when EFBIG occurs in the last
write() syscall before the loop ends.

So what is the best way to check for errors with write()?

Until I'm convinced it's wrong: according to the write(2) man page.

(I'm disregarding the fsync(2) angle, covered by others.)

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thursday, December 16, 2021 at 9:58:37 AM UTC-8, Steve Keller wrote:

A write() system call can return successfully and even the following
close() syscall may not return an error, even if there was an error
like EFBIG (File too large).

if ((fd = creat(name, 0666)) < 0) {
perror(name);
exit(1);
}
while (have_more_data) {
if (write(fd, buffer, count) < 0) {
perror("write");
exit(1);
}
}
if (close(fd) < 0) {
perror("close");
exit(1);
}

This may complete successfully, e.g. when EFBIG occurs in the last
write() syscall before the loop ends.

So what is the best way to check for errors with write()?

Uh, actually look at the return value and see if it matches the number of bytes you asked to write: on partial success, write() will return less than the requested count. Note that a short write won't change errno; if you need to know why it was unable
to write more than you need to keep calling write() until it completely fails (usually the next call) and check errno then.

To quote the (next issue) of the POSIX spec's description of write():
If a write( ) requests that more bytes be written than there is room for (for example, the file size
limit of the process or the physical end of a medium), only as many bytes as there is room for
shall be written. For example, suppose there is space for 20 bytes more in a file before reaching a
limit. A write of 512 bytes will return 20. The next write of a non-zero number of bytes would
give a failure return (except as noted below).

If the request would cause the file size to exceed the soft file size limit for the process and there
is no room for any bytes to be written, the request shall fail and the implementation shall
generate the SIGXFSZ signal for the thread.

And then in the ERRORS section:
ERRORS
These functions shall fail if:
...
[EFBIG] An attempt was made to write a file that exceeds the implementation-defined
maximum file size and there was no room for any bytes to be written.

[EFBIG] An attempt was made to write a file that exceeds the file size limit of the
process, and there was no room for any bytes to be written. A SIGXFSZ signal
shall also be sent to the thread.


Philip Guenther

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Philip Guenther <guenther@gmail.com> writes:

<snip>

but down below the ERRORS section, in the NOTES:

A successful return from write() does not make any guarantee that data
has been committed to disk. On some filesystems, including NFS, it
does not even guarantee that space has successfully been reserved for
the data. In this case, some errors might be delayed until a future
write(), fsync(2), or even close(2). The only way to be sure is to
call fsync(2) after you are done writing all your data.

Successfully writing to the file system isn't the same thing as
successfully writing to actual nonvolatile storage (ie the disk). Much
as we wish it were (and in all but a microscopic subset of cases we can
get away with pretending).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Philip Guenther <guenther@gmail.com> writes:

Uh, actually look at the return value and see if it matches the
number of bytes you asked to write: on partial success, write() will
return less than the requested count. Note that a short write won't
change errno; if you need to know why it was unable to write more
than you need to keep calling write() until it completely fails
(usually the next call) and check errno then.

To quote the (next issue) of the POSIX spec's description of write():
If a write( ) requests that more bytes be written than
there is room for (for example, the file size limit of the
process or the physical end of a medium), only as many
bytes as there is room for shall be written. For example,
suppose there is space for 20 bytes more in a file before
reaching a limit. A write of 512 bytes will return 20. The
next write of a non-zero number of bytes would give a
failure return (except as noted below).

If the request would cause the file size to exceed the soft
file size limit for the process and there is no room for
any bytes to be written, the request shall fail and the
implementation shall generate the SIGXFSZ signal for the
thread.

And then in the ERRORS section:
ERRORS
These functions shall fail if:
...
[EFBIG] An attempt was made to write a file that exceeds the
implementation-defined maximum file size and there was
no room for any bytes to be written.

[EFBIG] An attempt was made to write a file that exceeds the
file size limit of the process, and there was no room
for any bytes to be written. A SIGXFSZ signal shall
also be sent to the thread.

Ah, thanks. How to handle a short write, i.e. with no error return
code, is exactly what I was looking for. This information is not in
the Linux write(2) man page.

BTW, is it possible that after a short write the next write() syscall
for the remaining data will write some more data, possibly even the
whole remaining data successfully? I think of something like a pipe
or a socket where not the whole data can be sent immediately.

Or will the write() syscall always block until all data could be
written? I.e. will the next write syscall after a short write always
fail and return -1?

Steve

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Jorgen Grahn <grahn+nntp@snipabacken.se> writes:

The OpenBSD one is similar. Are you talking about other Unix-like
systems, or are you saying the man page is wrong? Because it seems to
say that in your scenario, the fatal write() will return -1 and set
EFBIG.

I should have written more clearly, that I was talking about a "short
write". I.e. the write() syscall doesn't return -1 and doesn't set
errno, but it could only write data partially.

I wasn't saying the man page is wrong, only that an error can occur
without write() returning -1 to indicate the error. Repeating the
write() with the rest of the data to get the error, as Philip
suggested, was obviously too simple for me to get that idea :-)

Until I'm convinced it's wrong: according to the write(2) man page.

(I'm disregarding the fsync(2) angle, covered by others.)

Yes, f*sync() is not what I was looking for.

Steve

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Steve Keller <keller.steve@gmx.de> writes:

Jorgen Grahn <grahn+nntp@snipabacken.se> writes:

The OpenBSD one is similar. Are you talking about other Unix-like
systems, or are you saying the man page is wrong? Because it seems to
say that in your scenario, the fatal write() will return -1 and set
EFBIG.

I should have written more clearly, that I was talking about a "short
write". I.e. the write() syscall doesn't return -1 and doesn't set
errno, but it could only write data partially.

I wasn't saying the man page is wrong, only that an error can occur
without write() returning -1 to indicate the error. Repeating the
write() with the rest of the data to get the error, as Philip
suggested, was obviously too simple for me to get that idea :-)

write generally needs to be called in a loop until either everything was written or an error ocurred, subject to whatever special properties the
"output thing" the file descriptor refers to might have (eg, atomicity
of PIPE_BUF writes to a pipe, or preservation of message boundaries by
datagram sockets). But that's possibly not enough to detect errors
reliably: I've encountered people who believed that a write call which
cannot write anything due to the disk being full should return 0 and not
-1 and an ENOSPC error. I don't know if implementations exist which
actually work in this way.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri, 2021-12-17, Steve Keller wrote:

Jorgen Grahn <grahn+nntp@snipabacken.se> writes:

The OpenBSD one is similar. Are you talking about other Unix-like
systems, or are you saying the man page is wrong? Because it seems to
say that in your scenario, the fatal write() will return -1 and set
EFBIG.

I should have written more clearly, that I was talking about a "short
write". I.e. the write() syscall doesn't return -1 and doesn't set
errno, but it could only write data partially.

Ah, and now I see this in write(2):

Note that a successful write() may transfer fewer than count
bytes. Such partial writes can occur for various reasons; for
example, because there was insufficient space on the disk device
to write all of the requested bytes, or [more well-known reasons]

This is something I've not noticed before. I assumed write() to disk
files used an "all or nothing" policy.

In my defense, I don't write to disk very often.

I wasn't saying the man page is wrong, only that an error can occur
without write() returning -1 to indicate the error. Repeating the
write() with the rest of the data to get the error, as Philip
suggested, was obviously too simple for me to get that idea :-)

Until I'm convinced it's wrong: according to the write(2) man page.

(I'm disregarding the fsync(2) angle, covered by others.)

Yes, f*sync() is not what I was looking for.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thursday, December 16, 2021 at 7:35:16 PM UTC-8, Steve Keller wrote:

Philip Guenther <guen...@gmail.com> writes:

Uh, actually look at the return value and see if it matches the
number of bytes you asked to write: on partial success, write() will return less than the requested count. Note that a short write won't
change errno; if you need to know why it was unable to write more
than you need to keep calling write() until it completely fails
(usually the next call) and check errno then.

To quote the (next issue) of the POSIX spec's description of write():
If a write( ) requests that more bytes be written than
there is room for (for example, the file size limit of the
process or the physical end of a medium), only as many
bytes as there is room for shall be written. For example,
suppose there is space for 20 bytes more in a file before
reaching a limit. A write of 512 bytes will return 20. The
next write of a non-zero number of bytes would give a
failure return (except as noted below).

If the request would cause the file size to exceed the soft
file size limit for the process and there is no room for
any bytes to be written, the request shall fail and the
implementation shall generate the SIGXFSZ signal for the
thread.

And then in the ERRORS section:
ERRORS
These functions shall fail if:
...
[EFBIG] An attempt was made to write a file that exceeds the implementation-defined maximum file size and there was
no room for any bytes to be written.

[EFBIG] An attempt was made to write a file that exceeds the
file size limit of the process, and there was no room
for any bytes to be written. A SIGXFSZ signal shall
also be sent to the thread.

Ah, thanks. How to handle a short write, i.e. with no error return
code, is exactly what I was looking for. This information is not in
the Linux write(2) man page.

BTW, is it possible that after a short write the next write() syscall
for the remaining data will write some more data, possibly even the
whole remaining data successfully? I think of something like a pipe
or a socket where not the whole data can be sent immediately.

If the condition that prevented writing more data has a change of state between the two calls, then sure, more data may be written successfully. For example, if some other process or thread freed up disk space between the calls than an ENOSPC or EDQUOT
condition that caused a short-write may go away, permitting more data to be written.

Could the system decide to do short-writes for no good reason and then permit more later, just because? Given the first sentence of the wording I quoted, I think the answer is "no", but also think I'm uninterested in arguing about it and would just call
such an implementation "low quality" and not use it.


Philip Guenther

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Philip Guenther <guenther@gmail.com> writes:

On Thursday, December 16, 2021 at 7:35:16 PM UTC-8, Steve Keller wrote:

Philip Guenther <guen...@gmail.com> writes:=20
=20

Uh, actually look at the return value and see if it matches the=20
number of bytes you asked to write: on partial success, write() will=20
return less than the requested count. Note that a short write won't=20
change errno; if you need to know why it was unable to write more=20
than you need to keep calling write() until it completely fails=20
(usually the next call) and check errno then.=20
=20
To quote the (next issue) of the POSIX spec's description of write():= >=20
If a write( ) requests that more bytes be written than=20
there is room for (for example, the file size limit of the=20
process or the physical end of a medium), only as many=20
bytes as there is room for shall be written. For example,=20
suppose there is space for 20 bytes more in a file before=20
reaching a limit. A write of 512 bytes will return 20. The=20
next write of a non-zero number of bytes would give a=20
failure return (except as noted below).=20
=20
If the request would cause the file size to exceed the soft=20
file size limit for the process and there is no room for=20
any bytes to be written, the request shall fail and the=20
implementation shall generate the SIGXFSZ signal for the=20
thread.=20
=20
And then in the ERRORS section:=20
ERRORS=20
These functions shall fail if:=20
...=20
[EFBIG] An attempt was made to write a file that exceeds the=20
implementation-defined maximum file size and there was=20
no room for any bytes to be written.=20
=20
[EFBIG] An attempt was made to write a file that exceeds the=20
file size limit of the process, and there was no room=20
for any bytes to be written. A SIGXFSZ signal shall=20
also be sent to the thread.

Ah, thanks. How to handle a short write, i.e. with no error return=20
code, is exactly what I was looking for. This information is not in=20
the Linux write(2) man page.=20
=20
BTW, is it possible that after a short write the next write() syscall=20
for the remaining data will write some more data, possibly even the=20
whole remaining data successfully? I think of something like a pipe=20
or a socket where not the whole data can be sent immediately.=20

If the condition that prevented writing more data has a change of state bet= >ween the two calls, then sure, more data may be written successfully. For = >example, if some other process or thread freed up disk space between the ca= >lls than an ENOSPC or EDQUOT condition that caused a short-write may go awa= >y, permitting more data to be written.

Could the system decide to do short-writes for no good reason and then perm= >it more later, just because? Given the first sentence of the wording I quo= >ted, I think the answer is "no", but also think I'm uninterested in arguing=
about it and would just call such an implementation "low quality" and not =
use it.

It helps to remember that the write system call was used for more than
just disk-based files. Consider a line printer, for example, that
supports 132 column lines where the write system call writes more than
132 characters - the driver might report that as a short write.

Short writes on a socket file descriptor are also possible.

Short reads are common on serial devices such as magnetic tape or
other unit record-style devices.

Granted most of those cases are now shrouded in the mists of history.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Scott Lurndal wrote:

One may also catch the SIGXFSZ signal to ensure one detects file
size resource limit violations (see setrlimit(2), signal(7))

Simpler just to set SIGXFSZ to SIG_IGN so that write() will fail
with EFBIG (which can then be handled in the normal way).

--
Geoff Clare <netnews@gclare.org.uk>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)