Discussion:
pipe() file descriptor value ordering?
Matthew Dempsky
2014-04-09 21:39:13 UTC
Permalink
pipe() says "Their [filedes[0] and filedes[1]] integer values shall be
the two lowest available at the time of the pipe() call."

My reading is filedes[0] and filedes[1] together need to be the two
lowest available descriptors, but there doesn't seem to be any
normative requirement about which of the two will be lower. E.g., is
it valid for filedes[0] to end up greater than filedes[1]? Does it
make a difference either way if we allow for other threads making
concurrent calls to close()?
Eric Blake
2014-04-09 22:04:00 UTC
Permalink
Post by Matthew Dempsky
pipe() says "Their [filedes[0] and filedes[1]] integer values shall be
the two lowest available at the time of the pipe() call."
My reading is filedes[0] and filedes[1] together need to be the two
lowest available descriptors, but there doesn't seem to be any
normative requirement about which of the two will be lower. E.g., is
it valid for filedes[0] to end up greater than filedes[1]? Does it
make a difference either way if we allow for other threads making
concurrent calls to close()?
I see no reason why things wouldn't work if filedes[0] > filedes[1],
even though that is not the traditional behavior.

pipe() and close() are required to act atomically with regards to each
other - either both or none of the file descriptors have been allocated
at the time pipe() completes. I don't see how this can affect your
question.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
Matthew Dempsky
2014-04-09 22:19:22 UTC
Permalink
Post by Eric Blake
I see no reason why things wouldn't work if filedes[0] > filedes[1],
even though that is not the traditional behavior.
Yep, that's my impression as well.
Post by Eric Blake
pipe() and close() are required to act atomically with regards to each
other - either both or none of the file descriptors have been allocated
at the time pipe() completes. I don't see how this can affect your
question.
My thought line was suppose a hypothetical implementation of pipe()
that looks something like:

int pipe(int *filedes) {
int rfd = open("/dev/pipemux", O_RDONLY);
if (rfd == -1) return -1;

int wfd = open(write_end_of_pipe(rfd), O_WRONLY);
if (wfd == -1) { close(rfd); return -1; }

filedes[0] = rfd;
filedes[1] = wfd;
}

(I.e., the two file descriptors are allocated sequentially and without
locking the file descriptor table.)

In such an implementation the file descriptors would be guaranteed to
be in increasing order in a single-threaded app, but a multi-threaded
app might subvert that guarantee by concurrently calling close() and
freeing up a lower file descriptor.
Geoff Clare
2014-04-10 08:47:46 UTC
Permalink
Post by Eric Blake
pipe() and close() are required to act atomically with regards to each
other - either both or none of the file descriptors have been allocated
at the time pipe() completes.
Where does the standard say that?
--
Geoff Clare <g.clare-7882/***@public.gmane.org>
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Eric Blake
2014-04-10 12:15:07 UTC
Permalink
Post by Geoff Clare
Post by Eric Blake
pipe() and close() are required to act atomically with regards to each
other - either both or none of the file descriptors have been allocated
at the time pipe() completes.
Where does the standard say that?
Oh, I guess it doesn't. I was thinking that since both functions were
async-signal-safe (XSH 2.4.3) that someone calling pipe() in the main
program and then close() in the signal handler could not cause the
second pipe descriptor to be lower than the first in a single-threaded
program.

But now that I look, I see XSH 2.9.7, which lists the set of functions
that MUST be atomic with respect to one another; close() is on that
list, but pipe() is not. So close() in the middle of pipe() could
indeed make pipe() appear to behave differently. Or is it that pipe()
is not on that list because that list only talks about operations on
regular files or symlinks, but pipe() is neither?
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
Geoff Clare
2014-04-10 13:12:30 UTC
Permalink
Post by Eric Blake
Post by Geoff Clare
Post by Eric Blake
pipe() and close() are required to act atomically with regards to each
other - either both or none of the file descriptors have been allocated
at the time pipe() completes.
Where does the standard say that?
Oh, I guess it doesn't.
Okay, just wanted to check I hadn't missed something.
Post by Eric Blake
I was thinking that since both functions were
async-signal-safe (XSH 2.4.3) that someone calling pipe() in the main
program and then close() in the signal handler could not cause the
second pipe descriptor to be lower than the first in a single-threaded
program.
But now that I look, I see XSH 2.9.7, which lists the set of functions
that MUST be atomic with respect to one another; close() is on that
list, but pipe() is not. So close() in the middle of pipe() could
indeed make pipe() appear to behave differently. Or is it that pipe()
is not on that list because that list only talks about operations on
regular files or symlinks, but pipe() is neither?
I'm sure the reason pipe() is not on that list is just because the list
relates to operations on regular files and symlinks.
--
Geoff Clare <g.clare-7882/***@public.gmane.org>
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Casper.Dik-QHcLZuEGTsvQT0dZR+
2014-04-10 07:28:37 UTC
Permalink
Post by Matthew Dempsky
pipe() says "Their [filedes[0] and filedes[1]] integer values shall be
the two lowest available at the time of the pipe() call."
My reading is filedes[0] and filedes[1] together need to be the two
lowest available descriptors, but there doesn't seem to be any
normative requirement about which of the two will be lower. E.g., is
it valid for filedes[0] to end up greater than filedes[1]? Does it
make a difference either way if we allow for other threads making
concurrent calls to close()?
In a threaded application, it is not possible to determine whether the two
file descriptors just returned are the lowest available as files might
have been closed before the check is done.

In a single threaded application it can be determined, so the guarantee
can only been verified in single threaded application.

I don't see a reason why we can't get fd[0] > fd[1]; seems like a
implementation detail.

Casper
Wojtek Lerch
2014-04-10 13:28:32 UTC
Permalink
Post by Matthew Dempsky
My reading is filedes[0] and filedes[1] together need to be the two
lowest available descriptors, but there doesn't seem to be any
normative requirement about which of the two will be lower. E.g., is
it valid for filedes[0] to end up greater than filedes[1]? Does it
make a difference either way if we allow for other threads making
concurrent calls to close()?
In a threaded application, it is not possible to determine whether the two file
descriptors just returned are the lowest available as files might have been
closed before the check is done.
If a program has two threads that call pipe() while no other thread calls close(), can one thread get fds 3 and 6 and the other 4 and 5?
Geoff Clare
2014-04-10 13:57:54 UTC
Permalink
Post by Wojtek Lerch
Post by Matthew Dempsky
My reading is filedes[0] and filedes[1] together need to be the two
lowest available descriptors, but there doesn't seem to be any
normative requirement about which of the two will be lower. E.g., is
it valid for filedes[0] to end up greater than filedes[1]? Does it
make a difference either way if we allow for other threads making
concurrent calls to close()?
In a threaded application, it is not possible to determine whether the two file
descriptors just returned are the lowest available as files might have been
closed before the check is done.
If a program has two threads that call pipe() while no other thread
calls close(), can one thread get fds 3 and 6 and the other 4 and 5?
I don't think this behaviour would conform to the requirement: "Their
integer values shall be the two lowest available at the time of the
pipe() call."
--
Geoff Clare <g.clare-7882/***@public.gmane.org>
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Wojtek Lerch
2014-04-10 14:15:48 UTC
Permalink
Post by Geoff Clare
Post by Wojtek Lerch
If a program has two threads that call pipe() while no other thread
calls close(), can one thread get fds 3 and 6 and the other 4 and 5?
I don't think this behaviour would conform to the requirement: "Their
integer values shall be the two lowest available at the time of the
pipe() call."
I don't know -- can't the two calls happen *at the same time*? Note that pipe() is not on the list of "atomic" functions in 2.9.7; is there another promise of atomicity elsewhere that applies to pipe()?

http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_09_07
Casper.Dik-QHcLZuEGTsvQT0dZR+
2014-04-10 14:16:13 UTC
Permalink
Post by Geoff Clare
Post by Wojtek Lerch
Post by Matthew Dempsky
My reading is filedes[0] and filedes[1] together need to be the two
lowest available descriptors, but there doesn't seem to be any
normative requirement about which of the two will be lower. E.g., is
it valid for filedes[0] to end up greater than filedes[1]? Does it
make a difference either way if we allow for other threads making
concurrent calls to close()?
In a threaded application, it is not possible to determine whether the two file
descriptors just returned are the lowest available as files might have been
closed before the check is done.
If a program has two threads that call pipe() while no other thread
calls close(), can one thread get fds 3 and 6 and the other 4 and 5?
I don't think this behaviour would conform to the requirement: "Their
integer values shall be the two lowest available at the time of the
pipe() call."
Are you suggesting that a threaded application can't call pipe or that
two concurrent calls to pipe return the same descriptors?

By your argument two concurrent calls could also not get 3 & 4 and 5 & 6
as the second concurrent pipe call does not return the two lowest
available fds.

Let me suggest that the "lowest available descriptor" makes little sense
in a process with more than one thread; as you cannot even determine
what the lowest available descriptor is before or after calling the
function as other threads can be do anything.

Casper
Geoff Clare
2014-04-10 14:34:30 UTC
Permalink
Post by Casper.Dik-QHcLZuEGTsvQT0dZR+
Post by Geoff Clare
Post by Wojtek Lerch
If a program has two threads that call pipe() while no other thread
calls close(), can one thread get fds 3 and 6 and the other 4 and 5?
I don't think this behaviour would conform to the requirement: "Their
integer values shall be the two lowest available at the time of the
pipe() call."
Are you suggesting that a threaded application can't call pipe or that
two concurrent calls to pipe return the same descriptors?
Neither.
Post by Casper.Dik-QHcLZuEGTsvQT0dZR+
By your argument two concurrent calls could also not get 3 & 4 and 5 & 6
as the second concurrent pipe call does not return the two lowest
available fds.
Let me suggest that the "lowest available descriptor" makes little sense
in a process with more than one thread; as you cannot even determine
what the lowest available descriptor is before or after calling the
function as other threads can be do anything.
Interesting point. I think the only reasonable interpretation of
"the [two] lowest available descriptor(s)" is that when two threads
call pipe() (or open() or whatever) at the same time, the competition
for the lowest file descriptor(s) is "won" by one of them and the
winner is treated as as if it was called before the loser. Perhaps
we should add some words to that effect (but more formal) somewhere.
--
Geoff Clare <g.clare-7882/***@public.gmane.org>
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Casper.Dik-QHcLZuEGTsvQT0dZR+
2014-04-10 14:55:24 UTC
Permalink
Post by Geoff Clare
Interesting point. I think the only reasonable interpretation of
"the [two] lowest available descriptor(s)" is that when two threads
call pipe() (or open() or whatever) at the same time, the competition
for the lowest file descriptor(s) is "won" by one of them and the
winner is treated as as if it was called before the loser. Perhaps
we should add some words to that effect (but more formal) somewhere.
But should the standard be written such that returning 3 & 4 and 5 & 6 is
the only valid outcome?

I'd suggest that 3 & 5 and 4 & 6 as well as 3 & 6 and 4 & 5 should be
equally valid.

It makes little sense to force a lock on "get a new file descriptor"
during a part of the call of pipe() when new file descriptors are
allocated.

In Solaris, it seems that all possible outcomes listed here look possible.
I see no reason why the standard should insist on a single-threaded view
of the world in those cases were a program cannot reliably detect whether
the standard is being followed or not.

Casper
Geoff Clare
2014-04-11 09:30:32 UTC
Permalink
Post by Casper.Dik-QHcLZuEGTsvQT0dZR+
Post by Geoff Clare
Interesting point. I think the only reasonable interpretation of
"the [two] lowest available descriptor(s)" is that when two threads
call pipe() (or open() or whatever) at the same time, the competition
for the lowest file descriptor(s) is "won" by one of them and the
winner is treated as as if it was called before the loser. Perhaps
we should add some words to that effect (but more formal) somewhere.
But should the standard be written such that returning 3 & 4 and 5 & 6 is
the only valid outcome?
I'd suggest that 3 & 5 and 4 & 6 as well as 3 & 6 and 4 & 5 should be
equally valid.
It makes little sense to force a lock on "get a new file descriptor"
during a part of the call of pipe() when new file descriptors are
allocated.
In Solaris, it seems that all possible outcomes listed here look possible.
I see no reason why the standard should insist on a single-threaded view
of the world in those cases were a program cannot reliably detect whether
the standard is being followed or not.
This was discussed in yesterday's teleconference and the consensus was
in agreement with your views here. I.e. each individual fd allocation
should be atomic and there should be no requirement for pipe() to
allocate both fds in one atomic operation.
--
Geoff Clare <g.clare-7882/***@public.gmane.org>
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Michael Kerrisk
2014-04-11 13:33:10 UTC
Permalink
I've only just caught up with this thread...
Post by Geoff Clare
Post by Casper.Dik-QHcLZuEGTsvQT0dZR+
Post by Geoff Clare
Interesting point. I think the only reasonable interpretation of
"the [two] lowest available descriptor(s)" is that when two threads
call pipe() (or open() or whatever) at the same time, the competition
for the lowest file descriptor(s) is "won" by one of them and the
winner is treated as as if it was called before the loser. Perhaps
we should add some words to that effect (but more formal) somewhere.
But should the standard be written such that returning 3 & 4 and 5 & 6 is
the only valid outcome?
I'd suggest that 3 & 5 and 4 & 6 as well as 3 & 6 and 4 & 5 should be
equally valid.
It makes little sense to force a lock on "get a new file descriptor"
during a part of the call of pipe() when new file descriptors are
allocated.
In Solaris, it seems that all possible outcomes listed here look possible.
I see no reason why the standard should insist on a single-threaded view
of the world in those cases were a program cannot reliably detect whether
the standard is being followed or not.
This was discussed in yesterday's teleconference and the consensus was
in agreement with your views here. I.e. each individual fd allocation
should be atomic and there should be no requirement for pipe() to
allocate both fds in one atomic operation.
I wonder if things are so straightforward. The following point came up
in a conversation I had this week.

There's a lot of code out there that has sequences like this to set
stdout to be the write end of a pipe (or analogously, stdin to be the
read end of a pipe):

int pfd[2];
pipe(pfd);

if (pfd[1] != STDOUT_FILENO) { /* Defensive check */
dup2(pfd[1], STDOUT_FILENO);
close(pfd[1]);
}

If file descriptors 0 and 1 are closed before the call to pipe(), then
the above sequence is broken if pipe() returns the file descriptors in
unnatural order--that is, 1 in pfd[0] and 0 in pfd[1], since we end up
doing the following

dup2(0, 1); /* Close FD 0, duplicate it on FD 1 */
close(0); /* No-op */

Is it really meant that POSIX is saying that all of the existing code
that follows the above pattern is broken, or is there perhaps some
oher point in the standard that I am missing?

Thanks,

Michael
Geoff Clare
2014-04-14 09:25:44 UTC
Permalink
Post by Michael Kerrisk
Post by Geoff Clare
Post by Casper.Dik-QHcLZuEGTsvQT0dZR+
Post by Geoff Clare
Interesting point. I think the only reasonable interpretation of
"the [two] lowest available descriptor(s)" is that when two threads
call pipe() (or open() or whatever) at the same time, the competition
for the lowest file descriptor(s) is "won" by one of them and the
winner is treated as as if it was called before the loser. Perhaps
we should add some words to that effect (but more formal) somewhere.
But should the standard be written such that returning 3 & 4 and 5 & 6 is
the only valid outcome?
I'd suggest that 3 & 5 and 4 & 6 as well as 3 & 6 and 4 & 5 should be
equally valid.
It makes little sense to force a lock on "get a new file descriptor"
during a part of the call of pipe() when new file descriptors are
allocated.
In Solaris, it seems that all possible outcomes listed here look possible.
I see no reason why the standard should insist on a single-threaded view
of the world in those cases were a program cannot reliably detect whether
the standard is being followed or not.
This was discussed in yesterday's teleconference and the consensus was
in agreement with your views here. I.e. each individual fd allocation
should be atomic and there should be no requirement for pipe() to
allocate both fds in one atomic operation.
I wonder if things are so straightforward. The following point came up
in a conversation I had this week.
There's a lot of code out there that has sequences like this to set
stdout to be the write end of a pipe (or analogously, stdin to be the
int pfd[2];
pipe(pfd);
if (pfd[1] != STDOUT_FILENO) { /* Defensive check */
dup2(pfd[1], STDOUT_FILENO);
close(pfd[1]);
}
If file descriptors 0 and 1 are closed before the call to pipe(), then
the above sequence is broken if pipe() returns the file descriptors in
unnatural order--that is, 1 in pfd[0] and 0 in pfd[1], since we end up
doing the following
dup2(0, 1); /* Close FD 0, duplicate it on FD 1 */
close(0); /* No-op */
Should be:

dup2(0, 1); /* Close FD 1, duplicate FD 0 on FD 1 */
close(0); /* Close FD 0 */
Post by Michael Kerrisk
Is it really meant that POSIX is saying that all of the existing code
that follows the above pattern is broken, or is there perhaps some
other point in the standard that I am missing?
While I agree that code like the above is common, I'm doubtful that
it is ever used with fds 0 and 1 both closed. The usual situation
where a pipe is set up with fds 0 and 1 as its read and write ends
is to connect two processes so that one reads the output of the other.
The first process has the write end as fd 1; the second has the read
end as fd 0; there is no process that has the two ends of the pipe
on fd 0 and fd 1, so closing them before calling pipe() seems like a
very odd thing to do. Can you point to any real code that actually
does that?

I suppose if we do want to cater to applications that do it, then all
the standard would need to do is require that pipe() allocates pfd[0]
before it allocates pfd[1]. In a single-threaded process with no
signal handlers that manipulate file descriptors, this would
guarantee that closing fds 0 and 1 before calling pipe() puts 0 in
pfd[0] and 1 in pfd[1].
--
Geoff Clare <g.clare-7882/***@public.gmane.org>
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Eric Blake
2014-04-14 13:27:57 UTC
Permalink
Post by Geoff Clare
I suppose if we do want to cater to applications that do it, then all
the standard would need to do is require that pipe() allocates pfd[0]
before it allocates pfd[1]. In a single-threaded process with no
signal handlers that manipulate file descriptors, this would
guarantee that closing fds 0 and 1 before calling pipe() puts 0 in
pfd[0] and 1 in pfd[1].
We already state in TC1 in quite clear terms that any attempt to start a
process with fd 0, 1, or 2 closed results in undefined behavior. I
think this is yet another case of that. No need to mandate the order
between pfd[0] and pfd[1], because any app closing fd 0 deserves what
they get with regards to undefined behavior.

Regarding quality of implementation, any implementation where pfd[0] >
pfd[1] _should_ take care to properly merge those fds into fd 0 and 1
during popen() as appropriate - but that's not something we can or
should mandate.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
Geoff Clare
2014-04-14 14:05:50 UTC
Permalink
Post by Eric Blake
Post by Geoff Clare
I suppose if we do want to cater to applications that do it, then all
the standard would need to do is require that pipe() allocates pfd[0]
before it allocates pfd[1]. In a single-threaded process with no
signal handlers that manipulate file descriptors, this would
guarantee that closing fds 0 and 1 before calling pipe() puts 0 in
pfd[0] and 1 in pfd[1].
We already state in TC1 in quite clear terms that any attempt to start a
process with fd 0, 1, or 2 closed results in undefined behavior. I
think this is yet another case of that. No need to mandate the order
between pfd[0] and pfd[1], because any app closing fd 0 deserves what
they get with regards to undefined behavior.
In the scenario in question, the app does not do anything with fds 0
and 1 closed except call pipe() to open them again. It isn't affected
by that stuff we added in TC1 about closing fd 0, 1, or 2.
--
Geoff Clare <g.clare-7882/***@public.gmane.org>
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Michael Kerrisk
2014-04-15 05:33:25 UTC
Permalink
Eric,
Post by Eric Blake
Post by Geoff Clare
I suppose if we do want to cater to applications that do it, then all
the standard would need to do is require that pipe() allocates pfd[0]
before it allocates pfd[1]. In a single-threaded process with no
signal handlers that manipulate file descriptors, this would
guarantee that closing fds 0 and 1 before calling pipe() puts 0 in
pfd[0] and 1 in pfd[1].
We already state in TC1 in quite clear terms that any attempt to start a
process with fd 0, 1, or 2 closed results in undefined behavior. I
think this is yet another case of that. No need to mandate the order
between pfd[0] and pfd[1], because any app closing fd 0 deserves what
they get with regards to undefined behavior.
I realize that Geoff pointed out that this text in TC1 probably does
not apply, but I am curious: where is that text in TC1?

Thanks,

Michael
Nick Stoughton
2014-04-16 20:00:24 UTC
Permalink
Page 778, lines 26071-26076 in the combined base+TC1 standard read:

If file descriptor 0, 1, or 2 would otherwise be closed after a successful
call to one of the exec
family of functions, implementations may open an unspecified file for the
file descriptor in the
new process image. If a standard utility or a conforming application is
executed with file
descriptor 0 not open for reading or with file descriptor 1 or 2 not open
for writing, the
environment in which the utility or application is executed shall be deemed
non-conforming,
and consequently the utility or application might not behave as described
in this standard.
Post by Michael Kerrisk
Eric,
Post by Eric Blake
Post by Geoff Clare
I suppose if we do want to cater to applications that do it, then all
the standard would need to do is require that pipe() allocates pfd[0]
before it allocates pfd[1]. In a single-threaded process with no
signal handlers that manipulate file descriptors, this would
guarantee that closing fds 0 and 1 before calling pipe() puts 0 in
pfd[0] and 1 in pfd[1].
We already state in TC1 in quite clear terms that any attempt to start a
process with fd 0, 1, or 2 closed results in undefined behavior. I
think this is yet another case of that. No need to mandate the order
between pfd[0] and pfd[1], because any app closing fd 0 deserves what
they get with regards to undefined behavior.
I realize that Geoff pointed out that this text in TC1 probably does
not apply, but I am curious: where is that text in TC1?
Thanks,
Michael
Michael Kerrisk
2014-04-17 11:04:57 UTC
Permalink
Thanks, Nick.


On Wed, Apr 16, 2014 at 10:00 PM, Nick Stoughton
Post by Nick Stoughton
If file descriptor 0, 1, or 2 would otherwise be closed after a successful
call to one of the exec
family of functions, implementations may open an unspecified file for the
file descriptor in the
new process image. If a standard utility or a conforming application is
executed with file
descriptor 0 not open for reading or with file descriptor 1 or 2 not open
for writing, the
environment in which the utility or application is executed shall be deemed
non-conforming,
and consequently the utility or application might not behave as described in
this standard.
Post by Michael Kerrisk
Eric,
Post by Eric Blake
Post by Geoff Clare
I suppose if we do want to cater to applications that do it, then all
the standard would need to do is require that pipe() allocates pfd[0]
before it allocates pfd[1]. In a single-threaded process with no
signal handlers that manipulate file descriptors, this would
guarantee that closing fds 0 and 1 before calling pipe() puts 0 in
pfd[0] and 1 in pfd[1].
We already state in TC1 in quite clear terms that any attempt to start a
process with fd 0, 1, or 2 closed results in undefined behavior. I
think this is yet another case of that. No need to mandate the order
between pfd[0] and pfd[1], because any app closing fd 0 deserves what
they get with regards to undefined behavior.
I realize that Geoff pointed out that this text in TC1 probably does
not apply, but I am curious: where is that text in TC1?
Thanks,
Michael
Michael Kerrisk
2014-04-15 05:31:39 UTC
Permalink
Hi Geoff,
Post by Geoff Clare
Post by Michael Kerrisk
Post by Geoff Clare
Post by Casper.Dik-QHcLZuEGTsvQT0dZR+
Post by Geoff Clare
Interesting point. I think the only reasonable interpretation of
"the [two] lowest available descriptor(s)" is that when two threads
call pipe() (or open() or whatever) at the same time, the competition
for the lowest file descriptor(s) is "won" by one of them and the
winner is treated as as if it was called before the loser. Perhaps
we should add some words to that effect (but more formal) somewhere.
But should the standard be written such that returning 3 & 4 and 5 & 6 is
the only valid outcome?
I'd suggest that 3 & 5 and 4 & 6 as well as 3 & 6 and 4 & 5 should be
equally valid.
It makes little sense to force a lock on "get a new file descriptor"
during a part of the call of pipe() when new file descriptors are
allocated.
In Solaris, it seems that all possible outcomes listed here look possible.
I see no reason why the standard should insist on a single-threaded view
of the world in those cases were a program cannot reliably detect whether
the standard is being followed or not.
This was discussed in yesterday's teleconference and the consensus was
in agreement with your views here. I.e. each individual fd allocation
should be atomic and there should be no requirement for pipe() to
allocate both fds in one atomic operation.
I wonder if things are so straightforward. The following point came up
in a conversation I had this week.
There's a lot of code out there that has sequences like this to set
stdout to be the write end of a pipe (or analogously, stdin to be the
int pfd[2];
pipe(pfd);
if (pfd[1] != STDOUT_FILENO) { /* Defensive check */
dup2(pfd[1], STDOUT_FILENO);
close(pfd[1]);
}
If file descriptors 0 and 1 are closed before the call to pipe(), then
the above sequence is broken if pipe() returns the file descriptors in
unnatural order--that is, 1 in pfd[0] and 0 in pfd[1], since we end up
doing the following
dup2(0, 1); /* Close FD 0, duplicate it on FD 1 */
close(0); /* No-op */
dup2(0, 1); /* Close FD 1, duplicate FD 0 on FD 1 */
close(0); /* Close FD 0 */
(Yup, thanks,)
Post by Geoff Clare
Post by Michael Kerrisk
Is it really meant that POSIX is saying that all of the existing code
that follows the above pattern is broken, or is there perhaps some
other point in the standard that I am missing?
While I agree that code like the above is common, I'm doubtful that
it is ever used with fds 0 and 1 both closed.
(Ack.)
Post by Geoff Clare
The usual situation
where a pipe is set up with fds 0 and 1 as its read and write ends
is to connect two processes so that one reads the output of the other.
The first process has the write end as fd 1; the second has the read
end as fd 0; there is no process that has the two ends of the pipe
on fd 0 and fd 1, so closing them before calling pipe() seems like a
very odd thing to do. Can you point to any real code that actually
does that?
No, I know of no real world example. But I was standing up in front of
a group explaining the technique above, when someone asked, "what
if...". And it got me thinking: what does the standard allow/deny?
Post by Geoff Clare
I suppose if we do want to cater to applications that do it, then all
the standard would need to do is require that pipe() allocates pfd[0]
before it allocates pfd[1]. In a single-threaded process with no
signal handlers that manipulate file descriptors, this would
guarantee that closing fds 0 and 1 before calling pipe() puts 0 in
pfd[0] and 1 in pfd[1].
I suppose the only argument I have in favor of a change to the
standard is that code like this:

if (pfd[1] != STDOUT_FILENO) { /* Defensive check */
dup2(pfd[1], STDOUT_FILENO);
close(pfd[1]);
}


is already predicated on handling the possibility that FD1 may have
been closed before the pipe call, which implies that FD 0 could also
have been closed before the pipe() call (and therefore indeed pfd[0]
might be using FD 0). It's not a compelling argument I agree. On the
other hand, this does seem like a loophole in the standard, and
perhaps your suggested change would be a suitable fix.

Cheers,

Michael
Ranjit Singh
2014-04-27 15:49:15 UTC
Permalink
Post by Geoff Clare
I suppose if we do want to cater to applications that do it, then all
the standard would need to do is require that pipe() allocates pfd[0]
before it allocates pfd[1]. In a single-threaded process with no
signal handlers that manipulate file descriptors, this would
guarantee that closing fds 0 and 1 before calling pipe() puts 0 in
pfd[0] and 1 in pfd[1].
I think that's a perfectly reasonable requirement. A note summarising
the discussion in Rationale would be useful too, so people know that
in a multi-threaded situation, it's on them, but in a single-threaded
process the traditional behaviour is guaranteed.

Regards,
Ranjit.
--
"One can be a gentleman, without being a push-over."
Joerg Schilling
2014-04-10 14:55:56 UTC
Permalink
Post by Casper.Dik-QHcLZuEGTsvQT0dZR+
Post by Geoff Clare
Post by Wojtek Lerch
If a program has two threads that call pipe() while no other thread
calls close(), can one thread get fds 3 and 6 and the other 4 and 5?
I don't think this behaviour would conform to the requirement: "Their
integer values shall be the two lowest available at the time of the
pipe() call."
Are you suggesting that a threaded application can't call pipe or that
two concurrent calls to pipe return the same descriptors?
By your argument two concurrent calls could also not get 3 & 4 and 5 & 6
as the second concurrent pipe call does not return the two lowest
available fds.
A pipe() call takes some time and on a threaded environment, two pipe() calls
could happen at the same time as one or more close() calls.

I see no reson why the first pipe fd could not be allocated before a close()
call that then frees a lower fd for the second fd pof the pipe array.

Jörg
--
EMail:joerg-3Qm2Liu6aU2sY6utFDHCwYAplN+***@public.gmane.org (home) Jörg Schilling D-13353 Berlin
js-CFLBMwTPW48UNGrzBIF7/***@public.gmane.org (uni)
joerg.schilling-8LS2qeF34IpklNlQbfROjRvVK+***@public.gmane.org (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Loading...