what does XCU 2.9.1.1 part 1.d.i.a. mean? Does anyone follow it?

Discussion:

Philip Guenther

2014-07-01 07:54:32 UTC

XCU 2.9.1.1 describes how the shell does command seach and execution. To
quote:
------
1. If the command name does not contain any <slash> characters, the first
successful step in the following sequence shall occur:
a. If the command name matches the name of a special built-in
utility, that special built-in utility shall be invoked.
b. If the command name matches the name of a function known to
this shell, the function shall be invoked as described in
Section 2.9.5. If the implementation has provided a standard
utility in the form of a function, it shall not be recognized
at this point. It shall be invoked in conjunction with the path
search in step 1.d.
c. If the command name matches the name of a utility listed in the
following table, that utility shall be invoked.
alias false jobs read wait
bg fc kill true
cd fg newgrp umask
command getopts pwd unalias
d. Otherwise, the command shall be searched for using the PATH
environment variable as described in XBD Chapter 8:
i. If the search is successful:
a. If the system has implemented the utility as a regular
built-in or as a shell function, it shall be invoked at this
point in the path search.
b. Otherwise, the shell executes the utility in a separate utility
environment (see Section 2.12) with actions equivalent to
calling the execl() function as defined in <...>
...
ii. If the search is unsuccessful, the command shall fail with an
exit status of 127 and the shell shall write an error message.
------

Okay, so consider a shell which has 'test' as a built-in. It's not
special, let's assume there's no function definition, and its not in the
list in 1.c, so the shell has to do a search of $PATH per 1.d.

Question 1)
Part 1.d.i.a. *doesn't* refer to the search finding the utility in any
particular directory, such as a directory in the PATH required for
standards compliance. So it doesn't matter if it found a completely
different version of the utility; it can/should still invoke the
(standard-behaving) built-in version. Is that correct?

Question 2)
Part 1.d.ii says that if you override PATH with one that doesn't
include (versions of) the standard utilties, then built-ins not listed
in part 1c must not be invoked by the shell. For example, assuming
$HOME/empty doesn't exist and can be created, then this:
mkdir $HOME/empty
( unset -f test; PATH=$HOME/empty; test -d $HOME/empty )
echo $?
must output "127". Is that correct?

Checking of a few systems (linux, solaris 10, Mac OSX) find they seem to
answer my questions 1 and 2 with "yes" and "no", respectively. That is,
they treat all built-ins like the list in 1c, skipping the $PATH search
completely and rendering part 1.d.i.a. superfluous.

(If it matter neither whether the path search is successful, nor what it
finds when it is successful, then the search is pointless.)

Can we delete part 1.d.i.a ?

Philip Guenther

Nick Stoughton

2014-07-01 19:37:56 UTC

Permalink

As far as I can see all of the following shells fail to conform as you
describe:

ksh
bash
dash
tcsh
zsh

(some of these fail for lots of other reasons too!)

The only shell I have been able to find that *does* implement the standard
as required is "yash" (and then only in posixly-correct mode).

However, XRAT (page 3693, line 126346 onwards) has this to say:
--------------
The command search shown here does not match all historical
implementations. A more typical
sequence has been:
â¢ Any built-in (special or regular)
â¢ Functions
â¢ Path search for executable files
But there are problems with this sequence. Since the programmer has no idea
in advance which
utilities might have been built into the shell, a function cannot be used
to override portably a
utility of the same name. (For example, a function named cd cannot be
written for many
historical systems.) Furthermore, the PATH variable is partially
ineffective in this case, and only
a pathname with a <slash> can be used to ensure a specific executable file
is invoked.

After the execve( ) failure described, the shell normally executes the file
as a shell script. Some
implementations, however, attempt to detect whether the file is actually a
script and not an
executable from some other architecture. The method used by the KornShell
is allowed by the
text that indicates non-text files may be bypassed.

The sequence selected for the Shell and Utilities volume of POSIX.1-2008
acknowledges that
special built-ins cannot be overridden, but gives the programmer full
control over which
versions of other utilities are executed. It provides a means of
suppressing function lookup (via
the command utility) for the userâs own functions and ensures that any
regular built-ins or
functions provided by the implementation are under the control of the path
search. The
mechanisms for associating built-ins or functions with executable files in
the path are not
specified by the Shell and Utilities volume of POSIX.1-2008, but the
wording requires that if
either is implemented, the application is not able to distinguish a
function or built-in from an
executable (other than in terms of performance, presumably). The
implementation ensures that
all effects specified by the Shell and Utilities volume of POSIX.1-2008
resulting from the
invocation of the regular built-in or function (interaction with the
environment, variables, traps,
and so on) are identical to those resulting from the invocation of an
executable file.

------------
So it appears that this is deliberate, if largely un-implemented. I would
encourage you to file a bug at http://austingroupbugs.net for further
deliberation.

Post by Philip Guenther
XCU 2.9.1.1 describes how the shell does command seach and execution. To
------
1. If the command name does not contain any <slash> characters, the first
a. If the command name matches the name of a special built-in
utility, that special built-in utility shall be invoked.
b. If the command name matches the name of a function known to
this shell, the function shall be invoked as described in
Section 2.9.5. If the implementation has provided a standard
utility in the form of a function, it shall not be recognized
at this point. It shall be invoked in conjunction with the path
search in step 1.d.
c. If the command name matches the name of a utility listed in the
following table, that utility shall be invoked.
alias false jobs read wait
bg fc kill true
cd fg newgrp umask
command getopts pwd unalias
d. Otherwise, the command shall be searched for using the PATH
a. If the system has implemented the utility as a regular
built-in or as a shell function, it shall be invoked at this
point in the path search.
b. Otherwise, the shell executes the utility in a separate utility
environment (see Section 2.12) with actions equivalent to
calling the execl() function as defined in <...>
...
ii. If the search is unsuccessful, the command shall fail with an
exit status of 127 and the shell shall write an error message.
------
Okay, so consider a shell which has 'test' as a built-in. It's not
special, let's assume there's no function definition, and its not in the
list in 1.c, so the shell has to do a search of $PATH per 1.d.
Question 1)
Part 1.d.i.a. *doesn't* refer to the search finding the utility in any
particular directory, such as a directory in the PATH required for
standards compliance. So it doesn't matter if it found a completely
different version of the utility; it can/should still invoke the
(standard-behaving) built-in version. Is that correct?
Question 2)
Part 1.d.ii says that if you override PATH with one that doesn't
include (versions of) the standard utilties, then built-ins not listed
in part 1c must not be invoked by the shell. For example, assuming
mkdir $HOME/empty
( unset -f test; PATH=$HOME/empty; test -d $HOME/empty )
echo $?
must output "127". Is that correct?
Checking of a few systems (linux, solaris 10, Mac OSX) find they seem to
answer my questions 1 and 2 with "yes" and "no", respectively. That is,
they treat all built-ins like the list in 1c, skipping the $PATH search
completely and rendering part 1.d.i.a. superfluous.
(If it matter neither whether the path search is successful, nor what it
finds when it is successful, then the search is pointless.)
Can we delete part 1.d.i.a ?
Philip Guenther

Jilles Tjoelker

2014-07-02 21:29:29 UTC

Permalink

ksh93 implements 1.d.i.a for some builtins, although it also has some
builtins (POSIX and other) that are not found via PATH.

For example, ksh93 Version AJM 93u+ 2012-08-01 from FreeBSD ports can do
things like (after creating an executable named /opt/ast/bin/cat):

$ command -V cat
cat is a tracked alias for /bin/cat
$ (PATH=/opt/ast/bin:$PATH; command -V cat)
cat is a shell builtin version of /opt/ast/bin/cat
$ (PATH=/opt/ast/bin:$PATH; cat -\?)
Usage: cat [-bdenstuvABDERST] [file ...]
$ cat -\?
cat: illegal option -- ?
usage: cat [-belnstuv] [file ...]
$ PATH=/var/empty cat -\?
ksh93: cat: not found [No such file or directory]

I think this is what is meant by 1.d.i.a.

This ksh93 feature allows scripts to opt-in to builtins that may not be
compatible with the utilities provided with the system (for example, the
builtin version does not support FreeBSD's -l option). For a shell that
is more tightly coupled with the rest of the system, this seems less
useful.

The apparent goal of allowing applications to be certain of invoking
their own executables is not achieved by almost all shells, and may not
be desirable either (how about calling utilities implemented as shell
scripts using a non-standard PATH?).

--
Jilles Tjoelker