$((x)) vs $(($x)) and -/+ operators

[Stephane Chazelas]

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04

If the shell variable x contains a value that forms a valid
integer constant, optionally including a leading plus or
minus sign, then the arithmetic expansions "$((x))" and
"$(($x))" shall return the same value.
* The sizeof() operator and the prefix and postfix "++" and "--"
operators are not required

That would seem to imply that

$((a-b)) should work the same as $(($a-$b)) as well.

However, that's not the case for values of $b like "-1" in all the
shells I have tried (yash, zsh, ksh93, mksh, dash, bash, zsh).

In those shells,

a=1 b=-1; "$((a-b))"

is treated as $((1 - -1))

While $(($a-$b)) which is expanded as $((1--1)) is considered as
a syntax error because "--" is treated as the decrementing unary
operator.

I'd argue it's a bug in those shells.

If not, then maybe the standard should make it a requirement
that the "-" and "+" binary operators be surrounded by blank
characters (or braces used) if operands might start with "-" or
"+".

Interestingly. $((1--(1+1))) doesn't fail in bash or dash but
fails in ksh93, zsh, mksh and yash.

GNU and Solaris bc, gawk, mawk, nawk also fail on 1--1 and
1--(1+1)

oawk:

$ awk 'BEGIN{print 1--1;exit}'
10

There's a problem with the "-" unary operator as well:

a=-1; echo "$((-$a))"

fails in everything except dash and bash.

In any case, I suspect the specification for sh, bc and awk
should probably be updated as from my reading, it seems they
only say blanks are ignored and don't make a difference wrt the
meaning of the arithmetic expression, while obviously something
has to be done to be able to disambiguate --x between the unary
-- and the unary - applied twice (so --x vs - -x or -(-x)).

All shells except bash and dash complain on $((1---a))

Some more fun:

$ a=1 dash -c 'echo "$((----a)) $a"'
1 1
$ a=1 bash -c 'echo "$((----a)) $a"'
0 0
$ a=1 zsh -c 'echo "$((----a)) $a"'
-1 -1

More generally I'd argue that saying $(($x)) and $((x)) expand
to the same value is not enough to say how one may use one in
place or the other. For instance $(($x = 2)) would obviously not
work. But would a=foo; : "$(( a = 1 ))" be specified?

[Geoff Clare]

Post by Stephane Chazelas
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04

If the shell variable x contains a value that forms a valid
integer constant, optionally including a leading plus or
minus sign, then the arithmetic expansions "$((x))" and
"$(($x))" shall return the same value.
* The sizeof() operator and the prefix and postfix "++" and "--"
operators are not required

That would seem to imply that
$((a-b)) should work the same as $(($a-$b)) as well.

No, that doesn't follow. The point of specifying that $((x)) and
$(($x)) return the same value was to clarify that $((x)) has to
treat leading 0 as octal and 0x as hex. Specifying it for just
the simple $((x)) case is enough to ensure that bare variables
are always treated this way. It was never meant to imply any
more than that.

Post by Stephane Chazelas
However, that's not the case for values of $b like "-1" in all the
shells I have tried (yash, zsh, ksh93, mksh, dash, bash, zsh).
In those shells,
a=1 b=-1; "$((a-b))"
is treated as $((1 - -1))
While $(($a-$b)) which is expanded as $((1--1)) is considered as
a syntax error because "--" is treated as the decrementing unary
operator.
I'd argue it's a bug in those shells.

No, that's exactly the behaviour I would expect in a shell that
supports the "--" operator.

Post by Stephane Chazelas
If not, then maybe the standard should make it a requirement
that the "-" and "+" binary operators be surrounded by blank
characters (or braces used) if operands might start with "-" or
"+".

Yes, the onus is on applications to ensure that if they use expansions
within arithmetic expressions, the results will have the syntax they
intended. I.e. if $b might start with a "-" they should use
$(($a - $b)) or $(($a-($b))).

[...]

Post by Stephane Chazelas
More generally I'd argue that saying $(($x)) and $((x)) expand
to the same value is not enough to say how one may use one in
place or the other.

I agree, we should update the requirement to cover the general
case instead of only the specific $((x)) case. Something like:

If the shell variable x contains a value that forms a valid
integer constant, optionally including a leading plus or
minus sign, then when the variable name x is used in an
arithmetic expansion and is not being assigned to, it shall
yield the same value as ($x). For example, if x=-010 then
$((10-x)) is equivalent to $((10-(-010))) which expands to 18.

Post by Stephane Chazelas
For instance $(($x = 2)) would obviously not
work. But would a=foo; : "$(( a = 1 ))" be specified?

$(( a = 1 )) sets the shell variable a to the value 1, regardless
of whether it previously had a value (assuming it is not readonly).

[Stephane Chazelas]

2014-10-21 12:46:44 +0100, Geoff Clare:
[...]

Post by Geoff Clare

Post by Stephane Chazelas
While $(($a-$b)) which is expanded as $((1--1)) is considered as
a syntax error because "--" is treated as the decrementing unary
operator.
I'd argue it's a bug in those shells.

No, that's exactly the behaviour I would expect in a shell that
supports the "--" operator.

My thinking was that I could not find anything in the spec that
said that 1--1 was any different from 1 - -1 (especially
considering that the "--" operator is *not* in the sh arithmetic
syntax (though allowed by interpreters as an extension)) like
1-+1 is the same as 1 - +1, but now, thinking more about it, I
suppose that's addressed somewhere in the spec or possibly the C
spec by the way tokenisation is done.

I could not find a reference to that, but I'd expect at least
the C spec to specify for instance how a---b is tokenised (like
"a" "--" "-" b or "a" "-" "--" b or unspecified (to allow
implementations to implement a "---" operator))

So I agree it's a non-issue.

[...]

Post by Geoff Clare
Yes, the onus is on applications to ensure that if they use expansions
within arithmetic expressions, the results will have the syntax they
intended. I.e. if $b might start with a "-" they should use
$(($a - $b)) or $(($a-($b))).

[...]

Such a note might be worth adding to the spec, as it's not
necessarily obvious. Might even be worth saying that $((a - b))
should be prefered over $(($a - $b)).

[...]

Post by Geoff Clare

Post by Stephane Chazelas
For instance $(($x = 2)) would obviously not
work. But would a=foo; : "$(( a = 1 ))" be specified?

$(( a = 1 )) sets the shell variable a to the value 1, regardless
of whether it previously had a value (assuming it is not readonly).

[...]

OK. The reason I mentions that is that the only place that
specifies that using variable names (without the leading "$") in
arithmetic expressions is valid is that $((x)) == $(($x)) above,
with the restriction that $x has to contain integer constants.

Also, it's not all variable names that are allowed. Obviously
$-, $*, $?, $! (without "$") are not, nor is $@, $# is only
recognised by zsh (in mksh, it seems to be ignored).

[Nick Stoughton]

FYI: The C standard does specify how a---b is handled: postfix operators
have higher precedence than unary operators, so it will be evaluated as a--
- b. If you read the C standard, the order of clauses in 6.5 (Expressions)
is the order of precedence (see footnote 85 in C11). For that reason,
"a--b" is not a valid C expression;

On Tue, Oct 21, 2014 at 6:20 AM, Stephane Chazelas <

Post by Stephane Chazelas
[...]

Post by Geoff Clare

Post by Stephane Chazelas
While $(($a-$b)) which is expanded as $((1--1)) is considered as
a syntax error because "--" is treated as the decrementing unary
operator.
I'd argue it's a bug in those shells.

No, that's exactly the behaviour I would expect in a shell that
supports the "--" operator.

My thinking was that I could not find anything in the spec that
said that 1--1 was any different from 1 - -1 (especially
considering that the "--" operator is *not* in the sh arithmetic
syntax (though allowed by interpreters as an extension)) like
1-+1 is the same as 1 - +1, but now, thinking more about it, I
suppose that's addressed somewhere in the spec or possibly the C
spec by the way tokenisation is done.
I could not find a reference to that, but I'd expect at least
the C spec to specify for instance how a---b is tokenised (like
"a" "--" "-" b or "a" "-" "--" b or unspecified (to allow
implementations to implement a "---" operator))
So I agree it's a non-issue.
[...]

Post by Geoff Clare
Yes, the onus is on applications to ensure that if they use expansions
within arithmetic expressions, the results will have the syntax they
intended. I.e. if $b might start with a "-" they should use
$(($a - $b)) or $(($a-($b))).

[...]
Such a note might be worth adding to the spec, as it's not
necessarily obvious. Might even be worth saying that $((a - b))
should be prefered over $(($a - $b)).
[...]

Post by Geoff Clare

Post by Stephane Chazelas
For instance $(($x = 2)) would obviously not
work. But would a=foo; : "$(( a = 1 ))" be specified?

$(( a = 1 )) sets the shell variable a to the value 1, regardless
of whether it previously had a value (assuming it is not readonly).

[...]
OK. The reason I mentions that is that the only place that
specifies that using variable names (without the leading "$") in
arithmetic expressions is valid is that $((x)) == $(($x)) above,
with the restriction that $x has to contain integer constants.
Also, it's not all variable names that are allowed. Obviously
recognised by zsh (in mksh, it seems to be ignored).
--
Stephane

[Philip Guenther]

Post by Nick Stoughton
FYI: The C standard does specify how a---b is handled: postfix operators
have higher precedence than unary operators, so it will be evaluated as a--
- b. If you read the C standard, the order of clauses in 6.5 (Expressions)
is the order of precedence (see footnote 85 in C11). For that reason, "a--b"
is not a valid C expression;

Careful: this is a lexing/tokenization issue, not a precedence issue.
The ruling clause for this is 6.4p4:

If the input stream has been parsed into preprocessing tokens up
to a given character, the
next preprocessing token is the longest sequence of characters
that could constitute a
preprocessing token. <...>

6.4p6 is a relevant example, showing that even though prefix ++ has
higher precedence than addition, "a+++++b" is tokenized as "a++ ++ +b"
(and fails to parse).


Philip Guenther

[Schwarz, Konrad]

-----Original Message-----
Sent: Dienstag, 21. Oktober 2014 18:36
To: Nick Stoughton
Subject: Re: $((x)) vs $(($x)) and -/+ operators
On Tue, Oct 21, 2014 at 8:19 AM, Nick Stoughton

Post by Nick Stoughton
FYI: The C standard does specify how a---b is handled: postfix
operators have higher precedence than unary operators, so it will be
evaluated as a--
- b. If you read the C standard, the order of clauses in 6.5
(Expressions) is the order of precedence (see footnote 85 in C11).

For that reason, "a--b"

Post by Nick Stoughton
is not a valid C expression;

Careful: this is a lexing/tokenization issue, not a precedence issue.
If the input stream has been parsed into preprocessing tokens up to
a given character, the
next preprocessing token is the longest sequence of characters that
could constitute a
preprocessing token. <...>
6.4p6 is a relevant example, showing that even though prefix ++ has
higher precedence than addition, "a+++++b" is tokenized as "a++ ++ +b"
(and fails to parse).

Colloquially, this is known as the "maximal munch" rule.

Konrad Schwarz

[Geoff Clare]

Post by Stephane Chazelas
Also, it's not all variable names that are allowed. Obviously
recognised by zsh (in mksh, it seems to be ignored).

In the terminology of the standard, those are not variables; they
are (special) parameters. According to XBD chapter 3 a variable is
a named parameter, and names contain only "underscores, digits, and
alphabetics from the portable character set".

[Chet Ramey]

Post by Stephane Chazelas
a=-1; echo "$((-$a))"
fails in everything except dash and bash.

For different reasons, I suspect. I don't think dash chose to implement
the prefix and postfix increment and decrement operators, and bash makes
sure that there is an identifier preceding or following the operator in
order for it to be interpreted as such instead of a series of unary
minuses.

[Stephane Chazelas]

2014-10-21 11:26:48 -0400, Chet Ramey:
[...]

Post by Chet Ramey
I don't think dash chose to implement
the prefix and postfix increment and decrement operators

[...]

Oh yes, you're right.

And

dash -c 'a=1 b=-1; echo "$(($a-$b))"'

does work contrary to what I said earlier. Which is what you'd
expect from a shell that doesn't implement the "--" operator.

[Stephane Chazelas]

2014-10-20 21:20:45 +0100, Stephane Chazelas:
[...]

$ oawk 'BEGIN{print 1--1;exit}'
10

[...]

Actually, funnily enough, that's parsed as 1-- 1. So the
concatenation of "1" and a decremented "1" ("0")!

oawk 'BEGIN{print 4--; print 4; exit}'
4
3

Not relevant here since POSIX specifies nawk, just thought I'd
mention it as a fun fact.

Also:

$ oawk 'BEGIN{print 1=2,1; exit}'
2 2
$ oawk 'BEGIN{1=2; print 1; exit}'
2

(that's the oawk from the heirloom toolchest so probably the
same as Solaris' /bin/awk and historical awk).