[1003.1(2008)/Issue 7 0000519]: Add to make macro variable pattern substitution (expansion), e.g., $(foo:%.o=%.c)

A NOTE has been added to this issue.
======================================================================
http://austingroupbugs.net/view.php?id=519
======================================================================
Reported By: dwheeler
Assigned To: ajosey
======================================================================
Project: 1003.1(2008)/Issue 7
Issue ID: 519
Category: Shell and Utilities
Type: Enhancement Request
Severity: Objection
Priority: normal
Status: Under Review
Name: Your Name Here
Organization:
User Reference:
Section: make
Page Number: 2915
Line Number: 95808
Interp Status: ---
Final Accepted Text:
======================================================================
Date Submitted: 2011-11-27 23:11 UTC
Last Modified: 2012-04-16 14:44 UTC
======================================================================
Summary: Add to make macro variable pattern substitution
(expansion), e.g., $(foo:%.o=%.c)
======================================================================

----------------------------------------------------------------------
(0001206) geoffclare (manager) - 2012-04-16 14:44
http://austingroupbugs.net/view.php?id=519#c1206
----------------------------------------------------------------------
[Rewording of http://austingroupbugs.net/view.php?id=519#c1196]

After the paragraph:

Macro expansions in string1 of macro definition lines shall be
evaluated when read. Macro expansions in string2 of macro definition
lines shall be performed when the macro identified by string1 is
expanded in a rule or command.

add the following new paragraphs:

Macro expansions using the forms $(string1:[op]%[os]=[np][%][ns]) or
${string1:[op]%[os]=[np][%][ns]} are called pattern macro expansions,
where "op" is the old prefix, "os" is the old suffix, "np" is the new
prefix and "ns" is the new suffix. Any item inside square brackets
is optional. With this form, when the macro string1 is expanded each
whitespace separated word that completely matches the "[op]%[os]"
pattern on the left hand side of the <equals-sign> ('='), where the
<percent> ('%') character matches zero or more characters, shall be
replaced by the right hand side of the <equals-sign> and shall then
be further modified according to the use of <percent> characters as
described below. Any words that do not match shall be unmodified in
the expansion.

If more than one <percent> character appears on the left hand side
of the <equals-sign> ('='), the second and subsequent <percent>
characters shall be treated as literal characters in "os".

If no <percent> character appears on the right hand side of the
<equals-sign>, no further modification of the word shall be performed.
If a single <percent> character appears on the right hand side, the
<percent> character in the word shall be replaced with the characters
matched by the <percent> on the left hand side. If more than one
<percent> character appears on the right hand side, it is unspecified
whether the first <percent> character in the word is replaced with the
characters matched by the <percent> on the left hand side and all
remaining <percent> characters are left unchanged, or each <percent>
character is replaced with the characters matched by the <percent> on
the left hand side.

In both macro expansion forms, any macro expansions on the right hand
side of the colon shall be recursively expanded before further
examination. If this results in more than one <equals-sign> after the
<colon>, the first one shall be considered to be the separator.

If the original value of the variable named by string1 is an empty
string, the final result shall be an empty string. Matching shall be
case-sensitive.

Issue History
Date Modified Username Field Change
======================================================================
2011-11-27 23:11 dwheeler New Issue
2011-11-27 23:11 dwheeler Status New => Under Review
2011-11-27 23:11 dwheeler Assigned To => ajosey
2011-11-27 23:11 dwheeler Name => Your Name Here
2011-11-27 23:11 dwheeler Section => make
2011-11-27 23:11 dwheeler Page Number => 2915
2011-11-27 23:11 dwheeler Line Number => 95808
2011-11-28 14:15 joerg Note Added: 0001057
2011-11-28 14:46 joerg Note Added: 0001058
2011-11-29 00:07 dwheeler Note Added: 0001063
2011-11-29 01:14 dwheeler File Added: makefile-patterns.tar.gz

2011-11-29 09:56 joerg Note Added: 0001066
2011-11-29 10:06 joerg Note Added: 0001067
2011-11-29 10:32 joerg Note Added: 0001068
2011-11-30 04:58 dwheeler Note Added: 0001069
2011-11-30 16:56 joerg Note Added: 0001071
2012-04-12 14:43 joerg Note Added: 0001196
2012-04-12 15:59 joerg Note Edited: 0001196
2012-04-12 16:16 joerg Note Edited: 0001196
2012-04-12 16:19 joerg Note Edited: 0001196
2012-04-12 16:21 joerg Note Edited: 0001196
2012-04-12 16:26 joerg Note Edited: 0001196
2012-04-16 14:44 geoffclare Note Added: 0001206
======================================================================

Antoine Leca

2014-08-08 16:21:00 UTC

Sorry to revisit an old one.

Post by Austin Group Bug Tracker
======================================================================
http://austingroupbugs.net/view.php?id=519
======================================================================
(0001206) geoffclare (manager) - 2012-04-16 14:44
http://austingroupbugs.net/view.php?id=519#c1206
----------------------------------------------------------------------
[Rewording of http://austingroupbugs.net/view.php?id=519#c1196]

<snip>

Post by Austin Group Bug Tracker
Macro expansions using the forms $(string1:[op]%[os]=[np][%][ns]) or

^ ^

Post by Austin Group Bug Tracker
${string1:[op]%[os]=[np][%][ns]} are called pattern macro expansions,

^ ^

Post by Austin Group Bug Tracker
where "op" is the old prefix, "os" is the old suffix, "np" is the new
prefix and "ns" is the new suffix. Any item inside square brackets
is optional. [...]

<snip>

Post by Austin Group Bug Tracker
If no <percent> character appears on the right hand side of the
<equals-sign>, no further modification of the word shall be performed.
If a single <percent> character appears on the right hand side, the
<percent> character in the word shall be replaced with the characters
matched by the <percent> on the left hand side. If more than one
<percent> character appears on the right hand side, it is unspecified

<rest is irrelevant>

It is my understanding that, from a syntactic point of view, one cannot
drop the % without also specifying that either np or ns will also miss;
so it should really reads either
$(string1:[op]%[os]=[[np]%][ns])
or
$(string1:[op]%[os]=[np][%[ns]])
(along with the respective ${} form.)

Also, having either % or $ in subst1 in the $(string1:subst1=[subst2])
syntax, will lead to incompatible behaviour with preceding releases of
the Standard: % defers to the new feature, while $ would trigger macro
expansion which /could/ result in the new feature to be used.
Perhaps this should be recorded somewhere in the informative part?

Antoine

Geoff Clare

2014-08-11 09:12:08 UTC

Post by Austin Group Bug Tracker
======================================================================
(0001206) geoffclare (manager) - 2012-04-16 14:44
http://austingroupbugs.net/view.php?id=519#c1206
----------------------------------------------------------------------
[Rewording of http://austingroupbugs.net/view.php?id=519#c1196]

<snip>

Post by Austin Group Bug Tracker
Macro expansions using the forms $(string1:[op]%[os]=[np][%][ns]) or

^ ^

Post by Austin Group Bug Tracker
${string1:[op]%[os]=[np][%][ns]} are called pattern macro expansions,

^ ^

Post by Austin Group Bug Tracker
where "op" is the old prefix, "os" is the old suffix, "np" is the new
prefix and "ns" is the new suffix. Any item inside square brackets
is optional. [...]

<snip>

I see your point, but actually I prefer the current text to either of
these. If there is no % on the right of the =, then the right hand
side is a fixed string to be used as the replacement. It doesn't
matter which part of this string you consider to be np and which ns;
the result is the same.

If we change it to:

$(string1:[op]%[os]=[[np]%][ns])

this could be misread as implying that when there is no % on the
right, since the right hand side is entirely a "new suffix", only
the suffix part of the left hand side is replaced. Likewise for
the other suggestion and only the prefix part being replaced.

Post by Antoine Leca
Also, having either % or $ in subst1 in the $(string1:subst1=[subst2])
syntax, will lead to incompatible behaviour with preceding releases of
the Standard: % defers to the new feature, while $ would trigger macro
expansion which /could/ result in the new feature to be used.
Perhaps this should be recorded somewhere in the informative part?

I think you have identified a defect in the current standard. It
does not match existing practice if subst1 contains a % (either
directly or as a result of macro expansion). I think we should fix
it in TC2 by adding a statement that the behaviour is unspecified if
subst1 contains a % after any macros have been recursively expanded.
Then we will need to update bug 519 to include a modification of this
new statement (changing it to a reference to the later description of
the behaviour when there is a %).

Since you found the problem, would you like to be the one to submit a
new bug in Mantis to request this fix?

--
Geoff Clare <g.clare-7882/***@public.gmane.org>
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England

Antoine Leca

2014-08-12 08:56:29 UTC

I think you have identified a defect in the current standard. It
does not match existing practice if subst1 contains a % (either
directly or as a result of macro expansion). [...]

It is interesting that you wrote "either directly or as a result of
macro expansion", since it is not my understanding that the current
Posix standard prescribed macro expansion here :-)

Post by Geoff Clare
Since you found the problem, would you like to be the one to submit a
new bug in Mantis to request this fix?

Done, this is now http://austingroupbugs.net/view.php?id=865

Note that I miswrote <percent> when it should rather be <percent-sign>,
in the "desired action" to-be-inserted text.

Also I did not add the "2008-tc2" tag (based on the idea it is
Committee's job) nor did I add a link to bug to #519 (because I do not
have the authorizations to perform such action.)
Also the page and line numbers are speculated, since I do not have
access to the official PDF copy of the consolidated 2008+TC1 Standard.

Antoine

Geoff Clare

2014-08-12 11:26:43 UTC

I think you have identified a defect in the current standard. It
does not match existing practice if subst1 contains a % (either
directly or as a result of macro expansion). [...]

It is interesting that you wrote "either directly or as a result of
macro expansion", since it is not my understanding that the current
Posix standard prescribed macro expansion here :-)

I've posted a note about this in the new bug.

Post by Geoff Clare
Since you found the problem, would you like to be the one to submit a
new bug in Mantis to request this fix?

Done, this is now http://austingroupbugs.net/view.php?id=865
Note that I miswrote <percent> when it should rather be <percent-sign>,
in the "desired action" to-be-inserted text.
Also I did not add the "2008-tc2" tag (based on the idea it is
Committee's job) nor did I add a link to bug to #519 (because I do not
have the authorizations to perform such action.)
Also the page and line numbers are speculated, since I do not have
access to the official PDF copy of the consolidated 2008+TC1 Standard.

I have added the relationship to #519 and fixed the page and line numbers.
(The history says I also modified the desired action, but I didn't.
Perhaps firefox removes trailing whitespace when it submits or something.)

--
Geoff Clare <g.clare-7882/***@public.gmane.org>
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England

Antoine Leca

2014-08-12 16:45:58 UTC

Post by Geoff Clare
I think you have identified a defect in the current standard. It
does not match existing practice if subst1 contains a % (either
directly or as a result of macro expansion). [...]

It is interesting that you wrote "either directly or as a result of
macro expansion", since it is not my understanding that the current
Posix standard prescribed macro expansion here :-)

I've posted a note about this in the new bug.

It is http://austingroupbugs.net/view.php?id=865#c2346 (disregarding the
1st paragraph, a valid point against an oversimplification of mine!)

I see two arguments here: one is about the text of the Standard
: because line 97270 says "Macros can appear anywhere in the makefile"
combined with that there are no place which says it is unspecified to
have macro expansion in subst1 or subst2, while there is explicit text
about macro expansion in string1.
I did not quite get it the same way as you do; for me, "the makefile"
was the text file (see *Input files* section; see also the description
of -f option), that is, before any macro expansion takes place.
However I see your point (now!), particularly since it aptly explains
why the text "if string1 in a macro expansion contains a macro
expansion, the results are unspecified." is repeated twice, in lines
97280 and 97286; probably this is the result of several iterations
during the formation of the original standard in 1984-93.
But I feel like it requires to deduce it from what appears otherwise to
be a repetition, then comparing the part which it is not explicitly
written; and frankly this is a bit too far stretched for me.
I certainly would welcome making it crystal clear, for example with some
additional text in Examples or Rationale.

The second argument, about having no implementation without support from
such expansion /although/ they do support $(string1:subst1=subst2)
_for_suffixes_, as Posix specifies, seems to me a much more solid point.
I do not have enough experience to know such a case (the implementation
I am basing does not support neither substitution nor expansion at the
moment, so it does not count; other ancient implementations like System
III or SVR3 do not expand but they do not restrict themselves to
suffixes either, so they are likely discarded); but I am sure
knowledgeable people will comment on this issue.

Also, if I read your point correctly, since only the /string1/ part is
required to not be the result of macro expansion, any conforming
implementation, even if it does not handle recursive macro expansion (ie
expanding /string1/), is required to correctly handle

.POSIX:
SUBST = :.c=.o
SRCS = a.c b.c c.c# can be overridden from command-line!
target: $(SRCS$(SUBST))

in addition to the (less convoluted) similar form where the : is on the
last line.
/If/ this holds, then I wonder whether there are any compliant
implementation which does NOT support expansion of /string1/, since all
the hard work is very probably already done...

Antoine

Geoff Clare

2014-08-14 11:08:16 UTC

It is interesting that you wrote "either directly or as a result of
macro expansion", since it is not my understanding that the current
Posix standard prescribed macro expansion here :-)

I've posted a note about this in the new bug.

It is http://austingroupbugs.net/view.php?id=865#c2346 (disregarding the
1st paragraph, a valid point against an oversimplification of mine!)

[...]

Post by Antoine Leca
Also, if I read your point correctly, since only the /string1/ part is
required to not be the result of macro expansion, any conforming
implementation, even if it does not handle recursive macro expansion (ie
expanding /string1/), is required to correctly handle
SUBST = :.c=.o
SRCS = a.c b.c c.c# can be overridden from command-line!
target: $(SRCS$(SUBST))

No, that has a macro expansion where string1 is SRCS$(SUBST) and so
the result is unspecified as per line 97280. The unspecified
behaviour doesn't depend on what the nested macro expands to, only
that it is there.

Post by Antoine Leca
in addition to the (less convoluted) similar form where the : is on the
last line.

Yes, that is required to be supported (if my interpretation is right)
because string1 is now just SRCS and the nested expansion is after
the colon.

I tried the following makefile on Linux (GNU make), Solaris 10
(/usr/xpg4/bin/make and /usr/ccs/bin/make), Solaris 11, and HP-UX 11,
and it worked as expected (echoed a.o b.o c.o three times) on all of
them.

---- start ----
.POSIX:
SUBST = .c=.o
DOTC = .c
DOTO = .o
SRCS = a.c b.c c.c

all: target1 target2 target3

target1: $(SRCS:$(SUBST))
echo $?

target2: $(SRCS:$(DOTC)=.o)
echo $?

target3: $(SRCS:.c=$(DOTO))
echo $?

a.o b.o c.o:
---- end ----

However, those versions of make also accepted your

target: $(SRCS$(SUBST))

macro expansion, so this result doesn't support my point as
strongly as would a version that doesn't accept $(SRCS$(SUBST))
but does handle the above makefile correctly.

--
Geoff Clare <g.clare-7882/***@public.gmane.org>
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England

Austin Group Bug Tracker

2012-04-16 14:47:22 UTC

The following issue has been UPDATED.
======================================================================
http://austingroupbugs.net/view.php?id=519
======================================================================
Reported By: dwheeler
Assigned To: ajosey
======================================================================
Project: 1003.1(2008)/Issue 7
Issue ID: 519
Category: Shell and Utilities
Type: Enhancement Request
Severity: Objection
Priority: normal
Status: Under Review
Name: David A. Wheeler
Organization:
User Reference:
Section: make
Page Number: 2915
Line Number: 95808
Interp Status: ---
Final Accepted Text:
======================================================================
Date Submitted: 2011-11-27 23:11 UTC
Last Modified: 2012-04-16 14:47 UTC
======================================================================
Summary: Add to make macro variable pattern substitution
(expansion), e.g., $(foo:%.o=%.c)
======================================================================

Issue History
Date Modified Username Field Change
======================================================================
2011-11-27 23:11 dwheeler New Issue
2011-11-27 23:11 dwheeler Status New => Under Review
2011-11-27 23:11 dwheeler Assigned To => ajosey
2011-11-27 23:11 dwheeler Name => Your Name Here
2011-11-27 23:11 dwheeler Section => make
2011-11-27 23:11 dwheeler Page Number => 2915
2011-11-27 23:11 dwheeler Line Number => 95808
2011-11-28 14:15 joerg Note Added: 0001057
2011-11-28 14:46 joerg Note Added: 0001058
2011-11-29 00:07 dwheeler Note Added: 0001063
2011-11-29 01:14 dwheeler File Added: makefile-patterns.tar.gz

2011-11-29 09:56 joerg Note Added: 0001066
2011-11-29 10:06 joerg Note Added: 0001067
2011-11-29 10:32 joerg Note Added: 0001068
2011-11-30 04:58 dwheeler Note Added: 0001069
2011-11-30 16:56 joerg Note Added: 0001071
2012-04-12 14:43 joerg Note Added: 0001196
2012-04-12 15:59 joerg Note Edited: 0001196
2012-04-12 16:16 joerg Note Edited: 0001196
2012-04-12 16:19 joerg Note Edited: 0001196
2012-04-12 16:21 joerg Note Edited: 0001196
2012-04-12 16:26 joerg Note Edited: 0001196
2012-04-16 14:44 geoffclare Note Added: 0001206
2012-04-16 14:47 geoffclare Name Your Name Here => David
A. Wheeler
2012-04-16 14:47 geoffclare Interp Status => ---
======================================================================

Austin Group Bug Tracker

2012-04-19 15:09:58 UTC

Austin Group Bug Tracker

2012-04-19 15:26:12 UTC