Batch: For directories not working with delayed expansion

Question

I have a CI system that regularly publishes new files. In a script, I want to take the latest version of several (separately published) files. The general file structure is the following: C:\Path\to\publish\<token>\<flavor>\file. There are 3 variable parts in this, per published file. (And all 3 can in some cases have wildcards in it)

The setup I currently have, makes use of delayed expansion to make this configurable:

set A_PATH=C:\Path\to\publish\*
set A_SUBDIR=<flavor>
set A_NAME=file_*.txt

call :searchFile A
goto :logicUsingA

:searchFile
SETLOCAL ENABLEDELAYEDEXPANSION
set SEARCH_PATH=%~1_PATH
set SEARCH_NAME=%~1_NAME
set SEARCH_SUBDIR=%~1_SUBDIR
FOR /D %%I IN ('DIR !%SEARCH_PATH%! /O:-D /B') do (
 echo I:%%I
 FOR %%J IN (!%SEARCH_SUBDIR%!) do (
  echo J:%%J
  FOR /F "delims=" %%K IN ('DIR %%I\%%J\!%SEARCH_NAME%! /A:-D /O:-D /B') do (
   echo K:%%K
   ENDLOCAL
   set "%~1=%%I\%%J\%%K"
   GOTO :EOF
  )
 )
) 2> nul

This system almost did what it needed to do, namely, based on a configuration pick some files and fill a variable with them. Unfortunately, we discovered that something goes wrong with executing this script.

The FOR /D %%I IN ('DIR !%SEARCH_PATH%! /O:-D /B') should sort the directories with the newest first. However, the print below it, prints: (assume 001 was created before 002 ...)

I:'DIR
I:C:\Path\to\publish\001
I:C:\Path\to\publish\002
I:C:\Path\to\publish\003

This makes me suspect that the delayed expansion doesn't play nice with the for-loop, as this is the only noticeable difference with other solutions I find at stackoverflow, like https://stackoverflow.com/a/45104510/2466431

Would it be possible to instruct the for-loop to execute this dir-command iso treating it as a list of items?

EDIT: What the code is trying to achieve is a windows batch variant of the bash one-liner: ls -t C/Path/to/publish/*/<flavor>/file_*.txt

Answer 1

The loop for /D%%I in ('dir !%SEARCH_PATH%! /O:-D /B') do ( ... ) is wrong as you must use for /F to capture and parse the output of a command like dir in our situation. So you may either use for /D %%I in ("!%SEARCH_PATH%!") do ( ... ) or for /F "delims=" %%I in ('dir "!%SEARCH_PATH%!" /O:-D /B') do ( ... ) to be syntactically correct.

But neither will probably be useful for you, because:

• for /D does not access the file system unless wildcards are present;
• for /D does not provide sort options (it returns directories as it gets them from the file system);
• dir, when a directory name is given, lists its contents rather than just returning its name;

Nevertheless, there is a way to work around the said problem with dir, namely appending \ and trying to list the contents of the given directory; if wildcards are given, dir returns an error; if a dedicated directory name is given but it does not exist, dir returns an error too; conditional execution can be applied to make use of that behaviour:

dir /B /A:D "%SomeDir%\" && (echo "%SomeDir%") || dir /B /A:D /O:-D /T:C "SomeDir%"

This provides the following results:

• if %SomeDir% points to a dedicated existing directory, the first dir command lists its contents, so it succeeds, hence the echo command is executed, returning the (quoted) variable contents, but the second dir command is skipped; (quotation of the echo string is done here in order to protect whitespaces or other special characters; the surrounding quotation marks "" are no problem though as they can easily be removed later anyway;)
• if %SomeDir% points to a dedicated directory that cannot be found, the first dir command fails (error: The system cannot find the file specified.), hence the echo command is skipped, but the second dir command is executed, which also fails (error: File Not Found);
• if %SomeDir% contains a wildcard in the last path element, the first dir command fails (error: The filename, directory name, or volume label syntax is incorrect.), hence the echo command is skipped, but the second dir command is executed, which lists all matching directory names, sorted by creation date/time (newest first);

The listing returned by the first dir command as well as any error messages can be suppressed using redirection, so the remaining output is the one by echo or by the second dir:

> nul 2>&1 dir /B /A:D "%SomeDir%\" && (echo "%SomeDir%") || 2> nul dir /B /A:D /O:-D /T:C "SomeDir%"

This can now be applied to your script:

@echo off
set "A_PATH=C:\Path\to\publish\*"
set "A_SUBDIR=<flavor>"
set "A_NAME=file_*.txt"

call :searchFile A
echo A:%A%
rem goto :logicUsingA
goto :EOF

:searchFile
set "%~1="
setlocal EnableDelayedExpansion
set "SEARCH_PATH=%~1_PATH"
set "SEARCH_NAME=%~1_NAME"
set "SEARCH_SUBDIR=%~1_SUBDIR"
cd /D "!%SEARCH_PATH%!\.." || exit /B 1
for /F "delims=" %%I in ('
    ^> nul 2^>^&1 dir /B /A:D "!%SEARCH_PATH%!\" ^
        ^&^& ^(echo "!%SEARCH_PATH%!"^) ^
        ^|^| 2^> nul dir /B /A:D /O:-D /T:C "!%SEARCH_PATH%!"
') do (
    echo I:%%~nxI
    for /F "delims=" %%J in ('
        ^> nul 2^>^&1 dir /B /A:D "%%~nxI\!%SEARCH_SUBDIR%!\" ^
            ^&^& ^(echo "!%SEARCH_SUBDIR%!"^) ^
            ^|^| 2^> nul dir /B /A:D /O:-D /T:C "%%~nxI\!%SEARCH_SUBDIR%!"
    ') do (
        echo J:%%~J
        for /F "delims=" %%K in ('
            ^> nul 2^>^&1 dir /B /A:-D "%%~nxI\%%~J\!%SEARCH_NAME%!\" ^
                ^|^| 2^> nul dir /B /A:-D /O:-D /T:C "%%~nxI\%%~J\!%SEARCH_NAME%!"
        ') do (
            echo K:%%K
            endlocal
            set "%~1=%CD%\%%~nxI\%%~J\%%K"
            goto :EOF
        )
    )
)

goto :EOF

In the inner-most loop that enumerates files rather than directories, I applied the same technique, but I skipped the echo command as it should never be executed anyway; the only reason why I kept the two dir commands here is to handle the situation when you specify a certain file name (no wildcards) but there is actually a directory with that name (a single dir would then unintentionally return its contents).

There have also a few other things changed:

• the variable %~1 is initially cleared (in the sub-routine :searchFile, just in case no file could be found at all);
• the quoted set syntax is consistently used (set "VAR=Value");
• all paths are now quoted to avoit trouble with whitespaces or other special characters;
• the dir option /T:C is added to regard the creation date/time rather than the date/time of the last modification; just remove it if you want to regard the latter;
• cd /D"!%SEARCH_PATH%!\.." ||exit/B 1 has been added to change to the parent directory, because the later used dir /B commands just return directory or file names but not full paths; that is also the reason why the ~nx-modifier is used in %%~nxI, so there is no more difference whether the value comes from dir /B or echo; the exit /B 1 part makes sure to skip the remaining code in case the parent directory could not be found/accessed;
• ~-modifiers are used in %%~J to ensure unquoted directory names (remember that I put quotes in the echo command lines);
• 2> nul has been removed (from the end of the body of the outer-most loop) to not suppress error messages in general;