Number of tokens limit in a FOR command in a Windows batch script

Question

I was trying to process a text file in a Windows batch script and I ran into something that looks like a limitation to 31 tokens in a FOR loop. I isolated the issue in the c

Answer 1

I've created a typewriter function for batch (just for fun) and also encountered this limitation. Below a stripped-down version to show you how I fixed the problem. You can use up to 8,191 chars including space-separators when executed from the command line. When using a variable the maximum length is 32,767.

@echo off
::#############################################################################
 :_typeWriter  
::#############################################################################
::
:: Copyleft Denny Lenselink 2018
::

%= Set local environment =%
( call ) & 2>nul setlocal EnableDelayedExpansion || exit /b 99

:: Set vars
set "_CNT=0" && set "_LEN=0" && set "_STR= %~1" && set "_TOT=0"

:_typeWriter_Loop
set /a "_CNT+=1"

:: 31 tokens limit fix. Cut the used part of the string and set counter to 1
if !_CNT! equ 32 ( set "_CNT=1" && set "_STR=!_STR:~%_TOT%!" && set "_TOT=0" )

:: Go through string (seeking words)
for /f "tokens=%_CNT% delims= " %%* in ( "%_STR%" ) do (

    :: Set word var
    set "_WRD=#%%*"

    :: Calculate word length
    for /l %%I in ( 12, -1, 0 ) do (

        set /a "_LEN|=1<<%%I"

        for %%J in ( !_LEN! ) do ( if "!_WRD:~%%J,1!"=="" set /a "_LEN&=~1<<%%I" )
    )

    :: Strip first char (used before to calculate correct word length)
    set "_WRD=!_WRD:~1!"

    :: Count chars including spaces
    set /a "_TOT=!_TOT!+!_LEN!+1"

    :: Type word or use echo
    

Execute: C:\Temp> typewriter "this is a test that works"

Result:

this is a test that works

Changing the line to echo !_WRD! will result in:

this
is
a
test
that
works