Read a file, output each line to a new file, preserve all characters?

Microsoft Windows
Post Reply
Rekrul
Posts: 52
Joined: 2021-Aug-15, 11:29 pm

Read a file, output each line to a new file, preserve all characters?

Post by Rekrul »

I'm trying to write a script that will read a given file line by line and output each one to a new file. I want to do this, so that the script can make some conditional changes to some of the lines. It should be a simple process, but of course it's not because Batch is the most ass-backward, convoluted mess ever invented.

The main problem is all the symbols that you absolutely, positively cannot under any circumstances, have in a string, unless the whole thing is in quotes. Stuff like the ampersand, percent sign, greater than, less than, etc. Trying to echo such a line will cause all sorts of problems. OK, most of those can be escaped, but you can't do that when reading in lines of a file. And the ampersand can't be escaped at all. If there's a way to do it, I can't figure it out.

So you enclose it in quotes, problem solved, right? Not even close! You may have gotten those symbols into a string, but now every line is going to have quotes on it when you output it to the new file. So just echo the string without the first and last characters, right? Nope! Then you're back to having a problem with the above characters.

I know! use substring replace operations to escape those characters! Nope, that doesn't work. Well, it does if you use DelayedExpansion, and then use two variables as placeholders for the characters, since you STILL can't include them directly in the replace operation, even by escaping them.

So problem solved? Yeah, except for one tiny, insignificant little problem: DelayedExpansion makes it impossible to use exclamation points in strings. .They can contain them if it's defined BEFORE Delayed Expansion is enabled, but the second you do ANY string operations with it enabled, it immediately strips every exclamation point.

ARGH!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Is there any way that I can read each line of a file, which may contain ! % & < >, put it into a variable/string, then output it to a new file, WITHOUT having the line enclosed in quotes in the new file?

I've tried Googling this problem and come up empty. All the solutions assume you just want to remove the quotes, but don't include the problem of how to handle all the problem characters without the quotes.
User avatar
Simon Sheppard
Posts: 190
Joined: 2021-Jul-10, 7:46 pm
Contact:

Re: Read a file, output each line to a new file, preserve all characters?

Post by Simon Sheppard »

An approach I have used is to replace all the problem characters with something else, either a character like ~ or ` use a unique phrase "Gr33nTreez"

Then do all your processing and then rename the characters back to what they started as.

https://ss64.com/vb/syntax-replace.html
Rekrul
Posts: 52
Joined: 2021-Aug-15, 11:29 pm

Re: Read a file, output each line to a new file, preserve all characters?

Post by Rekrul »

Hmm, I thought I saved my reply...

Anyway, thanks for replying.
Simon Sheppard wrote: 2022-Dec-19, 12:16 am An approach I have used is to replace all the problem characters with something else, either a character like ~ or ` use a unique phrase "Gr33nTreez"

Then do all your processing and then rename the characters back to what they started as.
I guess that's what I'll have to do. I was hoping to do it all in Batch to make it as portable as possible, but I guess there's no avoiding the use of additional programs to get the job done.
bluesxman
Posts: 10
Joined: 2021-Jul-26, 3:41 pm

Re: Read a file, output each line to a new file, preserve all characters?

Post by bluesxman »

Have you tried something like this approach?

Code: Select all

setlocal disabledelayedexpansion
set /a count=0
for /f "useback tokens=* delims=" %%L in ("your_file.txt") do (
  set "line=%%L"
  call :write_file
)

goto :EOF

:write_file
set /a "count+=1"
REM your conditional changes would go here
REM example
set "line=%line:foo=bar%" 
(
  <nul set /p "out=%line%"
  echo:
) > your_file_line_%count%.txt
goto :EOF
But I'll caveat this by saying that CMD is really not well suited to this sort of thing, there are various pitfalls. The majority can probably be handled, but you'd have to take them on a case by case basis.
Last edited by bluesxman on 2023-Jan-05, 10:04 am, edited 1 time in total.
GOD
Posts: 1
Joined: 2023-Jan-03, 1:00 pm

Re: Read a file, output each line to a new file, preserve all characters?

Post by GOD »

bluesxman wrote: 2022-Dec-26, 10:42 am Have you tried something like this approach?

Code: Select all

setlocal disabledelayeexpansion
set /a count=0
for /f "useback tokens=* delims=" %%L in ("your_file.txt") do (
  set "line=%%L"
  call :write_file
)

goto :EOF

:write_file
set /a "count+=1"
REM your conditional changes would go here
REM example
set "line=%line:foo=bar%" 
(
  <nul set /p "out=%line%"
  echo:
) > your_file_line_%count%.txt
goto :EOF
But I'll caveat this by saying that CMD is really not well suited to this sort of thing, there are various pitfalls. The majority can probably be handled, but you'd have to take them on a case by case basis.
You really do write some adorable cute code.
Shadow Thief
Posts: 12
Joined: 2021-Aug-03, 1:45 am

Re: Read a file, output each line to a new file, preserve all characters?

Post by Shadow Thief »

GOD wrote: 2023-Jan-03, 1:04 pm
bluesxman wrote: 2022-Dec-26, 10:42 am Have you tried something like this approach?

Code: Select all

setlocal disabledelayeexpansion
set /a count=0
for /f "useback tokens=* delims=" %%L in ("your_file.txt") do (
  set "line=%%L"
  call :write_file
)

goto :EOF

:write_file
set /a "count+=1"
REM your conditional changes would go here
REM example
set "line=%line:foo=bar%" 
(
  <nul set /p "out=%line%"
  echo:
) > your_file_line_%count%.txt
goto :EOF
But I'll caveat this by saying that CMD is really not well suited to this sort of thing, there are various pitfalls. The majority can probably be handled, but you'd have to take them on a case by case basis.
You really do write some adorable cute code.
What a weird thing to say to someone
Simon_Weel
Posts: 34
Joined: 2021-Dec-13, 3:53 pm

Re: Read a file, output each line to a new file, preserve all characters?

Post by Simon_Weel »

As you discovered, the Windows SMD Shell isn't exactly suited for this kind of jobs. You may want to look into something like Autohotkey https://www.autohotkey.com/?
bluesxman
Posts: 10
Joined: 2021-Jul-26, 3:41 pm

Re: Read a file, output each line to a new file, preserve all characters?

Post by bluesxman »

GOD wrote: 2023-Jan-03, 1:04 pm You really do write some adorable cute code.
Thanks, I was going for "functional and concise, with a hint of 'inoffensive'" but these days I'll take whatever compliments I can get.
jeb
Posts: 12
Joined: 2023-May-10, 1:28 pm

Re: Read a file, output each line to a new file, preserve all characters?

Post by jeb »

To be bullet proof, you need the delayed toggling technique.

Code: Select all

@echo off
set "filename=%~1"

SETLOCAL DisableDelayedExpansion
FOR /F "delims=" %%a in ('findstr /n "^" "%filename%"') do (
    set "var=%%a"
    SETLOCAL EnableDelayedExpansion
    set "var=!var:*:=!"
    echo(!var!
    ENDLOCAL
)
The findstr is used to prefix each line with a line number, this is necessary to avoid the removal of empty lines or lines beginning with the EOL character (default is semi colon).

The delayedexpansion has to be disabled in:

Code: Select all

set "var=%%a"
Because with enabled mode, exclamation marks and carets would be removed.

But to remove the line number in front, the delayedexpansion mode has to be enabled
The line number is removed by the

Code: Select all

set "var=!var:*:=!"
You can't use the delims=: here, because this would also remove all colons at the beginning of lines.

Code: Select all

echo(!var! 
Is essential, to avoid problems with lines like "OFF", "/?" or "..\..\windows\system32\calc.exe".
echo(!var! is the only echo that can output any content even empty content.
Post Reply