Modulus Returning Incorrect Value

Microsoft Windows
Post Reply
User avatar
MigrationUser
Posts: 336
Joined: 2021-Jul-12, 1:37 pm
Contact:

Modulus Returning Incorrect Value

Post by MigrationUser »

12 Aug 2012 18:11
AYColumbia


Hi,
I need to find the numeric value for FPS from this format, e.g., 24000/1001 which is equivalent to 23.976. Here's a simple example script that is returning the incorrect mod of 977 rather than 976.

Code: Select all

@echo off
setlocal EnableDelayedExpansion
set a=24000
set b=1001
set /a div=!a! / !b!
set /a mod=!a! %% !b!
echo !div! - !mod!
endlocal
If you use a claculator, the result is actually: 23.97602397602398

Also, is there a way to tell the batch file how to treat the mod value, i.e., not to round up and precision? Thanks for any help.

----------------------------

#2 13 Aug 2012 15:13
dbenham


I'm not sure if you did your math wrong, or if you misunderstand what the modulous operation does. But the batch script is giving the correct answers.

The result of integer divsion: 24000 / 1001 = 23
The remainder of integer division (modulous): 24000 %% 1001 = 977

Put another way, (23*1001)+977 = 24000

BTW - You don't need to expand variables when used in a SET /A computation, and you can perform multiple assignments with one command. You also don't require delayed expansion in this code.

Code: Select all

@echo off
setlocal
set a=24000
set b=1001
set /a div=a/b, mod=a%%b
echo %div% - %mod%
endlocal
----------------------------

#3 16 Aug 2012 04:21
AYColumbia


Thanks for your reply and the tip.

Yes, you're right, it's correct. After I posted this, I actually did the math by hand using long division and had to laugh. lol

So here's my situation. I actually need the 976 from the calculated 23.97602397602398. So precision of 3 and essentially throwing out the 02397602398. This is used for video frames per second and it has to be accurate. Do you have any suggestions for how to get this part in a batch file? I know there are limits to what can be done here. I've been putting off moving to WHS due to this batch file is pretty large, but I may finally have to do that as I prefer C# anyway. TIA.

----------------------------

#4 16 Aug 2012 12:14
RG

Code: Select all

set mod=%mod:~0,3%
will yield only the first 3 characters.
See SET /?

Windows Shell Scripting and InstallShield

----------------------------

#5 16 Aug 2012 18:16
dbenham


You can use the following to simulate floating point division with 3 decimal digits of precision (the 3rd decimal digit will be rounded up if the 4th is 5 or greater)

Code: Select all

@echo off
setlocal
set a=24000
set b=1001
set /a "whole=a/b, fract=(a%%b*10000/b+5)/10"
set "fract=000%fract%"
set "fract=%fract:~-3%"
echo %a%/%b% = %whole%.%fract%

Dave Benham

Last edited by dbenham (16 Aug 2012 18:52)

----------------------------

#6 18 Aug 2012 00:25
AYColumbia


Thanks so much all. Very much appreciated.
Post Reply