## Modulus Returning Incorrect Value

Microsoft Windows
MigrationUser
Posts: 336
Joined: 2021-Jul-12, 1:37 pm
Contact:

### Modulus Returning Incorrect Value

12 Aug 2012 18:11
AYColumbia

Hi,
I need to find the numeric value for FPS from this format, e.g., 24000/1001 which is equivalent to 23.976. Here's a simple example script that is returning the incorrect mod of 977 rather than 976.

Code: Select all

``````@echo off
setlocal EnableDelayedExpansion
set a=24000
set b=1001
set /a div=!a! / !b!
set /a mod=!a! %% !b!
echo !div! - !mod!
endlocal``````
If you use a claculator, the result is actually: 23.97602397602398

Also, is there a way to tell the batch file how to treat the mod value, i.e., not to round up and precision? Thanks for any help.

----------------------------

#2 13 Aug 2012 15:13
dbenham

I'm not sure if you did your math wrong, or if you misunderstand what the modulous operation does. But the batch script is giving the correct answers.

The result of integer divsion: 24000 / 1001 = 23
The remainder of integer division (modulous): 24000 %% 1001 = 977

Put another way, (23*1001)+977 = 24000

BTW - You don't need to expand variables when used in a SET /A computation, and you can perform multiple assignments with one command. You also don't require delayed expansion in this code.

Code: Select all

``````@echo off
setlocal
set a=24000
set b=1001
set /a div=a/b, mod=a%%b
echo %div% - %mod%
endlocal``````
----------------------------

#3 16 Aug 2012 04:21
AYColumbia

Yes, you're right, it's correct. After I posted this, I actually did the math by hand using long division and had to laugh. lol

So here's my situation. I actually need the 976 from the calculated 23.97602397602398. So precision of 3 and essentially throwing out the 02397602398. This is used for video frames per second and it has to be accurate. Do you have any suggestions for how to get this part in a batch file? I know there are limits to what can be done here. I've been putting off moving to WHS due to this batch file is pretty large, but I may finally have to do that as I prefer C# anyway. TIA.

----------------------------

#4 16 Aug 2012 12:14
RG

Code: Select all

``set mod=%mod:~0,3%``
will yield only the first 3 characters.
See SET /?

Windows Shell Scripting and InstallShield

----------------------------

#5 16 Aug 2012 18:16
dbenham

You can use the following to simulate floating point division with 3 decimal digits of precision (the 3rd decimal digit will be rounded up if the 4th is 5 or greater)

Code: Select all

``````@echo off
setlocal
set a=24000
set b=1001
set /a "whole=a/b, fract=(a%%b*10000/b+5)/10"
set "fract=000%fract%"
set "fract=%fract:~-3%"
echo %a%/%b% = %whole%.%fract%``````

Dave Benham

Last edited by dbenham (16 Aug 2012 18:52)

----------------------------

#6 18 Aug 2012 00:25
AYColumbia

Thanks so much all. Very much appreciated.