Rounding with precision problem

Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

All Forums

SQL Server 2000 Forums

SQL Server Development (2000)

Rounding with precision problem

Author

Topic

ronstone
Starting Member

32 Posts

Posted - 2005-01-30 : 18:35:11

Ok, this may be a trivial task, but I can't get the results I am looking for.

I have a user function, that returns a decimal (18,4).

The number is: .04950

I need it to pass .0496

I've tried every rounding combination with convert(decimal(x,x)) with no luck.

Am I missing something obvious? The only rounding I was able to get, was .0500 (or 0).

Thanks,
Ron

derrickleggett
Pointy Haired Yak DBA

4184 Posts

Posted - 2005-01-30 : 19:51:41

It can't round that to .0496. That doesn't even make sense. You can round then add .0001 if you want. "0" does not round UP. Matter of fact, it doesn't even round down. It is zero.

MeanOldDBA
derrickleggett@hotmail.com

When life gives you a lemon, fire the DBA.

madhivanan
Premature Yak Congratulator

22864 Posts

Posted - 2005-01-31 : 05:15:11

Try this

Declare @n decimal(18,4)
Set @n=.04950
select left(Round(@n+0.00005,4),len(@n))

The Result is 0.0496

Madhivanan

derrickleggett
Pointy Haired Yak DBA

4184 Posts

Posted - 2005-01-31 : 08:01:18

well, duh

This works also:


Declare @n decimal(18,4)
select @n = .04950+0.00005
select @n

Now, my point is that he can't ROUND the number he gave us to what he wants. This is called addition, not rounding. Or, did they change it since I was in school?

I missed something really obvious here, didn't I? (cough, cough)

MeanOldDBA
derrickleggett@hotmail.com

When life gives you a lemon, fire the DBA.

ronstone
Starting Member

32 Posts

Posted - 2005-01-31 : 12:10:54

quote:
Originally posted by derrickleggett

It can't round that to .0496. That doesn't even make sense. You can round then add .0001 if you want. "0" does not round UP. Matter of fact, it doesn't even round down. It is zero.

Correct. My post was obviously not proofed when I posted it. I meant .04956.

Subscribe to SQLTeam.com

SQLTeam.com Articles via RSS

SQLTeam.com Weblog via RSS

- Advertisement -

Resources