ASP Hosting sale!
Double disk space and transfer for FREE!
Limited time offer! Act Now!

aspdev | articles | tutorials | forums

 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

calculation with numbers with comma

 
Post new topic   Reply to topic    ASPdev.org Forum Index -> ASP programming
View previous topic :: View next topic  
Author Message
GinoTheCop



Joined: 30 Dec 2005
Posts: 9

PostPosted: Mon Jan 30, 2006 8:02 am    Post subject: calculation with numbers with comma Reply with quote

Hi everyone.. I have a form page with 16 different fields. When user inputs or selects from dropdown lists, the page has to do a calculation.

The calculations are like
Y= 0.120218 * 0.010208 * 0.036481 (...16 different numbers)
sY = Y X 100
sX = 1 - sY

I usually get this
Y = 0,000000000000000000000011 (22 zeros after comma)
sY = 0,000000000000000000001092

its fine upto this point

but i get
sX= 1,000000000000000000000000

seems after the 16th digit after comma my code is ignoring numbers
how can i fix this?
_________________
Cuneyt Yetgin
Back to top
View user's profile Send private message
administrator
Site Admin


Joined: 01 Oct 2004
Posts: 183

PostPosted: Mon Jan 30, 2006 12:12 pm    Post subject: Reply with quote

You can use type-conversions using CDbl VBScript function. For example:


sY = CDbl(0.000000000000000000001092)
sX = CDbl(1.0) - sY

this should force double precision calculation and return a sub-type double result.

Try it and let me know if it worked?
_________________
Peter
ASP & ASP.NET Articles and Tutorials
Back to top
View user's profile Send private message
GinoTheCop



Joined: 30 Dec 2005
Posts: 9

PostPosted: Tue Jan 31, 2006 2:54 am    Post subject: Reply with quote

I have converted every variable that takes place in the calculation to CDbl type but no luck... still can not substract from 1 The result is still 1 after substraction...
_________________
Cuneyt Yetgin
Back to top
View user's profile Send private message
GinoTheCop



Joined: 30 Dec 2005
Posts: 9

PostPosted: Tue Jan 31, 2006 5:17 am    Post subject: Reply with quote

Below is the sample code that i get the error with..


<%

a = CDbl(0.02744)
b = CDbl(0.02016)
c = CDbl(0.04466)
d = CDbl(0.013068)
e = CDbl(0.123084)
f = CDbl(0.028561)
g = CDbl(0.008134)
h = CDbl(0.023056)
i = CDbl(0.014884)
j = CDbl(0.009234)
k = CDbl(0.079422)
l = CDbl(0.104006)
m = CDbl(0.003468)
n = CDbl(0.075808)
o = CDbl(0.02262)

sY = a*b*c*d*e*f*g*h*i*j*k*l*m*n*o
sYY = CDbl(sY) * CDbl(100.0)
sX = CDbl(1.0) - CDbl(sYY)
sXX = sX * 100

sV = FormatNumber(sY, 24)
sVV = FormatNumber(sYY, 24)
sW = FormatNumber(sX, 24)
sWW = FormatNumber(sXX, 24)

Response.Write "Y=<b>" &sY& "</b><br>" & "Y * 100=<b>" &SYY& "</b><br>" & "1 - (Y * 100)=<b>" &sX& "</b><br>" & "<b>%" &sXX& "</b><hr>"
Response.Write "Y=<b>" &sV& "</b><br>" & "Y * 100=<b>" &SVV& "</b><br>" & "1 - (Y * 100)=<b>" &sW& "</b><br>" & "<b>%" &sWW& "</b><hr>"
%>
_________________
Cuneyt Yetgin
Back to top
View user's profile Send private message
jon



Joined: 01 Oct 2004
Posts: 40

PostPosted: Tue Jan 31, 2006 9:38 am    Post subject: Reply with quote

I'll try your code later and I'll let you know if I get the same problem.
Back to top
View user's profile Send private message
GinoTheCop



Joined: 30 Dec 2005
Posts: 9

PostPosted: Tue Jan 31, 2006 3:59 pm    Post subject: Reply with quote

Ok.. Thanks a lot.. hope we can find a solution for this.. By the way the code is to calculate the frequency of a DNA sample
_________________
Cuneyt Yetgin
Back to top
View user's profile Send private message
GinoTheCop



Joined: 30 Dec 2005
Posts: 9

PostPosted: Fri Feb 03, 2006 3:22 am    Post subject: Reply with quote

Any ideas? Istill couldnt find a solution for this.. I also tried the same calculation with MS Excel.. Guess what, Excel can not make the same calculation either...
_________________
Cuneyt Yetgin
Back to top
View user's profile Send private message
raven



Joined: 31 Jan 2006
Posts: 41

PostPosted: Thu Feb 23, 2006 5:31 pm    Post subject: Reply with quote

Quote:
sX = CDbl(1.0) - CDbl(sYY)


Do you need to double that?

why not just sx=CDbl(1.0)-sYY
_________________
Programmers do it in code
Back to top
View user's profile Send private message
GinoTheCop



Joined: 30 Dec 2005
Posts: 9

PostPosted: Fri Feb 24, 2006 1:54 am    Post subject: Reply with quote

No I just wanted to make sure everything is double.. Does not matter whether you double it or not.. You can not get the correct result after calculation... Still couldnt solve this..
_________________
Cuneyt Yetgin
Back to top
View user's profile Send private message
administrator
Site Admin


Joined: 01 Oct 2004
Posts: 183

PostPosted: Fri Feb 24, 2006 9:53 am    Post subject: Reply with quote

GinoTheCop wrote:
No I just wanted to make sure everything is double.. Does not matter whether you double it or not.. You can not get the correct result after calculation... Still couldnt solve this..


I found where your problem is Smile

The CDbl function and VBscript support up to 15 digits of precision, so everything after the 15th digit from the decimal point is ignored.

To test this do the following:

Open Notepad and type this:

MsgBox(1-0.000000000000000000001092)
MsgBox(1-0.000000000000001)

Save the file as test.vbs somewhere on your hard drive and then double-click it to run, to see what I mean.
_________________
Peter
ASP & ASP.NET Articles and Tutorials
Back to top
View user's profile Send private message
GinoTheCop



Joined: 30 Dec 2005
Posts: 9

PostPosted: Fri Feb 24, 2006 11:30 am    Post subject: Reply with quote

Yes I did notice that ofter 15th digit it is ignored. But when you do this calculation with the default calculator that comes with windows it is possible.. So i thought maybe there is some way to go around this...
_________________
Cuneyt Yetgin
Back to top
View user's profile Send private message
administrator
Site Admin


Joined: 01 Oct 2004
Posts: 183

PostPosted: Fri Feb 24, 2006 11:41 am    Post subject: Reply with quote

VBScript is a scripting language, which has limited capabilities. You can move your calculations into a custom COM object (DLL) and use this object within your ASP pages. This is the only work around I see in your case.

Of course you can move directly to ASP.NET, which will save you a lot of time and effort.
_________________
Peter
ASP & ASP.NET Articles and Tutorials
Back to top
View user's profile Send private message
GinoTheCop



Joined: 30 Dec 2005
Posts: 9

PostPosted: Fri Feb 24, 2006 11:45 am    Post subject: Reply with quote

No this is a project I ve been running for 4 years now.. Thousands of lines of coding is involved and I do not have that much time to re write (and learn asp.net) everything. I was asked if i could add a frequency calculation function the the code. So I assume as you suggested I will have to look into a custom dll option... Thanx for your time and effort to help..
_________________
Cuneyt Yetgin
Back to top
View user's profile Send private message
administrator
Site Admin


Joined: 01 Oct 2004
Posts: 183

PostPosted: Fri Feb 24, 2006 11:47 am    Post subject: Reply with quote

No problem Smile

Use VB6 for the DLL, it should be very easy to create it and it should handle the calculations nicely.

Good luck and keep us posted!
_________________
Peter
ASP & ASP.NET Articles and Tutorials
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    ASPdev.org Forum Index -> ASP programming All times are GMT - 5 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2002 phpBB Group

SQL Tutorial