Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 SQL Server 2008 Forums
 SQL Server Administration (2008)
 Long, long, long running transactions

Author  Topic 

Michael Valentine Jones
Yak DBA Kernel (pronounced Colonel)

7020 Posts

Posted - 2012-01-20 : 15:50:48
I have an odd problem I never saw before. I have a server where all databases are set to read committed snapshot. There is an application that is leaving transactions open for long periods of time (weeks) and the version store in the tempdb database grows until tempdb runs out of space.

The programmer refuses to fix the problem or admit it is a problem. I don’t think there is anything I can do from the server side, except to kill the connections when it becomes a problem. I don’t think there is a command that I can use to force the connection to commit or rollback the open transaction.

Any suggestions that don't involve violence against the programmer?




CODO ERGO SUM

tkizer
Almighty SQL Goddess

38200 Posts

Posted - 2012-01-20 : 15:59:58
We have this same exact problem! When the transaction is left open for one particular database and stays that way for a period of time, it starts seriously impacting performance of the others due to the size of the version store.

In our case, the developers have been unable to find the bug. They initially didn't believe us until we got Microsoft involved, and MS was able to show exactly what the spid had been doing and that it never ran commit/rollback and never again ran anything again on that spid. They are using Hibernate and other things, which handle their transaction logic. The bug is believed to be in those 3rd party tools, since the Java code doesn't handle that layer. I'm not a developer, so I have no idea if they are right or not.

Anyway, what we have done is written a script to check for this condition via sysprocesses and then killed the connection after a period of time. I've also considered launching that script when the version store reaches a certain point.

Here's the query I use to check for the size of the version store (stolen from somewhere on the web):

SELECT
user_object_perc = CONVERT(DECIMAL(6,3), u*100.0/(u+i+v+f)),
internal_object_perc = CONVERT(DECIMAL(6,3), i*100.0/(u+i+v+f)),
version_store_perc = CONVERT(DECIMAL(6,3), v*100.0/(u+i+v+f)),
free_space_perc = CONVERT(DECIMAL(6,3), f*100.0/(u+i+v+f)),
[total] = (u+i+v+f)
FROM (
SELECT
u = SUM(user_object_reserved_page_count)*8,
i = SUM(internal_object_reserved_page_count)*8,
v = SUM(version_store_reserved_page_count)*8,
f = SUM(unallocated_extent_page_count)*8
FROM sys.dm_db_file_space_usage
) x;

And here's part of the script that checks sysprocesses:

--Open transactions, last batch an hour or more ago
select
convert(varchar(20), db_name(dbid)) as DatabaseName,
datediff(mi, last_batch, getdate()) as TransactionDuration,
convert(varchar(30), hostname) as HostName,
spid
from master..sysprocesses
where open_tran <> 0 and last_batch < dateadd(hh, -1, getdate())
order by 1 asc, 2 desc



Tara Kizer
Microsoft MVP for Windows Server System - SQL Server
http://weblogs.sqlteam.com/tarad/

Subscribe to my blog
Go to Top of Page

tkizer
Almighty SQL Goddess

38200 Posts

Posted - 2012-01-20 : 16:01:42
I should add that RCSI is only enabled for like 3 databases on this instance. Only the other RCSI databases are affected when the culprit RCSI database has this condition occur due to the application bug. So the non-RCSI databases don't experience any performance problems, and the ones using RCSI experience a performance problem because of how big the version store is.

Tara Kizer
Microsoft MVP for Windows Server System - SQL Server
http://weblogs.sqlteam.com/tarad/

Subscribe to my blog
Go to Top of Page

robvolk
Most Valuable Yak

15732 Posts

Posted - 2012-01-20 : 16:04:56
Can you alter the database to turn off read committed snapshot? Is the benefit it provides worth the misuse you're encountering?

If this is happening in production I think you're justified in killing the developer if he/she won't fix it. Make sure their boss understands that production stops when this occurs. (It might be useful to have this happen while you're on vacation and no one else can fix it)

In the past I've set up jobs that killed connections that were open too long (e.g. last batch > 48 hours ago). It's easy to ignore system SPIDs.
Go to Top of Page

tkizer
Almighty SQL Goddess

38200 Posts

Posted - 2012-01-20 : 16:05:56
In my case, the benefit of RCSI definitely outweighs the problems this bug is causing. RCSI is awesome!

Tara Kizer
Microsoft MVP for Windows Server System - SQL Server
http://weblogs.sqlteam.com/tarad/

Subscribe to my blog
Go to Top of Page

Michael Valentine Jones
Yak DBA Kernel (pronounced Colonel)

7020 Posts

Posted - 2012-01-20 : 16:26:13
In my case, I know exactly what is happening. The application is making a connection with autocommit set to off.

When a connection is made with autocommit set off, the database engine automatically starts an implicit transaction when it first executes any of these statements:
INSERT, DELETE, SELECT, UPDATE, OPEN, FETCH, TRUNCATE TABLE, CREATE, DROP, REVOKE, or GRANT

The transaction remains in effect until you issue a COMMIT or ROLLBACK statement. If the transaction is not committed or rolled back, it just stays open, and that is what we are seeing.

Implicit Transactions
http://msdn.microsoft.com/en-us/library/ms188317.aspx

API Implicit Transactions
http://msdn.microsoft.com/en-us/library/ms175182.aspx


You may want to look at my second query below to see more detail on the transactions.
-- Check size of version store
select
[Version Store Pages Used] =
sum(version_store_reserved_page_count),
[Version Store Space MB] =
convert(numeric(10,2),round(((sum(version_store_reserved_page_count)*1.0)/128.00),2))
from
sys.dm_db_file_space_usage;


-- Get info on open snapshot transactions
select
a.transaction_id,
a.session_id,
a.is_snapshot,
d.login_time,
b.transaction_begin_time,
last_batch_time = d.last_batch,
Login_Hours =
convert(numeric(8,2),round(datediff(ss,d.login_time,getdate())/3600.0000,2)),
Transaction_Hours =
convert(numeric(8,2),round(datediff(ss,b.transaction_begin_time,getdate())/3600.0000,2)),
Last_Batch_Hours =
convert(numeric(8,2),round(datediff(ss,d.last_batch,getdate())/3600.0000,2)),
b.name,
b.transaction_type,
transaction_type_desc =
case b.transaction_type
when 1 then 'Read/write transaction'
when 2 then 'Read-only transaction'
when 3 then 'System transaction'
when 4 then 'Distributed transaction'
else ''end,
b.transaction_state,
transaction_state_desc =
case b.transaction_state
when 0 then 'Transaction has not been completely initialized yet.'
when 1 then 'Transaction has been initialized but has not started.'
when 2 then 'Transaction is active. '
when 3 then 'Transaction has ended. Used for read-only transactions.'
when 4 then 'Commit process has been initiated on the distributed transaction.'
when 5 then 'Transaction is in a prepared state and waiting resolution.'
when 6 then 'Transaction has been committed.'
when 7 then 'Transaction is being rolled back. '
when 8 then 'transaction has been rolled back.'
else '' end,
hostname = rtrim(d.hostname),
Program_name = rtrim(d.Program_name),
loginame= rtrim(d.loginame),
SQL_text = e.text
from
sys.dm_tran_active_snapshot_database_transactions a
join
sys.dm_tran_active_transactions b
on a.transaction_id = b.transaction_id
join
master.dbo.sysprocesses d
on a.session_id = d.spid
cross apply
sys.dm_exec_sql_text( d.sql_handle ) e
order by
a.transaction_id







CODO ERGO SUM
Go to Top of Page

Michael Valentine Jones
Yak DBA Kernel (pronounced Colonel)

7020 Posts

Posted - 2012-01-23 : 22:32:06
I was able to get a resolution for this problem.

I guessed that the problem might be due to having unbalanced BEGIN TRANSACTION/COMMIT, especially with the implict transaction that is started when AUTOCOMMIT is off.

I got the developer to use this code every time they do a COMMIT, and I no longer see open transactions:

-- Make sure transaction is completely committed
while @@trancount > 0 begin commit end






CODO ERGO SUM
Go to Top of Page

tkizer
Almighty SQL Goddess

38200 Posts

Posted - 2012-01-24 : 00:03:59
Thanks! I will give that suggestion to our developers. Problem is that some of these apps are in sustaining mode and no longer have developers for them.

Tara Kizer
Microsoft MVP for Windows Server System - SQL Server
http://weblogs.sqlteam.com/tarad/

Subscribe to my blog
Go to Top of Page

Kristen
Test

22859 Posts

Posted - 2012-01-24 : 02:24:31
"-- Make sure transaction is completely committed
while @@trancount > 0 begin commit end
"

I think this should be

while @@trancount > @trancountAtStartOfProc begin commit end[/i]"

otherwise you risk committing the "callers" level(s) too, and any ROLLBACK the caller tries to do will then fail (and the data will be "gold" already )
Go to Top of Page

Michael Valentine Jones
Yak DBA Kernel (pronounced Colonel)

7020 Posts

Posted - 2012-01-24 : 11:24:29
quote:
Originally posted by Kristen

"-- Make sure transaction is completely committed
while @@trancount > 0 begin commit end
"

I think this should be

while @@trancount > @trancountAtStartOfProc begin commit end[/i]"

otherwise you risk committing the "callers" level(s) too, and any ROLLBACK the caller tries to do will then fail (and the data will be "gold" already )



The code is not in a proc, so that doesn't apply.

My objective was just to get to the point where the transaction is completely committed, and not holding it open for a long time.






CODO ERGO SUM
Go to Top of Page

Michael Valentine Jones
Yak DBA Kernel (pronounced Colonel)

7020 Posts

Posted - 2012-01-24 : 11:28:37
quote:
Originally posted by tkizer
...Problem is that some of these apps are in sustaining mode and no longer have developers for them....



That's a big problem for me.

Someone calls with a problem, I suggest that they contact the developer (or vendor) for help, and hear that the developer left the company two years ago (or the vendor went out of business years ago).






CODO ERGO SUM
Go to Top of Page

Kristen
Test

22859 Posts

Posted - 2012-01-24 : 12:27:40
"The code is not in a proc, so that doesn't apply"

<ThumbsUp>

"...Problem is that some of these apps are in sustaining mode and no longer have developers for them...."

As a point of interest, do you have Escrow on Source Code that kicks in at that point?
Go to Top of Page
   

- Advertisement -