Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 SQL Server 2000 Forums
 SQL Server Development (2000)
 Bulk insert problems

Author  Topic 

smeghead
Starting Member

3 Posts

Posted - 2005-07-14 : 19:38:59
Hi,

Im trying to bulk insert to multiple tables from the same file using the firstrow and lastrow options. I have position information at the top of the file so I know where to start and end for each block of data to be bulk inserted. It also includes a locationId line which allows existing stock on hand figures to be removed for that location before the new ones are added.

Now when the data is inserted, I am missing some of the data at the start of the block. Ive checked and double checked the code to find the position information, and all the positions that are returned are correct. Well I used notepad's goto line number function and plugged in the numbers that each block should start and end at, and it matched.

From testing, it seems that the position information at the start is not included as a row in the line. So if the position information takes up 5 lines (then a locationId on 1 line), then the actual start and end positions are the ones I came up with, -6. This actually returned the correct results. The next problem is that this only happens on the 2nd block, and the first block is correct.

Bulk insert data is [LocationId],[StockItemId] OR [PluId],[SOH],[Date]

SOH = Stock On Hand
Plu = Product LookUp

Some data from the file:

0
0
1
19
18
36f36d30-5a66-4417-b072-39ec010b111b
36f36d30-5a66-4417-b072-39ec010b111b,0012d0a3-06c2-4b21-8bb2-1f9e90ae7503,1.0000,2005-06-18 23:59:59
36f36d30-5a66-4417-b072-39ec010b111b,00190ddf-0f0e-47ec-9913-cc97ce76c96e,0,2005-07-11 14:26:26
36f36d30-5a66-4417-b072-39ec010b111b,003f2c32-a5d8-4967-ab68-932d0b2bc3e5,1.0000,2005-07-05 15:43:30
36f36d30-5a66-4417-b072-39ec010b111b,0043da1b-5804-457c-8b1d-101db9bf0846,0,2005-07-04 11:58:06
36f36d30-5a66-4417-b072-39ec010b111b,0056e931-ea6b-43a0-ae96-eefef6cdc740,0,2005-07-07 15:06:10
36f36d30-5a66-4417-b072-39ec010b111b,005c94df-967f-4c54-83ce-27626b67c40a,0,2005-07-01 14:29:03
36f36d30-5a66-4417-b072-39ec010b111b,005e708f-22ea-4bdb-8662-1a4ecf469595,0,2005-07-05 16:41:22
36f36d30-5a66-4417-b072-39ec010b111b,006bc47b-ad15-49ac-a018-0daa528e6365,2.0000,2005-06-18 23:59:59
36f36d30-5a66-4417-b072-39ec010b111b,0089fb0b-03b7-442c-9cc4-13315571b0fc,0,2005-07-01 16:49:03
36f36d30-5a66-4417-b072-39ec010b111b,008b496c-f03f-48ab-9b5d-caf12fca29c2,2.0000,2005-07-09 14:25:01
36f36d30-5a66-4417-b072-39ec010b111b,009edcfa-6c07-4349-b74c-0f0de4b05277,0,2005-06-28 14:22:42
36f36d30-5a66-4417-b072-39ec010b111b,00a655c5-89ed-41f4-b337-dc90b3a3689e,1.0000,2005-07-02 13:47:34
36f36d30-5a66-4417-b072-39ec010b111b,00bedb7d-3c5a-4b9a-931a-e529e2762b06,1.0000,2005-07-08 16:38:01
36f36d30-5a66-4417-b072-39ec010b111b,00d550fb-c8b7-4f5c-985e-c74384c9b938,0,2005-07-09 16:12:17
36f36d30-5a66-4417-b072-39ec010b111b,00e73892-09d1-4bc6-a78e-42050d4a9a77,0,2005-06-29 16:01:07
36f36d30-5a66-4417-b072-39ec010b111b,00e90b6f-86db-43bc-8f91-c8f9a3f3bb99,3.0000,2005-06-18 23:59:59
36f36d30-5a66-4417-b072-39ec010b111b,00ea415a-9f2f-483d-ac18-7ae8fe540328,1.0000,2005-06-28 16:01:53
36f36d30-5a66-4417-b072-39ec010b111b,00eb7ec8-e8e3-4046-a8a8-20e51fed4ae9,0,2005-07-05 16:36:29
36f36d30-5a66-4417-b072-39ec010b111b,00fd8a77-7181-432b-973f-07103f342ea6,-1.0000,2005-07-08 19:24:42
36f36d30-5a66-4417-b072-39ec010b111b,fb69f67f-0670-4170-9233-ad3b26099062,4.0000,2005-07-07 17:15:44
36f36d30-5a66-4417-b072-39ec010b111b,fbcb8cd8-95e9-4f4b-b277-2d580d242024,8.0000,2005-07-05 12:45:57
36f36d30-5a66-4417-b072-39ec010b111b,fbd125f6-b9a9-4d8e-8382-a4b2e47e8e57,10.0000,2005-07-08 19:50:28
36f36d30-5a66-4417-b072-39ec010b111b,fc421ec1-88e2-4720-88e3-188db745fa2d,11.0000,2005-07-08 17:58:14
36f36d30-5a66-4417-b072-39ec010b111b,fc8909b4-bede-47db-ae93-043c47813736,1.0000,2005-06-18 23:59:59
36f36d30-5a66-4417-b072-39ec010b111b,fcf6288d-093d-437c-865a-88873702ba89,2.0000,2005-07-02 13:40:08
36f36d30-5a66-4417-b072-39ec010b111b,fd20e1ed-13fd-475d-b5d5-fe0b33fec957,0,2005-07-07 15:07:15
36f36d30-5a66-4417-b072-39ec010b111b,fd24a577-d3f9-4ffc-87fb-69b8ab2e41bf,0,2005-07-12 14:20:26
36f36d30-5a66-4417-b072-39ec010b111b,fde37a29-8ccd-420c-9c82-cddf3ed59358,1.0000,2005-06-18 23:59:59
36f36d30-5a66-4417-b072-39ec010b111b,fdeaab00-c6d2-411d-8547-5dbf51bfe4ed,0,2005-07-09 16:13:56
36f36d30-5a66-4417-b072-39ec010b111b,fdf671ce-5254-48eb-8aad-72c14189d469,8.0000,2005-07-14 13:07:44
36f36d30-5a66-4417-b072-39ec010b111b,fe1a7274-9d3b-498d-ba1c-2ef551f8ba89,-1.0000,2005-07-08 11:36:30
36f36d30-5a66-4417-b072-39ec010b111b,fe268597-47c0-4e2b-943a-569d310f3a33,1.0000,2005-07-08 20:53:47
36f36d30-5a66-4417-b072-39ec010b111b,fe6f3bbd-d369-4c1d-a079-5ba5c5f7d2cb,2.0000,2005-06-18 23:59:59
36f36d30-5a66-4417-b072-39ec010b111b,fec740b3-e051-471e-be32-a674fc31d52f,2.0000,2005-07-08 17:43:49
36f36d30-5a66-4417-b072-39ec010b111b,fed09442-9943-410e-bf49-ed71c0775446,5.0000,2005-06-18 23:59:59
36f36d30-5a66-4417-b072-39ec010b111b,ff424db2-6d5b-42f6-b916-146a19a9d7f5,5.0000,2005-07-12 16:15:24
36f36d30-5a66-4417-b072-39ec010b111b,ff4adaa3-2c9c-4160-b886-6055157dc691,0,2005-07-03 15:51:23

And the sql query which bulk inserts the data is:


IF EXISTS (
SELECT *
FROM tempdb.dbo.sysobjects
WHERE [name]='##StockOnHand'
AND type='u'
)
DROP TABLE ##StockOnHand

CREATE TABLE ##StockOnHand (LocationId uniqueidentifier NOT NULL, PluId uniqueidentifier NOT NULL,
SOH decimal(19,6) NOT NULL, [Date] datetime NOT NULL)

BULK INSERT ##StockOnHand
FROM 'c:\\temp\\SOH Check\\SmallTest.txt'
WITH
(
FIRSTROW =7,
LASTROW =25,
FIELDTERMINATOR = ','
)

DELETE
FROM StockOnHand
WHERE EXISTS (
SELECT *
FROM ##StockOnHand sohT
WHERE sohT.PluId=StockOnHand.PluId
AND sohT.LocationId=StockOnHand.LocationId
)

INSERT INTO StockOnHand (LocationId,PluId,SOH,[Date])
SELECT LocationId,PluId,SOH,[Date]
FROM ##StockOnHand

DROP TABLE ##StockOnHand

The second query which runs has the following bulk insert code:

BULK INSERT ##StockOnHand_si
FROM 'c:\\temp\\SOH Check\\SmallTest.txt'
WITH
(
FIRSTROW =26,
LASTROW =43,
FIELDTERMINATOR = ','
)

Note: this first and second query are basically the same, apart from the table name and one column name being changed (StockItemId changes to PluId). Ive just shown the bulk insert code for the second query, and not the entire query.

Now in the first query, I get 19 rows added, which is fine. The second query I get 12 rows added, which is wrong. It should be 18! The difference is 6, which happens to be the data at the top (5 position information lines, 1 locationId line).

Does anyone know why this would happen? I think I will have to split the file up, but it would be nice to do this all in one file as it is sent around to stores to keep their stock on hand figures accurate (which im sad to say they are not!!).

Cheers,

Trevor

nosepicker
Constraint Violating Yak Guru

366 Posts

Posted - 2005-07-14 : 21:00:21
In my experience with this kind of data, you have no choice but to remove the first 6 lines. What is happening is that when you tell BULK INSERT to start at line 7, it is actually starting at the seventh line that it can cleanly bulk insert. Therefore, if you check your data, I'll bet you'll see that the first line of the first block is actually the one with the date "2005-07-05 16:41:22", not the one with the date "2005-06-18 23:59:59". Your second block is missing 6 lines because the BULK INSERT actually started at something like line 32 instead of 26, because it started at the point where the data was bulk insertable.

If you experiment, you will find that you will never be able to get the first line of insertable data into your table (this will be line 7 in your file). If your FIRSTROW is 1 or 2 or 6 or 7, this line will always be skipped. Therefore, the best thing is just to clean out the first 6 lines and leave a clean set of data. BULK INSERT or the BCP function is not that intelligent - it simply looks for column delimiters and row delimiters.

Another option I just thought of is that you could potentially put those 6 lines at the end of the file instead of the beginning. The BULK INSERT will give you an error when it reaches those last 6 lines, but at least you will be able to get all your other data inserted correctly.
Go to Top of Page

smeghead
Starting Member

3 Posts

Posted - 2005-07-14 : 23:50:31
Thanks for the reply nosepicker, I tried it without the data at the start and it looks like it worked fine. It shouldnt be too hard to extract the position and location information from the file, I hope. ;)

Cheers,

Trevor
Go to Top of Page
   

- Advertisement -