Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 SQL Server 2005 Forums
 Other SQL Server Topics (2005)
 Find Missing Data

Author  Topic 

kenoki007
Starting Member

3 Posts

Posted - 2012-05-01 : 20:31:04
i have a table with 3 columns Fieldunitserial, Average and timestamp

the data stores something like this

88053 71.6478805541992 2012-05-01 01:40:59.377
88053 70.8151550292969 2012-05-01 01:41:59.510
88053 70.8151550292969 2012-05-01 01:42:59.657
88053 70.8151550292969 2012-05-01 01:43:59.727
88053 70.8151550292969 2012-05-01 01:44:59.940
88053 72.4079208374023 2012-05-01 01:46:00.097
88053 72.1670303344727 2012-05-01 01:47:00.310
88053 73.0599746704102 2012-05-01 01:48:00.420

the data for 01:45:00 is missing in 1 day i got 1440 registers how can i find the data like 1:45:00 with a query, and after find it how can i insert data to fullfill the gaps. thanks in advance

visakh16
Very Important crosS Applying yaK Herder

52326 Posts

Posted - 2012-05-01 : 23:18:22
is there a master table from which you populate values? or is there a formula based on which values vary?

------------------------------------------------------------------------------------------------------
SQL Server MVP
http://visakhm.blogspot.com/

Go to Top of Page

kenoki007
Starting Member

3 Posts

Posted - 2012-05-01 : 23:35:53
the data come from an OPC datalogger, but i will look the gaps from a day before, i mean tomorrow i check the today's data for gaps.
Go to Top of Page

RL
Starting Member

15 Posts

Posted - 2012-05-07 : 00:36:59
(NOTE: I'm answering this several days after the original post because it raises an interesting question.)

In the data provided there actually isn't a row missing for 1:45, its timestamp is 1:44:59.727. Log entries prior to 1:46 all have timestamps less than one second before the target time, then it switched to less than one second after. To account for this, the SQL should allow for a "gap" of 1 or more seconds before and after the target time.

Suggested approach:

1. Create a permanent table, daily_minutes, containing one row for each minute in a day (60 * 24 = 1440 rows).
2. Write a query that returns missing rows by comparing log timestamp to daily_minutes (allowing before/after gap).

NOTE: INSERT of missing rows is more complicated, you have to figure out what "average" to use, and deal with things like multiple consecutive missing rows.

-----------------
-- STEP 1: CREATE and load table daily_minutes (only need to do this once)
-----------------
-- DROP table if it already exists
IF EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_name = 'daily_minutes')
DROP TABLE daily_minutes;

CREATE TABLE daily_minutes
(minute_num INT, minute_ts TIME);

DECLARE @minute_num INT, @minute_ts TIME, @NTot INT;
SELECT @minute_num = 1,
@minute_ts = '00:00:00', -- start at 12AM
@NTot = (60 * 24); -- 1440 minutes in a day

-- Loop to INSERT a row for each minute
WHILE (@minute_num <= @NTot)
BEGIN
INSERT INTO daily_minutes SELECT @minute_num, @minute_ts;
SELECT @minute_num = @minute_num + 1; -- increment by 1
SELECT @minute_ts = DATEADD(minute, 1, @minute_ts); -- increment by 1 minute
END;

-----------------
-- STEP 2: SQL to identify missing rows
-- (NOTE: Log table name = log_table, columns = serial, advance, log_ts)
-----------------
DECLARE
@log_day DATETIME, -- the day of the log rows
@gap INT; -- allowed gap before/after each minute

SELECT
@log_day = CONVERT(CHAR(10), getdate()-1, 120), -- yesterday 12AM
@gap = 1; -- 1 second before or after


-- CTE includes rows for target minutes that DO exist in the log table
-- (DATEADDs allow for the before/after gap)
;WITH CTE AS
(
SELECT DM.minute_num, DM.minute_ts, LT.serial, LT.average, LT.log_ts
FROM daily_minutes DM INNER JOIN log_table LT
ON LT.[log_ts] BETWEEN DATEADD(s, -@gap, @log_day + DM.minute_ts)
AND DATEADD(s, +@gap, @log_day + DM.minute_ts)
)
-- Query returns rows for minutes that are NOT in CTE
SELECT (@log_day + DM.minute_ts) AS missed_target_ts,
DM.minute_num AS DM_minute_num,
DM.minute_ts AS DM_minute_ts
FROM daily_minutes DM
WHERE DM.minute_num
BETWEEN (SELECT MIN(minute_num) FROM CTE)
AND (SELECT MAX(minute_num) FROM CTE)
AND NOT EXISTS
(SELECT 1 FROM CTE
WHERE minute_num = DM.minute_num);

Go to Top of Page
   

- Advertisement -