Author |
Topic |
Kristen
Test
22859 Posts |
Posted - 2014-10-09 : 09:24:19
|
Hi gang,I want to generate some parametrised dynamic SQL to execute with sp_ExecuteSQL.I have a table with a list of Columns to use in the WHERE clause, and whether each column is a Start, End, Range, Match or Contains condition. so lets assume:MyColumn1 STARTMyColumn1 ENDMyColumn2 CONTAINSwhich should generate a WHERE clause along the lines of:WHERE MyColumn1 >= @MyColumn1_STARTAND MyColumn1 <= @MyColumn1_ENDAND MyColumn2 LIKE '%' + @MyColumn2 + '%' So I want to mechanically generate the SQL string for that WHERE clause. I can do that easily (and generically) from the table which defines the columns / conditions.However ...If the user does not provide a value for a parameter - say @MyColumn1_END - then that test should be left out. I could generate the SQL as:WHERE (@MyColumn1_START IS NULL OR MyColumn1 >= @MyColumn1_START)AND (@MyColumn1_END IS NULL OR MyColumn1 <= @MyColumn1_END)AND (@MyColumn2 IS NULL OR MyColumn2 LIKE '%' + @MyColumn2 + '%') but I suspect that will be very inefficient, and what I would like to do is to leave the test out all together if the @ConditionColumn is not provided (NULL)).Problem is ... that means that I cannot use generic code to construct the SQL string, as that code will need to be able to "see" the individual parameters:SELECT @strWHERE = ... string concatenation ...FROM MyDefinitionTableWHERE (ColumnName = 'MyColumn1_START' AND @MyColumn1_START IS NOT NULL)OR (ColumnName = 'MyColumn1_END' AND @MyColumn1_END IS NOT NULL)OR (ColumnName = 'MyColumn2' AND @MyColumn2 IS NOT NULL) (I was hoping that the code to construct the SQL string could be centralised in its own SProc, and not need to be specific to each query Sproc.Any ideas and/or advice on efficiency if I go down the(@MyColumn1_START IS NULL OR MyColumn1 >= @MyColumn1_START)... route?Thanks |
|
gbritton
Master Smack Fu Yak Hacker
2780 Posts |
Posted - 2014-10-09 : 09:40:26
|
I can't imagine that the extraneous test will impact performance in any measurable way. i do suggest you read this article though:http://sqlinthewild.co.za/index.php/2009/03/19/catch-all-queries/ |
|
|
Kristen
Test
22859 Posts |
Posted - 2014-10-10 : 08:01:56
|
Thanks, I'm familiar with Gail's article - it makes the point that the query plan will be cached and the plan will use whichever index is most suitable for the first time the query is used. Thus if I include the extraneous tests in the WHERE clause the exact same SQL statement will be used each time, and thus the cached query plan will be used based on the originally selected most-appropriate-index, regardless of which parameters are NULL, whereas if I craft specific SQL for each scenario the exact WHERE clause will vary and thus the SQL statement will be separately cached for each scenario, which is why I think it is important not to carry the extraneous tests [if I can!]However, that has reminded me of an old trick- assuming it has not been optimised out in recent versions of SQL - which is to add some dummy conditions to the WHERE clause (actually, probably any part of the SQL statement works OK, possibly even a comment) to make each variant unique.So I could add even more extraneous tests for each parameter that was actually used / NOT NULL. For example I could form a Binary value for the parameters that are used:SELECT @MyUsedColumns = CASE WHEN @MyColumn1_START IS NOT NULL THEN 1 ELSE 0 END + CASE WHEN @MyColumn1_END IS NOT NULL THEN 2 ELSE 0 END + CASE WHEN @MyColumn2 IS NOT NULL THEN 4 ELSE 0 END and then append to the SQL StringSELECT @strWHERE = @strWHERE + ' AND ' + CONVERT(varchar(10), @MyUsedColumns) + ' <> -1'so I would wind up with something like:WHERE (@MyColumn1_START IS NULL OR MyColumn1 >= @MyColumn1_START)AND (@MyColumn1_END IS NULL OR MyColumn1 <= @MyColumn1_END)AND (@MyColumn2 IS NULL OR MyColumn2 LIKE '%' + @MyColumn2 + '%')AND 7 <> -1 |
|
|
gbritton
Master Smack Fu Yak Hacker
2780 Posts |
Posted - 2014-10-10 : 08:53:27
|
So...just how bad is the performance? you can also influence the optimizer by setting up different plans for different combinations of arguments. that may help with the heaviest, most-frequently used queries. also, consider the cost of recompiling the query if you try to fool the optimizer into thinking that it hasn't seen some combination before. If the queries are fast anyway, why incur the recompilation overhead? |
|
|
Kristen
Test
22859 Posts |
Posted - 2014-10-10 : 10:46:03
|
I don't know how bad the performance might be, but as this is a generic tool what I don't want is to discover we have a crisis once it is in Production, at some unknown future point in time, so we want to program the issue out in Development so that it is properly optimised. For that reason I also don't want to use recompile - ideally there will be a cached query plan, and if that is the case I want to reuse it! and for it to be "appropriate" for the actual query the user is trying to run.Most of the time performance is not going to be an issue, even with the wrong index, but the cost of "improving" this code once it is in production will be huge, so I don't want to have that future threat |
|
|
gbritton
Master Smack Fu Yak Hacker
2780 Posts |
Posted - 2014-10-10 : 11:04:23
|
If you discover problems related to sub-optimal plans after you promote to production, you can still set up custom plans for the troublesome queries. It sounds like you don't yet know what the problems might be. You can't anticipate everything unfortunately. |
|
|
Kristen
Test
22859 Posts |
Posted - 2014-10-10 : 11:48:35
|
I can't set up custom plans as the whole crux is that the SQL is generated dynamically and outwith the control of the Sproc that is running the query.We don't change anything in Production without a full QA process, and with something like this, which is related to dynamic queries (which are in effect constructed by the end user), the breath of the testing is huge and expensive.Hence I don't want to be using queries that are open to possible poor query plans.I appreciate your help, but you are coming at this from a different angle to the one I feel comfortable adopting. |
|
|
gbritton
Master Smack Fu Yak Hacker
2780 Posts |
Posted - 2014-10-10 : 11:54:42
|
Well, good luck then! You have a two big problems:1. You can't anticipate how your query will be used2. You can't code for all contingencies.Surely you monitor production and fix incorrect or poorly performing queries? That is a key part of a DBA's job. So you have to cycle back through QA? I understand that. It's no big deal, just a little tedious. In your case it's probably necessary. |
|
|
ScottPletcher
Aged Yak Warrior
550 Posts |
Posted - 2014-10-10 : 12:39:05
|
quote: which should generate a WHERE clause along the lines of:WHERE MyColumn1 >= @MyColumn1_STARTAND MyColumn1 <= @MyColumn1_ENDAND MyColumn2 LIKE '%' + @MyColumn2 + '%'So I want to mechanically generate the SQL string for that WHERE clause. I can do that easily (and generically) from the table which defines the columns / conditions.
Hmm, so you know the names of the local variables that will be used? If so, you can actually generate code to dynamically read those values, which you could then test as you were dynamically constructing the SQL string. |
|
|
Kristen
Test
22859 Posts |
Posted - 2014-10-10 : 13:04:19
|
quote: Originally posted by gbritton You have a two big problems:1. You can't anticipate how your query will be used
No, I can't, but already I know that if I do nothing the query will be cached and use the wrong index for alternative scenarios. So no point me rolling out code that has that "known" flawquote: 2. You can't code for all contingencies.
Indeed ... sadly! We have an incredibly low failure rate in production. I have no idea if that is a good thing (maybe we spend far too much money in Development in order to achieve that), but it does mean that when we say "Deliver by February" that we are not having to cater for unexpected and unpredictable amounts of downtime patching up and mending production systems.quote: Surely you monitor production and fix incorrect or poorly performing queries?
We do (our code self monitors its own performance so we get an alert if any SProcs start performing adrift from long term average), but it is incredibly rare for us to have to modify production code because of a poor performing query. We aim to take care of that in DEV not as part of Production. |
|
|
gbritton
Master Smack Fu Yak Hacker
2780 Posts |
Posted - 2014-10-10 : 13:22:31
|
There's an old saying, "In theory, theory is the same as practice, but in practice, it isn't" Paraphrasing: In theory, development is the same as production, but in practice, it isn't.DEV environments frequently run with a subset of real data or even totally simulated data. also, they rarely get stressed (CPU, IO) since they're not getting hit by as many users. You can do a lot in DEV (I sure do!) but sometimes the way things get used in PROD is so different. You just can't anticipate it. Its why you need a feedback loop for continuous improvement.If you "know that if I do nothing the query will be cached and use the wrong index for alternative scenarios" then develop custom query plans for the ones you DO know about before you promote. |
|
|
Kristen
Test
22859 Posts |
Posted - 2014-10-10 : 13:26:03
|
quote: Originally posted by ScottPletcher Hmm, so you know the names of the local variables that will be used? If so, you can actually generate code to dynamically read those values, which you could then test as you were dynamically constructing the SQL string.
I have trialled that, and it works fine - along the lines of the code that I posted earlierSELECT @strWHERE = ... string concatenation ...FROM MyDefinitionTableWHERE (ColumnName = 'MyColumn1_START' AND @MyColumn1_START IS NOT NULL)OR (ColumnName = 'MyColumn1_END' AND @MyColumn1_END IS NOT NULL)OR (ColumnName = 'MyColumn2' AND @MyColumn2 IS NOT NULL) as this will only concatenate "snippets" of the WHERE clause for parameters that are not null (assume that ColumnName is the column in MyDefinitionTable which defines the column, and thus the parameter, being tested).But ... it means that I would have to build the WHERE cause @variable in the Query Sproc itself - i.e. every single Query SProc will have a duplicate of this code, in order that it can have a where clause specific to the parameters that are in use.What I would like to build is a generic solution - the MyDefinitionTable would be used by every Query Form we have to allow an "Advance Mode" query that is outside the scope of the regular "Fill in this Criteria Form and then press QUERY" approach that we normally use.For that to work I want the regular Query Sproc to be able to get some Dynamic SQL from a central SProc that can generate the code without knowing about the parameters. That way if we want to introduce new "abilities" into MyDefinitionTable the central Dynamic SQL code generation routine can be adapted to handle that, and leave the individual Query Sprocs to just run the SQL and deal with the resultset - and of course if we find a bug in the SQL Code Generation we only have to fix that in one place too.Hence I think my idea of a AND 7 <> -1 artificial comparison test might be the easiest solution, as that is easy for each SProc to generate, and unlikely that the code will ever need to be changed.I have been toying with generating the SQL in the Front End application code - i.e. passing the SQL for the WHERE clause as a parameter to the Sproc, but any changes to front end code represent a huge rollout effort for us, compared to any changes in SQl / Sprocs, hence I'd prefer to find a SQL-side solution if I can. |
|
|
Kristen
Test
22859 Posts |
Posted - 2014-10-10 : 13:33:54
|
quote: Originally posted by gbritton There's an old saying, "In theory, theory is the same as practice, but in practice, it isn't"
Well ... to me the fact that for us to change something in Production is incredibly rare suggests that our test environment is tight enough that I can rely on it. We have put a lot of effort, over the years, into getting our QA act together with that end in mind.quote: If you "know that if I do nothing the query will be cached and use the wrong index for alternative scenarios" then develop custom query plans for the ones you DO know about before you promote.
I'm struggling with how I would do that:1) I mechanically generate a SQL statement. Lets assume it has known potential inefficiencies (e.g. using the wrong index in some instances).2) The SQL will be executed using a single sp_ExecuteSQL - i.e. ALL paths through the SProc will use that same single sp_ExecuteSQL (albeit that the SQL string may vary)How do I apply a custom query plan? (And before I get to that point how do I determine that a particular query needs Custom Query Plan #X?)You may well be envisaging something that I haven't yet got my head around :)My plan is to make sure that the SQL String is "unique" to the query so that it gets its own cached query plan (rather that sharing one with all other queries, including using whichever index "whoever was first" dictates) |
|
|
gbritton
Master Smack Fu Yak Hacker
2780 Posts |
Posted - 2014-10-10 : 13:38:32
|
Why not have the Query Sprocs call an inline function or view that returns the WHERE clause variable? The function can return a default (e.. WHERE 1 = 1 or WHERE 1 <> 1 depending on what you want) so that sprocs that don't need to test anything run find with the code returned (either always true or always false (though that seems usesless!))Just trying to stay DRY! |
|
|
Kristen
Test
22859 Posts |
Posted - 2014-10-11 : 06:31:55
|
A Function would do (as an alternative to calling an SProc). As an aside we find functions inefficient (as a general sweeping statement!) but for this job the function will be getting half a dozen rows from a table with only has, at most, a few thousand rows, so "inefficient" is not something to need to worry about, but that aside given a choice I would use an Sproc instead, from preference.So I suppose the way this would work is:My SQL to generate the SQL Where Clause statement needs to be something like this:SELECT @strSQL = @strSQL + ' AND ' + MyColumnName + ' = @' + MyColumnNameFROM MyDefinitionTableWHERE SomePrimaryKey = @MyID and then in order for that to only include Columns where the parameter is NOT NULL I need the WHERE clause to test MyColumnName against the actual parameter. Can't hand-off that capability to the function, so I suppose my Function should return the SQL snippet for each row in MyDefinitionTable and the MyColumnName so that I could then make the test back in the Sproc:SELECT @strSQL = @strSQL + FunctionSQLSnippetFROM dbo.MyFunction(@MyID)WHERE (FunctionColumnName = 'MyColumn1_START' AND @MyColumn1_START IS NOT NULL)OR (FunctionColumnName = 'MyColumn1_END' AND @MyColumn1_END IS NOT NULL)OR (FunctionColumnName = 'MyColumn2' AND @MyColumn2 IS NOT NULL) and the function would look something like this:CREATE FUNCTION dbo.MyFunction( @DefinitionID int -- ID of the definition to use in MyDefinitionTable)RETURNS @tblArray TABLE( S_ID int IDENTITY(1,1) NOT NULL PRIMARY KEY, FunctionSQLSnippet varchar(8000) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL, FunctionColumnName varchar(255) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL)ASBEGIN INSERT @tblArray(FunctionSQLSnippet)-- NOTE: FunctionSQLSnippet will actually use some sort of "Action" column to generate -- StartsFrom, EndsAt, Range, Contains, Equals and other variations of SQL code! SELECT [FunctionSQLSnippet] = ' AND ' + MyColumnName + ' = @' + MyColumnName, [FunctionColumnName] = MyColumnName FROM dbo.MyDefinitionTable WHERE SomePrimaryKey = @DefinitionID RETURNGO I'm still bothered that the relationship between MyColumnName and the SQL Snippet will be "more complex" than I can currently envisage, such that communication between Function and SProc will require too much code [i.e. duplicated in every Sproc and not DRY] ... basically just a BTDTGTTS nagging doubt / forward-worry.But I could have a crack at using that approach as a First Attempt and see how far it gets me ... :) |
|
|
gbritton
Master Smack Fu Yak Hacker
2780 Posts |
Posted - 2014-10-11 : 09:36:50
|
why not have a master proc that auto-generates the query procs on demand? SO, write the master proc once. Every query calls it. If there exists a custom query proc, master calls custom; if not, master builds the custom proc and calls it. I suppose that may violate some production rules, though. |
|
|
Kristen
Test
22859 Posts |
Posted - 2014-10-12 : 04:57:38
|
I may not be describing what I want very well - what I want is a Master Proc. How would that be able to see that individual parameters are NULL, or not? |
|
|
gbritton
Master Smack Fu Yak Hacker
2780 Posts |
Posted - 2014-10-12 : 09:16:59
|
I suppose the parameters would be passed to the master? Anyway the idea I had was for a factory proc, in OOP terms |
|
|
Kristen
Test
22859 Posts |
Posted - 2014-10-12 : 20:48:21
|
quote: Originally posted by gbritton I suppose the parameters would be passed to the master?
The intention is that it is a generic Sproc, but the parameters will be different every time. Is there a way I can code that (within the master proc) in a "generic" manner? |
|
|
gbritton
Master Smack Fu Yak Hacker
2780 Posts |
Posted - 2014-10-13 : 08:25:42
|
OK but the parameter types are consistent. So the master can see them and act on null values, right? |
|
|
gbritton
Master Smack Fu Yak Hacker
2780 Posts |
|
Next Page
|