I have just completed a largish project designing and building a back-end system that is implemented across a series of processes, some of which are multithreaded. I have effectively designed and built this system from database design through to external interface implementation. A relatively complex system by requirement, but it has all come together fairly nicely (pats self on back ;-)).
What I have come to realise however in looking back at this experience, and in stepping back and looking at the 'complete' system, is that the part where things get a little twisted is in my sproc implementation. Although it all works nicely, I find that development of TSQL and SQL scripts in general, when compared to .NET (and OO) development tends to get a little 'hairy'. I find that TSQL and SQL don't really lend themselves to structured software design techniques, so I just tend to develop scripts (using as much foresight and caution as is practical of course) as the need develops.
I've studied databases to stage III at university, and have worked closely with databases throughout my IT career, but have found that there is not much documentation out there (or perhaps I've just not come across much) that goes into in-depth theoretical detail as to how sprocs can be organised in a structured manner.
I'd like to open a discussion about this to investigate what peoples thoughts are in this regard.
"…further to my previous email…
For example – you have then need to access/update/insert lots of small bits and pieces of seemingly unrelated data – some scalar, some that would return/update/insert only a few 'small' records. Lump them all together in a single API type sproc ( i.e. pass thru a parameter, and receive back the appropriate data/confirmation), or make a separate sproc for each?
Another scenario - you have large chunks of data that are being pushed and pulled through a system. The data has potentially been modified as it is shifted around. The database code needs to ( i.e. it is appropriate for the db code to) analyse it to be able to determine if a change has occurred, then marshal the data accordingly. The data may be passed in this way though several different marshalling 'gates' before it finds it's way to it's home in the db table(s). Is it best to generate a tree-like series of sprocs that the data percolates though before it reaches the tables, or try to lump the whole thing into a single script?
What I'd like to be able to develop I think is a more mature way of managing the problem of data manipulation through the sproc 'layer'. Ideally I'd like to be able to apply some methodology - in the same way I can organise my .NET code into manageable portions by employing OO methodologies.
After some development experience it does seem to become clearer to distinguish which section of a data manipulation procedure should be performed in db-side code and which shouldn't. Some discussion on this topic would also be warranted though I think, especially with the onslaught of technologies such as DLINQ which are set to dissolve these db-side/custom-app-side data manipulation boundaries even further.
Edit - 20101016:
When I originally made this post on the NZ .NET user group mailing list, several senior developers came back to me and said bluntly "don't put that business logic in your database" and/or "the RDBMS is designed for data, not code"...all I can say is, they were so, so right...