I recently noticed that Red Gate have launched themselves into the SSIS component market by releasing a new Data Cleanser component, albeit in beta for now. It seems to be quite a simple component, bringing together several features that you can find elsewhere, but with a suitable level  polish that you’d expect from them. String operations include find and replace with regular expressions, case formatting and trim, all of which are available today in one form or another, but will the RedGate factor appeal to people? Benefits include ease of use, all operations in one place, versus ... [More]

SQLBits 9 has now been announced as 29th September to 1st October 2011 at the Adelphi Hotel in Liverpool. This will follow the now familiar three day format, a training day, and two full days of concurrent sessions. Saturday 1st October will of course be the free community day. Despite growing the event quite dramatically over the past four years, this is something we are all very proud to have maintained and is a key factor when planning the events. Plenty of more info to come, but in the meantime session submission is now open so why not submit an abstract?

On Wednesday 18th May 1Pm –2PM Eastern Time I am going to be helping Attunity deliver a webcast.  In it I want to highlight some of the challenges we face in getting data around our environments.  I want to highlight the Microsoft products that can help us realise Business Intelligence. It promises to be a good hour.  Hope to see you all there.  Here is the link Attunity Webcast

Our friend Alistair Aitchison has been busy doing some crazy stuff with spatial data in SQL Server.  He just pinged me with his latest creation.  In the words of Rolf Harris “Can you tell what it is yet?”

Whilst getting some courseware ready I was playing around writing some code and I decided to very simply show when a window starts and ends based on you asking for a TumblingWindow of n time units in StreamInsight.  I thought this was going to be a two second thing but what I found was something I haven’t yet found documented anywhere until now.   All this code is written in C# and will slot straight into my favourite quick-win dev tool LinqPad   Let’s first create a sample dataset   var EnumerableCollection = new [] { new {id = 1, StartTime = DateTime.Parse(&q... [More]

I had an hour spare this afternoon so I wanted to have another play with Reactive Extensions in .Net and StreamInsight.  I also didn’t want to simply use a console window as a way of gathering events so I decided to use a windows form instead. The task I set myself was this. Whenever I click on my form I want to subscribe to the event and output its location to the console window and also the timestamp of the event.  In addition to this I want to know for every mouse click I do, how many mouse clicks have happened in the last 5 seconds. The second point here is really int... [More]

In his blogpost Roman from the StreamInsight team asked if we could create a Reactive Framework version of what he had done in the post using StreamInsight.  For those who don’t know, the Reactive Framework or Rx to its friends is a library for composing asynchronous and event-based programs using observable collections in the .Net framework.  Yes, there is some overlap between StreamInsight and the Reactive Extensions but StreamInsight has more flexibility and power in its temporal algebra (Windowing, Alteration of event headers) Well here are two alternate ways of doing what Roman... [More]

I am giving a full day seminar on StreamInsight at the SQLBits conference in Brighton, UK.  The seminar is happening on 7th April 2011.  Early bird discounts are available so get over to the site and register.  During this day I will be explaining exactly what StreamInsight is and we’ll be looking at how to solve some very interesting business problems with streaming data.  A complete rundown of what it is I will be covering is here.

I have been thinking a lot recently about what it would be like to have StreamInsight and SSIS working together.  Well the CAT team have produced a paper on some of our options here. Here are some of my thoughts. There is of course a slight mismatch in their types of usage.  StreamInsight is an Event Stream processing engine capable of operating on new data in the sub second timeframe.  The engine allows you to do real time analytics and take decisions on events that have potentially only just happened.  SSIS on the other hand is a batch processing engine.  In... [More]

Whilst writing some courseware I wanted to be able to see the start and end times of Edge shaped events from within the debugger.  A quick recap on Edge events At the start of the event you do not know the end time and most probably cannot work it out or you should be using one of the other shapes. You enqueue an event (Start Edge) with the start time and payload of the event.  The end time of the event is set to infinity When you see the end edge come through, you enqueue another event (End Edge) with the previous start time and payload and restate the event’s end time.&... [More]

This is a pre-article to one I am going to be writing on adjusting an event’s time and duration to satisfy business process requirements but it is one that I think is really useful when understanding the way that Hopping/Tumbling windows work within StreamInsight.  A Tumbling window is just a special shortcut version of  a Hopping window where the width of the window is equal to the size of the hop Here is the simplest and often used definition for a Hopping Window.  You can find them all here public static CepWindowStream<CepWindow<TPayload>> HoppingWindow<TPa... [More]

I have mentioned before about doing development for StreamInsight in Linqpad.  I have it installed on two separate PCs and I have enabled autocompletion on only one of them.  Whilst both versions are an excellent tool, the one with autocompletion enabled is so much easier to use.  After enabling autocompletion you can see I now get parameter listing