Row Count Plus Transformation

by Guest 15 Oct 2008 20:12

As the name suggests we have taken the current Row Count Transform that is provided by Microsoft in the Integration Services toolbox and we have recreated the functionality and extended upon it. There are two things about the current version that we thought could do with cleaning up

  1. Lack of a custom UI
  2. You have to type the variable name yourself

In the Row Count Plus Transformation we solve these issues for you.

Another thing we thought was missing is the ability to calculate the time taken between components in the pipeline. An example usage would be that you want to know how many rows flowed between Component A and Component B and how long it took. Again we have solved this issue.

Credit must go to Erik Veerman of Solid Quality Learning for the idea behind noting the duration. We were looking at one of his packages and saw that he was doing something very similar but he was using a Script Component as a transformation. Our philosophy is that if you have to write or Copy and Paste the same piece of code more than once then you should be thinking about a custom component and here it is.

The Row Count Plus Transformation populates variables with the values returned from;

  1. Counting the rows that have flowed through the path
  2. Returning the time in seconds between when it first saw a row come down this path and when it saw the final row.

It is possible to leave both these boxes blank and the component will still work.

 

All input columns are passed through the transformation unaltered, you are not permitted to change or add to the inputs or outputs of this component.

Optionally you can set the component to fire an event, which happens during the PostExecute phase of the execution. This can be useful to improve visibility of this information, such that it is captured in package logging, or can be used to drive workflow in the case of an error event.

Properties

Property Data Type Description
OutputRowCountVariable String The name of the variable into which the amount of row read will be passed (Optional).
OutputDurationVariable String The name of the variable into which the duration in seconds will be passed. (Optional).
EventType RowCountPlusTransform.EventType The type of event to fire during post execute, included in which are the row count and duration values.

RowCountPlusTransform.EventType Enumeration

Name Value Description
None 0 Do not fire any event.
Information 1 Fire an Information event.
Warning 2 Fire a Warning event.
Error 3 Fire an Error event.

Installation

The component is provided as an MSI file which you can download and run to install it. This simply places the files on disk in the correct locations and also installs the assemblies in the Global Assembly Cache as per Microsoft’s recommendations.

You may need to restart the SQL Server Integration Services service, as this caches information about what components are installed, as well as restarting any open instances of Business Intelligence Development Studio (BIDS) / Visual Studio that you may be using to build your SSIS packages.

Finally you will have to add the transformation to the Visual Studio toolbox manually. Right-click the toolbox, and select Choose Items.... Select the SSIS Data Flow Items tab, and then check the Row Count Plus Transformation in the Choose Toolbox Items window. This process has been described in detail in the related FAQ entry for How do I install a task or transform component?

We recommend you follow best practice and apply the current Microsoft SQL Server Service pack to your SQL Server servers and workstations, and this component requires a minimum of SQL Server 2005 Service Pack 1.

Downloads

The Row Number Transformation is available for both SQL Server 2005 and SQL Server 2008, with full support for seamless upgrades. Please choose the version to match your SQL Server version, or you can install both versions and use them side by side if you have both SQL Server 2005 and SQL Server 2008 installed.

Row Count Plus Transformation for SQL Server 2005

Row Count Plus Transformation for SQL Server 2008

Version History

SQL Server 2008

Version 2.0.0.5 - SQL Server 2008 release.
(15 Oct 2008)

SQL Server 2005

Version 1.1.0.43 - Bug fix for duration. For long running processes the duration second count may have been incorrect.
(8 Sep 2006)

Version 1.1.0.42 - SP1 Compatibility Testing. Added the ability to raise an event with the count and duration data for easier logging or workflow.
(18 Jun 2006)

Version 1.0.0.1 - SQL Server 2005 RTM. Made available as general public release.
(20 Mar 2006)

Screenshot

Row Count Plus Transformation Editor dialog

Troubleshooting

Make sure you have downloaded the version that matches your version of SQL Server. We offer separate downloads for SQL Server 2005 and SQL Server 2008.

If you get an error when you try and use the component along the lines of The component could not be added to the Data Flow task. Please verify that this component is properly installed.  ... The data flow object "Konesans ..." is not installed correctly on this computer, this usually indicates that the internal cache of SSIS components needs to be updated. This is held by the SSIS service, so you need restart the the SQL Server Integration Services service. You can do this from the Services applet in Control Panel or Administrative Tools in Windows. You can also restart the computer if you prefer. You may also need to restart any current instances of Business Intelligence Development Studio (BIDS) / Visual Studio that you may be using to build your SSIS packages.

Once installation is complete you need to manually add the task to the toolbox before you will see it and to be able add it to packages - How do I install a task or transform component?

(Updated installation and troubleshooting sections August 2009)

Comments

2/19/2009 6:50:40 PM #

This looks excelent.
I need something like this.

Btw, do you have a plan to extend its functionality to something else.
On more then one place I need Sum, Max, Min...

Dejan

Dejan Canada

2/20/2009 7:54:04 AM #

without knowing more of your requirements I would have to say that the Aggregate Transform is probably what you need.

Allan Mitchell United Kingdom

6/1/2009 10:14:58 AM #

Hey, there is a tiny copy/paste error in the "How to install" text:

"The component is provided as an MSI file, however to complete the installation, you will have to add the transformation to the Visual Studio toolbox manually. Right-click the toolbox, and select Choose Items.... Select the SSIS Control Flow Items tab, and then check the File Watcher Task from the list. "

File Watcher? Must be Row Count Plus, right? :)

Ceduljko Netherlands

6/1/2009 9:48:55 PM #

I get the Row Count in the current Row Count Transformation after the Data Flow Task is complete meaning not while the Data Flow Task is running.  Does Row Count Plus Transformation resolve that issue?  Thanks.

Ed United States

6/2/2009 10:42:04 PM #

Ceduljko, thanks for pointing out the typo, that is now corrected.

Ed, The Row Count Plus Transformation only updates the variable during the post execute event for the Data Flow Task, in a similar manner to the stock Row Count Transformation. This is because updating the variable row by row would be very expensive, a major performance overhead. It also doesn’t make much sense, no stock transforms can use a variable row by row, so would have to be a custom transformation (or Script Component with explicit variable locking) that could even use the variable. Even then it would be a very bad design pattern as it would assume some form of synchronisation between transformations, which again is not the case.

Darren Green United Kingdom

8/20/2009 7:41:55 PM #

I installed your component - it was just what I was looking for - unfortunately it doesn't work.  All I get is an error message telling me it's not installed correctly.   I get a big kick out of the "Help" that the link points to - These always say the about same thing "Yeah it don't work - too bad ain't it".

The error message says

"===================================

The component could not be added to the Data Flow task.
Please verify that this component is properly installed. (Microsoft Visual Studio)

===================================

The data flow object "Konesans.Dts.Pipeline.RowCountPlusTransform.RowCountPlusTransform, Konesans.Dts.Pipeline.RowCountPlusTransform, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b2ab4a111192992b" is not installed correctly on this computer. (Microsoft.DataTransformationServices.Design)

------------------------------
For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft%u00ae+Visual+Studio%u00ae+2005&ProdVer=8.0.50727.42&EvtSrc=Microsoft.DataTransformationServices.Design.SR&EvtID=PipelineObjectNotInstalled&LinkId=20476

------------------------------
Program Location:

   at Microsoft.DataTransformationServices.Design.DtsBasePackageDesigner.GetPipelineInfo(String creationName, IServiceProvider serviceProvider)
   at Microsoft.DataTransformationServices.Design.DesignUtils.GetNewPipelineComponentObjectName(IDTSComponentMetaDataCollection90 parentCollection, String clsid, IDTSComponentMetaData90 componentMetadata, PipelineComponentInfo& pipelineComponentInfo)
   at Microsoft.DataTransformationServices.Design.PipelineTaskDesigner.AddNewComponent(String clsid, Boolean throwOnError)"

If you could give me a clue how to do this I would really appreciate it - I've tried everything I can think of and all I get is error messages or all zeros.

Thanks

Charles Duwel United States

8/21/2009 8:55:53 AM #

Charles, an error along the lines of The component could not be added to the Data Flow task. Please verify that this component is properly installed.  ... The data flow object "Konesans ..." is not installed correctly on this computer  usually indicates that the internal cache of SSIS components needs to be updated. This is held by the SSIS service, so you need restart the the SQL Server Integration Services service. You can do this from the Services applet in Control Panel or Administrative Tools in Windows. You can also restart the computer if you prefer. You may also need to restart any current instances of Business Intelligence Development Studio (BIDS) / Visual Studio that you may be using to build your SSIS packages.

Darren Green United Kingdom

8/21/2009 12:35:29 PM #

Thanks for your reply and sure enough if I'd read the instructions fully it would have installed correctly - I did restart Visual Studio but missed the part about restarting SQL SSIS.  Now my only problem is that it doesn't do what I need to do.  I need to add a incrementing ID column to each row in tables as they go by.  I would think this would be a common need but it seems impossible - at least for me.  I could just dump the data in a new table and use the Identity property but there has to be a better way.  Any ideas?  I am used to DTS and while some of this new stuff is great some of it leaves me wondering if I am speaking the same language as the people who wrote the stuff.
Thanks again.

Charles Duwel United States

8/21/2009 1:06:46 PM #

I found what I was looking for - the Row Number Transformation component.  Just what the doctor ordered.  That and a Derived column task to make it a string like "CL1"...
See www.sqlis.com/post/Row-Number-Transformation.aspx.
Thanks Again

Charles Duwel United States

9/16/2009 4:00:50 PM #

Is it possible that I could use this component to do a row count then include that row count number to my destination flat file name? e.g. Error Rowcount = 100, my filename would be = Error_100_DimSkill.txt

Anthony San Juan Republic of the Philippines

9/16/2009 10:37:13 PM #

Anthony, you cannot count the rows and write to a named the file in the same Data Flow, it just does not work. A Data Flow source can still be reading rows whilst you're already writing the Destination, so obviously the file has been created well before you have seen all the rows to determine the name. Actually the filename must be known at the start of the task, as that is when it is created, before any data has been read.

A simple way around this is to count the rows (you can use the stock Row Count transform), and store the result in a variable as normal. Then have a second task that renames the file, see the File System Task, which can Rename a file. Perhaps use a variable with an expression to build the new filename with the row count variable.

Darren Green United Kingdom

10/6/2009 10:38:43 AM #

Hi! Thanks for developing Row count plus. It helps me to improve my SSIS Logs. I have one question: We need to log the count of datasets affected by a select, insert... statement. For that case I used Row count plus. But if we have dataflows with more than one data source it's not possible to log the way I want. (BTW: I used that instuction for logging: consultingblogs.emc.com/.../...ent-Handlers.aspx). We decided to use a row count plus for every data source an fire an information event. But the problem is that we can't see the name of the row count plus, just the information that a row count plus information with xx rows was fired. Is it possible to name this information. I need something like "Row count plus NAME information: xx. Is that possible? Thanks!

Tobias Müller Germany

Add comment




  Country flag

biuquote * Required fields. Your email address will not shown, check the preview for what you see. We use it to send you notifications on new comments if you ask us to below.
  • Comment
  • Preview
Loading




Welcome to SQLIS.com our free SQL Server Integration Services (SSIS) resource site.

MVP

RecentComments

Comment RSS