Henderson Consulting » Mike

String manipulations with the SSIS Script Component

Mike — Fri, 13 Jan 2012 02:55:26 +0000

Problem: Your source data contains a column that should be split in two. In this case there is a name column in an Excel file that contains both the first and last name. The destination database has the first and last name as separate columns.

Solution: Use the script the component and split apart the column data into a string array.

Step I: In a Data Flow add an Excel source and then add a Script Component with the component type “Transformation.” Add a First Name, Middle Name and Last Name as column outputs. Be sure to add the name column from the source data as an input column.

Step II: Open the script editing window and edit the ProcessInputRow method. In this example were going to assume that each name is separated by a space. Using the String.Split() method, break up the name column into a string array. If the array length is 3, then we have a first, last and middle name. If the array length is only 2, then only the first and last name is present.

Step III: Assign the output columns with the appropriate item from the index.

publicclass ScriptMain : UserComponent
{
    public override void InputCustomers_ProcessInputRow(InputCustomersBuffer Row)

    {

        var names = Row.Name.Split(‘ ‘);
        //if lenght is two, not middle skip the middle name

        if (names.Length == 2)

        {

            Row.FirstName = names[0];

            Row.LastName = names[1];

        }

        else if (names.Length == 3)

        {

            Row.FirstName = names[0];

            Row.MiddleName = names[1];

            Row.LastName = names[2];

        }
    }

}

We now have three columns ready to be passed to the destination data source.

String manipulations with the SSIS Script Component

XML and Me

Mike — Fri, 06 Jan 2012 12:29:15 +0000

Starting with the Spring Semester at Northwestern Michigan College, I will be teaching an introductory course in XML (CIT 185 – XML Programming).

Teaching in some capacity has been an item in my “bucket list” for some time. So I am excited to begin sharing the love with XML, XSLT, XQuery and friends. And teaching at NMC has special significance. Fourteen years ago when I was just starting out in the IT world I took a database course at NMC. Now I hope to help someone else get started as a programmer.

SSAS Database Backup Utility with AMO

Mike — Sun, 11 Dec 2011 03:36:31 +0000

There are few, and I do mean a few, examples of backing up your SSAS databases in an automated fashion. For a couple of years I have used a method of looping through a list in SSIS and executing an XMLA script. In this post I will demonstrate create a console application using Analysis Management Objects (AMO).

Step 1: Create a console application

Create a console application in Visual Studio and add Microsoft.AnalysisServices to your references.

Step 2: Add a new class Argument Parser class.

The code in this class is used almost verbatim from a handy post at .Net Scraps.

Step 3: Add a new class Processing.

This class will have all the code doing the backup. The method BackupSsasDatabases connects to the server identified in arguments passed in. Once connected to the server, we then loop through the SSAS databases enumerated in the DatabaseCollection. There is a helper method called GetBackupInfo that returns an instance of the BackupInfoClass and is used as an argument for the Backup method.

The GetFilePath method uses an argument passed into the application indicating the destination for the backup file, and builds the file name from the database name and the date and time of the backup.

Step 4: Arguments

There are four arguments the application uses. The first two are required and the second two are optional:

ServerName : The name of the server you’re connecting to
Path: The destination directory for the backup file
AllowOverwrite: Used in BackupInfo class
BackupRemotePartitions: Used in BackupInfo class

class Program
{
staticvoid Main(string[] args)
{
            if (args.Length > 0)

            {

                Hashtable _args = ArgumentParser.Parse(args);
                //if we don’t have the required arguments, call it quits

                //if we do have the required arguments, proceed.

                if (_args.ContainsKey("ServerName") && _args.ContainsKey("Path"))

                {

                    var proc = new Processing();

                    proc.BackupSsasDatabases(_args);

                }

                else

                {

                    //write out info on missing arguments

                    if (!_args.ContainsKey("ServerName"))

                    {

                        Console.WriteLine("ServerName argument is missing");

                    }
                    if (!_args.ContainsKey("Path"))

                    {

                        Console.WriteLine("Path argument is missing");

                    }
                    Console.WriteLine("Press any key to continue");

                    Console.ReadKey(true); 

                }
            }

            else

            {

                Console.WriteLine("No arguments passed");

                Console.WriteLine("Press any key to continue");

                Console.ReadKey(true); 

            }            

        }
    } // class

class Processing
{
#region Backup SSAS Databases.
        /// 


        /// Backup SSAS Databases.

        /// 
/// 
publicvoid BackupSsasDatabases(Hashtable args)
{

            var _serverName = args["ServerName"].ToString();
            using(Server srv = new Server())

            {

                srv.Connect(_serverName);
                foreach(Database db in srv.Databases)

                {

                    Console.WriteLine("Backing up " + db.Name);

                    Console.WriteLine("Start " + DateTime.Now.ToShortTimeString());
                    var backupInfo = GetBackupInfo(db.Name, args);
                    db.Backup(backupInfo);
                    Console.WriteLine("Done " + DateTime.Now.ToShortTimeString());

                }
                srv.Disconnect();

            }

        }
        #endregion
        #region Get File Path
        /// 


        /// build file path for backup.

        /// 
/// 
/// 
string GetFilePath(string name, string path)
{

            var c = DateTime.Now;

            var f =new StringBuilder();
            f.Append(path);

            f.Append(name.Replace(" ","_"));

            f.Append("_");

            f.Append(c.Year.ToString());

            f.Append(c.Month.ToString());

            f.Append(c.Day.ToString());

            f.Append("_");

            f.Append(c.Hour.ToString("D"));

            f.Append(c.Minute.ToString("D"));

            f.Append(".abf");
            return f.ToString();

        }
        #endregion
        #region Get Backup Info
        /// 


        /// Get Backup Info

        /// 
/// 
/// 

        BackupInfo GetBackupInfo(string name, Hashtable args)
{

            var backupInfo =new BackupInfo();
            backupInfo.AllowOverwrite = args.ContainsKey("AllowOverwrite")? Convert.ToBoolean(args["AllowOverwrite"]) : true;

            backupInfo.BackupRemotePartitions = args.ContainsKey("BackupRemotePartitions") ? Convert.ToBoolean(args["BackupRemotePartitions"]) : false;

            backupInfo.File = GetFilePath(name, args["Path"].ToString());
            return backupInfo;

        }
        #endregion

    } // class

Step 4: Create a batch file and backup your SSAS databases.

Summary

There are other approaches to backing up your SSAS databases (Powershell comes to mind). This is one is offered as a backup processing solution tied together in an .exe file.

Download Backup SSAS Database Code (33)

Knee Deep in Azure Table Storage

Mike — Tue, 06 Dec 2011 02:29:29 +0000

I have been trying to work up a demo using SSIS and Azure Table Storage. I keep getting distracted with other minutia of Azure that is interesting, but I have not been able to contain it in a blog post.

Just the same, Azure table storage is quite an interesting animal. Not a relational store like SQL Azure. Not really a NoSQL alternative like Mongo DB (read steep learning curve). I am hoping to have something more substantive to share soon because once the get it down, I can see a whole new world of data storage and revival opening up.

Another upcoming topic: fun with HTML5/ASP MVC/Razor.

Working with MongoDB and SSIS

Mike — Mon, 14 Nov 2011 09:54:02 +0000

A little over a year ago I saw a presentation on NoSQL (i.e. Not Only SQL), and specifically MongoDB. MongoDB is an open source document oriented database, and like many things related to NoSQL it seems to have a buzz about it. In this blog post I will demonstrate using SSIS and MongoDB.

The Basics

I am not going to detail installing MongoDB and related drivers, the MongoDB already does fair job and I recommend following their documentation. For the purposes of this demonstration I installed MongoDB and the C# drivers.

Once the MongoDB prerequisites have been installed, create an SSIS package and add a Data Flow. In the data flow I added a flat file source that will extract a list of countries from a CSV file. Then add a Script Component and select “Destination” as the Script Component Type.

Next open the script editor and add the following references and usings:

MongoDB.Bson
MongoDB.Driver

The Code

I created a Country class as part of my ETL, but this is not necessary and I would refer one to the C# driver tutorial for other approaches. In a pretty concise manner, I was able to connection the server, database and add the country data.

usingSystem;
usingSystem.Data;
usingMicrosoft.SqlServer.Dts.Pipeline.Wrapper;
usingMicrosoft.SqlServer.Dts.Runtime.Wrapper;
using MongoDB.Bson;

using MongoDB.Driver;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]

public class ScriptMain : UserComponent

{
    public override void InputCountry_ProcessInputRow(InputCountryBuffer Row)

    {

        var connectionString = "mongodb://localhost";

        var mongo = MongoServer.Create(connectionString);
        var db = mongo.GetDatabase("MyMongoDb");
        MongoCollection<Country> countries = db.GetCollection<Country>("Countries");
        var country = new Country { CountryCode = Row.CountryCode, CountryName = Row.CountryName };
        countries.Insert(country);

    }

}
public class Country

{

    public string CountryCode { get; set; }

    public string CountryName { get; set; }

}

In this demonstration I showed how to use SSIS and create an ETL with MongoDB. This turned out pretty easy as I was able to download, install, and create this proof of concept on a Sunday. Not to mention juggling other weekend chores.

Using Windows Azure Marketplace Data with SSIS

Mike — Sat, 05 Nov 2011 18:49:52 +0000

In this blog post I will describe using Microsoft Windows Azure Marketplace Data and SSIS. The Azure Marketplace allows one to buy and sell Software-as-a-Service applications and datasets. Most of the services are available for a subscription fee, but there are also several offered for free.

For the purposes of this demonstration I will use data from the Practice Fusion Medical Research Data, which is available for free. This data is available for research purposes and contains HIPPA compliant de-identified data. I have a personal interest in Autism and want to direct some after-work energy seeking out and analyzing Autism data. Unfortunately what I extracted from the sample dataset thus far does not have any Autism related data. Just the same it has been a good exercise in doing ETL with data services.

Add variables

Add three variables to your package. One is for the account key and one for the User ID, which can found in the My Account section. The third variable is the service root. This can be found in the details section of the service one is subscribing to.

Add Data Flow & Script Component

Add a Data Flow to a package. Inside the Data Flow, add a Script Component and select Script Component Type of Source.

Add the appropriate columns you want to extract from the service.

Add a service reference and use the Service Root URL.

Consume the Data with Managed Code.

At this point the data can be queried with a Linq expression and output rows generated. For more information see the MSDN documentation on creating application with Azure Marketplace Data.

usingSystem;
usingSystem.Collections;
usingSystem.Data;
usingSystem.Linq;
usingSystem.Net;
usingMicrosoft.SqlServer.Dts.Pipeline.Wrapper;
usingMicrosoft.SqlServer.Dts.Runtime.Wrapper;
using SC_c0e4421be5a54826af3fe68d3227f2be.PracticeFusion;

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]

public class ScriptMain : UserComponent

{

    public override void CreateNewOutputRows()

    {

        var serviceUri = new Uri(Variables.ServiceRootUrl);

        var context = new PracticeFusionMedicalResearchDataContainer(serviceUri);
        context.Credentials = new NetworkCredential(Variables.UserID, Variables.Key);

        var diagnosis = context.SyncDiagnosis.ToList();
        foreach (SyncDiagnosis d in diagnosis)

        {

            OutputDiagnosisBuffer.AddRow();

            OutputDiagnosisBuffer.DiagnosisGuid = d.DiagnosisGuid;

            OutputDiagnosisBuffer.PatientGuid = d.PatientGuid;

            OutputDiagnosisBuffer.ICD9Code = d.ICD9Code;

            OutputDiagnosisBuffer.DiagnosisDescription = d.DiagnosisDescription;

            OutputDiagnosisBuffer.StartYear = (int)(d.StartYear > 0 ? d.StartYear : 0);

            OutputDiagnosisBuffer.StopYear = (int)(d.StopYear > 0 ? d.StopYear : 0);

            OutputDiagnosisBuffer.Acute = d.Acute == true ? true : false;

        }

    }

}

In this blog post I demonstrated how to consume data made available using the Window Azure Marketplace. I see this as useful where one would like to extract external data for the Business intelligence needs.

Generating Rows for Date Dimension

Mike — Sat, 22 Oct 2011 23:00:12 +0000

I was recently looking up scripts for populating date dimension tables. The scripts I reviewed were cribbed or otherwise stolen from other how-to documents. They all followed a sequence of establishing a start and end date range, and populating rows via a while loop. Generally speaking, I try to avoid while loops in SQL as it smacks of RBAR, or Row by Agonizing Row.

I started wondering if there was a better way, and then remembered a trick I saw demonstrated in a webinar. The demo showed how to efficiently populate tables with large amounts of test data. The trick is to query system tables to artificially return large lots of rows.

To start I want to return enough rows to cover the date range to be inserted in the table illustrated above. This is done by selecting from the system table sys.all_objects. To get the number of rows I need, I return the Top n number of rows selected. The exact number is determined by doing a DATEDIFF using the minimum and maximum dates.

DECLARE @date date =’12/31/1999′;

DECLARE @stop date =’12/31/2020′;
SELECT  TOP (DATEDIFF(DAY,@date,@stop))

@date
FROM sys.all_objects AS O

Now add an incrementing number using the ROW_NUMBER() function. Combined with the DATEADD function I now have all the dates I need for my dimension table.

At this point start filling out the other columns with the appropriate date related functions.

DECLARE @date date =’12/31/2009′;

DECLARE @stop date =’12/31/2010′;
SELECT  TOP (DATEDIFF(DAY,@date,@stop))

DATEADD(DAY,ROW_NUMBER() OVER (ORDER BY O.object_id), @date) AS CalendarDate

, DATEPART(DW,DATEADD(DAY,ROW_NUMBER() OVER (ORDER BY O.object_id), @date)) AS DayNumberOfWeek

, DATENAME(WEEKDAY,DATEADD(DAY,ROW_NUMBER() OVER (ORDER BY O.object_id), @date)) AS DayNameOfWeek

, DATEPART(MONTH,DATEADD(DAY,ROW_NUMBER() OVER (ORDER BY O.object_id), @date)) AS MonthNumber

, DATENAME(MONTH,DATEADD(DAY,ROW_NUMBER() OVER (ORDER BY O.object_id), @date)) AS [MonthName]

, DATEPART(QUARTER,DATEADD(DAY,ROW_NUMBER() OVER (ORDER BY O.object_id), @date)) AS CalendarQuarter

, DATEPART(YEAR,DATEADD(DAY,ROW_NUMBER() OVER (ORDER BY O.object_id), @date)) AS CalendarYear
FROM    sys.all_objects AS O

And there, dynamically generated rows for a date dimension and now while loop in sight.

SSIS: Dynamically Configuring the Data Connection Manager

Mike — Fri, 21 Oct 2011 01:34:20 +0000

In the last post I described using the WMI Data Reader Task. The data collected is by nature dynamic. Servers are stood up and decommissioned, as are databases. In this post I’ll demonstrate retrieving data from a changing list of servers.

Create Variables

To start, create a variable with a type of Object. Make sure the scope is package wide. The object is going to be created in one Data Flow and then used later in a For Each Loop container. Variables are by default created in the current scope. This caused me several headaches when I first started doing SSIS. A nice change in SQL Denali is the scope of variables will be package in scope by default.

Next create a variable of type string to hold the name of the server. Again, be sure the variable scope is package wide.

Populate the Server List Variable

Create a Data Flow and add a Data Source that retrieves your list of services. For the destination add a Recordset Destination set the variable to the server list variable.

Create Data Connection Manager

Create a WMI Data Connection Manager. In the last post I described using the WMI Data Reader Task. Now I am going to place this task inside a For Each Loop container and loop through a list of servers.

In the Data Connection Manager properties, click the Properties Expressions Editor. Select the Connection String and Server Name properties. Now add expressions to build the strings for these properties:

Connection String: “ServerName=\\\\” + @[User::ServerNameWmi] + “;Namespace=\\root\\cimv2;UseNtAuth=True;UserName=;”

Server Name: “\\\\” + @[User::ServerNameWmi]

Now add a For Each Loop Container to a Control Flow. Set the Enumerator property to Foreach ADO Enumerator. Set the ADO Object Source Variable to the object variable created earlier. In the variable mappings set the server name variable.

And now one can gather information on however many servers returned in the initial query.

Using the WMI Reader Task

Mike — Wed, 19 Oct 2011 02:07:07 +0000

This is another post in my Database BI series. In past posts I described using SMO objects to retrieve data about the SQL Server instances and databases. This post will describe using the WMI Reader Task to gather information about a server’s disk drive. This will be used to track size and growth of databases relative to the rest of the server, and plan for future growth.

To start, create a WMI Connection Manager. The server name is defaulted to localhost, you may have to edit this to \\ as needed.

Add a variable with a data type of object. The WMI Reader task will output a DataTable to this object.

Add a WMI Data Reader Task and a Data Flow Task to your control flow.

Configure the WMI Reader.

Below is the WQL Query. There is much more to be gleaned, but for the purposes of this application, these columns will do.

SELECT Caption, Description, DriveType, FileSystem, FreeSpace, Name, Size, STATUS, SystemName, VolumeName

FROM Win32_LogicalDisk

In the Data Flow task create a script component and select the script component type of source.

Add output columns based on what’s in the WQL Query. Pass in the object variable as a read only variable.

Then in the code, loop through the data table to create output rows.

/* Microsoft SQL Server Integration Services Script Component

*  Write scripts using Microsoft Visual C# 2010.

*  ScriptMain is the entry point class of the script.*/
using System;

using System.Data;

using Microsoft.SqlServer.Dts.Pipeline.Wrapper;

using Microsoft.SqlServer.Dts.Runtime.Wrapper;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]

public class ScriptMain : UserComponent

{
    public override void CreateNewOutputRows()

    {

        var wmi = (DataTable)Variables.ServerDataWmi;
        foreach (DataRow row in wmi.Rows)

        {

            OutputWmiBuffer.AddRow();

            OutputWmiBuffer.Caption = row[0].ToString();

            OutputWmiBuffer.Description = row[1].ToString();

            OutputWmiBuffer.DriveType = Convert.ToInt32(row[2]);

            OutputWmiBuffer.FileSystem = row[3].ToString();
            if(row[4] != System.DBNull.Value)

            {

                OutputWmiBuffer.FreeSpace = Convert.ToInt64(row[4]);

            }

            OutputWmiBuffer.Name = row[5].ToString();
            if (row[6] != System.DBNull.Value)

            {

                OutputWmiBuffer.Size = Convert.ToInt64(row[6]);

            }
            OutputWmiBuffer.Status = row[7].ToString();

            OutputWmiBuffer.SystemName = row[8].ToString();

            OutputWmiBuffer.VolumeName = row[9].ToString();
            OutputWmiBuffer.DriveLetter = row[5].ToString().Substring(0, 2).ToUpper();

        }
        OutputWmiBuffer.SetEndOfRowset();

    }
}

Now the data retrieved from the WQL Query can be transformed and loaded into SQL table to track the growth patterns of your servers.

SSIS: Using the Script Component to Insert Rows and Retrieve Newly Created Keys

Mike — Mon, 10 Oct 2011 19:07:55 +0000

In the last post I demonstrated using SQL Server Management Objects (SMO) to extract information on installed SQL Server instances. In this post the script component will be used as a transformation task, addressing a common scenario of inserting rows in a parent table, retrieve the newly created keys, and pass it along in the pipeline to be used with child tables.

Data Model

The data extracted will go into the database data warehouse I am developing. There are two tables involved in this ETL: a Data Store Object table and a SQL Server table. Similar to an order and order detail table, the Data Store Object table will contain some core data about the server such as name and create date. We’re talking information that’s not going to change once created. The SQL Server table will store details that could change over time.

Using the Lookup Component

After extracting the server information, use the lookup component to check if a server has already been added to the Data Store Object table. After the script component extracts the server information will, sort the data, and then add the Lookup component. In the transformation editor, specify non matches to redirect in the general section. In column section of the transformation editor, use the name column to join the two data sources, and add the Data Store Object table primary key. Rows that have been added in previous run will get assigned a key, and rows that need to be added will be null.

Add Script Component.

Add a script component to the data flow and select transformation as the component type. Send the Lookup No Match Output to this script component. In the input column select the columns needed to add a row to the Data Store Object table. Add the Data Store Object Table key as an output.

What this transformation is going to do the take the data on a new server and call a store procedure that inserts a new row in the database. The stored procedure will retrieve the newly created key and pass it back to SSIS in the form of an output parameter. The output column will be assigned the value of the output parameter.

using System;

using System.Data;

using System.Data.SqlClient;

using Microsoft.SqlServer.Dts.Pipeline.Wrapper;

using Microsoft.SqlServer.Dts.Runtime.Wrapper;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]

public class ScriptMain : UserComponent

{

    IDTSConnectionManager100 connMgr;

    SqlConnection conn;

    SqlCommand cmd;

    SqlParameter name;

    SqlParameter createDate;

    SqlParameter dataStoreObjectID;

    public override void AcquireConnections(object Transaction)

    {

        base.AcquireConnections(Transaction);

        connMgr = this.Connections.DatabaseRepository;

        conn = (SqlConnection)connMgr.AcquireConnection(null);

    }
    public override void PreExecute()

    {

        base.PreExecute();

        cmd = new SqlCommand("dso.DataStoreObject_InsertServer", conn);

        cmd.CommandType = CommandType.StoredProcedure;
        name = new SqlParameter("@Name", SqlDbType.NVarChar, 128);

        name.Direction = ParameterDirection.Input;

        cmd.Parameters.Add(name);
        createDate = new SqlParameter("@CreateDate", SqlDbType.DateTime);

        createDate.Direction = ParameterDirection.Input;

        cmd.Parameters.Add(createDate);
        dataStoreObjectID = new SqlParameter("@DataStoreObjectID", SqlDbType.Int);

        dataStoreObjectID.Direction = ParameterDirection.Output;

        cmd.Parameters.Add(dataStoreObjectID);

    }
    public override void InputServer_ProcessInputRow(InputServerBuffer Row)

    {

        cmd.Parameters["@Name"].Value = Row.Name;

        cmd.Parameters["@CreateDate"].Value = Row.CreateDate;

        cmd.ExecuteNonQuery();
        Row.DataStoreObjectID = (int)cmd.Parameters["@DataStoreObjectID"].Value;
    }
    public override void ReleaseConnections()

    {

        base.ReleaseConnections();

        connMgr.ReleaseConnection(conn);

    }

Note:

I picked up many of the database interaction particulars from this post and would like recognize the author (Greg Van Mullem).

Add Merge Component

After the new objects have been added, combine with the existing objects using the merge component.

Summary

In the previous post I discussed using the script component in the extract portion of an ETL. In this post I demonstrated using the script component in the transformation, or T, in ETL. More importantly, the example steps through a common scenario of inserting rows in a database, get the newly created keys, and move along.