Henderson Consulting » SQL Azure

Knee Deep in Azure Table Storage

Mike — Tue, 06 Dec 2011 02:29:29 +0000

I have been trying to work up a demo using SSIS and Azure Table Storage. I keep getting distracted with other minutia of Azure that is interesting, but I have not been able to contain it in a blog post.

Just the same, Azure table storage is quite an interesting animal. Not a relational store like SQL Azure. Not really a NoSQL alternative like Mongo DB (read steep learning curve). I am hoping to have something more substantive to share soon because once the get it down, I can see a whole new world of data storage and revival opening up.

Another upcoming topic: fun with HTML5/ASP MVC/Razor.

Using Windows Azure Marketplace Data with SSIS

Mike — Sat, 05 Nov 2011 18:49:52 +0000

In this blog post I will describe using Microsoft Windows Azure Marketplace Data and SSIS. The Azure Marketplace allows one to buy and sell Software-as-a-Service applications and datasets. Most of the services are available for a subscription fee, but there are also several offered for free.

For the purposes of this demonstration I will use data from the Practice Fusion Medical Research Data, which is available for free. This data is available for research purposes and contains HIPPA compliant de-identified data. I have a personal interest in Autism and want to direct some after-work energy seeking out and analyzing Autism data. Unfortunately what I extracted from the sample dataset thus far does not have any Autism related data. Just the same it has been a good exercise in doing ETL with data services.

Add variables

Add three variables to your package. One is for the account key and one for the User ID, which can found in the My Account section. The third variable is the service root. This can be found in the details section of the service one is subscribing to.

Add Data Flow & Script Component

Add a Data Flow to a package. Inside the Data Flow, add a Script Component and select Script Component Type of Source.

Add the appropriate columns you want to extract from the service.

Add a service reference and use the Service Root URL.

Consume the Data with Managed Code.

At this point the data can be queried with a Linq expression and output rows generated. For more information see the MSDN documentation on creating application with Azure Marketplace Data.

usingSystem;
usingSystem.Collections;
usingSystem.Data;
usingSystem.Linq;
usingSystem.Net;
usingMicrosoft.SqlServer.Dts.Pipeline.Wrapper;
usingMicrosoft.SqlServer.Dts.Runtime.Wrapper;
using SC_c0e4421be5a54826af3fe68d3227f2be.PracticeFusion;

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]

public class ScriptMain : UserComponent

{

    public override void CreateNewOutputRows()

    {

        var serviceUri = new Uri(Variables.ServiceRootUrl);

        var context = new PracticeFusionMedicalResearchDataContainer(serviceUri);
        context.Credentials = new NetworkCredential(Variables.UserID, Variables.Key);

        var diagnosis = context.SyncDiagnosis.ToList();
        foreach (SyncDiagnosis d in diagnosis)

        {

            OutputDiagnosisBuffer.AddRow();

            OutputDiagnosisBuffer.DiagnosisGuid = d.DiagnosisGuid;

            OutputDiagnosisBuffer.PatientGuid = d.PatientGuid;

            OutputDiagnosisBuffer.ICD9Code = d.ICD9Code;

            OutputDiagnosisBuffer.DiagnosisDescription = d.DiagnosisDescription;

            OutputDiagnosisBuffer.StartYear = (int)(d.StartYear > 0 ? d.StartYear : 0);

            OutputDiagnosisBuffer.StopYear = (int)(d.StopYear > 0 ? d.StopYear : 0);

            OutputDiagnosisBuffer.Acute = d.Acute == true ? true : false;

        }

    }

}

In this blog post I demonstrated how to consume data made available using the Window Azure Marketplace. I see this as useful where one would like to extract external data for the Business intelligence needs.

Checking in on New Year’s Resolutions

Mike — Sun, 30 Jan 2011 14:44:44 +0000

At the end of January how am I doing with my resolutions?

Personal Development

A good start. I am reading a book called Mindfulness in Plain English, and joined a local group to help get me started. I may have tapped into some kind of meme. After deciding to explore meditation, I heard a panelist on the podcast Geek Speak remark that he too had resolved to meditate. Then I learned about a new medical study that shows meditation can encourage growth and change in the brain.

Geek Resolutions For 2011

After reaching level one in Project Euler I am taking a short break.
Iron Python: I’ve downloaded some things from the official site, but mostly reading rather than coding at this point.
SSIS: I am continuing to read and develop best practices in SSIS.
My Azure resolution is the focus at this point. The Code Project announced a programming contest and I mean to prepare an entry.
The phone app I hope will dovetail with what I am planning for the programming contest.

New Year’s Resolution: More Blogging?

Mike — Sat, 01 Jan 2011 21:59:04 +0000

It’s a new year and among my potential resolutions is to blog more. This space has taken a back seat to Twitter and Facebook, which is fine because I enjoyed limiting my input to 140 characters. Now I am feeling motivated to do something more expansive, say two paragraphs.

Geek Resolutions For 2011 (in no particular order or level of commitment)

Get to level level 2 in Project Euler. I have been making my way through the Project Euler problems using F#. I would like to keep it going in 2011
Learn Iron Python. Learning F# has got me interesting in some of the other .Net languages.
Keep up to date with SSIS. At my workplace I am the subject matter expert for all things SSIS. The next version of SQL Server appears to have some significant changes and I need to keep up.
Write an Azure application and publish it. I am quite interested in the Azure space and I aim to have a real project launched in Azure.
Write a Windows 7 Phone application and publish it. It would probably be sexier to do an iPhone/iPad or Android app, but the Visual Studio Express edition for Windows 7 Phone is free, so that’s a good place to start.

Personal Growth Resolutions

Meditation: I been feeling the need to clear up the clutter in my brain and this seems like a good place to start.

Everything in its right place.

Mike — Tue, 08 Sep 2009 01:15:12 +0000

Watched a presentation on what’s in it for the DBA (http://bit.ly/Xpmt5) in SQL Azure. The big take away for me is SQL Azure manages data file, and file group optimization under the covers. Whether it has to be this way because the nature of the service architecture, or because MS wants to in more control, I like the idea right now. I like not having to evaluate and optimize what object goes in what file group, or whether I need to add another data file or inflate existing ones.

Trying to fit together pieces like network architecture, legacy database design, and growth patterns into the right place can be pretty daunting. Doing it right and seeing the improved performance is satisfying. Not getting it right has a long tail.

Newer data types not supported in SQL Azure.

Mike — Sun, 06 Sep 2009 13:21:54 +0000

Some of the newer data types introduced in SQL 2008 are not supported in the SQL Azure CTP (http://bit.ly/ti7mZ). More specifically hierarchyid and spacial data types.

For me that’s not a big deal because I have only tinkered with these data types, and am not working on any production applications that use them.

I understand adding support for spatial data types is on the way, but the SQL Azure team will prioritize this feature on the basis of demand.

Getting my head in the SQL Azure Cloud.

Mike — Sat, 05 Sep 2009 02:00:00 +0000

Last night I got an email invitation to participate in the CTP release of SQL Azure. I have no idea how exclusive this is or is not, but it was nice just the same.

A short while ago I filled out a form requesting an invitation. At one point I had to gave a brief description as to why I should be included. I wrote a couple of sentences describing my intentions and submitted. The last night the invitation arrived.

I filled out some more information regarding my account and created my first database. It was a little quirky, but I did manage to make a connection to the Azure database from SSMS.

I have been neglecting this blog for a while, but I plan to post a little more frequently as I take a deeper dive into Azure.