Author: sunnywiz

Me.

Having Fun with SSIS

My new project involves getting data from 500+ remote sites into a target database; in parallel, quickly.  Thanks to some proof of concept work done by my coworker Mike “The Devastator” Hurd (I made up the nickname), I got to learn SSIS, and I’m cursing that I had not picked it up earlier. 

What Is SSIS?

  • Imagine you wanted to write a generic program to get data from point A to point B, but modify it along the way.
  • You wanted pluggable components that could talk to each other.
  • You wanted anything that could be done in parallel to be done in parallel.  Ie, read from two tables at the same time if you have bandwidth.
  • You want it to work buffered rather than row by row.  But dynamically figure out buffer sizes so that they fit well to the page size on the computer (to minimize swapping)
  • You want a GUI that you can configure these components with.
  • You want every component to have a configurable “if things fail” state, to redirect failed rows to other components, etc.
  • You want it to be solid – no leaking memory, etc
  • You want to parameterize the stuff getting into the components
  • You want it to be runnable from a command line, with ability to override parameters and connection strings.
  • You want the whole configuration to be saveable as an XML file, or hosted in a SQL Server
  • You want to be able to schedule it to run from SQLAGent (SQL Server’s “Cron” thing)
  • You want to be able to run it directly from C# via managed code.

Then you would write SSIS.

What Does It Look Like?

image

image

image

image

Initial Gotchas

SSIS requires some level of licensing, so this does NOT work:

  • Install SQLEXPRESS
  • install Visual Studio 2012 Premium
  • Install SSIS add-on
  • Install SSDT (“database tooling for VS2012”)

It successfully loads, and you can debug packages within Visual Studio, but you cannot run them outside via dtexec and dtexecui (the command line thingies for SSIS).  Instead, you need to add:

  • Install SQL Server Developer Edition (which includes SSIS)  (or standard edition)
    • Select Integration Services & Data Tools
    • Do NOT have to install the engine.
    • Might have to install an updated SSDT to get VS2012 support

Then everything runs fine.

The other thing I quickly learned is it is very Schema-dependent.  The reason for this is it allocates row counts into buffer sizes based on the size of the rows – so a Varchar(MAX) field is very different from a Varchar(250) field – and the schema will fail to validate if the underlying data source it is reading from does not match perfectly.

The solution I had to apply was to do a select cast(x as varchar(250)) as x to extract the data to guarantee the schema coming in to SSIS.  

Conclusion

Very glad to have this tool in my arsenal.   Thanks Mike!

Day at the Office: Using vi’s Macros

I had a method which was too long.  It was a switch/case statement, with a bunch of processing at each case label.    There were a lot of local variables that were being used: 

var foo=something; 
var bar=yetsomethingelse; 
var scooby=another thing; 
var fred=yetanotherthing; 

switch(thingy) { 
  case A: 
    stuff with foo and bar; 
    break; 
    
  case B: 
    stuff with foo and scooby; 
    break; 
    
  case C:
    stuff with foo, bar, and scooby; 
    break;
    
  case D:
    stuff with foo, bar, and fred; 
    break;
    
  default:
    throw new exception(); 
    break;
}

In this mockup I only have 4 variables; the real one had 12.   And each case label was probably 20-30 lines of code.

Apply Refactoring: Replace Local Variables with a Class

And the name of the class shall be context. 

So I had a lot of errors, where this: 

if (something(fred)) anotherthing(barney)

had to become this: 

if (something(c.Fred) anotherthing(c.Barney);

The Vi Macro

The exact keystrokes were:

  Begin with cursor on fred
qn Begin Recording a Macro to register n.  I use n because its on the right hand, and the @ symbol to apply a macro is on the left hand.
~ Change case (now we have Fred)
h Go left one (because ~ moves you forward)
i Enter insert mode
c. <Esc> type in “c.” and exit insert mode
l move the cursor right one.
q Finish recording

 

Then, I hit rebuild (F6), looked at the list of errors, and then primed myself:

/fred @n  / enter  (find fred, fix fred macro, find next fred)

@n / enter   (fix fred, find next fred)

Apply Refactoring: Extract Methods

Now that I had a reasonable list of variables (one mega-variable, the “context”, and some other minor ones), I could take the meat of each case block and extract out a method for just that case block.

The result: cleaner code… and in one case, I could pass my context variable one layer deeper and extract out some other meaningful stuff from it (which I had not pulled out into a variable before).

Shapeways: Color Sandstone Dan

Its here!

image image
 

[youtube=http://www.youtube.com/watch?v=BwftC16fRmc&w=448&h=252&hd=1]

I was impressed at how well it was packed:

image

That’s some kid of primo bubble wrap where the air pocket extends across multiple cells; so any one cell receiving a blow.. it does NOT pop. 

It cost me $8.50 + $6.50 Shipping and handling.

The process to put this together, roughly:

  • Create a Hi-res and a low-res model (500 faces or so) from AgiSoft
  • Fix any glaring problems with the low res model
  • Add a Subdivision Surface modifier to the low-res model in Blender to give it back some curves, while retaining a simple geometry that you can edit by hand.
  • Toggling back and forth between high and low, add cuts and edge creases to get the shape correct.  (Usually: Ears, Nose, and Mouth; perhaps Eyes)
  • Export the subsurfed model back to Agisoft for texturing; then back to blender for sizing, and sending to Shapeways.

There’s an additional step around making holes to remove material; I will explain that more on the Lamont model next week.

Save the Wrists! Ordered a new keyboard / Code Dictation?

My wrists have been numbing more lately.  At my wife’s suggestion, I’ll bring it up with my GP at my appointment next Wednesday.  Meanwhile:

  • Switch to using the trackball with my left hand.
  • Switch keyboards.

This is the keyboard I’ve been using for the last 2 years or so – Microsoft Wireless Comfort Keyboard 5000.   It has some level of curve in it, however, I still have Ulnar Deviation and Pronation when using it.  The keyboard action is very light, and its been good to me for a while.   It is wireless .. less cords to deal with.

image

This is my older, wired, USB keyboard.  Microsoft Ergonomic Keyboard 4000 v1.0.  I switched back to it, but what I find is that the keys take too much pressure to press.. and my hands are tired.   I couldn’t do it.    The keyboard is also larger, which means I need to stretch my fingers further to press the keys.  However, Ulnar deviation is a bit better.

image

Just for fun, I tried using both at the same time:

image

This worked surprisingly well.  It was fluid, except when I needed to switch to the mouse.   I’m used to my keyboard being a single keyboard, and something about having two keyboards confused me; I didn’t know which hand to move.  I guess I know my “home” position based on the distance between my hands, and if I move one hand, I cannot find home easily without looking. 

I did some research and decided to order this keyboard, it should arrive next week sometime:

 

In the mean time some of the things I can do are:

  • type at a deliberate rate. (not look at keyboard)
  • type with one hand only.   (requires looking at keyboard)
  • I can also just use one finger from each hand.  By doing this the entire hand moves, and ulnar deviation is removed.   It is also quite fast.  (requires looking at keyboard)
[youtube=http://www.youtube.com/watch?v=xLR0Nwh1l6o&w=448&h=252&hd=1]
What did I type?

This is me typing with one hand.

In light of Hanselman’s new app which allows dictation from an iPhone, I wonder if there are any programmer-specific dictation tools?  I have looked at vimspeak, however that mostly addresses the control aspects of using vi.   Think of it: 

public int Add(int a, int b) { return a+b; }

“public int shift A d d paren open int a comma int b paren close curly open return a plus b semi close curly” ?

Other folks have talked about it over here: http://productivity.stackexchange.com/questions/3605/how-can-we-use-dragon-naturallyspeaking-to-code-more-efficiently

The problem is one of lost context.  In programming we are trying to be specific about contexts – usually using parens and curly’s as visual queues – for example, that “int a” and “int b” are parameters of Add.   The solution seems like it would be codifying contexts into spoken word, by adding extra keywords to tell the parser what we are trying to do.  For example:

“function <name>” => starts adding function, puts you in function mode; creates defaults; additional things make sense in this mode:

  • “parameter x”
    • [of type] y
  • “returns [type] ”
  • [visibility] “public” | “private” | “internal”   // keywords can infer that we meant visibility
  • “body”  // go to editing body mode

So theoretically, “function Shift a d d visibility public returns int parameter a of type int parameter b of type int body return a plus b”, or more specifically:

What is said What the code looks like (using underscore to denote cursor)
function object a01() { }
Shift A object A() { }
d d object Add() {  }
visibility object Add()  {  }
public public object Add() {  }
returns public object Add() { }
int public int Add() {  }
parameter public int Add() {  }
a public int Add(object a) { }
of type public int Add(object a) {  }
int public int Add(int a) { }

etc.    Ie, you map the spoke word into meanings that then get applied to the editor.

You could even navigate “into” and “out of” contexts by using clicks and clucks – sounds that are easy to make yet are definitely not letters.   “down” and “up” would apply at the context level – ie, statements, or functions.

It is an interesting problem.  It could be a fun thing to solve.  Any takers?

Day at the Office: Visual Studio Express 2013

Customer has requested sufficient documentation that after we leave the project, they could bring on another developer and they could get up to speed efficiently.

imageSo, here’s my table of contents that I’m filling out (sure to change as it progresses).    I’m currently working on the “Build From Scratch”, which, to avoid any future awkwardness, means “Create a blank windows 7 virtual machine, Install visual studio and Compile.”

Just to see if it was possible, I tried installing the free Visual Studio Express 2013 for Desktop instead of the paid version.

It works.

One of my resentments against Microsoft – from back when I was a Unixite and a Micro$oft Hat3r – was that the average joe can’t afford the tools.   (Whereas, Perl and Python and gnu C++ were free).   This is no longer true. 

You can download and run VS2013 Express for Web, or Desktop.   Limitation:  no code analysis stuff; possibly harder to integrate 3rd party tools.   Includes: NuGet package manager, Intellisense, Multiple projects per solution, etc.

You can download and run SQLExpress.  Its pretty much the full blown SQL except it limits itself to 1 core and 1G of memory.  

I think that pretty much covers most of the software development I’ve done in the last 14 years, and gotten paid for.

Good Job, Microsoft.  I think I can be proud of you now.

Time Use May 2014

One of those sanity moments where I look at “what am I doing”, and are my priorities in line. (TL;DR: they are not)

My wife and kid went out of town for a week, so I had the opportunity to schedule myself any way I choose.  I have a comparison of “with wife and kid” and “bacheloring it”:

Wife And Kid  (Monday-Sunday)

image

Bacheloring (Tuesday-Saturday)

image

Ob ser v at io n u s

(observation + obvious; black=both; purple=1; blue=2)

  • My work hours were more scattered when the wife and kid were away
  • Work spills over to the weekend if I can’t get all my hours in during the week.   And that is what happens when sleep intrudes into work; which is what happens when entertainment intrudes into sleep.
  • I had more “white space” – time that I’m not doing anything in particular, just being – when the family was away.
  • I spend a significant amount of time in pink (hanging with wife) – I like this.
  • I spend a significant amount of time in green (entertainment, hobbies) – I like this.
  • I do stay up too late doing hobby stuff and watching netflix (entertainment) – at the expense of squeezing sleep.
  • I spent more time eating when the family was away:
    • Mostly, I was cooking up (measuring) batches of soylent and grilling

Not So Obvious

The timeline is not zoomed in enough to see some of the small stuff:

  • I walked the dogs every day while the family was gone.   But it only took 15 minutes.
  • I didn’t get to the gym while they were away.  That’s because my gym time and dog feeding and peeing time conflicted.
  • I spent a lot more time doing errands – cleaning stuff, fixing stuff – while the family was gone.
  • I did not nap at work while the family was gone.

Soylent: Not So Good News

I bought a week’s supply of PeopleChow 3.01 from Doug, with the intent of living on it while the family was away.  I did two trials, with my glucometer:

Trial #1: 1/3 the batch; 100g carb:

  • before: 89 mm/l
  • 30 minutes after:  138
  • 60 minutes after:  168
  • 120 minutes after:  158

Trial #2:  1/6 the batch, 50g carb:

  • before: 93
  • 30 minutes after:  158

The goal? My goal? is to be under 120 after 2 hours after a meal.   I couldn’t do it – too many carbs.

I tried altering the formula to use less corn flour; the result was unpalatable (puke worthy).  I gave up on it.

But what this did was, I started measuring my blood sugar again.

Me: Not So Good News

I’m a lot less able to withstand a carb load than I used to be able to.   Or so it seems.   Today was 40g of carb in some Indian Lentils:

  • Before: 110
  • 2 hours after, even with a walk:  158

And, I can feel it.  I feel puffy, flabby, out of energy, tired.

The Twin Cycle Hypothesis of Etiology of Type 2 Diabetes

imageI did some reading to see what’s new these days in diabetes stuff.  I came across this article, with this pretty cool picture (I could not find a public link, so this is a screenshot):

In a nutshell, it gives me an answer to “what the heck is going on” – and a glimmer that it gives me is, “loose enough weight, the cycle becomes less worse.”   It focuses on some T2 Diabetics who got gastric bypass surgery and radically altered their body fat content and wham! some of the diabetic cycle vanished.

In the past, I had gotten down to 165, which for me is a BMI of about 22.    I felt a lot better then, I was running, working out, having a blast.    But I did not cease to be diabetic. 

Then I saw this line here:

image

For me a BMI of 19 would be 130 lbs.   That’s 50 lbs less than I am right now.      30 lbs less than what I had aspired to get down to at my best.   I have to get down there, AND STAY DOWN THERE, because I’m pretty sure that the last bits of fat to get used up are going to be the ones which are the most troublesome.   Or, I could continue to be reasonably happy yet declining.

Maybe I didn’t drop enough weight that time?

How Hard should I strive?  After all, I could say I’m an old man.  I’m past my prime.   I’m beyond the life expectancy that humans had in the middle ages…

Bottom Line:  I have a Choice to make

My former sponsor’s favorite word – Choices.  Ah, the bliss of not knowing you have a choice.  

I could make a choice to get healthy again.

Which would mean, I need to put exercise back in my schedule.   And logging food.

Which would mean, something has to go.

What Goes?

  • It won’t be sleep.
  • It won’t be work – at least, not yet.  I’m not independently wealthy yet.
  • It won’t be [all of] hanging with the wife.
  • It won’t be Recovery work.
  • It would have to be entertainment and hobby.   There isn’t anything else to let go of.

I’ve been overdoing it.  My brain gets so tired, I just want to numb out with mindless TV watching.. or, my brain gets so obsessed, I have to solve this problem now! (3d printing, blender, and shapeways – I’m looking at you). 

So, sometime soon, expect that all my 3D printing stuff will come to a stop.  Or, it will be relegated to one experiment per weekend (or some other healthy amount).   A check at the posting queue for this blog – actually, the queue is empty right now.   When I post this post, there are no others in the queue after it.

I guess I’ll write one more post for “Shelving the Hobby” – making a list of the irons I have in the fire, so that I can let them go temporarily.   Or not, I can list them here:

  • I have a solution for Agisoft to Blender to Shapeways which involves decimation down to 500, subsurf, and edge creasing.  I have a color print ordered of that.
  • I have a solution for printing initials cubes without supports; I have to slice it, but basically I print it at a 45 degree angle so that all letters are facing “up” (kinda).   I have not actually done this yet.   I have ordered a small (20cm) cube from Shapeways to see how well their printers do the job.
  • I might be doing some silver jewelry via Shapeways involving people’s initials.

Done.  Shelved, will post pictures when they arrive.

I think I have a date with a gym tomorrow morning.   And if my work hours suffer, well, that’s what Sunday afternoons are for.