Code – Page 2 – A Geeky Gulati

Using Timesheet data from Harvest to create a Force Directed Graph Animation

This is too complicated to try to put into words, so there’s a screencast instead.

Video of the final product:

https://www.youtube.com/watch?v=_TJi9yAm_kM

Video explaining how to do it:

Source: https://github.com/sunnywiz/HarvestToGephi/blob/master/HarvestToGephi/Program.cs

In text form: C# code to convert a Harvest CSV extract into a node and edges CSV file. Then in Gephi, import the two files, convert the start/end dates into an interval, and set up the prettyness. Record a long video with lots of stabilization and then speed it up.

Sorting Watch Later by Video Length

EDIT – does not work anymore see comments

Something I do often – I sort my Watch Later list in Youtube by how long the videos are. I think its “shortest video to get the quickest entertainment bang for my time” (which is semi sad).

I wanted to automate it. Found several dead ends: Youtube doesn’t let you use their App Scripts API to modify the Watch Later list, and the page doesn’t let you run a bookmarklet to load an external js file.

However, Chrome allows for snippets.. and some folks pointed out that I could use various other softwares (Greasemonkey? another name was given) for running arbitrary javascript on a page.

Ctrl-Shift-I, Sources, >>, Snippets

https://gist.github.com/sunnywiz/064a4d07f8469074356e73eba5dc1215

That’s my solution for now. It gets what I need done. I wish I could have done that more generically.

Thanks to Matt H. for pointing me down the path of Chrome snippets. That dude is bright.

Tree of Directions Revisited

We are in the process of possibly changing residences, so (during my coding fun times on Wednesday mornings with a friend) I revisited this project. When I left it, it looked like this:

In fact, that was so not-cute that I had tried to go 2D with it instead.

I revisited it, and went through several stages.

First, I had to re-do how I called Google Maps – they no longer had a “free” tier that was usable, but they now had a “pay as you go” tier which equated to $5 for 1000 route requests. No problem. But, to keep costs low, I added a cache strategy so I didn’t ask for the same thing twice. (Caveat: I think I forgot to turn off traffic, so different requests at different times were delayed by different amounts).

Then, I tried to create a surface underneath the plot. I did this using some electrical engineering stuff I once helped somebody with – Here’s the plot + a surface underneath it. Better, but not the awesome I was hoping:

I called that the “Minecraft” look. (That’s a very rough draft with only 10×10, Its about the same at 50×50 except the squares are smaller). I went for something where I calculated the midpoint and drew a polyhedron from each of the four vertices, it looked a lot better:

(The previous were github previews of STL files; this is 3D Model Viewer built in to Windows).

But, its still to .. Jaggedy. So, I did a few things: First, I made it so rather than a spiderweb, I did a “ramp” effect (filling in underneath the path), as well as, I trimmed off all the residential streets (<30mph) at the ends of the routes. This gave me a much better print, which is closer to what I had in mind when I started: something that showed “Which were the best ways to get places”:

However, I didn’t realize it, but I had done something even better. Here’s Just the Ramps:

This is what I had been going for! To heck with the surface print part! You can see the mountain AND the detail! This is commit a75a67 at https://github.com/sunnywiz/TreeOfDirection/.

Next direction — where I do the same thing for both addresses, and then figure out a way to do stuff in two colors (ie, two prints, but that join together). In order to do this, I have to first force the bounding boxes to be exactly the same.

Also other things learned – 3D slicing has come a long way, I don’t need things to be perfectly combined as long as they don’t have holes in them.

pruning promotions in GMail

I finally got around to writing some code – turns out its super easy, there’s something called script.google.com, and you Resources|Advanced Google Resources and add the Gmail resource, and then you can write functions to prune your inbox arbitrarily:

function prunePromotions() { 
  var response = Gmail.Users.Messages.list('me',{'q':'label:promotions', 'includeSpamTrash':false});
  if (response && response.messages && response.messages.length > 0) { 
    // messages seem to come in from most recent to least recent
    Logger.log('inspecting '+response.messages.length+' messages'); 
    var collected = {};
    for (var i=0; i < response.messages.length; i++) { 
      var message = response.messages[i]; 
      if (message.id) { 
        var message = Gmail.Users.Messages.get('me',message.id);           
        if (message && message.payload && message.payload.headers && message.payload.headers.length > 0) { 
          for (var j=0; j < message.payload.headers.length; j++) { 
            var header = message.payload.headers[j]; 
            if (header.name=='From') { 
              var from = header.value; 
              if (!collected[from]) collected[from] = []; 
              collected[from].push(message); 
              break; 
            } // if header is from
          } // for each header
        } // if message.payload.headers
      } // if message.id
    } // every message
    for(var from in collected) {
      if (collected[from].length > 1) { 
        for (var i=1; i < collected[from].length; i++) { 
          var message = collected[from][i];
          Logger.log('trashing '+message.snippet); 
          Gmail.Users.Messages.trash('me',message.id);
        }
      }
    }
  } 
}

Unfortunately, I get a lot of promotions, and this only works on the most recent 100 messages or so, so it might be of limited use in this state. But its a start.

Learned some IoT today

My head is swimming with new information, so dumping it here in a bit, but first, a meta- of what’s going on.

1. I often feel “left behind” and “not having time to play with things”. I decided to try to make it a priority by committing with a friend to spend an hour learning new stuff.. together. So the idea is we spend half an hour on my project, and half an hour on his project, in a pair-program kind of way.

2. We did the first non-planning session of it today. Well, it was still a lot of planning. And what happened is: the 30 minutes were just enough to get me unblocked and to learn which directions to go.. and then we switched over to his project.

3. I’m looking forward to revisiting this next week.

3.97. if this works well over time, we might polish it up a bit and open it up to more folks.

The Details.

First, His Project: (topic not revealed, that’s his story)

I saw Jupyter for the first time in action. It was amazing. I think I’ll use it the next time I need to go spelunk and chart in a RDBMS (instead of query, copy-to-excel, create chart)
Revisiting OpenCV for image manipulation and stuff
There might be some ML in that project’s future. That’s up to him, of course. My job is to provide interest and provide support and be a rubber duck.
I hope to learn more about running things in docker containers from him.

My Project – Temperature logging for my A/C, Furnace — what I learned

I got an LED to blink and read a photoresistor via a WIFI connection (via particle.io and Tinker)
Kinda-sorta why we need resistors going to ground to make circuits work. At least, I saw that repeated pattern.
What a breadboard is, and how it is wired
Digital vs Analog ~= 3.3V and ranges, and HI=3.3V. Some specific pins have better A-to-D converters and can read ranges up to 4096.
The temperature sensor I have uses something called a One-Wire protocol, and I have to do a library-add to get the right library to read it. It involves scanning for devices, multiple devices can be on the same wire. However, there’s two ways of powering it – parasitic and non-parasitic.
It might be that when I plug power into the photon, if the other end is my computer, i might be able to debug print via a serial connection. That will help with the one-wire scan.
Turns out particle.io does all kinds of cloudy-mc-cloud stuff so once i have my number, all i need to do is publish(“temperature”, value) and it heads up to the cloud. Then if I add the integration to Azure IoT, it goes into Azure where I can do other things with it. AWS doesn’t seem to have that integration. There are other folks who do have ways to graph things. I need to analyze patterns over a 6-month time period, involving the drop or rise in temperature from ambient to heated/cooled.
For one-wire stuff, i need a 4.7k ohm resistor, and i’ll probably need breadboard cables. I didn’t have them yet. I ordered them. Should be coming along with a few more thermistors.
Using one-wire, i might be able to have a separate sensor for “ambient temperature” so that i can diff the two easier and just take the delta.
I read up on the spec for the “Wiring” language, which is basically C/C++ but scaled down.
I learned there’s a web, a desktop, a CLI, as well as a VsCode development environment. The vscode one seems to have a debugger. Don’t think I’ll need that, as long as I can get serial.print() to work.

Side note – I didn’t have LetsEncrypt set up to auto-renew my cert, so the https cert for this blog expired. I had to go find my blog post to find the link to find the commands to renew it. Done.

AWS RDS SqlServer Native Backup and Restore

Had to learn this yesterday to clone a production environment down to a lower environment. Figured it qualified for a blog post.

exec msdb.dbo.rds_backup_database 
         @source_db_name='xxxProd', 
         @s3_arn_to_backup_to='arn:aws:s3:::xxx-sql-native-backup/xxxProd.bak',
         @overwrite_S3_backup_file=1,
         @type='full';

exec msdb.dbo.rds_task_status;   -- till lifecycle=SUCCESS

ALTER DATABASE xxxUAT SET SINGLE_USER WITH ROLLBACK IMMEDIATE;

drop database xxxUAT;
exec msdb.dbo.rds_restore_database 
         @restore_db_name='xxxUAT', 
         @s3_arn_to_restore_from='arn:aws:s3:::xxx-sql-native-backup/xxxProd.bak';

exec msdb.dbo.rds_task_status;   -- till lifecycle=SUCCESS

delete from xxxUAT.dbo.SensitiveTableStuff;

The Gotcha’s were:

Had to set up an option group that added SqlServer Native Backup and Restore to the RDS instance. It took a few minutes to apply, the RDS instance did not reboot or go offline during this process.
Could not restore over an existing database.
Learned the hard way that while you can detach, you can’t re-attach a database using SSMS. Reattaching uses a custom stored procedure. And detaching and attaching had nothing to do with deleting.

From Certifiable to Certified

Yesterday, I passed my AWS Certified Developer Associate exam. I started studying for it two weeks ago.

Actually, no, I started studying for it a year ago when I started using AWS. And at the time, I thought I was going to take the Certified Solutions Architect Associate exam, but we went for this one because its easier, and I had two weeks to study for it.

Why two weeks? My company has been going for an AWS Partnership thing, and for that we need two certs. It wasn’t a immediate priority… until some stuff came up which really affected some of our value. We raised the flag, this needs to be a priority. Schedule it and take it, and lets send two people so that if either one passes, we’re good. We both passed.

What did I learn in those two weeks about how to study for the exam? If I were to go back in time to younger self, what would I tell him?

ACloud.Guru, but not like that

https://acloud.guru is VERY good. I’m keeping my subscription to it so I can listen in to some of the master-level courses during my commute. However, I had prescribed something like:

Listen and watch all the videos
Take all the Quizzes
Take the practice exam
Go read other boring stuff.

Inefficient and Fear Based.

Turns out, if you get a subscription, they have this thing called an Exam simulator (beta). And at the end of this you get to see what you did wrong (as well as what you did right), and an explanation of the thing, and a link to more resources:

The suggestion is to attack the problem from a different angle

Thing is, both the real exam and the practice exam asked some pretty in depth questions that the video instruction does not directly cover. Ryan does say, “Make sure you read the FAQs” and that’s 100% for real. You gotta read the FAQ’s.

However, the FAQ’s themselves reveal the core ideas central to the service you are reading about. As do the videos. My suggestion is that once you are comfortable, you get denser information faster from the FAQ’s.

I also found that the developer exam really wanted to know if you had used something. Called some methods. And if you hadn’t .. like me, I just barely used scan and query mostly I was being a sysop and using terraform to set things up for others .. then reading through the actions, attributes, headers etc – gives you that feel of what actually working with the thing is like. (Assuming you have plenty of experience and have done many things in the past). You’ll get into the mind of the designers of the system, and from that, you can infer all kinds of stuff.

Some Of My Notes

I started out taking notes on Paper. Then I took notes in Google Docs. Then I switched over to sheets.

I’m a visual person, and I like organization, and having a big grid where to “store” information (human brain is optimized for location tracking) was very helpful for me. If I got a concept wrong, I could go back to the place where that information was hiding on my sheet… it would probably help to have a background “image” to place information even better. I’ll do that next time.

The Actual Exam

Here’s what was different from what I expected: Location: Downtown Louisville, Jefferson Tech I think

Parking was easy to find early enough in the morning. I vavigated straight to a parking lot, and paid for the day (less worry!), about 2 blocks away.
The proctors were very nice friendly people who gently guided my anxiety stricken self through what I needed to do.
Much to my surprise, I got there an hour early, and they invited me to take my test early. I was done by the time my actual start time came around.
The actual exam was pretty much at the same level of detail as the acloud.guru exam was, except for more “you really gotta know this” choose all that apply type questions. I detected at least one stale question on the test, which I commented on.

My weakest areas: Federation, and Cloud Formation, I think. Makes sense, haven’t really had to do that, and we use Terraform for the same task.

Woot woot. Okay, gotta go run a race now.

Visualizing Bitcoin Adventures

I’ve been riding a Bitcoin (and Ethereum) roller coaster for a year .. and its been a fun little diversion. However, its hard to see the journey .. it feels like I’ve come out ahead, but have I really?

I doodled various (complicated) ways to try to show stuff via a 3D graph .. but when I actually went to play, this 2D version works just as well:

Vertical (down) drops == I transferred bitcoin (either to another account, or buying something with bitcoin).
To the top-left = I bought bitcoin for USD
To the bottom-right = I sold bitcoin for USD (for buying something usually)
Note: A similar chart would exist for my Ethereum account. No, i did not buy a switch for $90, it was more than that.
Steep curve vs Shallow curve gives a feeling for price. buy Steep, sell Shallow is the desire.
I could beat myself up .. if only I hadn’t spent the bitcoin, I’d have so much more now…
Currently, my experiment is net-positive. Even if bitcoin goes to $0 right now.

How I created this

Download transaction report from Coinbase. This is a .CSV file, which I then open in excel.
Their report has a “Amount” (Column C) which has + for bitcoin added, – for bitcoin removed.
Their report has a “Transfer Total” (Column H) .. but it isn’t signed.
Their report also has a Transfer Fee column .. I’m ignoring that for this graph. I did sum it up, I’ve paid $30 in fees this year.
Add a new column, “Signed Transfer Total” formula is something like “=IF(H27<>0,H27*SIGN(-C27),0)” – to get a signed USD column. Note that I’m reversing the sign so that a plus in bitcoin is a minus in USD.
Add a new column, “USD Balance”, which sums up all the Signed Transfer Total to date. Something like “=SUM(J$6:J27)”. For simplicity I added it right next to their bitcoin balance column
Grab those two columns and chart it, scatterplot, lines
Adjust the vertical size of the graph till you get something that works for you.

In summary

I do not regret this Bitcoin experiment.

I do hope to leave stuff in Bitcoin, i think it will continue to grow for a while.

Learning New Stuff–Terraform, AWS, Lambda, DotNetCore

This last week has been a crash course in new stuff for me. I’m helping with the scripts that manage the infrastructure around a project – freeing up the developer to work on user stories, I’m taking care (or trying to take care) of the deployment aspects of it. In a way, its a big catch-up to other folks who have been charging ahead into newer technologies – so its not like I’m having to discover things on my own. On the other hand, everything is already evolved to N+2, and I’m at N-1, so its a bit of a firehose.

Here goes though, stuff I’ve picked up this week:

Teamcity build calling a powershell script to do deployment stuff.

New to me: I didn’t know PSPROJ was a thing – that I could step debug through powershell in Visual Studio now. Come a long way since Powershell 1.0.
Dotnet lambda package, zipping, sending to S3… Somebody else whose first name rhymes with “Miss” and last name rhymes with “Aye Lee” wrote this part for something else, I get to adapt it for the current project.

AWS API Gateway => AWS Lambda => C# NetCore1.0 => MVC chain

Got to learn about the “Version Hell” that happens in NetCore1.0. It will probably be much nicer by the time we get to 2.0 or better.. just the 1.0 to 1.1 is pretty rough at the moment. Get the intersection of the bleeding edge of NetCore as it was 7 months ago with the bleeding edge of where AWS is taking their Amazon Linux. We had to do a deviation and host some stuff via EB rather than Lambda.
I’ll be playing more with this on Monday as I try to debug something into not giving me a 500 internal server error.

Terraform as a way of deploying AWS Resources

Modules, and Variables, and Data sources, oh my.
Debugging Terraform – I found the GET/POST requests.. the problem was a Content/Type for a resource in an S3 bucket. Can’t get .body that way, so couldn’t get the hash value.
Partial apply’s because sometimes you don’t recognize a change and don’t want to mess up somebody else’s experimentation
I got to copy what Miss Aye Lee did, nice job Dude.

Rewrapping my brain around Build Configurations

Thanks to previous training, Build Config = Debug (PDB) vs Release, but also = XSLT Config Transforms to get configuration values per environment.
Now, Build Config = just Debug vs Release for “how debuggable do you want this”
There’s another avenue for “which settings do you want to use” which is completely different.
More playing with this on Monday.

AWS Security stuff

IAM User’s for local access from visual studio while developing
Roles for when running in Lambda, EC2, etc. (Built by Terraform)
Policy documents describing what access available to what (built by Terraform), shared by the IAM and Role.
All the stuff that was actually built by Terraform using a Terraform runner credential
The Terraform Runner’s policy that allows it to create all the things
All running in another account that we cross-account assume roles into.
Somebody whose first name does not sound like XML and whose last name might have to do with Whiskey is a good teacher and dreamer.

The end result:

If starting from scratch – done by human.

cd env-shared; terraform plan & apply to create shared resources, like S3 buckets, VPC’s, RDS’s, etc
Any further environment changes, also applied by human via script file. No clicky the mouse.

New environment – like QA1 or QA2 or other – done by human

cd env-qa1 (or mkdir, if starting new)
copy and edit a file that says what the environment name is
terraform plan and apply to create all the things

DynamoDB tables
Queues

Every build to be deployed – automated, not done by human.

powershell to get stuff up to S3
powershell to call terraform to deploy

Lambda
API Gateway hangs out with this.

Pretty powerful stuff. Glad I’m learning it. It will feel better end of next week when I actually have something completely checked in that completely works.

Tree of Direction == Claws of Doom

https://github.com/sunnywiz/TreeOfDirection/commit/8058bc3598c189ded425f044bde39528e335d4fd

This was supposed to look pretty, but its not.

If I look at the simpler version in an isometric top down view, it might make more sense:

This is taking a grid of points on a map, and asking Google for the optimal route from home to that point. This ends up creating .. I don’t know what to call it. Its a decision tree, says Lamont .. yes, that’s a good name for it. It also points out that Google has forgotten that a certain street doesn’t go through, and is incorrectly routing me down Old Henry Rd.

I’ve added something in red .. its the boundaries between “neighborhoods”. Also note that 3 routes almost touch where my current work is, which is why there’s so much indecision about the “best” route to work. Whereas, if you’re in the middle of a particular neighborhood, then there’s little choice. This says the best way for me to get to the new work place will be 146->HurstBourne.

I think this is better done staying as a 2D print. I might need to learn.. PdfKit? Something to generate PDF’s with. I can get a poster-sized PDF printed for $5 via Office Max. However, what I might do is overlay a few maps .. like “with heavy traffic” vs “without”, with transparency. And a lot more detail.

I did learn a lot about using Google’s Directions API. Very nice. Polylines are pretty amazing – took a small string and it decoded into 22 points!!! https://developers.google.com/maps/documentation/utilities/polylinealgorithm.

More to follow, I’m sure.