sunnywiz – Page 21 – A Geeky Gulati

Tired of PhotoScan, I just want it to Work: 2/N

Since the last Blog post, I finished going around the object and selecting enough cameras to get a decent set of dense point cloud going. I did this in four chunks; I was trying to quarter the model with each chunk:

9 cameras
1553 points

17 cameras
3260 points

10 cameras
2332 points

11 cameras
2575 points

Lets Align Some Chunks, Shall We?

Align Chunks, Camera Based, 1 Camera overlap

Nothing. It needs more cameras? I could see it being in 3 dimensions, it would need 3 cameras minimum. I’ll come back to this with more cameras in overlap.

Align Chunks, Point Based

Not Quite. And as far as I can tell, there are no manual align controls anywhere (in the non-professional version).

I could take this out to Meshlab and try to align it there, however, I won’t later be able to map a texture; I have to solve this in PhotoScan.

Add More Cameras, Align Chunks, Camera Based, 3 Camera Overlap

While I’m at it, I also add in a few more cameras in some of the gaps that I see. And this is what I get:

Queue Darth Vader Imperial March

I can tell it got the cameras correct. However, my fear is realized: I think as I walked around the subject, he moved slightly. Or, my distance from the subject was not constant, so I ran into some lens calibration issues, and thus the resultant object was not mapped at the correct size. Either way, what we have now is a FrankenDan.

I cannot resist. Going all the way through to a model and texturing this beauty – and learning how to do an animation in blender at the same time —

[youtube=http://www.youtube.com/watch?v=-V4Yam2yCU8&w=448&h=252&hd=1]

Franken Dan Murphy

Attempting a single chunk with the same 55 cameras

What is happening is either a) the model moved, or b) I changed distance from the model (and the camera alignment is wonky), and it just cannot get the math to work. FrankenDan is actually a better representation of the reality that was captured.

So.. I don’t think there is a solution here, with a GoPro Hero3 walking around a subject. There are several directions I could go, though:

Start from the back of the subject, so that the seam would be in the back.
Put markers on the person’s back (so that there is something to “fix” on), or give them a “garland” of some sort.
Use the 120fps to capture the model quicker; but I need to find a reliable way to spin the camera around the subject and hopefully not invoke motion blur. (Hula Hoop Mount?)
Use a better camera (not a GoPro); perhaps a DSLR; with a ring laid out for distance from the subject (see teaser solution below)
Use multiple cameras! (so many people have had success with this – and they don’t have to be good cameras either)

Teaser Solution: DSLR

In comparison, here is me, taken via a DSLR camera with a 50mm fixed (prime) lens. Its not quite printable, as the (shiny? homogeneous?) back of my head failed to capture. There’s definitely something to be said for not using a GoPro.

Soylent IS [for] People!

I am eagerly awaiting my first batch of Soylent.me. Turns out I’m not the only person .. coworker Doug is awaiting it as well. But coworker Doug, being very interested in the subject, went a step further and bought all the ingredients for the most popular DIY.soylent.me recipe: People Chow 3.0.1 – “Tortilla Perfection”. He brought some in today. I had some (a mugful) for brunch.

Taste: very corn-tortilla-ish
Texture: A little chalky, about the same as a slim-fast meal replacement thingy from wal-greens,
Burpiness: Yes, I burped. And I seem to continue to want to burp.
Hunger removal: Yes, definitely removing hunger. Past hunger, now at the “wow I just had 2 bean burritos from Taco Bell” stage. from 1 mug.
Wierdness:

One of the muscles in my back (leading up the left side of my neck) decided to cramp shortly after consuming the beverage. Probably unrelated.

I would definitely have more.

Day at the Office: LINQ vs Stored Procedure

I love using LINQ (to EF to SQL). It takes care the plumbing of getting data out of the database. But there was one case where I had to swap it out for using a stored procedure.

The Setup

There were several screens that showed similar data, but constrained in different ways. They all shared common code along these lines:

Base:

iQueryable Q  // defines the base query that all screens need

Overrides 1..N:

IQueryable Filter(IQueryable Q) { 
    return Q.Where(… additional clauses…); 
    // specific to the screen 
}

and then calling back to some Shared code:

IQueryable Filter(IQueryable Q, FilterSettings fs) {
  // apply standard filters that user could set at the app global level
}

The app used to then take this IQueryable and send it over to a ProjectAintoB() function that did the actual query (.ToList()) and then mapped that stuff out to a viewmodel / list item thing to shove into a grid:

return new GridItem { 
    Description = Q.Description, 
    ClientName = Q.Client.Name, 
    ManagerName = Q.Client.Manager.FullName, 
    Exceptions = Q.Exceptions, 
    NotesCount = Q.Notes.Count()
};

Note: this is simplified. The original GridItem class had somewhere around 35 members, across 8-9 tables.

When we first started using the app, this worked well enough. It returned quickly enough to not be noticeable. However, as the app started to grow, things started to get noticeably slower, so we went onsite and did some profiling, and this is what we found:

Actual screenshot/image from original analysis document around the problem

At the client site, it was a lot longer, with CPU pegged at 100% for 2-3 seconds.
Turns out, most of that CPU was LINQ attempting to build the query, not actually executing the query.
- I didn’t notice this on my Dev machine because of superior hardware and multiple (8) processors. At the client, most of their processors were 2 cores, and the other core was busy doing client-ish stuff.
- This was with EF 5.0 with .Net 4.0, before they had “cached compilation of queries” (.Net 4.5). I tried upgrading to .Net 4.5; but still had problems.
After the query was executed, there was a network spike as the data returned from the server
- Turns out, by having 9 tables returning, and the way EF work(ed|s), there was a LOT of repetition – the same Manager FullName across 5,000 rows, etc.
- This lead to a large network payload of returned data
- This was a killer over VPN. (Which only affected me, the developer, but I am important!)

The Solution

We did it in two stages:

We kept the original concept of an IQueryable that gets modified by various layers to get down to the data that needs to be pulled back.
We executed it, but only pulled back a single item: select g.GridItemId. On some screens, this was 50 items; on other screens, 10000 items.
We called a stored procedure which took in a TVP (table valued parameter) of id’s, and returned multiple result sets:
- one result set for the grid items + external id’s to other tables
- one result set for all the Clients referenced by these grid items.
- one result set for all the managers referenced by these grid items.
- one result set for all the Notes referenced by these grid items.
We read these in, and directly constructed our grid item from these results.

And it worked wonderfully.

Average compilation time at the client site was down to 50 ms (most of the complexity was joining in to all the other tables to populate a grid item)
The second call to the sproc finished within 20-40ms or so
The payload across the wire was much smaller.

Here’s a screenshot of a paragraph I wrote in my excitement at getting it to run so much faster:

I also recorded a video of a before screen, and the after screen. I can’t show that to you, but it was a hit.

The Result

This, and other performance enhancements over the last 6 months – according to the client last week:

Client: Yes! Actually, 2.0 is faster than 1.0 now! We dislike going back to 1.0.
Me:

Tired of PhotoScan, I Just Want it to Work: 1/N

Subtitle: Going to ridiculous lengths to understand what doesn’t work with PhotoScan.

I took two sample videos with the GoPro a few days ago, of Dan and Rider. I want to print a color 3D model of them (shapeways, small), just to see it done, and to have a simple process to do it. But it keeps not quite working, and its annoying me. So, here goes another night of experimentation. What am I missing?

Here’s the precalibration stuff from Agisoft Lens, btw:

Check #1. How much does it matter how close or how far apart the frames are?

I extracted out at 60fps (the video is 120fps), so I have 1400 frames in a circle. That’s a lot of frames.

Here are sample reconstructions using just two frames – varying the number of frames apart. I’m using 4mm as the focal length, but I will play with that in the next section. Process: Align on High, Dense cloud on High. The picture on the right is what Frame # 0 looks like; the screen capture is “look through Frame 0”, zoom out to bring the head in frame, and rotate left (model’s right) about 45 degrees.

1 frame apart (cannot build dense cloud) 2 frames (cannot build dense cloud) 4 frames (cannot build dense cloud)	8 frames
16 frames	32 frames
64 frames	128 frames (cannot build dense cloud)
All 8 frames	Above view, to see how the cameras are aligned

Clearly, more pictures is not the answer. The best one was 0 to 32, which was about a 6 degree difference.

Check #2: Trying every 32 frames, how does adding more pictures improve quality?

This time I’m moving the camera up so I can see the “jaggies” around the edges

3 Frames combined (0,32,64):	4 frames combined:
6 frames combined:	7 frames combined:

The same 7 frames, this time with the wall in view, trying to line up the roof and the wall:

Check #3: Focal Length

Trying to solve for the wall jagginess.

2mm:	4mm:
6mm:	8mm: Cannot build dense cloud
5mm:
3mm:	4.5:

Okay, so .. 4.5 is wonky, but 4 and 5 are okay? Its very hard from this angle to see any difference in quality between 3,4,5 and 6. 2, 7, and 8 are clearly out**

Maybe another angle:

3mm:	4mm:
5mm:	6mm:

** Or maybe 7 is not quite out yet. Turns out, I can “align photos” once.. get one result.. then try aligning again .. and get a different result. So I retried 8 a couple of times over, and I got this:

None of this is making any sense to me. I guess I’ll stick with 4mm, for lack of a better idea. Do you see any patterns in this? Moving on.

Check #4: Low, Medium, High Accuracy?

I’ve bumped it up to 17 cameras (32 frames apart). Testing for “Align Photos” accuracy (Low, Medium, High) + Dense Cloud accuracy + Depth Filtering

High, High, Moderate	Low, Low: Cannot build dense cloud. Medium, Medium, Moderate
High, Medium, Moderate	High, High, Mild: (Mild took around 3 minutes)
High, Ultra-High, Aggressive: (12 minutes)	Close up of H/UH/A:

Aggressive is definitely the way to go; however, there are still way too many floaters!

Ah, but this image might clear that up a bit. It has to do with the velocity with which I was moving the camera. I slowed down. Hence several of the frames are not very far apart. I might need a different approach for frame selection.

Test #5: Compass Points Approach

I will attempt to bisect angles and derive frames in that manner. Note that I’m not going to try the full 360 – I suspect that the subject moves a bit, so it can’t connect 359 back to 0; instead, I’m hoping to get a nice 90 degree profile, and maybe merge chunks to get it down to a single model. So lets try to get a set of frames from the image on the Left (000) to the image on the Right (400).

0,200,400 – Aligns 2/3
0,100,200,300,400 – Aligns 5/5, but fails to build dense cloud
0,50,100,…,350,400:

I have to cut this blog post short here – it looks like I have WAY too many images, and Live Writer is freaking out. Doing a quick edit past, and then posting this as a part 1/N.

Day at the Office: Performance

I am working on a ticket where I have to augment data returned from an external system with some local overrides, without adversely affecting performance.

Running the screen, it seemed a bit tardy to begin with. So I pulled out one of the tools I love to use: a profiler.

I use the Ants Performance Profiler. I paid for it with part of my Christmas bonus. While I am a salaried employee, and my employer provides me with lots of stuff (awesome computer, pop, snacks, coffee, ReSharper, MSDN), there are additional things that I choose to use because I like who I am with them. These include Ants, TeamViewer, nDepend, Beyond Compare, Fences, and VmWare, and a 3rd monitor. Not all are paid for yet – two more christmas bonuses should cover it.

Well, that’s interesting. Curiosity strikes: What are those spikey areas about? What’s going on? How long is the initial query taking? How long are the webservice calls taking? If I add on to the end of the webservice call, how much would that grow?

Spike #1 – Query + Initial Screen Draw

Action: highlight the area and see where the time was spent. If you’ve never seen this before, there’s some Windows GUI app running stuff to get out of the way:

The green highlights in the top image are times when the method under the cursor was fired.
The stack trace is not so scary once you understand the internals of a running program a bit.

“A” is the part of the program which receives almost any kind of “this work needs to be done asynchronously” call. This means Execute.OnUIThread(), drawing events, bindings, etc – almost all get serviced here.
“B” is where Task.Factory.Start() things get serviced.
“C” is any point where input needed to be handled, and ended up making the app do something. Mouse movement, hover, clicks, etc – and part of whatever they ended up triggering. That is what is highlighted in green
“D” we haven’t drilled into yet.

Unfortunately, if you Async a call from A to B, they show up in different parts of the performance stack traces (at least as of Ants version 7)

Drilling further into “D”:

Ah, I’ve seen this before. “Measuring” usually means a grid has been set to “auto fit columns” mode – and as its drawing things, its figuring out how large they want to draw, how much to resize, and then redrawing.

What about the actual query to get the data? I found it way down in here (this time in bottom-up mode):

Wow, that was a lot of blurring. Point being: the query is fast, rendering the results in the grid is slow.

There’s another blog post that could be written here. The stack trace above goes from “Get me stuff” to a IQueryable to a Project to … What, a Stored procedure? Yep, it does, and it went from 25 seconds down to 2 seconds. Sometimes, EF does not win.

Spike #2: The Flat Plains of Update

This is the result of getting “new information” from the external service, and trying to update the grid.
What you are seeing is the NotifyOfPropertyUpdate() => saying “hey, UI, update yourself” .. and waiting for the update to happen.
-update- after a day of dinking around in this code, its simply a size issue. There are 15000 rows of data in the grid – and we are updating every single one, with 4 columns of new data – and each each update “paints” itself on the screen. This takes a while. I couldn’t find any way to optimize this other than delaying the entire screen till we have all the data (ie, don’t paint twice).

Lets Try To Remove AutoFit: After Changes

before:

after removing AutoFit:

Not very different.

Welcome to performance optimization. Sometimes, you think you have found a smoking gun, but you haven’t. I might have squeezed a 10% out of that, maybe.

I must now leave this blog post and really fix some things. And get on with the original task, which was to add some local database overrides to an external web service call without making the system significantly slower.

-update- Its now the end of the day, and I have the additional lookup code written. Here’s the impact:

The green highlight is the call to the external web service(s) (async) and the subsequent updating of the columns in the grid.
The pink/purple is the additional code that I wrote that patches up values from the local database.

Good enough.

Pepakura Lucy next to the Real One

This time, I printed things out on cardstock. Bad move – Cardstock doesn’t bend very well, I had to pre-bend every bend, and even so, the thickness of the paper caused some things to move out of place over time.

I think it would be better to make certain pieces from cardstock, and the rest from regular paper, however you would have to “open” the model just right to get the pieces just right and not obvious.

Also, the resulting struture was just too complicated to put together. There’s no way I can get the red piece on the right glued into her head correctly. Or, there’s a way, but its too frustrating to keep at it.

One thing I can point out though: In Pepakura, I said, “Model Height=160mm” .. and yes, the final model matches the original fairly well. That’s a win.

I’ve decided I’m not spending any more time on this particular model, too many other fun projects to play with.

Day at the Office: A Tale of Testing

TL;DR: Its really awesome when you understand the customer’s business language and can do thorough testing for them and document it.

I’ve been working on a bug for the last few days, which involves some math, and positive vs negative signs and such stuff. Basically:

Customer sent me a screenshot where the math was obviously not working.
We figured out what was going on (boundary case not handled), and handled it.

Normally, we’d be done at that point. Except, this customer does not have a dedicated QA department; we are the first line QA department. So.. we decided to be thorough. Hence, this Test Document (screenshot; intentionally small so that details are illegible)

We posted the testing results of 7 types of things + 5 states of things, and highlighted any suspicious behavior (in yellow) for confirmation from the client —

And we have a brilliant customer, who gave us prompt feedback: (in blue)

Which we then fixed, in purple.

Time Creep

This 2-3 hour bug has now become a 2-3 day bug. I feel a bit guilty about that, yet – if these situations had been discovered in the future, that would have been more angst, spread amongst more people; I’m glad we could figure this out now rather than later.

Also, by doing this testing and documenting it, I’m generating a snapshot-in-time of how the app is supposed to work.

Unit Tests?

The only thing that would make this even more awesome would be to do these as unit tests. But that would have been 3-4 days instead; also, we’re trying to wrap this project up. Also, the customer doesn’t speak “unit tests”, but the customer definitely speaks “screenshots of the app”, so we’re choosing to converse in the latter language.

But how did it get Borged up to begin with?

I had shared the content of this post with a friend, and he asked the obvious question – why wasn’t this caught sooner? The app has been used in production for over a year now.

Reason Part 1: this code (recalculating estimated cash balance in an account) was recently rewritten for performance optimization. The old code (before my time on the project) was huge, complex, not documented, and slow – it was actually used for generating charts forecasting cash balance out into the future, and was designed to handle one portfolio at a time; the new code could do things in batches of portfolios, maximizing lookups of shared stocks and options, etc. When we did the optimized calculations, we tested everything that we (and the client) knew of at the time. No regrets.

Reason Part 2: My client is also small enough that they don’t have a dedicated business guy who documents every rule around every calculation. There’s a wealth of “common shared knowledge” amongst several people. While we, as developers, try to understand that as much as possible, there’s a reason they do what they do very well – that stuff is complex. Instead, our goal is to get 80%-90% close, “usable”, and then if there’s gaps remaining, those will get revealed, and patched later. Reference concept Minimum Viable Product. (Only doable with in-house applications; externally facing applications in a marketplace need more attention on polish and completeness.)

More Photoscan Experiments

I need to wrap up this subject soon. But every time I visit a thread, I seem to open several others. This is about point and model reconstruction in Agisoft Photoscan.

My Goal

I want to be able to create realistic busts and maybe even poses of people, in full color, that I could 3d color print (via Shapeways, etc). (Here’s a professional company doing the same thing: Twinkind, $300, including visit to a studio.) In order to do this, I need to grab the frames very quickly, so keep trying to use a video camera to grab frames and then pull data out of the video. That’s not working too well for me.

Original, for reference, is on the right. Lucille Ball?

Two Attempts Contrasted: (Dense Cloud, because it shows imperfections well)

iPhone 5s, 26 pictures:

The subject is in focus
There’s a lot of resolution which leads to accuracy in 3D space
If you miss a shot from an angle, you are S.O.L. For example, I don’t have her right ear.
Takes a while to take the pictures This one took 2 minutes and 30 seconds. I had to click to focus each shot. And the subject must not move.
I don’t think I used calibration (taking picture of a grid, use Agisoft Lens to generate camera model) for this, it could get it from EXIF data, and it did a very good job at that.

JVC Everio HM1 1080p Video Camera, 0:47 seconds

Extracted at 5 frames per second to 247 frames.
I actually did two passes around the subject, so really about 20 seconds would have sufficed.
Lower resolution on the pictures, so the 3D is blockier (I think). And more depth errors.
a lot more processing time.
I did use Agisoft Lens to calibrate this.

GoPro Hero 3 Black 1080p Medium FOV

I don’t yet have decent results with this. I think I need to do some calibration with it first, then I might retry. Different post…

iPhone 5s Video

I also need to go back and try this using the quality estimation technique listed below to find the best frames. However, the data rate is much lower than the JVC, and I’m pretty sure artifacts will be a problem.

Red 5 Diamond

Heheh. I have a film friend who raves about this camera. Nope, I don’t have one.

Conclusion

I don’t yet have a solution for what I want, but I do have a solution for a model good enough to create a pepakura for. Maybe in the future I can have an array of cameras.. that’s what the pro’s like TwinKind use. I’m not a pro.

I might need to try this with “sport” mode + burst (click click click) using a DSLR camera.

Frustrated about 3D Stuff

General feeling of frustration tonight (time of writing: Friday 5/9). Hard to put into words. But I could draw it. (might need to zoom in)

The Green things are things that I have figured out.
The Red things are things that definitely have not worked for me.
The Yellow Things are things I want to be able to to.
The Orange things are things that I haven’t yet figured out. They depend on each other, there is usually a chain.
I forgot about Minecraft Prints. Those fall under “Color print that I might could afford”, pretty much.

Latest Frustration Tonight

It turns out a Agisoft PhotoScan project file (.PSZ) looks like a ZIP file, with a doc.xml, which has the below structure – so I could write code to hunt through 1420+ images, and select the best 50, spread out.

The code looked like this (sorry, no LINQ, going old school here, command line parameters, etc)

Except that the PSZ not really a Zip file. When I tried to stuff the modified doc.xml back into the .PSZ file, it came out as being corrupted. Dead End? Retry with 7Z? Extra metadata? older compression format? ?

I guess what I have is code that tells “which cameras I should enable”. That’s workable, except that I need to grab frames/image[@path] so that a human could identify it.

Future: Maybe I could write code to read the video, figure out the quality of the frames, and only extract the best frames?

Also: the $3000 version of Photoscan has a Python Scripting interface. Sorry, I’d rather buy a color 3D printer than that.

However, good news, it looks like I might finally have a lens profile for a GoPro Hero3 in 720p (x120FPS) Wide mode. I had to play with the focal length, for some reason numbers like 0.1mm and 2mm work way better than 7mm that folks advertise. More to be proven later.

Slicing and Dicing with OpenSCAD

In my quest to make a cool coaster, I wanted a way I could slice up a model so that each face could be printed, well, “face-up”, so I don’t run into problems with overhangs and supports and stuff gumming the works. I would then glue the model together later. (In the case of Coasters, I can also swap filaments and have the design “pop” on each side)…

In the process, I learned some OpenSCAD:

F5 = “quick”, F6 = slow and build
No such thing as “export this object”, it just builds and then you can choose to export everything.
variables are lexically scoped, ie, more like constants. There are some dynamic variables as well.
I had to remember some algebra.
I applied some D.R.Y.

Here’s the result as applied to http://www.thingiverse.com/thing:60728 – first in “ShowMe()” mode:

And then SliceEverything() mode without the full fledged CSG operations:

And then SliceEverything() in the way that I could export an STL:

Yeah, that crashed. It was too complicated. I hate to take out the // Slice Everything, and instead, here’s just the front and an ear, exported to STL, and viewed in NetFabb.

Note: Its NOT fast, when doing the full build (F6). It also crashes less if run from command line – apparently the crash is during rendering to the screen?)

Show Me The Code

Since its getting a bit long, I’m linking to github below; but this is what it looks like, approximately:

link to code at github

update: I’ve tried to use this approach for several things now, and … its very fragile. So fragile, that I have successfully used it to create a print. Almost every model is “too complex” and fails at render. I might need to try a different language.. something that is rock solid.