Backup Strategy Revisited: v2016.05

A while ago I wrote about using crashplan as my backup strategy. That didn’t last long. Crashplan (free) eventually stopped syncing. I did not have time to look into it.

Then I listened to Mac Power Users Episode MPU318: Backing Up and I got re-inspired to pick up the gauntlet again.

I recently re-acquired my desktop machine at home (it had become my wife’s in an emergency move when her machine started BSOD’ing), and I decided to try Yet.Another.Backup.Solution; this time going back to what some readers had suggested (see comment section here), using BitTorrent Sync.

So far so good

(click image for larger view)

I keep all active (personal) code bases in their own folder, c:\code,which rely on github, visual studio code, beanstalk, etc – various source code repositories – for their backups.
I keep a Dropbox folder somewhere on all my machines. Dropbox is my main “My Documents” folder, for ad hoc type stuff. But, it does not get stuff related to a project involving media – for example, a video edit from my GoPro. That’s too large for it.

I chose dropbox because I used them first, and I’ve stuck with them so far. I believe I’m at 7.5G, free. There are other services that provide similar…
I do an occasional 1-way copy from dropbox to a spot in my big share. Just in case.

I set up a Windows Storage Space on my big home machine, across the top three drives from my former home server – 2TB, 1.5TB, and 1TB … which combined give me a 2TB 2-way-mirrored array. Any one drive dies, no problem.
I set up BitTorrrentSync Pro (paid) ($40 personal one time) to set up a personal cloud:

1.2TB of data between my home storage space and an external hard drive hooked up to my work computer, over the internet.
Several per-year folders (2016, 2015, 2014, etc) that are much smaller that live on every machine I am productive with. (my convention is /yyyy/projectName for any given project, copy-forward if it goes between years).

I’m doing a lot less video work now, so the sizes are quite manageable. 2014 was the peak, that was 283G. That’s been pruned down and put in the big sync now. Most I’m doing for a year now is 15G, because of a GoPro video.

If I have a project that exceeds the size of one of the laptop drives, I’ll probably break it down into its own little BTSYNC cloud (“2016ProjectName”) and share that amongst the machines that care.

I’m still relying on Dropbox for my phone photo backup, although I am doing a iCloud backup as well.

I am currently not backing up my mom’s computers, because she has 4(!) of her own, with Dropbox set up to keep track of important stuff. Go mom! (I need to check on that soon, verify health).

On Seeding Large BtSync Share

I started out with a single copy of my files. USBHD1:\SHARE
Copied them all onto USBHD2:\SHARE
Walked USBHD2 to the office, bypassing the internet.
Set up a BitTorrentSync folder on USBHD1:\BTSYNC1 (initially empty)
Sync that to USBHD2:\BTSYNC1 (initially empty)
At home, copy USBHD1:\SHARE to USBHD1:\BTSYNC1 – 5 years!!!! (usb 2.0 is not great for disk speed)
At work, copy USBHD2:\SHARE to USBHD2:\BTSYNC1
Within 24 hours everything was synced.
I compared files against the original shares and there were no problems

Now that I’m more familiar with it, I could have just told the work computer, “really, use THIS folder to receive the files, yes, I know it already has stuff in it”, and it would have worked fine.

Moving the data from the USBHD to the StorageSpace was easy:

Copy the data
Stop BtSync
Unplug the Ext Hd
Start BtSync
Tell BtSync where to find the files again.
Profit.

What Did Not Work Well For Me With BtSync

Using my iPhone to browse my big share – Nope, it tried to download the big share to my phone (in selective sync mode) (which does one place holder file for every file) (too many files!)

Selective Sync in general – I did not like the idea of placeholder files. I’d rather have placeholder directories.

In Conclusion

So.. there you have it. I have my mirrored local, I have at least 1 offsite backup, several protocols, and an offline backup in case of ransomware.

We’ll see if its still working fine a month or two from now.

Figuring out a Sync strategy

At first I was going to draw this all out – I did, with VISIO, but the drawing isn’t quite right pretty enough. It does reflect the solution I’m going towards.

I have the following computers, with the following uses:

“Big” Computer – my main workhorse desktop at home. It does a lot of Video Editing, Audio Mixing, etc.

Currently has USB 3TB drive #1 mounted

Surface #1 – my most portable computer. Usually just edits things in the cloud, occasionally gets an offload of an SD card while I’m on the road.
Laptop #1 – My “I would program on this laptop” laptop, which I mostly use for Remote Desktop into my work computer
Laptop #2 – this is the laptop that controls the 3D printer. It has a lot of 3D printer tools on it.
Work Computer – for work stuff. Not my computer

Currently has USB 3TB drive #2 mounted

I pretty much use the pattern of:

C:\2014\<project Name>\ is where I put all files for a project.
I keep the same project name between computers if I move the project around. For example, when I was doing book design for Dad’s book, that started on Laptop #1, then moved to “Big” Computer
I consider “iTunes” to be a project, as well as my local instance of my Dropbox folder. Unfortunately, these are still in 2013.

My needs:

When I’m working in C:\2014, it needs to be fast and stable.
When I’m working in C:\2014, it eventually needs to get backed up to the big backup
Not all C:\2014 projects should be on all computers. In fact, almost NONE of them should be on the Surface, it only has a smallish SSD. Same deal with Laptop#2, that has 10G free or so after the O/S.
The Big Backup should be offsite-level backed up.

Limitations

A computer cannot btsync with itself (yet)
A R/O folder (like the phone backup destination) cannot be inside a R/W folder.

Options

I thought about something like this:

BigComputer 3TB \BtSync1 syncs to Work 3TB \BtSync1
I create a sub-sync of WORK:\BtSync1\users\sunny\2014 to BigComputer:c:\2014

It would work, but it would be a bit ugly. Lots of hops going offsite and then back home to get things backed up.

Winner?

I believe I’ve decided on the following:

The large 3TB drives maintain their own sync pool.
The local folders (C:\2014) MIGHT maintain their own sync pool, on a per-project basis. For example: 3dModels between my big computer and the 3d printer laptop.
Every project should end up at Sunny’s Big Computer
I’ll use Robocopy on a schedule to bridge the final gap from Sunny’s big computer c:\2014 to USB:\BTSync1\users\Sunny\2014
When I bring another machine online at home which can hold large drives, I’ll add it to the big sync pool as well to have a local (faster) backup. (Offsite can lag behind by days when we’re talking video files)

For things like backing up Mom’s important stuff, I’d probably create a USB:\BTSyncMom folder and have that be its own pool, not make it sub- or super- to any other sync pool. Or, continue to use Crashplan there.

Something I Don’t Yet Have

Windows Home Server gave me the ability to restore a machine to a specific state, covering all the installed software on a machine. I could do that using a copy of Ghost, or some other Hiren tools. I don’t yet have a plan for that. On the other hand .. reformatting a machine these days is “no big deal” for me, with all the data being out in the cloud somehow anyway.

Fun with Bittorrent Sync

BitTorrent Sync can be downloaded here: http://www.bittorrent.com/sync

Large Folders

I now have two external hard drives, one at home, one at work, synced with each other:

I first xcopy’ed one external drive to the other. This took about 20 arns.
I then hooked one drive up, and pointed BitTorrent at it. It started indexing.
I took the other drive to work, pointed Bittorrent at it there with the same shared folder secret, and it too started indexing.
The two instances of BitTorrent spent quite some time chatting with each other over the network: “I have this file!” “I do too!” .. getting to the point where they agreed that they were in sync with each other. This chatting and indexing phase took perhaps 2 days.
They are now in sync. They still “chat” at each other every now and then, but the network traffic is minimal. (I think)

Robustness

I then played with the robustness of the sync. First, I renamed a folder on one – did it transmit it as a rename? Why, yes it does:

I then turned BitTorrent Sync off on one of the sides (but not the other), and did some more renaming. I got a mixed bag of results:

What it looks like to me is, if BtSync is listening to a folder, it remembers actions taken and can catch up other clients to those actions; however if BtSync is turned off, it takes its new indexing data as “new stuff that happened” (ie, it doesn’t know there was a rename) and thus deletes and adds files as necessary. In the end, the two repositories are in sync, and the deleted files are copied over to the .SyncArchive folder.

iPhone Backup

I turned on the “synch my photos from my iPhone” feature. It creates a bitTorrent sync source, with only a readonly secret for others to consume.

If I delete a photo on my computer, it is NOT resynced from the phone.
If I delete a photo on my phone, it is NOT deleted from the computer
The files are not date/time stamped like they are with the Dropbox export.
It only synchronizes when I open the BitTorrent Sync app on the phone. There is no option for background refresh. (+1)

So far so good, I like it.

Nested Folders

The idea is this:

D:\BtSync1 is synched from A to B
On the same computer, can I additionally sync D:\BtSync1\Users\sunny\2014\3dmodels from A to computer C as C:\2014\3dmodels?

This way, I can keep specific projects (in this case, 3d printing stuff) synched between two computers, while having the data synced to the offsite backup as well?

Answer: Yes, as long as the parent (D:\BtSync1) and Child (+Users\Sunny\2014\3dmodels) are both Read-Write secrets. Ie, I could not place my phone backup folder in BtSync1, but I can do what I want above.

Testing it out:

Deleted file “LaundryBasketHandleV1.stl” on A (not shown by name in the log file below).

Deleted on B, and C

Deleted file “LaundryBasketHandleV1_fixed.stl” on C (“MOLLY-L2013”)

deleted on A and B

Created a new folder “coldwellbankercoaster” with a bunch of files on B (“SUNNYDESKTOP”)

copied to A and C.

I like it so far. Dude, I would pay $100 (one time, multi-install) or $15 (per computer) for this software. And for right now, its free!

I am brewing a plan for total world domination. My world, anyway.