The new device limit for free dropbox accounts has been around for awhile, but I only just discovered it when I got a new laptop because apparently, existing devices get grandfathered in. So in order for me to install Dropbox on the new laptop, I'd have to remove dropbox from 13 other devices (which means I apparently have 15 devices with Dropbox installed). A few of those devices may be defunct, but I definitely have more than 3 that are actively using Dropbox.
Here was my thought progression on working around this issue:
I started kicking around the idea of using rsync to keep a pseudo-dropbox folder on the new laptop up-to-date.
Then I thought, it would be nice if those syncs happened in real time as things change (as opposed to running on a cron job).
I discovered fswatch that could trigger rsync whenever anything changes in a computer with an active dropbox account.
Then I realized, I'd want the sync to trigger from the new computer (without dropbox) as well if things change there and realized, I would have problems with 2 instances of rsync running from 2 different computers...
I thought about adding a lock file in order to prevent different instances from running at the same time.
Then it occurred to me that I only need 1 instance of rsync to run and just need to trigger it from 1 computer with dropbox installed and from each computer without dropbox... Thus: rPi & node red!
I know which computer I would use with a dropbox instance installed. It runs all the time. And right now, I only have 1 device that I want to add a pseudo-dropbox instance to, but there may be others in the future. And I'd like to do this in a way that will scale... I never installed the Dropbox container on the rPi, but I guess I could use this strategy to apply to the rPi as well.
I haven't used rsync a lot, but I assume that it can handle this. It's just a question of how to handle overlapping/multiple triggering events. Is there a node red workflow design scheme for keeping track of the change events in such a way that redundant requests don't trigger useless syncs but also not miss differences encountered on multiple devices simultaneously?
Like, let's take 3 devices: A, B, & C. A is the rPi, B is the only device with Dropbox installed and C is my new laptop.
Example: Change occurs on B. B notifies node red running on A. A starts the sync between A and B. At the same time, A notifies node red that something has changed on A's dropbox copy (but this is because of the rsync that's running - and thus should be ignored). The A/B sync finishes and then A moves on to sync A with C. C notifies node red that something has changed on C's dropbox copy (but this is because of the rsync that's running - and thus should be ignored).
Then, what if an actual change occurs on C while the A/C sync is running - and that change is missed... We ignored that notification because the sync was running...
Questions:
How do I implement ignoring redundant sync notifications?
Are there files that you need to be able to change on more than one machine and therefore need to sync in both directions? If so then I wouldn't even contemplate trying to handle that myself. It is fiendishly complicated with very many edge cases that you may not have thought of. If any individual file is only ever changed on one device then you might be able to make it work, but it is not going to be easy.
Have you thought of the idea of having a dropbox account for each machine and using shared folders to sync between the accounts? I don't know if there are limits on the number of accounts you can share with on the free dropbox.
Probably the best solution is something like NextCloud, OwnCloud or Syncthing.
Yes, multi-way sync is a nightmare even for a dedicated sync tool. I would only consider a star approach with multiple devices syncing back to 1 master with no overlaps. Or the master pushing out to all slaves.
If you really need multi-way, multi-device sync, consider syncing to separate folders on the master and then having a periodic reconciliation process to automatically remove inconsistent files where you can and report where you need to do a manual reconciliation otherwise.
Alternatively, cough up for more space. I got grandfathered into more space on my Microsoft account so I tend to use that. I also pay for 1TB of Amazon storage since I already get unlimited photo storage via Prime. My NAS will back up files (pre-encrypted) to Amazon. So for me, everything goes back to the NAS and the NAS gets backed up. Office 365 Home or whatever they currently call it is also good value, you can get up to 4 TB off a single subscription by creating sub-accounts. 1TB per account.
Since rsync syncs everything incrementally and decides how to update based on things like lmds, I would think that all I need is to know when something has changed on one of the sources to know when to run the sync. Perhaps I could create a lock file and perform the sync with everything until no differences exist, then release the lock. During the lock, I would ignore new change notifications. Then I could also do an occasional cron'd sync just in case.
I don't use dropbox for photos, videos, or music. I have prime and have been thinking about employing something like that. My dropbox mainly has documents, scripts, vnclocs, project files, shares of project results, calendar & keychain backups, folder action folders that have applescripts that are triggered via IFTTT on different computers, etc..
I have a free personal and an unlimited company google drive, and a paid 50g iCloud Drive too. Mostly, it's my legacy IFTTT automations setup through Dropbox that make me want to setup a sync. But mainly (until recently) Dropbox is the easiest and least annoying to use. My free Google drive account is mostly taken up by motion detection security camera footage. But google drive is annoying because the trash fills up and you have to manually empty it. I finally implemented a workaround to that using a google workflow to automatically empty it. And google drive (until recently) didn't have a quick way to share a file from the Finder. And don't get me started on my iCloud drive. It's annoying as hell, but a necessary annoyance and is constantly low on space.
I've read up on the Dropbox device limitation and people have tried registering multiple accounts, but they apparently have a way of preventing that. Even if I were to implement this sync, it wouldn't work with my mobile devices...
So maybe I should just migrate all my stuff to google drive... In the meantime, in setting up this new computer, I just copied my dropbox over to it to get at all my scripts...
Right. I need 2-way sync. Basically, I want a pseudo-dropbox.
Just looked at Syncthing. It doesn't say that it syncs extended attributes or access control lists, which is necessary for HFS+ systems... Although, I admittedly didn't look any further than their main page.
Yeah, I was just thinking about that (and pouring through the rsync usage). This does happen with dropbox files, and what it does is it creates a "conflicted copy", i.e. it saves both. Then I typically look at them when I encounter them and resolve the conflict manually. This happens with my vncloc files frequently which are updated via cron (in one location - so I'm not sure why it ever conflicts - probably due to modifications in the sync).
But it looks like rsync doesn't quite work the way I thought it did... It will skip files whose lmd is newer, but otherwise it's only 1-way. Like I said, I haven't used rsync much (and in a long time).
I was just looking at the google drive plans. The free 15G google drive is a bit unpredictable since it saves gmail to that space as well. But you can get 100G for $20 a year. That's much better than the smallest 2Tb plan Dropbox offers for $10/mo. I just don't need that much. And it's more cost effective than my 50Gb iCloud plan is at $12/year.
On the other hand, maybe Syncthing would work after all. I think that the extended attributes and ACL may only be important for backups. After all, I definitely have different folder actions attached to different dropbox folders. I had discovered awhile back that you can't attach folder actions on different computers to the same folder.
However, syncthing says that its CPU intensive - so I wouldn't be able to run it on my rPi, which already gets too hot over the summer (in fact, I'm half way through a fan install - one of my many little projects).
Actually, I just discovered a work-around for accessing dropbox from a (new) iOS device. I have a WD backup drive that comes with MyCloud. It can access my Dropbox account and does so via a "browser" interface(!). Now I wonder how smoothly I can access that from my new laptop...
My understanding is that this is only during initial syncing as it has to work out the hashes of all the files as it initially indexes them. To quote from the FAQ where usage on a Pi is specifically mentioned:
Hashing, compression and encryption cost CPU time. Also, using the GUI causes a certain amount of extra CPU usage to calculate the summary data it presents. Note however that once things are in sync CPU usage should be negligible.
I have been contemplating trying it myself, with a Pi 4 so I my get to find out
All sync tools can get very intensive. The larger the number of files in a sync, the harder it gets no matter what tool you use. Often it is best to split the sync into smaller chunks of folders. This is what I do for most of my syncs as I have about a TB of data.
Incidentally, following up on this, I cannot use MyCloud to sync directories on a computer, nor does my MyCould drive keep a copy of the dropbox folder, unless I were to mount the drive and create a hard link to my Dropbox folder. If I did that, WD does provide a sync utility for syncing specified folders on other computers (even outside your local network via mycloud.com), but the software (called GoodSync) requires a minimum subscription of $30/year (and there are bandwidth limits implemented by folder size, so it's not even as large as my drive). I might as well pay for a service that has more functionality if I'm going to do that, because there's no way of sharing files and no IFTTT support.
I am underutilizing MyCloud however. I could be putting personal photos and media there for access anywhere for free. I've been meaning to do this for awhile, but the thing which makes me unenthusiastic about it is the limited browsing interface. I would have to use their app for playing music and searching for photos and that's very limiting. It's not like I can DJ from it or find tagged people in my photos...
