It is the easiest way to see if the architecture really does what it claims.
In the case of a native Google Drive backup, I want to verify a few things:
If you want the general explanation first, you can read Google Drive Is Not a Backup (Here’s How to Fix That) and Why Sync Is Not Backup, Technically.
This article is only about the logs.
Before reproducing the debug runs below, make sure you have:
gdrive-changes source--debug output enabled on a test setup rather than on production data you do not want to inspect liveHere is a representative cloudstic init --debug run:
$ cloudstic init --debug
[store #1] GET config 2074.6ms err=NoSuchKey
[store #2] LIST keys/ 99.4ms
[store #3] PUT keys/kms-platform-default 119.8ms 311B
[store #4] PUT config 123.6ms 63B
Created new encryption key slots.
Repository initialized (encrypted: true).
This is short, but it already tells a lot.
GET config confirms that the repository does not exist yet.
LIST keys/ checks the existing key slots.
PUT keys/... creates the encryption slot.
PUT config writes the repository marker.
So before backing up any file, the repository has an identity and key material.
Now the first Google Drive backup:
$ cloudstic backup -source gdrive-changes --debug
[store #8] GET index/snapshots 101.0ms err=NoSuchKey
[hamt] get node/... hit staging (158 bytes)
...
Scanning ... done! [20 in 790ms]
[store #14] PUT chunk/d2667... 807.7ms 1.2MB
[store #15] PUT chunk/3134f... 261.2ms 587.8KB
...
Uploading ... done! [45.65MB in 5.995s]
[store #51] PUT packs/d7596... 191.9ms 1.2MB
Backup complete. Snapshot: snapshot/6f70aa...
The main points are simple.
GET index/snapshots returns NoSuchKey, so there is no previous snapshot.
This means the source has to perform a full scan. This is expected, even with gdrive-changes.
PUT chunk/... means new file data is uploaded.
PUT packs/... means Cloudstic is bundling small metadata objects into packfiles instead of writing thousands of tiny objects individually.
That point is important for object stores like S3. It reduces API calls and makes metadata reads cheaper on later runs.
After this first backup, Cloudstic has more than just stored bytes.
It has created a real snapshot containing:
This is what makes the second backup interesting.
Here is a representative second run with no changes:
$ cloudstic backup -source gdrive-changes --debug
[store #8] GET index/snapshots 115.8ms 350B
[store #10] GET packs/d7596... 729.8ms 1.2MB
Scanning (incremental) ... done! [0 in 212ms]
...
Added to the repository: 286 B (315 B compressed)
Processed 0 entries in 1s
Snapshot 3eb699... saved
This is the part I care about most.
Scanning (incremental) ... done! [0 in 212ms]
This means Cloudstic reused the previous snapshot, read the stored change token, asked Google Drive for the delta, and found nothing to process.
So it did not walk the whole tree again.
This is the practical difference between a native integration and a mounted drive scanned like a local filesystem.
GET packs/... Is Good NewsOn the second run, there is still a metadata read:
[store #10] GET packs/d7596... 729.8ms 1.2MB
This is normal.
The engine still needs metadata to reason about the previous state.
The good sign is that it fetches a packfile, not hundreds of tiny metadata objects.
This is exactly why packfiles exist.
This line is also expected:
Added to the repository: 286 B (315 B compressed)
Even if the files did not change, the backup still produces a new checkpoint.
So Cloudstic writes a very small amount of metadata for the new snapshot and the updated incremental state.
That is what you want: a new restore point without re-uploading the source.
For me, the logs validate four things:
This is why the second backup can complete in about a second.
It is not doing the same work again. It is using the source’s own incremental mechanism and reusing the existing structure.
That is also why I find --debug useful. It lets you see the difference between a real native backup flow and a tool that only looks native from the outside.
If you want to try this yourself, start with the Cloudstic installation guide.
]]>The problem starts when you want to keep running it.
Before creating stores, auth entries, or profiles, make sure you have:
At the beginning, many people do something like this:
cloudstic backup \
-store s3:my-backups/cloudstic \
-password "correct horse battery staple" \
-source local:~/Documents
This works.
It is also how you end up with secrets in shell history, slightly different commands on each machine, and an automation story that becomes fragile very quickly.
This is why I added profiles.
For a backup workflow, the command itself should be short.
The complexity should live in configuration that you define once.
Profiles let you separate three things:
This is not about adding abstraction for the sake of abstraction.
It is about making the workflow repeatable.
Let us say you want:
~/Documents~/PicturesThe end result should look like this:
cloudstic backup -profile documents
cloudstic backup -profile pictures
cloudstic backup -profile work-drive
Or, if you want everything:
cloudstic backup -all-profiles
That is the practical value.
First define the destination.
For example, an S3-backed store:
cloudstic store new \
-name home-s3 \
-uri s3:my-backup-bucket/cloudstic \
-s3-region eu-west-1
At this point, the destination has a name.
You no longer need to repeat the bucket path and region in every command.
Configuration alone does not create the repository.
You still need to initialize it once:
cloudstic store init home-s3
This creates the encrypted repository and its key slots.
The important distinction is that repository encryption and runtime secret resolution are two different concerns.
The repository protects the backup data.
Profiles tell the CLI how to obtain the credentials needed to operate.
Now define what should be backed up.
For documents:
cloudstic profile new \
-name documents \
-source local:~/Documents \
-store-ref home-s3
For pictures:
cloudstic profile new \
-name pictures \
-source local:~/Pictures \
-store-ref home-s3
At this stage, daily commands already become simpler:
cloudstic backup -profile documents
cloudstic backup -profile pictures
The same idea also applies to cloud sources.
For example, define a Google auth entry once:
cloudstic auth new \
-name google-work \
-provider google \
-google-token-file ~/.config/cloudstic/tokens/google-work.json
Then authenticate once:
cloudstic auth login -name google-work
Now create a profile that uses it:
cloudstic profile new \
-name work-drive \
-source "gdrive-changes:/" \
-store-ref home-s3 \
-auth-ref google-work
The resulting command stays short:
cloudstic backup -profile work-drive
This is the part that matters most operationally.
Do not store raw secrets directly in profiles.yaml.
Store references instead.
The two practical patterns are:
env://VAR_NAMEkeychain://service/account on macOSFor example:
stores:
home-s3:
uri: s3:my-backup-bucket/cloudstic
s3_region: eu-west-1
s3_access_key_secret: env://AWS_ACCESS_KEY_ID
s3_secret_key_secret: keychain://cloudstic/home-s3/s3-secret-key
password_secret: keychain://cloudstic/home-s3/repo-password
This keeps the configuration file shareable while the actual secret values stay outside it.
On a server or in CI, environment variables are usually the easiest option.
On a personal macOS machine, Keychain is often more convenient.
With one store, one auth entry, and three profiles, the configuration stays readable:
stores:
home-s3:
uri: s3:my-backup-bucket/cloudstic
s3_region: eu-west-1
s3_access_key_secret: env://AWS_ACCESS_KEY_ID
s3_secret_key_secret: keychain://cloudstic/home-s3/s3-secret-key
password_secret: keychain://cloudstic/home-s3/repo-password
auth:
google-work:
provider: google
google_token_file: ~/.config/cloudstic/tokens/google-work.json
profiles:
documents:
source: local:~/Documents
store_ref: home-s3
pictures:
source: local:~/Pictures
store_ref: home-s3
work-drive:
source: gdrive-changes:/
store_ref: home-s3
auth_ref: google-work
This is the main idea:
Before trusting the setup, inspect it.
List profiles:
cloudstic profile list
Show one profile:
cloudstic profile show documents
Verify the store:
cloudstic store verify home-s3
This is especially useful after rotating credentials or moving to a new machine.
Once the setup exists, the real workflow becomes boring, which is exactly what I want from a backup system.
Run one profile:
cloudstic backup -profile documents
Run everything:
cloudstic backup -all-profiles
Inspect snapshots:
cloudstic list
Restore what you need:
cloudstic restore <snapshot-hash> -output ./restore.zip
For me, profiles are not a cosmetic CLI feature.
They are the point where backup commands stop being one-off experiments and become an actual workflow.
The benefit is simple:
If the command you run every day still contains every flag, every secret, and every path inline, the setup is not finished yet.
To get started, see the installation guide. For the complete command reference, see the Cloudstic CLI documentation. The CLI is also open source on GitHub.
]]>There is also a second problem that appears as soon as you try to back it up seriously: the drive moves.
On one machine it is mounted as /Volumes/WorkDrive.
On another one it becomes /media/alice/WorkDrive.
On Windows it may become E:\.
For a human, this is obviously the same drive, for many backup tools, it is not.
Many backup tools identify a source using some combination of:
This is acceptable for internal disks.
It is not acceptable for portable drives.
If the identity changes every time the drive moves to another machine, incremental history breaks. Even when deduplication prevents re-uploading all data, the tool still has to rebuild state as if it had discovered a new source.
In practice, this means more scanning, more metadata, and a fragmented backup history.
For a portable drive, the identity should come from the drive itself.
On modern drives, the right identifier already exists: the GPT partition UUID.
This UUID is:
This is the identifier I wanted Cloudstic to use for portable drives.
There are several good backup tools on the market. The issue here is not quality in general. The issue is portable-drive identity.
rsync is a very useful file copy tool.
It can be used to build snapshot-like workflows with --link-dest, but it does not have a real notion of encrypted repository, snapshot identity, or portable drive UUID.
For this use case, it requires scripting and discipline.
Time Machine is tied to macOS and works per machine.
If the same portable drive is backed up from two different Macs, you effectively get two different histories.
Restic and Borg are serious backup tools, and I like both.
However, their default source identity model is still centered on the machine and the path.
You can work around this with manual overrides such as --host, but then the correctness of the history depends on the user remembering to keep the override identical everywhere.
This works, but I do not consider it a satisfying solution for a portable drive.
Cloudstic treats a portable local source differently.
When a volume UUID is available, it takes precedence over hostname and path for snapshot matching.
This means that if you:
Cloudstic can continue the same snapshot lineage automatically.
This is the behavior I expected from the start.
The path inside the snapshot is stored relative to the volume root, so changing the mount point does not create an artificial identity change.
Suppose you have a 100 GB external SSD.
You back it up from machine A.
The next day you modify only 200 MB of files and plug the same drive into machine B.
The correct outcome is simple: the next backup should process the same source lineage and upload only the delta.
That is exactly what a UUID-based identity gives.
Without it, you still get deduplication benefits, but you lose the continuity of the source history.
Before running the examples below, make sure you have:
For example on local storage:
cloudstic init \
-store local:/Volumes/BackupDrive/cloudstic \
-recovery
Or on object storage, for example S3-compatible storage:
export CLOUDSTIC_STORE=s3:my-backup-bucket
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
cloudstic init -recovery
cloudstic backup \
-source local:/Volumes/WorkDrive \
-exclude ".Spotlight-V100/" \
-exclude ".fseventsd/" \
-exclude ".Trashes/" \
-exclude ".DS_Store" \
-tag work-drive
At this point Cloudstic detects the volume UUID and stores it in the snapshot metadata.
cloudstic backup \
-source local:/media/alice/WorkDrive \
-exclude ".Spotlight-V100/" \
-exclude ".fseventsd/" \
-exclude ".Trashes/" \
-tag work-drive
Same drive, different hostname, different mount point.
Cloudstic matches the volume UUID and continues the same incremental chain.
On macOS and Linux this UUID is detected automatically. When auto-detection is unavailable, you can still provide -volume-uuid manually.
This matters for retention as well.
If snapshots are grouped by volume UUID, a retention policy applies to the drive as one source, not once per machine.
So keeping 7 daily snapshots means 7 daily snapshots for the portable drive, not 7 on the Mac side and 7 on the Linux side.
This is a small detail, but it keeps history much cleaner.
Portable drives move between machines.
If the backup tool identifies them by hostname or mount point, it will eventually treat the same drive as multiple sources.
For me, this is the wrong model.
The identity should come from the drive itself.
Using the GPT partition UUID solves that problem and makes cross-machine incremental backups behave as expected.
That is the approach implemented in Cloudstic for portable local sources.
If you want to get started, see the installation guide. If you want the full command reference, see the Cloudstic CLI documentation.
]]>In this article, we look at the state of the art techniques used by modern storage engines and why the next generation of backup tech is moving toward a more flexible, flat architecture.
The biggest leap in modern backup architecture is abandoning files and folders at the storage layer. Instead, industry leaders use Content Addressable Storage (CAS). In a traditional system, data is found by its path. In CAS, data is found by its cryptographic hash such as SHA-256.
graph LR
subgraph "Traditional: Location-Based"
P1["/docs/report.pdf"] -->|path lookup| D1["Data Block A"]
P2["/backup/report.pdf"] -->|path lookup| D2["Data Block A (duplicate)"]
end
subgraph "CAS: Content-Based"
F1["report.pdf"] -->|SHA-256| H1["a8f5e..."]
F2["report_copy.pdf"] -->|SHA-256| H1
H1 -->|hash lookup| D3["Data Block A (stored once)"]
end
The CAS logic is simple. Same content leads to the same hash which is stored once. This is the foundation of deduplication. If you rename a file or move it to a different folder, a CAS based system realizes the content has not changed and uploads exactly zero bytes.
Standard CAS has a privacy hole. If a cloud provider knows the hash of a specific file, they can see if your bucket contains that hash even if they cannot read the encrypted content.
To solve this, one solution is to avoid using raw hashes for chunk names. Instead, we can use Keyed HMAC-SHA256. By using a secret key known only to you to salt the hashes, your data remains fully deduplicated but becomes cryptographically opaque to the host.
What if you have a 10GB database and change one single row?
Slicing it at fixed 1MB intervals is a trap. If you insert one byte at the very beginning, every 1MB boundary shifts. This changes the hash of every block.
Modern systems use algorithms like FastCDC. It rolls a sliding window across the data. When the data matches a specific mathematical pattern, it slices the chunk.
graph TD
subgraph "Fixed-Size Chunking"
FS["Original File: AABBCCDDEE"]
FS --> C1["AA | BB | CC | DD | EE"]
INS1["Insert 'X' at start: XAABBCCDDEE"]
INS1 --> C2["XA | AB | BC | CD | DE"]
C2 -.- NOTE1["⚠ Every chunk changed!"]
end
subgraph "Content-Defined Chunking (CDC)"
CDC["Original File: AABBCCDDEE"]
CDC --> C3["AAB | BCC | DDEE"]
INS2["Insert 'X' at start: XAABBCCDDEE"]
INS2 --> C4["XAAB | BCC | DDEE"]
C4 -.- NOTE2["✓ Only first chunk changed"]
end
Because the boundaries are determined by the content itself, an insertion at the start only changes the first chunk. The rest of the puzzle pieces realign perfectly.
Once you have thousands of content addressed chunks, you must remember which chunks belong to which files at a specific point in time. The industry standard is the Merkle Tree. This is a hierarchical structure where every node is the hash of its children. This enables Structural Sharing.
graph TD
subgraph "Snapshot 1"
R1["Root Hash: abc123"] --> A1["Dir A: def456"]
R1 --> B1["Dir B: 789abc"]
A1 --> F1["file1.txt: aaa"]
A1 --> F2["file2.txt: bbb"]
B1 --> F3["file3.txt: ccc"]
end
subgraph "Snapshot 2 (file2.txt changed)"
R2["Root Hash: xyz789"] --> A2["Dir A: new111"]
R2 -.->|"shared"| B1
A2 -.->|"shared"| F1
A2 --> F2new["file2.txt: ddd (new)"]
end
style B1 fill:#90EE90
style F1 fill:#90EE90
style F3 fill:#90EE90
style F2new fill:#FFB347
When you take a new backup:
The result is that every snapshot acts like a full backup for recovery, but it only consumes the storage space of an incremental update.
Most modern tools build their Merkle Trees by mirroring the computer’s natural directory structure. While intuitive, this path based approach creates two major headaches at scale.
Path Amplification: If you have a file buried ten folders deep, changing that one file forces the system to recalculate and reupload metadata for all ten parent directories.
The Rigid Hierarchy Problem: Modern data does not always live in a neat tree. Cloud drives like Google Drive allow a single file to have multiple parents, and flat data streams like database dumps do not have folders at all. Standard tools struggle to map these non tree structures efficiently, which often leads to metadata bloat.
I have been looking into these industry standards while building Cloudstic, a project aimed at solving these specific bottlenecks. To achieve both zero trust privacy and high performance, I realized the engine needed to be built from the ground up.
Cloudstic is a content addressable, encrypted CLI backup tool designed to be fully stateless. Unlike many traditional tools, it can back up from Google Drive and OneDrive using incremental APIs without ever needing to mount the drive.
Core Features:
If you want to try Cloudstic, start with the installation guide. For the full command reference and configuration options, see the Cloudstic CLI documentation.
]]>This distinction matters because many people use Google Drive as the place where they put important data and assume the problem is solved.
In practice, Google Drive protects you well against infrastructure failures on Google’s side. It does not protect you well against logical failures on your side.
For example:
In all these cases, sync can work exactly as designed and you can still lose access to your data.
The main difference is history.
With a real backup, you can go back to a previous point in time and restore the state that existed before the problem.
Without that, you only have the current state of the synchronized drive and whatever limited recovery features the provider exposes.
This is why file versioning is useful but not sufficient. In a real incident, you do not want to inspect documents one by one. You want to restore a coherent state.
If you want to protect Google Drive properly, there are a few possible approaches.
Google Takeout allows you to export your data.
This is useful as an export mechanism, but I do not consider it a practical backup strategy. It is manual, slow, and difficult to run frequently enough.
Another option is to mount Google Drive locally with a tool such as rclone and then use a traditional backup tool on top of that mount.
This works, but it is not ideal.
The backup tool sees a filesystem. Google Drive is not really a filesystem. It is an API-backed data source with its own semantics and its own incremental mechanisms.
As a consequence, this approach often ends up rescanning large trees, creating latency and local caching overhead even when very little changed.
The third option is to back up Google Drive natively through the provider API.
This is the approach I wanted.
Instead of pretending Google Drive is a local disk, the backup tool can use the change tracking exposed by Google itself and create real snapshots in an external repository.
I started working on Cloudstic because I did not find a solution that matched what I wanted.
I wanted a tool able to:
Cloudstic uses the Google Drive Changes API through the gdrive-changes source.
This means the first backup performs a full scan, but the following ones can request only what changed since the previous snapshot.
That is the main reason this approach is much faster than scanning a mounted drive again and again.
If you want the technical explanation behind this, I wrote a companion articles: Reading a Native Backup Through –debug.
Using a native backup flow, the result is straightforward:
This is, for me, the main value of a backup tool. It should give a predictable restore path.
This is not only for large companies or security teams.
It is useful for anyone who stores important documents in Google Drive, especially:
The more operational value your Google Drive contains, the less reasonable it is to treat sync as backup.
Before running the commands below, make sure you have:
Here is the minimal setup with Cloudstic.
If Cloudstic is not installed yet, start with the installation guide.
brew install cloudstic/tap/cloudstic
For example on S3-compatible storage:
export CLOUDSTIC_STORE=s3:my-backup-bucket
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
cloudstic init -add-recovery-key
This creates the repository and generates a recovery key.
On first use, Cloudstic opens a browser window for OAuth authentication and stores the token locally for later runs.
cloudstic backup -source gdrive-changes -tag cloud
The first run performs a full scan.
The following runs are incremental.
Google Drive is very useful.
I use it as a source of data, not as a backup destination.
If the data matters, I want snapshots, an external repository, and a restore path that does not depend on the current state of the synced account.
That is why I think backing up from Google Drive is just as important as backing up to it.
Cloudstic is open source on GitHub. If you want to get started, see the installation guide, and for the full command reference see the Cloudstic CLI documentation.
]]>Stay tuned for upcoming posts on cloud architecture, infrastructure as code, and modern engineering practices.
If you want to try Cloudstic, start with the installation guide.
]]>