You started importing a large archive before leaving for the day and came back to a crash dialog, or a laptop that had gone to sleep, or an update that restarted the machine. With most tools the only option is to start again from message one. mailin 2.0 checkpoints an import after every batch, so a crash costs you at most 200 messages of rework instead of hours. This post explains how that works and why the design is built around file identity rather than file names.
Why an interrupted email import usually means starting over
An import is a long sequence of small steps: read a message, parse it, store it, repeat. If the process is treated as one unit of work, an interruption at any point throws away everything before it. For a few hundred messages that is annoying. For a Gmail Takeout measured in tens of gigabytes, where an import can run for a long time, it means a single crash can cost you the whole session, and a second crash the next one.
The fix is not to make crashes impossible. Power fails, disks hiccup, and operating systems restart themselves. The fix is to make sure progress survives them.
Step one: give the file an identity
When you drop a file into mailin 2.0, the first thing it does is compute a SHA-256 hash of the file. That hash, not the filename or the path, is how mailin knows which file it is looking at from then on.
Doing this once is enough. Re-dropping the same file does not re-hash it, so returning to an interrupted import does not begin with another full pass over a huge file. The hash is also what makes everything that follows trustworthy: a checkpoint is only meaningful if you can be certain it refers to exactly the bytes you are now reading.
Step two: checkpoint after every batch
mailin's streaming ingest reads messages in batches of 200. After each batch is written to storage, a checkpoint is recorded against the file's hash: this file, this many batches, done. Then the next batch starts.
If the app crashes, or the Mac loses power, the checkpoint from the last completed batch is still on disk. When you drop the file again, mailin recognizes it, finds the checkpoint, and resumes from the batch after it. The messages in the batch that was in flight when the crash happened are processed again. That is the entire cost: rework is capped at one batch of 200 messages, no matter how far along the import was.
To put a number on it, an import that crashes 800 GB into a file resumes from the last batch, not from byte zero.
Why checkpoints are bound to identity, not names
Tying a checkpoint to a filename would be simpler, and wrong in both directions.
Suppose you copied the Takeout mbox to an external drive and renamed it before the retry. By name, it is a new file. By content, it is byte-for-byte the same, and its SHA-256 says so. Because the checkpoint is keyed to the content hash, mailin picks up where it left off.
Now suppose the reverse: a new export from the same account, saved with the same name as last month's. By name it is the same file. By content it is different, and resuming from an old checkpoint would skip messages that were never imported, or process the wrong ones. Because the identity is the hash, a different file gets a fresh start, and no checkpoint is ever applied to the wrong bytes.
What you see while it runs
A progress indicator tracks large imports. When the import finishes, the import report lists what came through and what did not. Messages that could not be parsed, including any single mbox message over the 100 MB ceiling, are counted as damaged and categorized rather than crashing the run.
One kind of interruption is deliberate. If the disk returns a read error partway through the file, mailin stops and reports the byte offset where it happened rather than pretending the import succeeded. That is not a crash to resume from; it is the file telling you something is wrong with the copy you have, and the offset tells you where to look. The post on very large archives goes into the rest of the streaming design.
A few practical notes
- Google Takeout delivers a ZIP. mailin 2.0 does not auto-extract ZIP files, so unzip first and import the mbox inside. See the Takeout guide for the export steps.
- You can drop several files at once; the format of each is detected automatically and each file is hashed and checkpointed on its own.
- Resumable ingest arrived with version 2.0. Existing users get it as a free in-place upgrade with their data intact, under the same interface.
- Resuming is about the file you are importing. Copies of the same message that already exist across two different archives are a separate job for the Duplicate Manager.
FAQ
How much work is lost when an import crashes?
At most one batch of 200 messages. mailin resumes from the last checkpoint and reprocesses only the batch that was in progress.
Does resuming re-hash the whole file?
No. The per-file SHA-256 is computed when the file is first dropped, and re-dropping the same file does not repeat it, so a retry does not start with another full pass over the file.
What if I renamed or moved the file before retrying?
The checkpoint is tied to the file's content hash, not its name or location, so a renamed or moved copy with identical content resumes from the same checkpoint.