A neat trick for transferring multiple small files quickly over the network
Transferring multiple small files over a network will always be slow due to overhead, but this trick will make the transfer a breeze.
Have you ever tried to copy thousands of small files to your NAS, to another computer in your network, copy to an FTP server or upload to the cloud?
It doesn't matter how fast your home network is, or how good your Internet connection is. You can have an insanely fast SSD in your system, and you will still face a long time before every single file is transferred.
Not only that, but if your connection breaks during this transfer you have to either start all over again, if you don't know how to skip the already transferred files.
On the other hand, when you transfer large files, the transfer rate is most likely to reach the maximum your network is capable of.
Why are network transfers slow?
Without getting too technical, this happens because of the overhead in communication when transferring a single file, multiplied by the number of files you are transferring.
A few dozen milliseconds for establishing the connection and starting the transfer may not mean much when transferring a single file, but it will quickly add up when you transfer thousands of files.
If you think this is bad, you haven't experienced a slow HDD back in the day, with fragmented data. That made network transfers even worse and made quite a noise (look up "HDD seeking sound" on YouTube to get an idea of how brutal this was).
That is the time when I started looking for a better solution, one that would speed up network transfers significantly.
The solution: archive your files first
I'm not going to show you how to do this because it all depends on what archiving program you are using, if any. Windows 11 now supports a lot of popular archive formats, but I still prefer to use 7-ZIP because it offers more flexibility.
One of the advantages that I'm looking specifically for this trick is to set the compression level to No Compression. This means the files are packed together inside the archive without any gains in final archive file size.
This makes little sense when it comes to typical archive usage, which is to save space, but in our case, it will speed up compression and decompression since there will be little demand on the CPU, who is the one doing all the calculations.
If you want a simplification of this solutions here are the steps:
Select the files and folders you wish to transfer.
Compress them into a single archive.
Transfer the archive to the destination.
Decompress the archive.
(optional) Test the archive to ensure data integrity.
Advantages:
You only need to transfer one big file.
Archives can be tested after the transfer to ensure data was not corrupted.
Disadvantages:
If you have limited disk space, you may not be able to create an archive or decompress at destination.
The destination platform must allow decompression of archives.
How fast is the single archive transfer method? It will all depend of course on the total size and the number of files; I wouldn't be surprised if you see 10x or even 100x faster transfers.
Previous articles you may have missed
Extracting multiple images from a Word document is slow and annoying, but there is a way to copy all images at the same time. There are actually two quick methods for doing this.