It happens that files should be moved e.g. from DropBox to GoogleDrive or to S3 and other cloud offerings. Of course it’s possible to download the total files from the source and upload them onto the target. Beside some cloud offerings, which get expensive for serious data volume (e.g. when data volumne is above 50 GB) there is a command line tool, which is able to do the same task: rclone
With rclone the task can run on a tiny cloud vm. As outgoing network traffic is expensive at cloud providers it might be a good idea to choose the correct provider, which works together with the selected target service. Sending data from a GCP vm to Google Services is for free and inbound transfer is for free at all providers
On a local machine we setup the rclone process. As a sample here we copy data from DropBox to Google Drive:
We register remotes for Google drive (MyGoogleDrive) as well as Dropbox (MyDropBox). For Google this requires for example the registration of a client id.
Than we test out the transfer on the desktop machine, which has a browser installed. We copy files from MyDropBox to MyGoogleDrive
rclone copyto MyDropBox:Misc MyGoogleDrive:DropBox/MiscWhen we see that this is working we can configure this on a headless console on VM. For Google we create a e2-micro VM (which is for free). Here we install the tool
sudo apt update
sudo apt install rcode screenWe configure new remotes but do not use the auto connection: Remote Setup (rclone.org)
This basically uses the authentication created on the main desktop machine. The command are the same as above. With screen we can use a shell, which continues even without active login.
After all files have been transferred the VM can be deleted again.