rclone
Sat, Aug 27, 2016 · 4 minute readlinux
Stumbled across rclone today; a command line program to sync files and directories to and from a wide variety of online storage systems including:
* OneDrive
* Backblaze B2
* Hubic
* Amazon Drive
Worth noting that OneDrive support does not seem to include OneDrive for Business (i.e. Office365).
Currently looking for a work round maybe using oauth2.
Install
Per the installation guide
lm180 ~ # wget http://downloads.rclone.org/rclone-current-linux-amd64.zip
--2016-08-27 12:55:02-- http://downloads.rclone.org/rclone-current-linux-amd64.zip
Resolving downloads.rclone.org (downloads.rclone.org)... 77.73.5.145
Connecting to downloads.rclone.org (downloads.rclone.org)|77.73.5.145|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3662290 (3.5M) [application/zip]
Saving to: ‘rclone-current-linux-amd64.zip’
rclone-current-linux-amd64.zip 100%[=====================================================================================================>] 3.49M 489KB/s in 8.9s
2016-08-27 12:55:12 (400 KB/s) - ‘rclone-current-linux-amd64.zip’ saved [3662290/3662290]
lm180 ~ # ls -l
total 3580
-rw-r--r-- 1 root root 3662290 Aug 24 23:16 rclone-current-linux-amd64.zip
lm180 ~ # unzip rclone-current-linux-amd64.zip
Archive: rclone-current-linux-amd64.zip
creating: rclone-v1.33-linux-amd64/
inflating: rclone-v1.33-linux-amd64/README.html
inflating: rclone-v1.33-linux-amd64/rclone
inflating: rclone-v1.33-linux-amd64/rclone.1
inflating: rclone-v1.33-linux-amd64/README.txt
lm180 ~ # cd rclone-v1.33-linux-amd64/
lm180 rclone-v1.33-linux-amd64 # cp rclone /usr/sbin/
lm180 rclone-v1.33-linux-amd64 # chown root:root /usr/sbin/rclone
lm180 rclone-v1.33-linux-amd64 # chmod 755 /usr/sbin/rclone
lm180 rclone-v1.33-linux-amd64 # mkdir -p /usr/local/share/man/man1
lm180 rclone-v1.33-linux-amd64 # cp rclone.1 /usr/local/share/man/man1/
lm180 rclone-v1.33-linux-amd64 # mandb
...
Checking for stray cats under /usr/local/man...
Checking for stray cats under /var/cache/man/oldlocal...
1 man subdirectory contained newer manual pages.
1 manual page was added.
0 stray cats were added.
32 old database entries were purged.Configure
This example assumes you have a Backblaze B2 account.
Also worth considering Hubic from OVH; 10TB for 50Euro/year.
Possibly worth trying Amazon Drive - but their service is rate limited, observed during an upload:
2016/08/27 18:09:15 fonts/glyphicons-halflings-regular.svg: Error detected after finished upload - waiting to see if object was uploaded correctly: HTTP code 429: "429 Too Many Requests", reponse body: {"logref":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","message":"Rate exceeded","code":""}Backblaze B2 configuration
alan@lm180 ~ $ rclone config
2016/08/27 13:27:12 Failed to load config file "/home/alan/.rclone.conf" - using defaults: open /home/alan/.rclone.conf: no such file or directory
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> backblaze
Type of storage to configure.
Choose a number from below, or type in your own value
1 / Amazon Drive
\ "amazon cloud drive"
2 / Amazon S3 (also Dreamhost, Ceph, Minio)
\ "s3"
3 / Backblaze B2
\ "b2"
4 / Dropbox
\ "dropbox"
5 / Encrypt/Decrypt a remote
\ "crypt"
6 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
7 / Google Drive
\ "drive"
8 / Hubic
\ "hubic"
9 / Local Disk
\ "local"
10 / Microsoft OneDrive
\ "onedrive"
11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
12 / Yandex Disk
\ "yandex"
Storage> 3
Account ID
account> xxxxxxxxxxxx
Application Key
key> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Endpoint for the service - leave blank normally.
endpoint>
Remote config
--------------------
[backblaze]
account = xxxxxxxxxxxx
key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
endpoint =
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
backblaze b2
e) Edit existing remote
n) New remote
d) Delete remote
s) Set configuration password
q) Quit config
e/n/d/s/q> qTest the configuration by running ls command.
alan@lm180 ~ $ rclone ls backblaze:/
2016/08/27 13:38:00
Transferred: 0 Bytes (0 Bytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 1.8srclone commands
All files in B2 are stored in buckets, create a bucket (BespokeITSolutions in these examples) online, then go ahead and create whatever directories you may want.
alan@lm180 ~ $ rclone mkdir backblaze:BespokeITSolutions/sites
2016/08/27 14:02:57
Transferred: 0 Bytes (0 Bytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 2sSync
Sync keeps source and destination directories in sync, deleting files only on the destination.
alan@lm180 ~ $ rclone sync ~/sites/bespoke-it.solutions backblaze:BespokeITSolutions/sites
2016/08/27 14:09:47 B2 bucket BespokeITSolutions path sites/: Waiting for checks to finish
2016/08/27 14:09:47 B2 bucket BespokeITSolutions path sites/: Waiting for transfers to finish
2016/08/27 14:09:49
Transferred: 2.347 MBytes (38.822 kBytes/s)
Errors: 0
Checks: 0
Transferred: 83
Elapsed time: 1m1.9s
Transferring:
* font-awesome/less/core.less
* font-awesome/less/icons.less
* font-awesome/less/variables.less
* font-awesome/scss/_variables.scss
2016/08/27 14:09:50 Waiting for deletions to finish
2016/08/27 14:09:50
Transferred: 2.347 MBytes (37.862 kBytes/s)
Errors: 0
Checks: 0
Transferred: 88
Elapsed time: 1m3.4s
alan@lm180 ~ $ rclone sync ~/sites backblaze:BespokeITSolutions/sites
2016/08/27 14:13:28 B2 bucket BespokeITSolutions path sites/: Waiting for checks to finish
2016/08/27 14:13:28 B2 bucket BespokeITSolutions path sites/: Waiting for transfers to finish
2016/08/27 14:13:28 Waiting for deletions to finish
2016/08/27 14:13:28
Transferred: 0 Bytes (0 Bytes/s)
Errors: 0
Checks: 88
Transferred: 0
Elapsed time: 3.3s