对象存储神器-rclone光速入门

1. 下载&安装

从下面的地址下载对应的二进制版本即可，之后安装在系统的/usr/local/bin目录下,或者其他用户指定的路径即可。二进制:https://rclone.org/downloads/源码:https://github.com/rclone/rclone

2. rclone简介

rclone是一个golang写的开源远程数据同步工具，后端支持文件系统以及多种类型的对象存储产品，是一个跨平台的数据同步工具，相比于s3cmd性能更加强大，同时还支持sync数据同步等功能实现差异数据的跨存储后台同步。并且还提供mount操作，可以直接将远端对象存储挂载到本地文件系统进行访问。官方的Features介绍如下

MD5/SHA-1 hashes checked at all times for file integrityTimestamps preserved on filesPartial syncs supported on a whole file basisCopy mode to just copy new/changed filesSync (one way) mode to make a directory identicalCheck mode to check for file hash equalityCan sync to and from network, e.g. two different cloud accountsOptional large file chunking (Chunker)Optional transparent compression (Compress)Optional encryption (Crypt)Optional FUSE mount (rclone mount)Multi-threaded downloads to local diskCan serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna

3. 初始配置与检查

查看软件版本

╭─user@Mac~╰─$rcloneversion1↵rclonev1.58.0-os/version:darwin10.15.7(64bit)-os/kernel:19.6.0(x86_64)-os/type:darwin-os/arch:amd64-go/version:go1.17.8-go/linking:dynamic-go/tags:cmount

通过命令rclone config file获取默认的配置文件路径，之后在对应的配置文件中填写对应的配置

╭─user@Mac~╰─$rcloneconfigfileConfigurationfileisstoredat:/Users/user/.config/rclone/rclone.conf╭─user@Mac~╰─$vi/Users/user/.config/rclone/rclone.conf╭─user@Mac~╰─$cat/Users/user/.config/rclone/rclone.conf[s3v2]type=s3provider=Cephaccess_key_id=xxxsecret_access_key=sssendpoint=https://s3.demo.comlocation_constraint=force_path_style=false

╭─user@Mac~/SourceCode/rclone‹master›╰─$rcloneconfigshow[s3v2]type=s3provider=Cephaccess_key_id=xxxsecret_access_key=sssendpoint=http://s3.demo.comlocation_constraint=force_path_style=false╭─user@Mac~/SourceCode/rclone‹master›╰─$rcloneconfigdump{"s3v2":{"access_key_id":"xxx","endpoint":"http://s3.demo.com","force_path_style":"false","location_constraint":"","provider":"Ceph","secret_access_key":"sss","type":"s3"}}%

通过rclone listremotes查看已经配置好的远端服务列表

╭─user@Mac~╰─$rclonelistremotess3v2:

通过rclone lsd查看bucket列表或者是远端的目录列表

╭─user@Mac~╰─$rclonelsds3v2:-12022-01-1011:11:52-1demo-storage╭─user@Mac~╰─$rclonelsds3v2:demo-storage02022-04-1110:42:45-1demo-storage02022-04-1110:42:45-1wallpapers-zelda

通过rclone backend features获取远端服务支持的功能特性列表

╭─user@Mac~/SourceCode/rclone‹master*›╰─$rclonebackendfeaturess3v2:{"Name":"s3v2","Root":"","String":"S3root","Precision":1,"Hashes":["md5"],"Features":{"About":false,"BucketBased":true,"BucketBasedRootOK":true,"CanHaveEmptyDirectories":false,"CaseInsensitive":false,"ChangeNotify":false,"CleanUp":true,"Command":true,"Copy":true,"DirCacheFlush":false,"DirMove":false,"Disconnect":false,"DuplicateFiles":false,"GetTier":true,"IsLocal":false,"ListR":true,"MergeDirs":false,"Move":false,"OpenWriterAt":false,"PublicLink":true,"Purge":false,"PutStream":true,"PutUnchecked":false,"ReadMimeType":true,"ServerSideAcrossConfigs":false,"SetTier":true,"SetWrapper":false,"Shutdown":false,"SlowHash":false,"SlowModTime":true,"UnWrap":false,"UserInfo":false,"WrapFs":false,"WriteMimeType":true}}

4. 数据上传下载与遍历

以上传本地bin目录到远端S3为例

╭─user@Mac~/SourceCode/rclone‹master›╰─$lsbinbisect-go-rclone.shcheck-merged.godecrypt_names.pymake_changelog.pynfpm.yamltest-repeat-vfs.shtest_proxy.pyupdate-authors.pybisect-rclone.shconfig.pyget-github-release.gomake_manual.pynot-in-stable.gotest-repeat.shtidy-betaupload-githubbuild-xgo-cgofuse.shcross-compile.gomake_backend_docs.pymake_rc_docs.shtest-all-commits-compile.shtest_independence.gotravis.rclone.confwin-build.bat╭─user@Mac~/SourceCode/rclone‹master›╰─$rclonecopybins3v2:demo-storage/bin-vv2022/04/1314:50:25DEBUG:rclone:Version"v1.58.0"startingwithparameters["rclone""copy""bin""s3v2:demo-storage/bin""-vv"]2022/04/1314:50:25DEBUG:Creatingbackendwithremote"bin"2022/04/1314:50:25DEBUG:Usingconfigfilefrom"/Users/user/.config/rclone/rclone.conf"2022/04/1314:50:25DEBUG:fscache:renamingcacheitem"bin"tobecanonical"/Users/user/SourceCode/rclone/bin"2022/04/1314:50:25DEBUG:Creatingbackendwithremote"s3v2:demo-storage/bin"2022/04/1314:50:25DEBUG:S3bucketdemo-storagepathbin:Waitingforcheckstofinish2022/04/1314:50:25DEBUG:S3bucketdemo-storagepathbin:Waitingfortransferstofinish2022/04/1314:50:25DEBUG:bisect-rclone.sh:md5=5476e61f31e1e2d697ce06a0f0100829OK2022/04/1314:50:25INFO:bisect-rclone.sh:Copied(new)...2022/04/1314:50:26DEBUG:upload-github:md5=a26b267a5c784152a6378457b217c3b9OK2022/04/1314:50:26INFO:upload-github:Copied(new)2022/04/1314:50:26INFO:Transferred:62.573KiB/62.573KiB,100%,0B/s,ETA-Transferred:25/25,100%Elapsedtime:0.8s2022/04/1314:50:26DEBUG:13goroutinesactive╭─user@Mac~/SourceCode/rclone‹master›╰─$rclonelsds3v2:demo-storage02022-04-1314:50:32-1bin02022-04-1314:50:32-1demo02022-04-1314:50:32-1demo-storage02022-04-1314:50:32-1wallpapers-zelda

通过lsd目录查看指定路径下的目录列表

╭─user@Mac~/SourceCode/rclone‹master›╰─$rclonelsds3v2:demo-storage02022-04-1315:02:41-1bin02022-04-1315:02:41-1demo02022-04-1315:02:41-1demo-storage02022-04-1315:02:41-1wallpapers-zelda

通过ls命令查看指定目录下的文件列表

╭─user@Mac~/SourceCode/rclone‹master›╰─$rclonelss3v2:demo-storage/bin253.ignore-emails382bisect-go-rclone.sh902bisect-rclone.sh...370travis.rclone.conf1224update-authors.py1034upload-github364win-build.bat

通过指定max-depth参数来设定ls命令所需遍历的目录层级深度

╭─user@Mac~/SourceCode/rclone‹master›╰─$rclonelss3v2:demo-storage--max-depth1↵382bisect-go-rclone.sh902bisect-rclone.sh142build-xgo-cgofuse.sh...370travis.rclone.conf1224update-authors.py1034upload-github364win-build.bat

同时支持以json方式输出，并输出对应文件的md5值信息。

╭─user@Mac~/SourceCode/rclone‹master›╰─$rclonelsjsons3v2:demo-storage/bin--hash1↵[{"Path":".ignore-emails","Name":".ignore-emails","Size":253,"MimeType":"application/octet-stream","ModTime":"2021-12-16T10:08:33.005446571+08:00","IsDir":false,"Hashes":{"md5":"8155885487321df7790cf63a9db7ba46"},"Tier":"STANDARD"},{"Path":"upload-github","Name":"upload-github","Size":1034,"MimeType":"application/octet-stream","ModTime":"2021-07-08T18:22:34.487717787+08:00","IsDir":false,"Hashes":{"md5":"a26b267a5c784152a6378457b217c3b9"},"Tier":"STANDARD"},{"Path":"win-build.bat","Name":"win-build.bat","Size":364,"MimeType":"application/x-msdownload","ModTime":"2021-07-08T18:22:34.487814575+08:00","IsDir":false,"Hashes":{"md5":"bf695fba05eb1ec8972eac29a59b21b4"},"Tier":"STANDARD"}]

同时支持按文件files-only或者目录dirs-only进行内容筛选

╭─user@Mac~/SourceCode/rclone‹master›╰─$rclonelsjsons3v2:demo-storage--dirs-only1↵[{"Path":"bin","Name":"bin","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T15:29:32.917088000+08:00","IsDir":true},{"Path":"demo","Name":"demo","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T15:29:32.917140000+08:00","IsDir":true},{"Path":"demo-storage","Name":"demo-storage","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T15:29:32.917145000+08:00","IsDir":true},{"Path":"wallpapers-zelda","Name":"wallpapers-zelda","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T15:29:32.917149000+08:00","IsDir":true}]

数据下载

╭─user@Mac/tmp/demo╰─$ls╭─user@Mac/tmp/demo╰─$pwd/tmp/demo╭─user@Mac/tmp/demo╰─$rclonecopys3v2:demo-storage/binbin╭─user@Mac/tmp/demo╰─$lsbinbisect-go-rclone.shcheck-merged.godecrypt_names.pymake_changelog.pynfpm.yamltest-repeat-vfs.shtest_proxy.pyupdate-authors.pybisect-rclone.shconfig.pyget-github-release.gomake_manual.pynot-in-stable.gotest-repeat.shtidy-betaupload-githubbuild-xgo-cgofuse.shcross-compile.gomake_backend_docs.pymake_rc_docs.shtest-all-commits-compile.shtest_independence.gotravis.rclone.confwin-build.bat

5. 使用sync数据同步

将远端数据同步到本地

╭─user@Mac/tmp/demo╰─$rclonesyncs3v2:demo-storage/binbin1↵╭─user@Mac/tmp/demo╰─$lsbin╭─user@Mac/tmp/demo╰─$lsbinbisect-go-rclone.shcheck-merged.godecrypt_names.pymake_changelog.pynfpm.yamltest-repeat-vfs.shtest_proxy.pyupdate-authors.pybisect-rclone.shconfig.pyget-github-release.gomake_manual.pynot-in-stable.gotest-repeat.shtidy-betaupload-githubbuild-xgo-cgofuse.shcross-compile.gomake_backend_docs.pymake_rc_docs.shtest-all-commits-compile.shtest_independence.gotravis.rclone.confwin-build.bat

模拟新建一个文件，同步到远端

╭─user@Mac/tmp/demo╰─$echonewfile1>bin/newfile1╭─user@Mac/tmp/demo╰─$rclonesyncbins3v2:demo-storage/bin╭─user@Mac/tmp/demo╰─$rclonelss3v2:demo-storage/bin253.ignore-emails382bisect-go-rclone.sh...370travis.rclone.conf1224update-authors.py1034upload-github364win-build.bat

模拟修改一个文件，同步到远端，可以看到只同步了差异部分

╭─user@Mac/tmp/demo╰─$echonew123>bin/newfile1╭─user@Mac/tmp/demo╰─$rclonesyncbins3v2:demo-storage/bin-vv2022/04/1315:39:01DEBUG:rclone:Version"v1.58.0"startingwithparameters["rclone""sync""bin""s3v2:demo-storage/bin""-vv"]2022/04/1315:39:01DEBUG:Creatingbackendwithremote"bin"2022/04/1315:39:01DEBUG:Usingconfigfilefrom"/Users/user/.config/rclone/rclone.conf"2022/04/1315:39:01DEBUG:fscache:renamingcacheitem"bin"tobecanonical"/tmp/demo/bin"2022/04/1315:39:01DEBUG:Creatingbackendwithremote"s3v2:demo-storage/bin"2022/04/1315:39:01DEBUG:S3bucketdemo-storagepathbin:Waitingforcheckstofinish...2022/04/1315:39:02DEBUG:upload-github:Unchangedskipping2022/04/1315:39:02DEBUG:win-build.bat:Sizeandmodificationtimethesame(differby0s,withintolerance1ns)2022/04/1315:39:02DEBUG:win-build.bat:Unchangedskipping2022/04/1315:39:02DEBUG:S3bucketdemo-storagepathbin:Waitingfortransferstofinish2022/04/1315:39:02DEBUG:newfile1:md5=f9f9a6b3363a401f566becf83a5fad1bOK2022/04/1315:39:02INFO:newfile1:Copied(replacedexisting)2022/04/1315:39:02DEBUG:Waitingfordeletionstofinish2022/04/1315:39:02INFO:Transferred:7B/7B,100%,0B/s,ETA-Checks:26/26,100%Transferred:1/1,100%Elapsedtime:0.3s2022/04/1315:39:02DEBUG:25goroutinesactive

6. 挂载S3目录到本地文件系统

rclone支持按指定目录作为根目录进行挂载,比如指定bucket名称为demo-storage下的bin目录作为根目录

╭─user@Mac/tmp╰─$rclonemounts3v2:demo-storage/bin~/mnt↵

之后就可以在本地文件系统上访问对应的目录数据了

╭─user@Mac~╰─$cd~/mnt╭─user@Mac~/mnt╰─$lsbisect-go-rclone.shcross-compile.gomake_manual.pytest-all-commits-compile.shtidy-betabisect-rclone.shdecrypt_names.pymake_rc_docs.shtest-repeat-vfs.shtravis.rclone.confbuild-xgo-cgofuse.shget-github-release.gonewfile1test-repeat.shupdate-authors.pycheck-merged.gomake_backend_docs.pynfpm.yamltest_independence.goupload-githubconfig.pymake_changelog.pynot-in-stable.gotest_proxy.pywin-build.bat

7. 其他操作

通过link命令生成文件的访问URL链接，支持通过expire设置URL有效时长。

╭─user@Mac~/mnt╰─$rclonelinks3v2:demo-storage/bin/win-build.bat--expire1h4↵http://demo-storage.s3.demo.com/bin/win-build.bat?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ZK4XIPVNW9PMOADEZMSR%2F20220413%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220413T080748Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b4d60414a5d13662ac7026821e0aa7ffa10f4654142b87ea8629d2ba98b05e8c

通过size命令，获取文件或者目录的大小

─user@Mac~/mnt╰─$rclonesizes3v2:demo-storage/binTotalobjects:26(26)Totalsize:62.580KiB(64082Byte)

通过delete命令，删除对应的文件

╭─user@Mac~/mnt╰─$rclonedeletes3v2:demo-storage/bin/win-build.bat

使用rmdir只能删除空目录，如果目录残留文件，需要使用purge命令删除数据，之后再执行rmdir才能彻底删除目录数据。

╭─user@Mac~/mnt╰─$rclonermdirs3v2:demo-storage/bin-v2022/04/1316:12:25INFO:S3bucketdemo-storagepathbin:Removingdirectory╭─user@Mac~/mnt╰─$rclonelss3v2:demo-storage/bin253.ignore-emails...1034upload-github╭─user@Mac~/mnt╰─$rclonepurges3v2:demo-storage/bin-v2022/04/1316:12:55INFO:build-xgo-cgofuse.sh:Deleted...2022/04/1316:12:55INFO:update-authors.py:Deleted2022/04/1316:12:55INFO:S3bucketdemo-storagepathbin:Removingdirectory╭─user@Mac~/mnt╰─$rclonelss3v2:demo-storage/bin╭─user@Mac~/mnt╰─$rclonermdirs3v2:demo-storage/bin╭─user@Mac~/mnt╰─$rclonelsds3v2:demo-storage02022-04-1316:13:16-1demo02022-04-1316:13:16-1demo-storage02022-04-1316:13:16-1wallpapers-zelda

关键词： 目录列表 配置文件 存储产品