Rclone
Learn how to use rclone to store data on the ScPrime distributed datacenter using the XNS Relayer
Overview
rclone is an open source, multi-threaded, command line computer program to manage or migrate content on cloud and other high latency storage. rclone can copy data to S3 protocol destinations, so it makes a good match for XNS storage.
Read this guide for how to configure rclone to store backups with XNS.
Prerequisites
- A working install of rclone
- A working instance of the XNS Relayer
- A configured XNS Relayer’s Endpoint Access Key and Endpoint Secret Key
Considerations
- rclone can work without a valid SSL certificate to use the XNS Relayer as an HTTPS endpoint by using the
--no-check-certificate
Application Version Used in Guide
rclone 1.60.1
Guide Revision Date
2023-03-20
Setup Guide
Step 1: Configure rclone
Run the rclone configuration tool by executing the following command. This command will present a series of prompts. The following documentation will step through these prompts to create a basic configuration of rclone that will work with XNS Relayer.
# rclone config
Step 1.1: Select configuration workflow
Type ‘n’ to initiate a new remote endpoint.
n) New remote
s) Set configuration password
q) Quit config
n/s/q> nStep 1.2: Name for the endpoint
Choose a name for your rclone endpoint.
Enter name for new remote
name> xnsrelayerStep 1.3: Select storage type
Select the option for S3-protocol storage. Many options will appear. The below was snipped for simplicity.
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
1 / 1Fichier
\ (fichier)
[...]
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi
\ (s3)
[...]
50 / seafile
\ (seafile)
Storage> 5Step 1.3: Select S3 provider
Select the option for generic S3-protocol compatible endpoint. Many options will appear. The below was snipped for simplicity
Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Amazon Web Services (AWS) S3
\ (AWS)
[...]
24 / Any other S3 compatible provider
\ (Other)
provider> 24Step 1.4: Select authentication environment
Select the option for entering S3 credentials.
Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
1 / Enter AWS credentials in the next step.
\ (false)
2 / Get AWS credentials from the environment (env vars or IAM).
\ (true)
env_auth> 1Step 1.5: Enter Endpoint Access Key
Enter the Endpoint Access Key from the XNS Relayer.
Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> YourEndpointAccessKeyFromXNSRelayerStep 1.6: Enter Secret Access Key
Enter the Secret Access Key from the XNS Relayer.
Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> YourSecretAccessKeyFromXNSRelayerStep 1.7: Select region
Enter the region option. ‘v4’ signatures must be used with XNS Realyer.
Option region.
Region to connect to.
Leave blank if you are using an S3 clone and you don't have a region.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
/ Use this if unsure.
1 | Will use v4 signatures and an empty region.
\ ()
/ Use this only if v4 signatures don't work.
2 | E.g. pre Jewel/v10 CEPH.
\ (other-v2-signature)
region> 1Step 1.8: Enter endpoint
Enter the address of the XNS Relayer. Make sure that the choice of ‘HTTP’ or ‘HTTPS’ aligns with the XNS Relayer configuration.
Option endpoint.
Endpoint for S3 API.
Required when using an S3 clone.
Enter a value. Press Enter to leave empty.
endpoint> https://10.10.0.11:9000Step 1.9: Select location constraint endpoint
Select the location constraint option. Please note that the below is empty.
Option location_constraint.
Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a value. Press Enter to leave empty.
location_constraint> [EMPTY, press Enter]Step 1.10: Select access control option
Access control lists (ACL) can be created for security. For the purposes of this guide, full control is selected. Many options will appear. The below was snipped for simplicity.
Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
/ Owner gets FULL_CONTROL.
1 | No one else has access rights (default).
\ (private)
/ Owner gets FULL_CONTROL.
[...]
6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
\ (bucket-owner-full-control)
acl> 1Step 1.11: Select whether to edit the advanced configuration
Additional configuration is not required.
Edit advanced config?
y) Yes
n) No (default)
y/n> nStep 1.12: Confirm configuration
Confirm your configuration parameters.
Configuration complete.
Options:
- type: s3
- provider: Other
- access_key_id: YourEndpointAccessKeyFromXNSRelayer
- secret_access_key: YourSecretAccessKeyFromXNSRelayer
- endpoint: http://10.10.0.11:9000
- acl: private
Keep this "xnsrelayer" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> yStep 1.13: Exit the configuration tool
Quit configuration of rclone.
Current remotes:
Name Type
==== ====
xnsrelayer s3
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> qStep 2: Verify rclone configuration is functional
Run the following command.
# rclone lsd --no-check-certificate xnsrelayer:
The buckets on the XNS Relayer will be listed, validating configuration of rclone:
-1 2023-03-20 07:07:13 -1 xnsrclone
2023 SCP Corp