Amazon Web ServicesCloud

AWS Transfer familyを使ってみた

Posted Cola🌀

AWS Transfer Family とは

ストレージサービスとの間でファイルを送受信できるフルマネージド型の転送サービスで、
現在はs3とEFSが対象として使用できます。
また、アクセス方法はFTP、FTPS、SFTPの3つのプロトコルで使用できます。

なぜfamilyという名称がついているかというと
元々、AWS Transfer for SFTP というサービスがあり
2020年にAWS Transfer for FTPとAWS Transfer for FTPSが提供されて
総称してAWS Transfer Familyというようになりました。

  • It can be used to transfer files into and out of S3 and EFS
  • SFTP, FTPS, and FTP protocols are supported

假如我们通过 AWS Transfer Family模拟SFTP服务器送信，有下面的情况，一定要要分清楚。

  • 第一种是我们系统A做为SFTP服务器，系统B来读取文件。这样的话，对与系统A来说，直接可以把文件放到Transfer Family对应的S3目录下，也就是系统A不需要接续Transfer Family，直接往s3 upload文件即可，但是系统B需要通过SFTP协议来接续Transfer Family 来从系统A取得文件。

第二种情况是系统B作为SFTP服务器，我们从系统A往系统B送信，这样的话，需要接续系统B的 Transfer Family uri，来发送文件。

所以一定要分清，是谁做SFTP服务器，文件存在哪一端。

How do we use it?

Demo

Visit https://ap-northeast-1.console.aws.amazon.com/transfer/home?region=ap-northeast-1#/
→ Click Create server

Set up configurations

Choose protocols

We can select one of the following options

  • SFTP (SSH File Transfer Protocol)
  • FTPS (File Transfer Protocol Secure)
  • FTP (File Transfer Protocol)
Check SFTP (SSH File Transfer Protocol) - file transfer over Secure Shell and click Next

Choose an identity provider

We can select one of the following protocols

  • Service managed
  • Custom
Check Service managed and click Next

Choose an endpoint

We need to configure endpoint type and custom hostname. We have the following options for an endpoint.

  • Publicly accessible
  • VPC hosted
Check Publicly accessible and click Next

Choose a domain

We can use S3 and EFS as the destination or starting point of transfer of files. 

Click Amazon S3 and click Next.

Configure additional details

We can attach an existing role or create a new role to allow the transfer family to put logs in CloudWatch Logs.

Choose to Create a new role and click Next

Review and create

Click Create server below.

Check on console

Add a service managed user

Click Server ID of the server that you want to add a user to. Click Add user.

User configuration

Attach to the role AWSTransferLoggingAccess
Input Username transfer-family-test, choose the role AWSTransferLoggingAccess. Then, choose S3 bucket where the files are transferred and
Generate SSH key

> ssh-keygen -P "" -m PEM -f ~/.ssh/transfer-family-test
Generating public/private rsa key pair.
Your identification has been saved in /Users/Kanata-Miyahana/.ssh/transfer-family-test.
Your public key has been saved in /Users/Kanata-Miyahana/.ssh/transfer-family-test.pub.
The key fingerprint is:
SHA256:zsBLQFnZn7PL+Q8ioX68yge7Nqo2lpCAgE4VcvDzM5A kanata-miyahana@ip-192-168-3-2.ap-northeast-1.compute.internal
The key's randomart image is:
+---[RSA 3072]----+
|.oo+oo.o         |
|o.+.o . .        |
|=  E.    . .     |
|o.  +o    +      |
|..   ++ S  o     |
|o    .+* ..      |
| . .  o++..o.    |
|  =  o+ +.+. .   |
| o.o.o=*.. ....  |
+----[SHA256]-----+

> cat ~/.ssh/transfer-family-test.pub | pbcopy

enter the public SSH key and click Add.

※ You can follow the instruction here to create an SSH public key.

Confirm that the user is successfully added

Install client application

Install Cyberduck

> brew install cyberduck --cask
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 3 taps (homebrew/cask-versions, homebrew/core and homebrew/cask).
==> New Formulae
snowpack
==> Updated Formulae
Updated 28 formulae.
==> New Casks
devbook                                 goldenpassport                          netnewswire-beta
==> Updated Casks
Updated 26 casks.

==> Downloading https://update.cyberduck.io/Cyberduck-7.8.2.34203.zip
==> Downloading from https://a48823c7ec3cf4539564-60c534a1284a12ce74ef84032e9b4e46.ssl.cf1.rackcdn.com/Cyberduck-7.8.2.
######################################################################## 100.0%
==> Installing Cask cyberduck
==> Moving App 'Cyberduck.app' to '/Applications/Cyberduck.app'
🍺  cyberduck was successfully installed!

Set up configurations

If Cyberduck is successfully installed, open the application. Click + button.

Set the following configuration

  • SFTP
  • endpoint → you can get it on AWS console

Test connection

Check that there is no file on S3
> aws s3 ls s3://transfer-family-test-miyahana-local
# There is nothing because we have never transferred any files

Open directory where the file you want to transfer locates, drag, and drop the file. You can copy the file from local to S3!!!
Check S3
> aws s3 ls s3://transfer-family-test-miyahana-local/
2021-03-09 00:04:58         12 test.txt

> aws s3 cp s3://transfer-family-test-miyahana-local/test.txt .
download: s3://transfer-family-test-miyahana-local/test.txt to ./test.txt

> cat test.txt
hello world