An array of beneficial features can be harnessed with Amazon S3 transfers made using FTP. The FTP S3 combination enhances data availability and improves access without any system limitations. This article deals with a detailed guide describing how to transfer data from FTP to S3 using two alternate solutions.
This article aims at providing you with a step-by-step guide to help you set up the FTP S3 integration with ease to help you transfer your data to Amazon S3 for a fruitful analysis securely. Upon a complete walkthrough of the content, you’ll be able to connect the FTP server to Amazon S3 easily. It will further help you build a customized ETL pipeline for your organization. Through this article, you will get a deep understanding of the tools and techniques & thus, it will help you hone your skills further.
What is FTP [File Transfer Protocol]?
FTP Working Mechanism
FTP is a robust and fast network protocol that allows large file transfers based on a client-server model. Features like block storage, Storage Area Networks, NAS, etc. can be used to optimize file transfer and storage strategies. This protocol, when used with Amazon S3, can greatly improve system performance.
What is Amazon S3?
Amazon’s Simple Storage Service renders a range of comprehensive and simple-to-implement services over the Internet. With a set of dedicated features for the storage, movement, security, configuration, and analytics of data, it is a popular choice among different businesses.
Creating an S3 FTP file transfer mechanism can present several significant benefits. The file storage can be minimized while also increasing the planning administration capacity. Using S3 FTP object storage as a file system you can carry out interactions at the application level but directly mounting S3 is not possible.
What are the 2 Methods to Implement FTP S3 Integration?
Hevo Data, an Automated No-code Data Pipeline, helps you directly transfer data from 150+ sources (including 40+ free sources) like FTP to a data warehouse or a destination of your choice in a completely hassle-free & automated manner. What’s more, the in-built transformation capabilities and the intuitive UI means even non-engineers can set up pipelines and achieve analytics-ready data in minutes.
Get Started with Hevo for Free
Method 1: FTP S3 Integration using S3FS
S3FS – Fuse is a FUSE-based file system that facilitates fully functional file systems, an S3 bucket can be directly mounted as a local filesystem. Read and write access is made available for the same along with fundamental file management commands for manipulation.
Procedure for FTP S3 Integration using S3FS
To install and setup S3FTP and carry out S3 transfer using FTP, you can implement the following steps:
Step 1: Create an S3 Bucket
The first thing to get started is to create an S3 bucket using the AWS console. This will serve as the final location for all files transferred through FTP.
Step A: Create a Bucket
The first thing to get started is to create an S3 bucket.
Image Source
Step B: Configure your Bucket
Set up your bucket. Give it a name and choose the AWS region, as well as access, versioning, and encryption options. After that, your S3 bucket should be ready to use.
Image Source
Step C: Create a New Server
Open AWS Transfer for SFTP and Create a New Server in your AWS interface.
Image Source
Step D: Choose a Protocol
Choose the protocol when configuring your new server (SFTP, FTPS, or FTP).
Image Source
Step 2: Create an IAM Policy and Role for S3 Bucket
Proceed to create an IAM [Identity and Access Management] Policy and Role that can facilitate read/write access to all previous S3 buckets. This can be done by running the following command:
aws iam create-policy
--policy-name S3FS-Policy
--policy-document file://s3fs-policy.json
Replace the bucket name “ca-s3fs-bucket” with the S3 bucket that you will be using within your own environment as shown below:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::ca-s3fs-bucket"]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": ["arn:aws:s3:::ca-s3fs-bucket/*"]
}
]
}
Finally, you need to create the S3FS role and attach the S3FS policy using the AWS console. This can be created using the AWS IAM console.
Step 3: Proceed to Launch FTP Server
In this step, you need to launch an EC2 instance on Amazon Linux 2 for hosting the FTP service.
Run the following command to launch the instance with the attached S3FS role:
aws ec2 run-instances
--image-id ami-0d1000aff9a9bad89
--count 1
--instance-type t3.micro
--iam-instance-profile Name=S3FS-Role
--key-name EC2-KEYNAME-HERE
--security-group-ids SG-ID-HERE
--subnet-id SUBNET-ID-HERE
--associate-public-ip-address
--region us-west-2
--tag-specifications
'ResourceType=instance,Tags=[{Key=Name,Value=s3fs-instance}]'
'ResourceType=volume,Tags=[{Key=Name,Value=s3fs-volume}]'
Within this command, you will use the –associate-public-ip-address parameter to temporarily assign a public IP address to the instance. A production-based environment would use an EIP address and facilitate the launch of the attached S3FS role.
Integrate FTP/SFTP to PostgreSQL
Integrate FTP/SFTP to MS SQL Server
Integrate FTP/SFTP to MySQL
Step 4: Installation and Building of S3FS
Install S3 FTP using the s3ftp.install.sh script. Update local OS packages and install additional packages required for the build and compilation as shown:
sudo yum -y update &&
sudo yum -y install
jq
automake
openssl-devel
git
gcc
libstdc++-devel
gcc-c++
fuse
fuse-devel
curl-devel
libxml2-devel
Use the following code to ensure the complete and correct installation of S3FS:
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse/
./autogen.sh
./configure
make
sudo make install
which s3fs
s3fs --help
Step 5: User Account and Home Directory Configuration
Create a user account for the authentication of the FTP service by using the following command:
sudo adduser ftpuser1
sudo passwd ftpuser1
Consequently, create a home directory that will be configured for use with the created user account. You can use the following command to create the same:
sudo mkdir /home/ftpuser1/ftp
sudo chown nfsnobody:nfsnobody /home/ftpuser1/ftp
sudo chmod a-w /home/ftpuser1/ftp
sudo mkdir /home/ftpuser1/ftp/files
sudo chown ftpuser1:ftpuser1 /home/ftpuser1/ftp/files
Step 6: Installation and Configuration of FTP
The next stage involves the installation and configuration of the actual FTP service. Install the vsftpd package and backup the default configuration as shown:
sudo yum -y install vsftpd
sudo mv /etc/vsftpd/vsftpd.conf /etc/vsftpd/vsftpd.conf.bak
Regenerate the configuration file by running the following script:
sudo -s
EC2_PUBLIC_IP=`curl -s ifconfig.co`
cat > /etc/vsftpd/vsftpd.conf << EOF
anonymous_enable=NO
local_enable=YES
write_enable=YES
local_umask=022
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
xferlog_std_format=YES
chroot_local_user=YES
listen=YES
pam_service_name=vsftpd
tcp_wrappers=YES
user_sub_token=$USER
local_root=/home/$USER/ftp
pasv_min_port=40000
pasv_max_port=50000
pasv_address=$EC2_PUBLIC_IP
userlist_file=/etc/vsftpd.userlist
userlist_enable=YES
userlist_deny=NO
EOF
exit
The following set of vsftp configuration and properties can be observed:
sudo cat /etc/vsftpd/vsftpd.conf
anonymous_enable=NO
local_enable=YES
write_enable=YES
local_umask=022
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
xferlog_std_format=YES
chroot_local_user=YES
listen=YES
pam_service_name=vsftpd
userlist_enable=YES
tcp_wrappers=YES
user_sub_token=$USER
local_root=/home/$USER/ftp
pasv_min_port=40000
pasv_max_port=50000
pasv_address=X.X.X.X
userlist_file=/etc/vsftpd.userlist
userlist_deny=NO
Add the user account to the vsftpd file by implementing echo "ftpuser1" | sudo tee -a /etc/vsftpd.userlist
. You can now initiate the FTP service by running sudo systemctl start vsftpd
.
Check the status of the FTP service and verify the startup by running sudo systemctl status vsftpd
.
The status can be observed as follows:
vsftpd.service - Vsftpd ftp daemon
Loaded: loaded (/usr/lib/systemd/system/vsftpd.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2019-08-13 22:52:06 UTC; 29min ago
Process: 22076 ExecStart=/usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf (code=exited, status=0/SUCCESS)
Main PID: 22077 (vsftpd)
CGroup: /system.slice/vsftpd.service
└─22077 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
Step 7: Run a Test with FTP Client
Finally, run a test with the FTP client to test the FTP service. Start by authentication of the FTP user account using the following script:
ftp 18.236.230.74
Connected to 18.236.230.74.
220 (vsFTPd 3.0.2)
Name (18.236.230.74): ftpuser1
331 Please specify the password.
Password:
230 Login successful.
ftp>
After successful authentication, switch to passive mode before performing the FTP upload and carry out the test as shown:
ftp> passive
Passive mode on.
ftp> cd files
250 Directory successfully changed.
ftp> put mp3data
227 Entering Passive Mode (18,236,230,74,173,131).
150 Ok to send data.
226 Transfer complete.
131968 bytes sent in 0.614 seconds (210 kbytes/s)
ftp>
ftp> ls -la
227 Entering Passive Mode (18,236,230,74,181,149).
150 Here comes the directory listing.
drwxrwxrwx 1 0 0 0 Jan 01 1970 .
dr-xr-xr-x 3 65534 65534 19 Oct 25 20:17 ..
-rw-r--r-- 1 1001 1001 131968 Oct 25 21:59 mp3data
226 Directory send OK.
ftp>
Terminate the FTP session after removing the remote files using ftp> del mp3dataftp> quit
to now proceed with the configuration of the S3FS mount.
Step 8: Initiate S3FS and Mount Directory
Implement the following script which involves bucket specifications and security specifications to initiate and launch S3FS:
EC2METALATEST=http://169.254.169.254/latest &&
EC2METAURL=$EC2METALATEST/meta-data/iam/security-credentials/ &&
EC2ROLE=`curl -s $EC2METAURL` &&
S3BUCKETNAME=ca-s3fs-bucket &&
DOC=`curl -s $EC2METALATEST/dynamic/instance-identity/document` &&
REGION=`jq -r .region <<< $DOC`
echo "EC2ROLE: $EC2ROLE"
echo "REGION: $REGION"
sudo /usr/local/bin/s3fs $S3BUCKETNAME
-o use_cache=/tmp,iam_role="$EC2ROLE",allow_other /home/ftpuser1/ftp/files
-o url="https://s3-$REGION.amazonaws.com"
-o nonempty
Now proceed with the debug process of the S3FS FUSE mounting with the following script if required:
sudo /usr/local/bin/s3fs ca-s3fs-bucket
-o use_cache=/tmp,iam_role="$EC2ROLE",allow_other /home/ftpuser1/ftp/files
-o dbglevel=info -f
-o curldbg
-o url="https://s3-$REGION.amazonaws.com"
-o nonempty
Check the status of the S3FS process and ensure that it’s running as required. This can be done as shown:
ps -ef | grep s3fs
root 12740 1 0 20:43 ? 00:00:00 /usr/local/bin/s3fs
ca-s3fs-bucket -o use_cache=/tmp,iam_role=S3FS-Role,allow_other
/home/ftpuser1/ftp/files -o url=https://s3-us-west-2.amazonaws.com
Once verified, your system is good to go for trying out an end-to-end transfer.
Step 9: Run S3 FTPS to Perform File Transfer
Set up FTPS and proceed to carry out your file transfer. You can test the connection using clients like FileZilla. Once corroborated, you can check the configuration of your AWS S3 web console. Initiate FTP transfers from your user directory automatically to the configured Amazon S3 bucket. Thus, your connection can facilitate the upload and synchronization using FTP to Amazon S3.
Limitations of FTP S3 Integration using S3FS
- Requires significant time investment and technical knowledge to implement.
- Potential errors with manual implementation could lead to data loss.
- The lack of an update process could lead to incompatibility in long run.
Method 2: FTP S3 Integration using FTP/SFTP Clients
Using an FTP/SFTP Client that supports Amazon S3 is the final best approach to connect FTP/SFTP to Amazon S3. FileZilla, WinSCP, and Cyberduck are some instances of these clients. You will not need to configure anything on the server if the FTP client is also an S3 client.
Using FileZilla, an open-source platform
Use FileZilla, which is a free, open-source, and cross-platform FTP server/client. FTP, SFTP, and FTPS are supported by the FileZilla client, although only FTP and FTPS are supported by the server. With the Pro edition, FileZilla additionally supports AWS S3 (among other cloud storage), making it easy to FTP to Amazon S3.
You can set the host and utilize S3 as a protocol when configuring the FileZilla client.
Image Source
- It’s important to remember that you’re configuring a client, not a server. As a result, you’ll be able to view but not alter the data.
- Outside of public networks, FTP is not advised for file transfers. For security concerns, you’ll need to set up an FTP server for S3 inside a VPC (Virtual Private Cloud) or through a VPN. If you want to access files over the Internet, use FTPS (for FileZilla).
How does Amazon S3 Transfer files?
The finest example of an object storage service is Amazon’s AWS S3 (Amazon Simple Storage Service). A single object in S3 might be anywhere from a few Kilobytes to a few Terabytes in size. Objects are categorized and stored in AWS S3 using a “buckets” structure.
S3 is a simple web services interface that allows you to store and retrieve any amount of data from anywhere at any time. The secure web-based protocol HTTPS and a REST Application Programming Interface (API), also known as RESTful API, are used to access S3.
With S3’s simple drag-and-drop tool, you may upload files, directories, or data.
Image Source
Storage Units: FTP vs S3 Buckets
FTP and SFTP were created for transferring files, while Amazon S3 buckets were created for storing things. Although both are used for remotely sharing and storing data, they operate in different ways.
Files are transactional-based units for file-based storage. FTP and SFTP (SSH File Transfer Protocol) are file transfer protocols that access data in storage at the file level. They have access to files that are organized in a hierarchy of folders and files. Objects are another form of storage unit that includes the data, its associated (expandable and customizable) metadata, and a graphical user interface (GUI) (Globally Unique Identifier). A file, a subfile, or a collection of unconnected bits and bytes are all examples of objects. Objects are used in Amazon S3 buckets.
Image Source
Blocks are a different sort of file format that is utilized in structured Database Storage and Virtual Machine File System (VMFS) volumes. The scope of this guide does not include blocks.
When it comes to storing large volumes of unstructured data, object storage surpasses file storage. Things are easy to access and deliver high stream throughput regardless of the amount of storage.
Cloud-based object storage, such as Amazon S3 buckets, scales significantly better than SFTP or FTP server storage in general.
Conclusion
This article teaches you how to set up the FTP S3 Integration with ease. It provides in-depth knowledge about the concepts behind every step to help you understand and implement them efficiently. These methods, however, can be challenging, especially for a beginner.
What if you want to move your data from FTP to any other data warehouse? You can breathe a sigh of relief because Hevo Data is here to help.
Visit our Website to Explore Hevo
Want to take Hevo for a spin? Sign Up for a 14-day free trial and experience the feature-rich Hevo suite first hand. You can also have a look at our unbeatable pricing that will help you choose the right plan for your business needs!
Share your experience of learning about the FTP S3 Integration! Let us know in the comments section below!
Sai Rao, a Customer Experience Engineer with two years of expertise, is dedicated to effective problem resolution and consistently delivering tangible results. His commitment to ensuring client satisfaction is reflected in his proven track record of success.
Aman Deep Sharma is a data enthusiast with a flair for writing. He holds a B.Tech degree in Information Technology, and his expertise lies in making data analysis approachable and valuable for everyone, from beginners to seasoned professionals. Aman finds joy in breaking down complex topics related to data engineering and integration to help data practitioners solve their day-to-day problems.