Boosting File Download Performance using aria2c

Recently, I faced a file transfer bottleneck while downloading GitLab artifacts from an on-premises GitLab server to an Oracle Cloud VM over an IPsec tunnel.

The challenge

Single-threaded downloads via curl were capped at just 1.5 MBps, far below the available bandwidth. By switching to the powerful aria2c tool, I dramatically improved the download speed and effectively utilized the network bandwidth. Here’s how you can achieve similar results.

The Problem: Single-Threaded Downloads with curl

While curl is a fantastic and versatile tool for HTTP downloads, it operates in a single-threaded manner by default. This limitation can lead to poor performance in high-latency or bandwidth-constrained environments like IPsec tunnels. In my case:

• Setup: GitLab artifacts stored on-premises located in US region, downloaded to Oracle Cloud VM located at Franfurt region over an IPsec tunnel.
• Observation: The single-threaded curl downloads peaked at 1.5 MBps, leaving most of the bandwidth underutilized.

The Solution: Parallel Downloads with aria2c

Enter aria2c, a lightweight, multi-threaded download utility that supports segmented downloads. Unlike curl, aria2c can split a file into multiple chunks and download them concurrently, significantly improving throughput in environments with high latency or limited per-thread bandwidth.

What is aria2c?

aria2c is an advanced download tool that supports:

• Multiple connections for a single file.
• HTTP, HTTPS, FTP, SFTP, BitTorrent, and Metalink protocols.
• Configurable connection limits and retries.

Steps to Use aria2c

Here’s how I used aria2c to resolve the download bottleneck:

1. Install aria2c on a Oracle Linux VM:

# dnf localinstall -y \
   https://yum.oracle.com/repo/OracleLinux/OL9/developer/EPEL/x86_64/getPackage/aria2-1.36.0-4.el9.x86_64.rpm

2. Command for Parallel Downloads

To download a file in multiple parts:

aria2c -x 16 -s 16 -o artifact.tar.gz http://<gitlab-server>/path/to/artifact

• -x 16: Sets the maximum number of connections per server.
• -s 16: Splits the file into 16 parts for concurrent downloading.
• -o artifact.tar.gz: Specifies the output file name.

3. Monitor the Download Speed

aria2c provides a real-time display of download progress, including the speed for each connection. This helped confirm that the bandwidth was now fully utilized.

Using curl

# curl -O https://$url/$file_path_uri
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  2  559M    2 15.3M    0     0  1329k      0  0:07:11  0:00:11  0:07:00 1432k
### Note that speed, it was varying between few Kb/s to 1.5MBps

With multiple connections using aria2c

# aria2c -x16 -s16 https://$url/$file_path_uri
[#ff3c51 210MiB/559MiB(37%) CN:16 DL:22MiB ETA:15s]
[#ff3c51 559MiB/559MiB(99%) CN:2 DL:21MiB]
11/17 16:07:20 [NOTICE] Download complete: tesfile.tar
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
ff3c51|OK  |    21MiB/s|./tesfile.tar
Status Legend:
(OK):download completed.

Why aria2c Outperforms curl

1. Multiple Connections: By splitting the file into chunks, aria2c establishes multiple simultaneous HTTP connections, effectively bypassing single-threaded limitations.
2. Bandwidth Utilization: Each connection utilizes a portion of the bandwidth, collectively maximizing throughput.
3. Optimized for Latency: In high-latency environments like IPsec tunnels, parallelism reduces the impact of delays.

Results

After switching to aria2c, the download speed increased significantly, leveraging the full potential of the IPsec tunnel. Here’s a comparison:

ToolSpeedConnections Usedcurl1.5 MBps1aria2c~15 MBps16

Conclusion

If you’re facing download bottlenecks in Oracle Cloud or any environment with network latency or bandwidth limitations, aria2c is a game-changer. It’s easy to set up, significantly boosts performance, and is highly customizable. Whether you’re downloading files from GitLab, S3, or any HTTP server, aria2c is a must-have tool in your networking arsenal.

Pro Tips

• Experiment with different -x and -s values to find the optimal number of connections for your environment.
• Use aria2c with a configuration file for even more advanced settings like retries, timeouts, and logging.
• Always monitor the server-side impact when using many connections to avoid overloading the source.

By leveraging tools like aria2c, you can ensure smooth and efficient file transfers, even in challenging network conditions.