Automating the Download of CSV Files from Azure File Share to a Local Drive

In this article, we’ll walk through the process of automatically downloading a CSV file from an Azure File Share to a local drive. This solution can be used to automate the retrieval of files stored in Azure for various purposes, such as local processing, backups, or archiving.

Prerequisites

Before starting, ensure you have the following:

1. Azure Storage Account: A storage account with a file share where the CSV file is stored.
2. Azure Storage Account Key: The account key to authenticate and access the file share.
3. Windows PowerShell: Installed on your local machine or server.
4. Windows Task Scheduler: To automate the PowerShell script execution.

Step 1: Setting Up the Azure File Share

If you don’t already have an Azure File Share set up, follow these steps to create one:

1. Log in to the Azure Portal.
2. Navigate to your Storage Account.
3. Under Data storage, select File Shares.
4. Click + File share to create a new file share, provide a name, and click Create.
5. Upload the CSV file you want to retrieve to this file share.
6. Note the Connection String, File Share Name, and Storage Account Key as they will be needed in the PowerShell script.

Step 2: PowerShell Script to Download CSV File from Azure File Share

The following PowerShell script downloads a CSV file from an Azure File Share and saves it to a specified local directory.

PowerShell Script

# Azure Storage Information
$storageAccountName = "yourstorageaccount"
$storageAccountKey = "yourstorageaccountkey"
$fileShareName = "yourfileshare"
$localPath = "C:\Downloaded_CSV_Files\"
$csvFileName = "yourfile.csv"
$logFile = "C:\Downloaded_CSV_Files\LogFile.txt"
# Function to log messages
function Log-Message {
    param (
        [string]$message
    )
    $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
    $logEntry = "$timestamp - $message"
    Add-Content -Path $logFile -Value $logEntry
}
# Install Azure PowerShell module if not already installed
# Install-Module -Name Az -AllowClobber -Force
# Start logging
Log-Message "Starting the CSV download process..."
try {
    # Import Az module
    Import-Module Az
    # Create a context for accessing the Azure Storage
    $context = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey
    # Get reference to the Azure File Share
    $share = Get-AzStorageFileShare -Context $context -Name $fileShareName
    # Get reference to the root directory
    $rootDir = $share.CloudFileShare.GetRootDirectoryReference()
    # Download the CSV file from Azure File Share to local directory
    $file = $rootDir.GetFileReference($csvFileName)
    $localFilePath = Join-Path $localPath $csvFileName
    # Check if the local directory exists, if not, create it
    if (!(Test-Path $localPath)) {
        New-Item -Path $localPath -ItemType Directory
        Log-Message "Created directory $localPath"
    }
    # Download the file
    $file.DownloadToFile($localFilePath, [System.IO.FileMode]::Create)
    Log-Message "CSV file downloaded to $localFilePath"
} catch {
    # Log any errors that occur during the process
    Log-Message "An error occurred: $_"
}
Log-Message "CSV download process completed."

Script Breakdown:

• Azure Storage Information: Update the values of $storageAccountName, $storageAccountKey, $fileShareName, and $csvFileName with your specific Azure File Share details.
• Logging: The Log-Message function is used to log events (like successful downloads or errors) into a log file (C:\Downloaded_CSV_Files\LogFile.txt).
• Error Handling: If anything goes wrong during the download process, the error will be logged.

How to Use the Script:

1. Copy the script into a .ps1 file (e.g., DownloadCSV.ps1).
2. Update the Azure Storage Information variables in the script with your details.
3. Run the script to test if the file is successfully downloaded to your specified local directory.

Step 3: Automate the PowerShell Script Using Windows Task Scheduler

To automate the file download process, we will schedule the PowerShell script to run at regular intervals using Windows Task Scheduler.

3.1 Open Task Scheduler

1. Press Windows + R, type Taskschd.msc, and press Enter.
2. In the Task Scheduler window, click Create Basic Task in the right panel.

3.2 Create the Task

1. Name the task (e.g., “Download CSV from Azure File Share”).
2. Description: Provide a description (e.g., “Automates the download of CSV files from Azure File Share to local directory”).
3. Click Next.

3.3 Set the Trigger

1. Select how often you want the task to run (e.g., Daily, Weekly, or Monthly).
2. Set the Start date and time for the first run.
3. Adjust the frequency (e.g., every day, every week).
4. Click Next.

3.4 Set the Action

1. Select Start a program.
2. In the Program/script field, enter: powershell.exe
3. In the Add arguments field, specify the following:

-ExecutionPolicy Bypass -File "C:\Downloaded_CSV_Files\DownloadCSV.ps1"

• -ExecutionPolicy Bypass: This ensures that the PowerShell script runs without needing to change the system’s execution policy.
• -File: Path to the PowerShell script.

4. Click Next.

3.5 Finish the Task

1. Review the settings.
2. Check the box Open the Properties dialog for this task when I click Finish.
3. Click Finish.

3.6 Set Advanced Properties (Optional)

1. In the General tab of the task properties, check Run whether the user is logged on or not.
2. In the Settings tab, you can configure:

• Allow task to be run on demand.
• If the task fails, restart every X minutes (this is useful for fault tolerance).
• Stop the task if it runs longer than X hours (useful if the task hangs).
• Click OK.

Step 4: Verifying the Task

After the task is scheduled, you can monitor its execution:

• Task Scheduler History: Check Task Scheduler’s history to see if the task ran successfully.
• Log File: Review the log file (C:\Downloaded_CSV_Files\LogFile.txt) to verify if the file was downloaded successfully.
• Local Directory: Confirm that the CSV file is saved to the specified local path (e.g., C:\Downloaded_CSV_Files\).

Step 5: Automating Cleanup (Optional)

If you download files regularly, you may want to implement a cleanup routine to remove old files after a certain time period. You can add this step to the script.

For example, to delete files older than 30 days, you can add this to the PowerShell script:

# Cleanup old files (older than 30 days)
Get-ChildItem $localPath -Filter *.csv | Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-30) } | Remove-Item
Log-Message "Old files cleaned up from $localPath"

Conclusion

By following this guide, you can automate the process of downloading CSV files from an Azure File Share to a local directory using PowerShell and Windows Task Scheduler. This setup ensures that files are regularly updated and available locally for further processing, archiving, or backups, without requiring manual intervention. You can extend this solution by adding additional logic for file processing, archiving, or data manipulation, depending on your specific use case.