This article delves into how to upload large files to Azure Blob Storage with ease using Azure’s robust storage services. When developing an application, the initial step is to determine the appropriate storage solution for data retention.
Azure Storage offers diverse solutions for various storage needs, ensuring scalability, availability, and durability of data within Azure’s ecosystem. Comprising services like Blobs, Files, Queues, among others, Azure Storage allows interaction through various programming languages such as .NET, Java, Golang, JavaScript, facilitated by the SDKs available.
In this section, we explore Azure Blob Storage, utilized for housing unstructured data like text and binary files. Tailored for tasks like streaming audio and video, managing log files, and housing backups and images, Azure Blob Storage offers a versatile solution for diverse storage needs.
Prerequisites
You can upload large files to Azure Blob Storage by hiring .NET developers for your project. If you prefer to do it yourself, Vivasoft is here to help.
Before you can upload large files to Azure Blob Storage, it’s important to ensure that you have all the necessary prerequisites in place. Understanding these prerequisites will help streamline the upload process, ensuring a smooth and efficient experience.
In this guide, we’ll cover everything you need to get started with uploading large files to Azure Blob Storage. You will need the following to upload large files to Azure Blob Storage:
- Visual Studio, Visual Studio Code
- .NET 8 SDK
- Node.js (any version) click here to download
- An Azure Subscription. If you don’t have any subscription. You can sign up for a free azure account
- Azure Storage Explorer Download the desktop app
Overview
Upload large files to Azure Blob Storage with this step-by-step guide designed to simplify the process. Azure Blob Storage is an ideal solution for storing and managing vast amounts of unstructured data, such as text and binary files.
Whether you’re dealing with streaming media, managing log files, or storing backups and images, this guide will walk you through the entire process. From setting up your storage account to committing the uploaded blocks, you’ll learn how to ensure efficient and reliable uploads, making the most of Azure’s scalable and durable storage solutions.
Follow along to seamlessly integrate Azure Blob Storage into your application for handling large file uploads.
Initially, accessing our Azure account is essential. Next, we must create a resource to configure the Azure storage account. Follow the subsequent steps below:
Step 1: Find a resource group in your marketplace
Step 2: Create a resource and open the resource group
Here is my resource group window:
Step 3 : Find a storage account in your marketplace.
Step 4 : Next, proceed to establish a storage account. Utilize the red box guide as necessary.
Please wait a few seconds for the validation and review process to finalize. Once everything is correctly added, proceed by clicking on the “Create” button.
Resource creation will require some time, thus please be patient and wait for the deployment process to complete.
How to Upload Large Files to Azure Blob Storage with .NET
Our storage account has now been successfully created. Next, we’ll proceed with the step-by-step process of uploading files. Initially, we need to create a .NET application.
Step 1: Create a new web API project.
Step 2: Configure the new project.
Step 3: Provide additional information.
Step 4: Create an API controller.
Step 5: Install necessary NuGet Package.
Now the solution look like this:
Step 6: This code snippet represents an ASP.NET Core Web API controller named AzureFileController responsible for generating Shared Access Signature (SAS) tokens for Azure Blob Storage containers. And here the GetSasToken method is an HTTP GET endpoint within an ASP.NET Core Web API controller. Here’s a brief explanation of its functionality:
- Endpoint: The method is accessible via HTTP GET requests at the route “api/AzureFile/GetSasToken”.
- Parameters: It accepts a single query parameter named “container” from the request URL. This parameter specifies the name of the Azure Blob Storage container for which the SAS token will be generated.
- SAS Token Generation: Inside the method, a SAS token is generated using the provided container name, along with a start time and expiry time set to one minute ago and one day ahead, respectively. The permissions are set to grant all access permissions to the container.
- Response: The generated SAS token is returned as the content of an HTTP 200 OK response, allowing clients to obtain the SAS token for the specified container.
[Route("api/[controller]")]
[ApiController]
public class AzureFileController : ControllerBase
{
public AzureFileController(){}
[HttpGet("GetSasToken", Name = "GetSasToken")]
public IActionResult GetSasToken([FromQuery] string container)
{
DateTimeOffset startTime = DateTimeOffset.UtcNow.AddMinutes(-1);
DateTimeOffset expiryTime = DateTimeOffset.UtcNow.AddDays(1);
var storageSharedKeyCredential = new StorageSharedKeyCredential("my_account_name", "my_account_key");
var sasBuilder = new BlobSasBuilder()
{
BlobContainerName = container,
Resource = "c"
};
sasBuilder.StartsOn = startTime;
sasBuilder.ExpiresOn = expiryTime;
sasBuilder.SetPermissions(BlobContainerSasPermissions.All);
string sasTokenGenerated = sasBuilder.ToSasQueryParameters(storageSharedKeyCredential).ToString();
return Ok(sasTokenGenerated);
}
}
Step 7: Access the Azure Storage Explorer. If it’s not yet installed, you can find the installation link provided above. Then create a container. “mycontainer” is a container name for the test. Let’s follow some instructions.
- Sign in to Azure Portal: Go to https://portal.azure.com/, sign in with your Azure account.
- Create Storage Account: Search for “Storage accounts” in the Azure portal, select “Storage accounts – Create”, and follow the prompts to create a new Storage account.
- Navigate to Storage Account: After creating the Storage account, select it from the list of resources.
- Access Containers: In the Storage account, find the “Containers” section in the left menu and click on it.
- Create Container: Click on “+ Container” or “Add”, provide a unique name, and configure settings like Public access level and Encryption.
In my case, my storage explorer showed like the following:
Step 8: Next, execute the application and send a request to the GetSasToken endpoint, specifying your container name.
How to Upload Large Files to Azure Blob Storage with Vue.js
Let’s learn how to upload large files to Azure Blob Storage with Vue.js. Begin by creating a client application using Vue + Vite with typescript. First, install Node.js, then follow the steps to create the client application and demonstrate the file uploading UI
Step 1: Create a Vue.js + Vite app
Step 2: Select the Vue and press enter.
Step 3: Select Typescript and press enter.
Step 4: Your file-upload-client app is ready now. Let’s run the following commands:
>> cd file-upload-client
>> npm install
>> npm run dev
After following the instructions, ensure that everything is okay.
Go to the browser
Let’s now create a function for uploading files. We’ll outline it with strategies to ensure smooth performance. See below for details:
i) Define the chunk-size and process the blob chunk-by-chunk
ii) Determine the number of blocks by considering the blob size and chunk size
iii) Generate a unique identifier for each block
iv) Maintain a list before uploading a blob
v) Upload blocks using separate URLs
vi) Then commit the blocks using a separate URL. Subsequently, the blob is uploaded to Azure Storage
For better comprehension, let’s provide an example:
When dealing with large blob files exceeding 2 MB, they will be divided into chunks. For example, if we have a 50 MB file, it will be split into 25 chunks. The process of uploading large files to Azure takes place chunk by chunk, employing a blocklist approach, followed by a single final commit. Let’s illustrate this process with an example.
The remaining blocks will be triggered in the event of any unsuccessful block due to network interruption or any other issue. The block list requested by the endpoint is as follows, and some headers are required with it
Request Header:
Key | Value |
---|---|
x-ms-blob-type | BlockBlob |
Content-Length | {get_file_length} |
Endpoint:
Then the final commit request by endpoint looks like below and some header and body are required.
Request Header:
Key | Value |
---|---|
x-ms-blob-content-type | {get_file_content_type} |
Content-Length | {get_file_length} |
Request Body:
<?xml version=”1.0″ encoding=”utf-8″?>
<BlockList>${Your_BlockIds.map( (id) => `<Latest>${id}</Latest>`).join(“”)}</BlockList>
Endpoint:
After uploading the file successfully then before reloading the page keep track record of file status to the database. Now follow the actual code with some steps. Now in this client vue.js we have to do some steps before uploading.
Step 1: These TypeScript interfaces define the structure for managing file uploads in a web application:
- FileUpload: Represents the collection of file items to be uploaded, each containing metadata and blocks.
- FileItem: Represents an individual file item with properties like FileId, File (the actual file), UploadPath, Name, FileSize, FileItemBlocks (array of blocks), BlockIds (array of block IDs), NumberOfBlocks, and UploadStatus (status of upload).
- FileItemBlock: Represents a block of a file item with properties like Serial (order of the block), FileId, BlockId, BlockStart (start position of block in file), BlockEnd (end position of block in file), ContentSize (size of block content), and UploadStatus.
- FileItemRequest: Represents a request to upload a file item or its block, with properties like UploadPath, FileItem, FileItemBlock, and IsCommitBlock (indicating whether it’s a commit operation).
- RequestModel: Represents a collection of FileItemRequest objects and a flag indicating whether it’s a commit operation.
- UploadStatus: Enum defining various states of file upload, such as NotStarted, Uploading, Uploaded, and FailedToUpload.
export interface FileUpload {
FileItems: FileItem[];
}
export interface FileItem {
FileId: number;
File: File;
UploadPath: string;
Name: string;
FileSize: number;
FileItemBlocks: FileItemBlock[];
BlockIds: string[];
NumberOfBlocks: number;
UploadStatus: UploadStatus;
}
export interface FileItemBlock {
Serial: number;
FileId: number;
BlockId: string;
BlockStart: number;
BlockEnd: number;
ContentSize: number;
UploadStatus: UploadStatus;
}
export interface FileItemRequest{
UploadPath:string;
FileItem:FileItem;
FileItemBlock:FileItemBlock;
IsCommitBlock:boolean;
}
export interface RequestModel{
Requests:FileItemRequest[];
IsCommit:boolean;
}
export enum UploadStatus {
NotStarted = 1,
Uplaoding = 2,
Uploaded = 3,
FailedToUpload = 4
}
Step 2: In your vue component Establish the required return variable to handle file upload interfaces. We’ll place it within the ‘data’. And the data function initializes two objects:
- handleUpload: Contains properties related to file uploading, including progress, flags for upload progress and upload bar visibility, uploadedBytes, totalFileSize, and requestList (an array of RequestModel objects) for managing upload requests. It also contains an empty FileUpload object.
- handleFile: Contains properties related to file handling, including an array of files and an array of file paths.
data() {
return {
handleUpload: {
progress: "0%",
isUploadProgress: false,
isUploadBar: false,
uploadedBytes: 0,
totalFileSize: 0,
requestList: [] as RequestModel[],
fileUpload: {} as FileUpload,
},
handleFile: {
files: [] as File[],
filePaths:[] as string[],
}
};
}
Step 3: Implement essential shared methods for generating unique block IDs and displaying text indicating upload progress. These are utility functions in TypeScript:
- getPercentProgress: Calculates the percentage completion based on the partial size and total size of a task. It returns the progress percentage as a string formatted to display a whole number followed by a percentage sign. If the progress is 100% or more, it returns “100%” directly.
- pad: Converts a number to a string and pads it with leading zeros until it reaches the specified length. It returns the padded string.
getPercentProgress(partialSize:number,totalSize:number){
const percentComplete = ((partialSize / totalSize) *100).toFixed(0);
if(Number(percentComplete)>=100) return `100%`;
return `${percentComplete}%`;
},
pad(number: number, length: number) {
let str = "" + number;
while (str.length < length) {
str = "0" + str;
}
return str;
},
Step 4: This uploadBlocks function is responsible for uploading a block of data asynchronously to Azure Blob Storage. Here’s how it works:
- It constructs the URL for uploading the block by appending the SAS token, block component, and block ID
- It prepares headers for the HTTP request, including the content type and length
- It slices the file data into a chunk based on the block start and end positions
- It converts the chunk data into an array buffer
- It sends a PUT request to the Azure Blob Storage endpoint with the chunk data as the request body
- If the response indicates success, it updates the upload progress indicators and the status of the uploaded block
- If an error occurs during the upload, it handles network-related errors and retries the upload if necessary
async uploadBlocks(sasUrl: string,fileItem: FileItem,fileItemBlock: FileItemBlock) {
const url = `${sasUrl}&comp=block&blockid=${fileItemBlock.BlockId}`;
const headers = new Headers();
const chunk = fileItem.File.slice(
fileItemBlock.BlockStart,
fileItemBlock.BlockEnd
);
const requestData = await chunk.arrayBuffer();
headers.append("x-ms-blob-type", "BlockBlob");
headers.append("Content-Length", requestData.byteLength.toString());
const payload = { method: "PUT", headers: headers, body: requestData};
try {
const response = await fetch(url, { ...payload });
if (!response.ok) {
console.error(
`Error uploading chunk starting from ${fileItemBlock.BlockStart}: ${response.statusText}`
);
throw new Error(
`Error uploading chunk starting from ${fileItemBlock.BlockStart}: ${response.statusText}`
);
}
console.log(
`${fileItem.File.name}-(block-${fileItemBlock.Serial}) uploaded successfully`
);
this.handleUpload.isUploadProgress = true;
this.handleUpload.uploadedBytes += requestData.byteLength;
this.handleUpload.progress = this.getPercentProgress(this.handleUpload.uploadedBytes,this.handleUpload.totalFileSize);
fileItemBlock.UploadStatus = UploadStatus.Uploaded;
return response;
} catch (error: any) {
if (
error instanceof TypeError ||
error instanceof DOMException ||
!navigator.onLine
) {
// Check for network-related errors (TypeError or DOMException)
console.error(
`Error blocking Internet connection Retrying...`,
error
);
}
}
},
Step 5: Create the commitBlocks function that is responsible for committing the uploaded blocks to Azure Blob Storage to finalize the upload process. Here’s how it works:
- It constructs the URL for committing the blocks by appending the SAS token and the blocklist component
- It prepares headers for the HTTP request, including the blob content type and the content length
- It constructs the request body in XML format, specifying the list of block IDs to be committed
- It sends a PUT request to the Azure Blob Storage endpoint with the blocklist XML as the request body
- If the response indicates success, it updates the upload progress indicators and sets the upload status of the file item to “Uploaded”
- If an error occurs during the commit process, it handles network-related errors and retries the operation if necessary
async commitBlocks(sasUrl: string, fileItem: FileItem) {
const url = `${sasUrl}&comp=blocklist`;
const headers = new Headers();
const requestBody = `${fileItem.BlockIds.map(
(id) => `${id} `
).join("")} `;
headers.append("x-ms-blob-content-type", `${fileItem.File.type}`);
headers.append("Content-Length", `${requestBody.length}`);
const payload = { method: "PUT", headers: headers,body: requestBody};
try {
const response = await fetch(url, { ...payload });
if (!response.ok) {
throw new Error(`${response.statusText}`);
}
console.log(`${fileItem.File.name}>>commit uploaded successfully`);
this.handleUpload.isUploadProgress = true;
fileItem.UploadStatus = UploadStatus.Uploaded;
return response;
} catch (error: any) {
if (
error instanceof TypeError ||
error instanceof DOMException ||
!navigator.onLine
) {
console.error(
`Error blocking Internet connection Retrying...`,
error
);
}
}
},
Step 6: Let’s create the onPromisesUpload method that is responsible for handling the upload process asynchronously using promises. Here’s how it works:
- It iterates through the list of requests stored in this.handleUpload.requestList
- For each request, it creates an array of promises (requestPromises) to upload blocks or commit blocks, depending on whether the request is for uploading a block or committing a block
- Within each request, it iterates through the list of individual block requests (Requests)
- For each block request, it checks whether it needs to upload or commit the block based on the IsCommitBlock flag
- If the block needs to be committed, it checks if the file item’s upload status is not already “Uploaded,” and if not, it adds a promise to commit the blocks to the requestPromises array
- If the block needs to be uploaded, it checks if the block’s upload status is not already “Uploaded,” and if not, it adds a promise to upload the block to the requestPromises array
- It then awaits all promises in the requestPromises array to complete using Promise.all
async onPromisesUpload() {
for (let i = 0; i < this.handleUpload.requestList.length; i++) {
const requestPromises: Promise[] = [];
const itemRequest = this.handleUpload.requestList[i];
if (itemRequest) {
console.table(
itemRequest.Requests.map((i) => {
return {
FileId: i.FileItem.FileId,
BlockId: i.FileItemBlock.BlockId,
BlockStart: i.FileItemBlock.BlockStart,
BlockEnd: i.FileItemBlock.BlockEnd,
IsCommit: i.IsCommitBlock,
};
})
);
for (let j = 0; j < itemRequest.Requests.length; j++) {
const request = itemRequest.Requests[j];
if (request.IsCommitBlock) {
if (request.FileItem.UploadStatus != UploadStatus.Uploaded) {
requestPromises.push(
this.commitBlocks(
request.UploadPath,
request.FileItem
)
);
}
} else {
if (request.FileItemBlock.UploadStatus != UploadStatus.Uploaded) {
requestPromises.push(
this.uploadBlocks(
request.UploadPath,
request.FileItem,
request.FileItemBlock
)
);
}
}
}
try {
await Promise.all(requestPromises);
} catch (err: any) {
console.error(`Error uploading file: ${err}`);
}
}
}
},
Step 7: Let’s create the uploadFile method is responsible for uploading files to Azure Blob Storage. Here’s a breakdown of its functionality:
- Initialization:
- It sets flags to indicate that the upload progress is ongoing
- Initializes variables and arrays to store information about file blocks, requests, and upload status
2. Chunking Files:
- It iterates through each file selected for upload.
For each file, it calculates the number of blocks needed based on a predefined chunk size (2 MB by default) - It divides each file into blocks of equal size and assigns unique block IDs to each block
3. Preparing Requests:
- It organizes file blocks into requests for parallel uploads, considering a maximum number of parallel uploads (10 by default)
- Requests are created for both uploading blocks and committing blocks, depending on the current file block’s status
- Files with all blocks uploaded are marked for committing in the next request
4. Executing Uploads:
- It invokes the onPromisesUpload method to handle the asynchronous upload process using promises
async uploadFile(files: File[], blobUrl: string) {
this.handleUpload.isUploadBar = true;
this.handleUpload.isUploadProgress = true;
const allFileBlocks: FileItemBlock[] = [];
this.handleUpload.fileUpload = { FileItems: [] as FileItem[]} as FileUpload;
const MB = 2;
const chunkSize = MB * 1024 * 1024;
let blockSL = 1;
let fileId = 1;
files.forEach((file) => {
const sasIndex = blobUrl.indexOf("?");
let sasUrl = blobUrl.substring(0, sasIndex);
sasUrl = `${sasUrl}/${file.name}?${blobUrl.substring(sasIndex + 1)}`;
const blockNumbers =file.size % chunkSize == 0? file.size / chunkSize: Math.floor(file.size / chunkSize) + 1;
const fileItem = {
FileId: fileId,
File: file,
FileSize: file.size,
Name: file.name,
UploadPath: sasUrl,
NumberOfBlocks: blockNumbers,
UploadStatus: UploadStatus.NotStarted,
BlockIds: [] as string[],
FileItemBlocks: [] as FileItemBlock[],
} as FileItem;
Snippet 2:
let blockStart = 0;
let blockEnd = Math.min(chunkSize, fileItem.FileSize);
let blockSLFileWise = 1;
while (blockStart < fileItem.FileSize) {
const blockId = btoa("block-" + this.pad(blockSL, 6));
const fileItemBlock = {
Serial: blockSLFileWise,
FileId: fileId,
BlockId: blockId,
BlockStart: blockStart,
BlockEnd: blockEnd,
ContentSize: blockEnd - blockStart,
UploadStatus: UploadStatus.NotStarted,
} as FileItemBlock;
fileItem.BlockIds.push(blockId);
fileItem.FileItemBlocks.push(fileItemBlock);
allFileBlocks.push(fileItemBlock);
blockStart = blockEnd;
blockEnd = Math.min(blockStart + chunkSize, fileItem.FileSize);
blockSL++;
blockSLFileWise++;
}
this.handleUpload.fileUpload.FileItems.push(fileItem);
this.handleUpload.totalFileSize += file.size;
fileId++;
});
let currentRequestList: FileItemRequest[] = [];
this.handleUpload.requestList = [];
const filesIdToCommitNextRequest = new Set();
const parallelUploads = 10;
const _continue = true;
let skip = 0;
while (_continue) {
filesIdToCommitNextRequest.forEach((fileId) => {
const fileItem = this.handleUpload.fileUpload.FileItems.find(
(i) => i.FileId == fileId
) as FileItem;
currentRequestList.push({
UploadPath: fileItem.UploadPath,
FileItem: fileItem,
FileItemBlock: fileItem.FileItemBlocks[0],
IsCommitBlock: true,
});
});
filesIdToCommitNextRequest.clear();
while (_continue) {
const fileItemBlock = allFileBlocks[skip] as FileItemBlock;
if (fileItemBlock) {
const fileItem = this.handleUpload.fileUpload.FileItems.find(
(i) => i.FileId == fileItemBlock.FileId
) as FileItem;
currentRequestList.push({
UploadPath: fileItem.UploadPath,
FileItem: fileItem,
FileItemBlock: fileItemBlock,
IsCommitBlock: false,
});
if (fileItemBlock.Serial == fileItem.NumberOfBlocks) {
filesIdToCommitNextRequest.add(fileItem.FileId);
}
}
if (currentRequestList.length == parallelUploads || skip >= allFileBlocks.length) {
this.handleUpload.requestList.push({Requests: [...currentRequestList],IsCommit: false});
currentRequestList = [];
break;
}
skip++;
}
if (!(skip < allFileBlocks.length || filesIdToCommitNextRequest.size > 0)){
break;
}
}
await this.onPromisesUpload();
},
Step 8: Set up a request to dynamically generate a SAS token from the server. Start by adding a CORS policy for the client URL on your backend .net server.. this configuration allows cross-origin requests from a specific origin (http://localhost:5173) while enabling various features like credential support, wildcard subdomains, and unrestricted HTTP methods and headers.
Next, create a method for requesting the .net server.
builder.Services.AddCors(options =>
{
options.AddPolicy("AllowAllOrigins",
policyBuilder =>
{
policyBuilder
.AllowCredentials()
.WithOrigins("http://localhost:5173")
.SetIsOriginAllowedToAllowWildcardSubdomains()
.AllowAnyHeader()
.AllowAnyMethod();
});
});
}
Now let’s back to the client side. Then a function encapsulates the logic to obtain a SAS token for a given container from the server, handling both successful and failed responses appropriately.
async getSasToken(container: string) {
const response = await fetch(`https://localhost:7045/api/AzureFile/GetSasToken?container=${container}`);
if (!response.ok) {
console.error(`Error getting SAS token: ${response.statusText}`);
return;
}
const data = await response.text();
return data;
},
Step 9: uploadWork serves as the entry point for initiating the file upload process to the specified Azure Storage container after obtaining the necessary authentication token. Next, generate a method to invoke the uploadFile method.
async uploadWork() {
const container = "my_container";
const token = await this.getSasToken(container);
if(token){
const blobUrl = `https://my_storage_name.blob.core.windows.net/${container}?${token}`;
await this.uploadFile(this.handleFile.files, blobUrl);
}
},
Step 10: browseClickFilesStore facilitates the addition of selected files to the list of files to be uploaded (this.handleFile.files) and stores their respective paths (this.handleFile.filePaths) for reference during the upload process.
browseClickFilesStore(items: any) {
for (let i = 0; i < items.length; i++) {
const item = items[i];
if (item) {
this.handleFile.files.push(item);
const path = item.webkitRelativePath;
if (!path || path === "") {
this.handleFile.filePaths.push(item.name);
} else {
this.handleFile.filePaths.push(path);
}
}
}
},
Step 11: Together, these functions enable users to browse and select files, and upon selection, trigger the processing and subsequent uploading of those files.
browseFile() {
const file: any = this.$refs.fileInput;
if (file) file.click();
},
onChangeFilesSelect(event: any) {
const items = event.target.files;
this.handleFile.filePaths = [];
this.handleFile.files = [];
this.browseClickFilesStore(items);
this.uploadWork();
},
Step 12: Here’s how your UI code appears in its final form.
{{ handleUpload.progress }}
Step 13: Let’s go to the browser and see what happens while uploading.
Click the ‘Upload Files’ button to open the file explorer, where you can select one or multiple files.
Then it’s showing percentage
After reaching 100%, let’s navigate to the storage explorer to verify if any files have been uploaded.
Conclusion
In conclusion, learning to upload large files to Azure Blob Storage involves several key steps to ensure a smooth and efficient process. We began by creating an Azure Blob Storage resource through the Azure Portal.
Following this, we developed a .NET API to generate SAS tokens, which are essential for secure file uploads. We also built a web client using Vite+Vue.js with TypeScript, enabling the upload of multiple files in chunks. By managing promises in parallel, we significantly enhanced the upload performance.
Lastly, we explored the use of Azure Storage Explorer, providing additional insights into managing and monitoring your storage resources. By following these steps, you can efficiently handle large file uploads to Azure Blob Storage, leveraging its robust and scalable storage solutions.