Large file upload more than 1GB using HTML5 Web workers, Drag and Drop and XHR2



To know about HTML5

This tutorial provides a guide and code examples for leveraging the large file upload using File API inside of a Web Worker. While we are trying to upload large file, we getting the problems like browser showing kill pages or unresponsive page etc.. This are because of we are blocking the user agent and we are utilising lot of CPU, it will lead to performance down or user agent getting continuos processing. We need to avoid the blocking or performance utilization on document. Solution for that is simple, we need to upload file using background processing nothing but using webworkers. By using webworkers we can upload files in background so that no errors or problems will occur.



In some cases reading the entire file into memory isn't the best option. For example, say you wanted to write an async file uploader. One possible way to speed up the upload would be to read and send the file in separate byte range chunks. The server component would then be responsible for reconstructing the file content in the correct order. Lucky for us, the File interface supports a slice method to support this use case. The method takes a starting byte as its first argument, ending byte as its second, and an option content type string as a third.

I was demonstrated the upload using file input and drag and drop. you could use client-side logic to verify an upload's mimetype matches its file extension or restrict the size of an upload. The most straightforward way to load a file is to use a standard <input type="file"> element. JavaScript returns the list of selected File objects as a FileList. Here's an example that uses the 'multiple' attribute to allow selecting several files at once. Another technique for loading files is native drag and drop from the desktop to the browser. This example include drag and drop support. The following example demonstrates reading chunks of a file.

If you load models or textures from external files, due to browsers' "same origin policy" security restrictions, loading from a file system will fail with a security exception. To solve this Start Chrome executable with a command line flag: chrome --allow-file-access-from-files or else run from local server


This will solve the following problems:
  1. How to upload large file more than 1GB
  2. Synchronous File Uploading
  3. Large File upload with XHR2 uisng webworkers
  4. Drag and Drop File upload
  5. Multiple File Upload without blocking UI
  6. Increasing performance While File uploads
  7. Background Processing of file uploads
  8. Slicing the file
code:
HTML

  <input type="file" id="files" name="files[]" multiple />
  <div id="drop_zone">
   Drop files here
  </div> 

  <output id="list"></output>
  


JavaScript


 var worker = new Worker('fileupload.js');
 worker.onmessage = function(e) {
 alert(e.data);
}
worker.onerror =werror;
function werror(e) {
  console.log('ERROR: Line ', e.lineno, ' in ', e.filename, ': ', e.message);
 }
function handleFileSelect(evt) {
 evt.stopPropagation();
 evt.preventDefault();

 var files = evt.dataTransfer.files||evt.target.files;
 // FileList object.
 
 worker.postMessage({
 'files' : files
 });
 //Sending File list to worker
 // files is a FileList of File objects. List some properties.
 var output = [];
 for (var i = 0, f; f = files[i]; i++) {
  output.push('<li><strong>', escape(f.name), '</strong> (', f.type || 'n/a', ') - ', f.size, ' bytes, last modified: ', f.lastModifiedDate ? f.lastModifiedDate.toLocaleDateString() : 'n/a', '</li>');
 }
 document.getElementById('list').innerHTML = '<ul>' + output.join('') + '</ul>';
}

function handleDragOver(evt) {
 evt.stopPropagation();
 evt.preventDefault();
 evt.dataTransfer.dropEffect = 'copy';
 // Explicitly show this is a copy.
}

// Setup the dnd listeners.
var dropZone = document.getElementById('drop_zone');
dropZone.addEventListener('dragover', handleDragOver, false);
dropZone.addEventListener('drop', handleFileSelect, false);
 document.getElementById('files').addEventListener('change', handleFileSelect, false);


Worker (fileupload.js) :

var file = [], p = true;
function upload(blobOrFile) {
 var xhr = new XMLHttpRequest();
 xhr.open('POST', '/server', false);
 xhr.onload = function(e) {
 };
 xhr.send(blobOrFile);
}

function process() {
 for (var j = 0; j <file.length; j++) {
  var blob = file[j];

  const BYTES_PER_CHUNK = 1024 * 1024;
  // 1MB chunk sizes.
  const SIZE = blob.size;

  var start = 0;
  var end = BYTES_PER_CHUNK;

  while (start < SIZE) {

   if ('mozSlice' in blob) {
    var chunk = blob.mozSlice(start, end);
   } else {
    var chunk = blob.webkitSlice(start, end);
   }

   upload(chunk);

   start = end;
   end = start + BYTES_PER_CHUNK;
  }
  p = ( j = file.length - 1) ? true : false;
  self.postMessage(blob.name + " Uploaded Succesfully");
 }
}


self.onmessage = function(e) {

for (var j = 0; j < e.data.length; j++)
  files.push(e.data[j]);

 if (p) {
  process()
 }

}


Conclusion
I hope this article has help you to upload large files. Web Workers are an underutilized and under-appreciated feature of HTML5. We can minimize the work to upload a large file. The technique is to slice the upload into multiple chunks, spawn an XHR for each portion, and put the file together on the server. This is similar to how GMail uploads large attachments so quickly.

Further Reading
  1.  Mastering in webworkers 
  2. Reading files and chunking. 
  3. HTML5  

Share this

Related Posts

Previous
Next Post »

34 comments

comments
July 24, 2012 at 3:38 PM delete

Hi,
Thank you for this example.
Could you share the server script as well? I don't really get how to save the file server-side.
Thank you in advance,

Valerie

Reply
avatar
rob
January 16, 2013 at 9:30 AM delete

While this is a great overview, I wasn't able to implement due to a few bugs.

Reply
avatar
January 18, 2013 at 8:50 PM delete

Bro...change globally theme..it's hard read using mobile browsers

Reply
avatar
January 18, 2013 at 8:51 PM delete

Bro...change globally theme..it's hard read using mobile browsers

Reply
avatar
January 24, 2013 at 4:06 PM delete

I will share php script in a quick session

Reply
avatar
February 4, 2013 at 10:24 PM delete

consloe.log - in first javascript - must be console.log,
expect that little thing - Great Post
thanks

Reply
avatar
March 13, 2013 at 3:07 PM delete

Updates: webkitslice prefix removed so you can use as it as in specification.

Reply
avatar
April 19, 2013 at 2:32 PM delete

Hi. Thanks for sharing.

Dunno if I miss something but when I select file, I get this error in console "DataCloneError: The object could not be cloned.
'files' : files"

I search on the web but don't really find what can be the reason of this error.

As previously ask, could you share a server-side script (PHP ?) to rebuilt file on the server.

Thank you in advance.

Reply
avatar
June 11, 2013 at 5:14 AM delete

Great article.

Can you provide the php script to try this on my computer and test how much memory will consume?

Thanks in advance

Reply
avatar
June 27, 2013 at 2:10 PM delete

Great Article,

Hi can you please send the php script.

Reply
avatar
November 6, 2013 at 8:46 PM delete

Great! Could you please share the php script? Thanks in advance, Jan

Reply
avatar
January 26, 2014 at 11:11 AM delete

It would be nice if you could attach the working project for reference.

Reply
avatar
January 30, 2014 at 10:04 AM delete

okay, I would Place working project ASAP.

Reply
avatar
May 22, 2014 at 9:09 AM delete

Can you please share how to save the file on the server side?

Reply
avatar
June 26, 2014 at 7:01 PM delete

please share how to save the file on server side?

Reply
avatar
June 26, 2014 at 7:55 PM delete

please share how to save this file on server particular location i.e. shared location on server?

Reply
avatar
February 13, 2015 at 5:24 PM delete

The blob,webkitSlice or blob.WebSlice is not working in safari could you please suggest me any alternative.Tried few snippets nothing worked the method should return a blob object which has size and type of blob. Any help or pointer appreciated

Reply
avatar
February 24, 2015 at 3:47 PM delete

hi Rakesh,

webkitSlice deprecated use blob.slice

Reply
avatar
June 1, 2015 at 9:19 PM delete

i am also interested in seeing your serverSide script. was that posted @ some point? thank you for the share

Reply
avatar
September 17, 2016 at 3:12 AM delete

Such a great post! Any idea for the server side code using ASP.NET

Reply
avatar
Anonymous
September 24, 2016 at 12:55 PM delete

few days ago I was browsing and found a file upload site quickly, besides that we receive payments from the file that we upload it if there is downloading perfectly. file2up.net

Reply
avatar
Anonymous
June 29, 2017 at 8:56 AM delete

Hi. Can you post your server side code? I am having a difficulty on that side. Cause I can already transfer the data of the file to the server, but i cant transfer the file itself to the server. Im confused on how should i do it. I hope you can help me about this. Thankyou.

Reply
avatar
June 7, 2018 at 11:20 PM delete

Nice. Please attach sample project

Reply
avatar
June 15, 2018 at 1:13 PM delete

Hi, thanks for your great work.
if you can, can you upload server side php script please

Reply
avatar
May 17, 2019 at 11:31 AM delete

Scenario: Your worker script is running in the background and all is fine and the user is able to continue using the page while the worker is uploading. All of a sudden, the user decides to click on some link that happens to be on the page, and the page navigates away from the site. The worker is no longer accessible and dependent on the browser it may or may quit right away, but the fact is, the upload will be interrupted before it is finished.

Question: How would you go about detecting any navigation away from the page that would kill the worker, and presenting the user with a warning about it? Have you run into that scenario before and what was your solution if you did?

Reply
avatar
October 15, 2019 at 5:28 PM delete

Such a great post! Any idea for the server side code using ASP.NET

href="https://sisgain.com/webrtc"webrtc development

Reply
avatar
Anonymous
October 5, 2021 at 4:22 PM delete

(failed)net::ERR_CONNECTION_RESET

Reply
avatar
October 13, 2021 at 1:42 AM delete

In server side you have to collect the all the chunks and then use file stream to create file using content-type.

Reply
avatar
January 21, 2022 at 11:26 AM delete

"Nicely explained. To support the WebRTC app development, I would like to share that In terms of global coverage, the WebRTC market spans North America, Europe, Asia, the Middle East, South America, and Africa. Nowadays, Google puts great efforts into the development of Web Real-Time Communication. The top WebRTC development companies are growing rapidly with the increasing demand of the technology.

Reply
avatar
Anonymous
September 19, 2023 at 10:03 AM delete

For the PHP script
Just do file_get_contents("php://input");
That gets you the data.. from there,
You wanna use fopen, and fwrite to save it to a file and whatever the extension is.. PNG, mp4 etc.. but fuck this website and fuck this dude..

Reply
avatar
Anonymous
September 19, 2023 at 10:04 AM delete

Raju konga can suck dick

Reply
avatar