A complete guide on how to Upload a large amount of data to MySQL database using Laravel Queue Batch Processing

In this blog, I will discuss what is batch process. You can upload any large amount of data to MySQL database within minutes using Laravel Queue Batch Processing. So, here I will give an example that I have uploaded almost 5 Lakh data to the MySql database table within approx. 4 minutes.

But the question is what is batch processing?

There are some situations where you need to send thousands of email reports to the user or upload a large amount of data to the database within a minimum time. So, if you do perform all these times taken long-running tasks, then you will face a waiting or loading screen on the web page and its result will be your application slow down or you will sit idle until those tasks are finished.

For this complex situation, Laravel framework approaches a technique called Queue that allows you to run all these tasks asynchronously in the background of a web application without slowdown of your application.

What is Laravel Queue?

Laravel Queue is the driver where we store the list of tasks. So, there are different types of queue drivers Like a database, Redis, Beanstalkd, IronMQ Amazon, etc. If the task is heavy, then you should consider, Redis, Beanstalkd, IronMQ Amazon, etc. But for low-weight task lists, we can use a database drive as.

Queue configuration

To use the database driver, we have to run the below command which creates migration files to create all database tables schema that will be used to string jobs and failed jobs

PHP artisan queue: table and PHP artisan migrate

After executing the above command jobs and falied_jobs table will be created.

Then we need to set the queue driver value at .env file like below

QUEUE_CONNECTION=database

Job Creation

Laravel job files are located in the app/Jobs directory. So, by executing the below command you can create any job like the below example.

PHP artisan makes job MytestJob and you will find the job class structure below

The above handle() method is called when any job is processed by queue and generated class “Mytestjob” implement the “ShouldQueue” interface which indicates that the job should be pushed in to queue to execute asynchronously.

How to dispatch Job

We can use the any of below commands to dispatch the jobs.

Queue::push(new Mytestjob($options)); or

dispatch(new Mytestjob($options)); or

(new Mytestjob($options))->dispatch(); or

\App\Jobs\Mytestjob::dispatch($options);

Then after Dispatching a job, you need to process this queue; for this, you have to start employment by using a straightforward command.

PHP artisan queue: work

Batching of Jobs

There is another feature of job processing called a batch. A batch is a collection of jobs which processed all jobs simultaneously. Before using batch, you should create a database table that stores information about job batches. To migrate the batch table execute the below commands.

PHP artisan queue:batches-table

PHP artisan migrate

so, you see this table stores the total number of jobs, number of pending jobs, number of failed jobs, and more.

How To Dispatch Batch

Now Follow the below steps of a live example of Batch Processing:

Route:

Create an upload blade file as below:

Create controller as below:

Create the Job file as below:

Then run this command:

“PHP artisan queue: work”

And then upload a large CSV file by choosing the input type file of “upload_form.blade.php”.

So, when the above code will be executed the below processing will be shown at the command prompt.