Queues and scheduled work

This guide covers the two things that happen outside a web request: jobs that wait in a queue for a worker, and tasks the scheduler runs on a clock. You need it when a job is not running, when a deploy did not take effect in the background, or when you add work of your own.

The queue connection

config/queue.php reads QUEUE_CONNECTION, default database: jobs are rows in the jobs table, which the kit's migrations create, and failed jobs go to failed_jobs (driver database-uuids). No extra service is needed. The connections and their variables:

QUEUE_FAILED_DRIVER (default database-uuids) chooses where failed jobs are kept. Job batches use the job_batches table.

What the kit queues

What is not queued, so you know where to look when it is slow: notifications built on App\Notifications\Notification are sent during the request, database row, email and push included, because the base class uses Queueable without ShouldQueue; the three ticket notices above are the exception. The email verification code is sent synchronously with Mail::mailer(...)->send(), and the SMS code goes out synchronously too. Add implements ShouldQueue to a notification of your own when it fans out to many people; the announcement job is the pattern for doing that in bulk.

To create a queued job use the project stub, which already implements ShouldQueue:

php artisan make:job Orders/ArchiveOldOrders

Run the worker locally

composer dev starts four processes in one terminal: php artisan serve, php artisan queue:listen --tries=1, php artisan pail --timeout=0 (the log tail) and npm run dev. A job you dispatch runs a moment later and its output appears in the same terminal. queue:listen reloads the code on every job, so you do not restart it after editing a job class; that is why it is used here and not in production.

If you would rather not run a worker, set QUEUE_CONNECTION=sync in your local .env. Everything runs inline and exceptions surface in the request that caused them.

Run the worker in production

The worker is a long-lived process:

php artisan queue:work --sleep=3 --tries=3 --max-time=3600

queue:work loads the application once and keeps it in memory, which is what makes it fast and what makes it stale after a deploy. Keep it alive with Supervisor; the complete program block is in /help/deploy-your-project. --sleep=3 is the pause when the queue is empty, --tries=3 the attempts before a job is marked failed, and --max-time=3600 the lifetime after which the worker exits so Supervisor restarts it.

Add --queue=high,default to work several named queues in priority order, and --timeout=60 to kill a single job that runs longer than a minute (keep it below the connection's retry_after).

Restart it on every deploy

php artisan queue:restart

The command writes a timestamp into the cache; each worker checks it after every job and exits when it is newer than its own start. Supervisor then starts a new worker, which loads the new code. Two consequences: the cache store has to be one the worker and the web process share (database or redis, never array), and a job that is running when you call it finishes first. Without this command the worker keeps executing the previous release: a notification whose text you changed keeps arriving with the old text until the worker happens to die.

Failed jobs

A job that throws on every one of its --tries attempts lands in failed_jobs with its exception. The exception is also reported through the normal channels: the log, and Slack when SLACK_BOT_TOKEN is set.

php artisan queue:failed            # list them, with id, connection, queue and date
php artisan queue:retry 9f1c-...    # push one back onto the queue by its id
php artisan queue:retry all         # push all of them back
php artisan queue:forget 9f1c-...   # delete one without retrying
php artisan queue:flush             # delete them all
php artisan queue:prune-failed --hours=168   # delete the ones older than a week

Retry after fixing the cause and restarting the worker; retrying a job into the old code fails it again.

Scheduled tasks

The scheduler runs everything registered with Schedule:: in routes/console.php and in the service providers of the packages. Among the tasks that ship with the kit:

Times are in the application timezone, UTC in config/app.php. php artisan schedule:list prints the table with the next run of each.

Nothing runs unless cron calls the scheduler every minute:

* * * * * cd /var/www/your-project && php artisan schedule:run >> /dev/null 2>&1

schedule:run looks at the clock, runs whatever is due and exits. Locally, php artisan schedule:work does the same in the foreground once a minute, and php artisan schedule:test lets you pick a task and run it now.

Add your own

A job goes in app/Jobs, a command in app/Console/Commands (registered automatically), and its schedule in routes/console.php:

Schedule::command('orders:archive-old')->dailyAt('03:00');
Schedule::job(new ArchiveOldOrders)->weeklyOn(1, '04:00');

A scheduled command that takes long, or that must not overlap with itself, gets ->withoutOverlapping(); one that should run on one server only when you have several gets ->onOneServer(), which needs a shared cache store.