Queue system (BullMQ)

Source of truth for routing rules (repository): docs/AGENTS_BACKEND.md — section BullMQ: exports queue and workers (exports router, slideshow queue, WorkersQueueService).

Overview

Long-running work is offloaded to BullMQ queues backed by Redis. The HTTP API enqueues jobs; a separate worker process (RUN_MODE=worker) runs @Processor classes. Defaults and retention live in backend/src/queue/queue.config.ts; queue names in backend/src/queue/types/queue-names.types.ts.

API vs worker

Mode	Env	What runs
API	QUEUE_ENABLED=true, QUEUE_WORKERS_ENABLED=false	QueueClientModule: enqueue + QueueService; no BullMQ job consumers for worker-only modules
Worker	QUEUE_ENABLED=true, QUEUE_WORKERS_ENABLED=true	QueueWorkerModule + WorkerModule: processors consume jobs

Shared entrypoint backend/src/main.ts selects AppModule vs WorkerModule via RUN_MODE.

Recommended worker env (avoid duplicate schedulers):

QUEUE_ENABLED=true
QUEUE_WORKERS_ENABLED=true
SCHEDULE_ENABLED=false
EVENT_EMITTER_ENABLED=false

Redis: REDIS_HOST, REDIS_PORT, REDIS_PASSWORD (empty password must be unset, not "").

Queues and processors

Queue (QueueNames)	Redis name	Primary processors / notes
EXPORTS	exports	ExportsQueueRouterProcessor only — routes export:build, doc:ingest, doc:ocr, doc:extract_structured to injectable handlers
GALLERY_SLIDESHOW	gallery-slideshow	GallerySlideshowRenderProcessor — job slideshow:render
GOOGLE_CALENDAR_SYNC	google-calendar-sync	GoogleCalendarSyncProcessor
SCHEDULED_EMAILS	scheduled-emails	ScheduledEmailsProcessor router only — send-email (legacy) + delegates email:send to injectable EmailSendProcessor
WORKFLOW_TASKS	workflow-tasks	WorkflowTasksProcessor
RECURRING_SESSIONS	recurring-sessions	RecurringSessionsProcessor
AUDIT_CLEANUP	audit-cleanup	AuditCleanupProcessor
AI_INSIGHTS	ai-insights	AiInsightsProcessor router only — generate (legacy) + delegates ai:run to injectable AiRunProcessor
PYTHON_ML	python-ml	Multiple Ml* processors (worker-only; API uses PythonMlClientModule)
MOBILE_PUSH	mobile-push	MobilePushProcessor

Enqueue (versioned jobs): WorkersQueueService in backend/src/workers/workers-queue.service.ts with Zod payloads (job-payloads.schemas.ts).

Legacy: QueueService.addExportJob throws — use WorkersQueueService.enqueueExportBuild after creating an export_jobs row (including scheduled exports).

Module map

• QueueRuntimeModule — switches disabled / client / worker global queue mode
• QueueCoreModule — Bull root + registerQueue for all QueueNames
• QueueClientModule — API: QueueService + WebSocket progress emitter
• QueueWorkerModule — worker: no-op progress emitter + internal processors (e.g. audit cleanup)
• WorkerModule — worker-only imports: ExportsQueueRouterModule, email/AI/ML/doc-style modules, calendar, etc.
• ExportsQueueRouterModule — only place that registers @Processor(EXPORTS)

Bull Board

When enabled (QUEUE_BOARD_ENABLED / route via QUEUE_BOARD_ROUTE, default /ops/queues): dashboards per registered queue. See backend/src/queue/queue-board.module.ts. Protect in production (basic auth / private network).

WebSocket progress

API emits queue:job:progress when QueueService drives progress (legacy paths). Payload shape includes jobId, status, optional progress, message, result.

Debugging

• Bull Board: inspect waiting/active/failed jobs and payloads
• Ensure API and worker use the same Redis instance and key prefix
• For stuck exports jobs, confirm the worker process is running and logs show ExportsQueueRouterProcessor

References

• BullMQ docs
• NestJS BullMQ
• Module README: backend/src/queue/README.md