T157088: [EPIC] Develop a JobQueue backend based on EventBus

Phabricator Link	T157088
Status	resolved
Priority	Medium (orange)
Points
Phabricator Task Created	2017/02/02 07:11 PM
Wiki Page Created	2019/06/09 01:25 PM
Phabricator Task Last Modified	2020/11/23 09:50 PM
Wiki Page Last Updated	2021/10/01 08:06 PM
Phabricator Task Closed	2019/03/28 12:16 PM
Authored By	Pchelolo
Assigned To	Pchelolo
Projects	Analytics (Analytics-Radar) (Radar - Watching) ChangeProp (Backlog) Epic Event-Platform (Backlog) MediaWiki-Core-JobQueue Platform Team Workboards (Done with CPT) Services (done) User-mobrovac (Backlog) WMF-JobQueue (Untriaged)
Platform Engineering Initiative Column
Platform Team Workboards Column	Done with CPT
Subtasks	T171533: Document ChangeProp and EventBus monitoring (resolved) T166341: SSDs for main Kafka clusters (declined) T163380: Support posting Jobs to EventBus simultaneously with normal job processing (resolved) T163379: Create JobQueue implementation that posts to EventBus (resolved) T157093: Support regexes in topic names in ChangeProp (resolved) T157091: Make EventBus service support wildcards in schema definitions (resolved) T157090: Generalise deduplication in ChangeProp (resolved) T157089: Add storage to Change-Prop for deduplication (resolved) T178881: Set up ChangeProp for JobQueue in beta (resolved) T175281: Separate off ChangePropagation for JobQueue as a new deployment (resolved) T198220: Stop and remove old job runners (resolved) T190327: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus (resolved) T188947: Create an LVS endpoint for jobrunners on videoscalers (resolved) T188667: Support dynamic rates for ChangeProp (declined) T183744: FY17/18 Q3 Program 8 Services Goal: Migrate two high-traffic jobs over to EventBus (resolved) T183586: Separate ChangeProp and JobQueue Redis (declined) T182372: Make Kafka JobQueue use Special:RunSingleJob (declined) T175958: Add unit tests to EventBus extension (resolved) T175957: Options for implementing JobQueue statistics methods (declined) T175780: Requests for new JobQueue monitoring capabilities (resolved) T175727: Support topic arrays in ChangeProp config (declined) T175212: Services Q2 2017/18 goal: Migrate a subset of jobs to multi-DC enabled event processing infrastructure. (resolved) T175146: [RfC] Move RunJobs.php to the mediawiki (core) repository (stalled) T174600: Add the ability to sign and verify jobs (resolved) T174599: Set up LVS and VirtualHost for RunSingleJob.php (resolved) T172832: Investigate use-cases for delayed job executions (resolved) T169937: Services Q1 2017/18 goal: Begin migrating job queue processing to multi-DC enabled eventbus infrastructure. (resolved) T166342: New SCB nodes (declined) T157094: Create schema for Job event (resolved) T157092: Support per-topic configuration in EventBus service (declined) T182274: Create custom per-job metric reporters capability (declined) T175800: Allow easy tuning of the jobqueue concurrency. (open)
Parent Tasks

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
Analytics	⇐ Incoming	9 February 2017
Analytics	⇒ Radar	9 February 2017
⇑ WMF-JobQueue		10 July 2018
⇑ Platform Team Workboards (Doing)		20 December 2018
⇓ Platform Team Workboards (Doing)		28 March 2019
⇓ Services (doing)		28 March 2019
⇑ WMF-JobQueue		28 March 2019
⇑ Platform Team Workboards (Done with CPT)		28 March 2019
⇑ Services (done)		28 March 2019
⇓ Analytics		10 June 2020
⇑ Analytics (Analytics-Radar)		10 June 2020