Difference between revisions of "T157088: [EPIC] Develop a JobQueue backend based on EventBus"

Latest revision as of 19:06, 1 October 2021

Phabricator Link	T157088
Status	resolved
Priority	Medium (orange)
Points
Phabricator Task Created	2017/02/02 07:11 PM
Wiki Page Created	2019/06/09 01:25 PM
Phabricator Task Last Modified	2020/11/23 09:50 PM
Wiki Page Last Updated	2021/10/01 08:06 PM
Phabricator Task Closed	2019/03/28 12:16 PM
Authored By	Pchelolo
Assigned To	Pchelolo
Projects	Analytics (Analytics-Radar) (Radar - Watching) ChangeProp (Backlog) Epic Event-Platform (Backlog) MediaWiki-Core-JobQueue Platform Team Workboards (Done with CPT) Services (done) User-mobrovac (Backlog) WMF-JobQueue (Untriaged)
Platform Engineering Initiative Column
Platform Team Workboards Column	Done with CPT
Subtasks	T171533: Document ChangeProp and EventBus monitoring (resolved) T166341: SSDs for main Kafka clusters (declined) T163380: Support posting Jobs to EventBus simultaneously with normal job processing (resolved) T163379: Create JobQueue implementation that posts to EventBus (resolved) T157093: Support regexes in topic names in ChangeProp (resolved) T157091: Make EventBus service support wildcards in schema definitions (resolved) T157090: Generalise deduplication in ChangeProp (resolved) T157089: Add storage to Change-Prop for deduplication (resolved) T178881: Set up ChangeProp for JobQueue in beta (resolved) T175281: Separate off ChangePropagation for JobQueue as a new deployment (resolved) T198220: Stop and remove old job runners (resolved) T190327: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus (resolved) T188947: Create an LVS endpoint for jobrunners on videoscalers (resolved) T188667: Support dynamic rates for ChangeProp (declined) T183744: FY17/18 Q3 Program 8 Services Goal: Migrate two high-traffic jobs over to EventBus (resolved) T183586: Separate ChangeProp and JobQueue Redis (declined) T182372: Make Kafka JobQueue use Special:RunSingleJob (declined) T175958: Add unit tests to EventBus extension (resolved) T175957: Options for implementing JobQueue statistics methods (declined) T175780: Requests for new JobQueue monitoring capabilities (resolved) T175727: Support topic arrays in ChangeProp config (declined) T175212: Services Q2 2017/18 goal: Migrate a subset of jobs to multi-DC enabled event processing infrastructure. (resolved) T175146: [RfC] Move RunJobs.php to the mediawiki (core) repository (stalled) T174600: Add the ability to sign and verify jobs (resolved) T174599: Set up LVS and VirtualHost for RunSingleJob.php (resolved) T172832: Investigate use-cases for delayed job executions (resolved) T169937: Services Q1 2017/18 goal: Begin migrating job queue processing to multi-DC enabled eventbus infrastructure. (resolved) T166342: New SCB nodes (declined) T157094: Create schema for Job event (resolved) T157092: Support per-topic configuration in EventBus service (declined) T182274: Create custom per-job metric reporters capability (declined) T175800: Allow easy tuning of the jobqueue concurrency. (open)
Parent Tasks

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
Analytics	⇐ Incoming	9 February 2017
Analytics	⇒ Radar	9 February 2017
⇑ WMF-JobQueue		10 July 2018
⇑ Platform Team Workboards (Doing)		20 December 2018
⇓ Platform Team Workboards (Doing)		28 March 2019
⇓ Services (doing)		28 March 2019
⇑ WMF-JobQueue		28 March 2019
⇑ Platform Team Workboards (Done with CPT)		28 March 2019
⇑ Services (done)		28 March 2019
⇓ Analytics		10 June 2020
⇑ Analytics (Analytics-Radar)		10 June 2020

@@ Line 5: / Line 5: @@
 |points=
 |dateCreated=1486080671
-|dateModified=1574445115
+|dateModified=1606186201
 |dateClosed=1553789800
 |author={{Phabricator User
@@ Line 18: / Line 18: @@
 }}
 |projects={{Phabricator Project
-|name=Analytics
+|name=Analytics (Analytics-Radar)
-|column=Radar
+|column=Radar - Watching
-|entryDate=1486658976
+|entryDate=1486080671
 }}{{Phabricator Project
 |name=ChangeProp
@@ Line 26: / Line 26: @@
 |entryDate=1486080671
 }}{{Phabricator Project
-|name=Core Platform Team Workboards
+|name=Platform Team Workboards
 |column=Done with CPT
 |entryDate=1486080671
@@ Line 37: / Line 37: @@
 |entryDate=1486080671
 }}{{Phabricator Project
-|name=MediaWiki-JobQueue
+|name=MediaWiki-Core-JobQueue
 |entryDate=1486080671
 }}{{Phabricator Project
@@ Line 54: / Line 54: @@
 |subtasks=171533,166341,163380,163379,157093,157091,157090,157089,178881,175281,198220,190327,188947,188667,183744,183586,175958,175957,175780,175727,175212,175146,174600,174599,172832,169937,166342,157094,157092,182372,182274,175800
 |transitions={{Transition
+|date=1591771481
+|type=Removed Project
+|project=Analytics
+}}{{Transition
+|date=1591771481
+|type=Added Project
+|project=Analytics (Analytics-Radar)
+}}{{Transition
 |date=1553789800
 |type=Removed Project
-|project=Core Platform Team Workboards (Doing)
+|project=Platform Team Workboards (Doing)
 }}{{Transition
 |date=1553789800
@@ Line 68: / Line 76: @@
 |date=1553789800
 |type=Added Project
-|project=Core Platform Team Workboards (Done with CPT)
+|project=Platform Team Workboards (Done with CPT)
 }}{{Transition
 |date=1553789800
@@ Line 76: / Line 84: @@
 |date=1545309818
 |type=Added Project
-|project=Core Platform Team Workboards (Doing)
+|project=Platform Team Workboards (Doing)
 }}{{Transition
 |date=1531278153