Difference between revisions of "T157088: [EPIC] Develop a JobQueue backend based on EventBus"

Latest revision as of 19:06, 1 October 2021

Phabricator Link	T157088
Status	resolved
Priority	Medium (orange)
Points
Phabricator Task Created	2017/02/02 07:11 PM
Wiki Page Created	2019/06/09 01:25 PM
Phabricator Task Last Modified	2020/11/23 09:50 PM
Wiki Page Last Updated	2021/10/01 08:06 PM
Phabricator Task Closed	2019/03/28 12:16 PM
Authored By	Pchelolo
Assigned To	Pchelolo
Projects	Analytics (Analytics-Radar) (Radar - Watching) ChangeProp (Backlog) Epic Event-Platform (Backlog) MediaWiki-Core-JobQueue Platform Team Workboards (Done with CPT) Services (done) User-mobrovac (Backlog) WMF-JobQueue (Untriaged)
Platform Engineering Initiative Column
Platform Team Workboards Column	Done with CPT
Subtasks	T171533: Document ChangeProp and EventBus monitoring (resolved) T166341: SSDs for main Kafka clusters (declined) T163380: Support posting Jobs to EventBus simultaneously with normal job processing (resolved) T163379: Create JobQueue implementation that posts to EventBus (resolved) T157093: Support regexes in topic names in ChangeProp (resolved) T157091: Make EventBus service support wildcards in schema definitions (resolved) T157090: Generalise deduplication in ChangeProp (resolved) T157089: Add storage to Change-Prop for deduplication (resolved) T178881: Set up ChangeProp for JobQueue in beta (resolved) T175281: Separate off ChangePropagation for JobQueue as a new deployment (resolved) T198220: Stop and remove old job runners (resolved) T190327: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus (resolved) T188947: Create an LVS endpoint for jobrunners on videoscalers (resolved) T188667: Support dynamic rates for ChangeProp (declined) T183744: FY17/18 Q3 Program 8 Services Goal: Migrate two high-traffic jobs over to EventBus (resolved) T183586: Separate ChangeProp and JobQueue Redis (declined) T182372: Make Kafka JobQueue use Special:RunSingleJob (declined) T175958: Add unit tests to EventBus extension (resolved) T175957: Options for implementing JobQueue statistics methods (declined) T175780: Requests for new JobQueue monitoring capabilities (resolved) T175727: Support topic arrays in ChangeProp config (declined) T175212: Services Q2 2017/18 goal: Migrate a subset of jobs to multi-DC enabled event processing infrastructure. (resolved) T175146: [RfC] Move RunJobs.php to the mediawiki (core) repository (stalled) T174600: Add the ability to sign and verify jobs (resolved) T174599: Set up LVS and VirtualHost for RunSingleJob.php (resolved) T172832: Investigate use-cases for delayed job executions (resolved) T169937: Services Q1 2017/18 goal: Begin migrating job queue processing to multi-DC enabled eventbus infrastructure. (resolved) T166342: New SCB nodes (declined) T157094: Create schema for Job event (resolved) T157092: Support per-topic configuration in EventBus service (declined) T182274: Create custom per-job metric reporters capability (declined) T175800: Allow easy tuning of the jobqueue concurrency. (open)
Parent Tasks

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
Analytics	⇐ Incoming	9 February 2017
Analytics	⇒ Radar	9 February 2017
⇑ WMF-JobQueue		10 July 2018
⇑ Platform Team Workboards (Doing)		20 December 2018
⇓ Platform Team Workboards (Doing)		28 March 2019
⇓ Services (doing)		28 March 2019
⇑ WMF-JobQueue		28 March 2019
⇑ Platform Team Workboards (Done with CPT)		28 March 2019
⇑ Services (done)		28 March 2019
⇓ Analytics		10 June 2020
⇑ Analytics (Analytics-Radar)		10 June 2020

@@ Line 3: / Line 3: @@
 |status=resolved
 |color=orange
+|points=
+|dateCreated=1486080671
+|dateModified=1606186201
+|dateClosed=1553789800
 |author={{Phabricator User
 |type=author
@@ Line 14: / Line 18: @@
 }}
 |projects={{Phabricator Project
-|name=WMF-JobQueue
+|name=Analytics (Analytics-Radar)
-|column=Untriaged
+|column=Radar - Watching
+|entryDate=1486080671
+}}{{Phabricator Project
+|name=ChangeProp
+|column=Backlog
+|entryDate=1486080671
 }}{{Phabricator Project
-|name=Core Platform Team Workboards
+|name=Platform Team Workboards
 |column=Done with CPT
+|entryDate=1486080671
 }}{{Phabricator Project
-|name=Services
+|name=Epic
-|column=done
+|entryDate=1486080671
 }}{{Phabricator Project
-|name=MediaWiki-JobQueue
+|name=Event-Platform
+|column=Backlog
+|entryDate=1486080671
 }}{{Phabricator Project
-|name=Epic
+|name=MediaWiki-Core-JobQueue
+|entryDate=1486080671
 }}{{Phabricator Project
 |name=User-mobrovac
 |column=Backlog
+|entryDate=1486080671
 }}{{Phabricator Project
-|name=Analytics
+|name=WMF-JobQueue
+|column=Untriaged
+|entryDate=1486080671
+}}{{Phabricator Project
+|name=Services
+|column=done
+|entryDate=1486080671
+}}
+|subtasks=171533,166341,163380,163379,157093,157091,157090,157089,178881,175281,198220,190327,188947,188667,183744,183586,175958,175957,175780,175727,175212,175146,174600,174599,172832,169937,166342,157094,157092,182372,182274,175800
+|transitions={{Transition
+|date=1591771481
+|type=Removed Project
+|project=Analytics
+}}{{Transition
+|date=1591771481
+|type=Added Project
+|project=Analytics (Analytics-Radar)
+}}{{Transition
+|date=1553789800
+|type=Removed Project
+|project=Platform Team Workboards (Doing)
+}}{{Transition
+|date=1553789800
+|type=Removed Project
+|project=Services (doing)
+}}{{Transition
+|date=1553789800
+|type=Added Project
+|project=Services (done)
+}}{{Transition
+|date=1553789800
+|type=Added Project
+|project=Platform Team Workboards (Done with CPT)
+}}{{Transition
+|date=1553789800
+|type=Added Project
+|project=WMF-JobQueue
+}}{{Transition
+|date=1545309818
+|type=Added Project
+|project=Platform Team Workboards (Doing)
+}}{{Transition
+|date=1531278153
+|type=Added Project
+|project=WMF-JobQueue
+}}{{Transition
+|date=1486658976
+|type=Entered Column
+|project=Analytics
 |column=Radar
-}}{{Phabricator Project
+}}{{Transition
-|name=ChangeProp
+|date=1486658976
-|column=Backlog
+|type=Exited Column
-}}{{Phabricator Project
+|project=Analytics
-|name=EventBus
+|column=Incoming
-|column=Backlog
 }}
-|subtasks=171533,166341,163380,163379,157093,157091,157090,157089,178881,175281,198220,190327,188947,188667,183744,183586,182372,175958,175957,175780,175727,175212,175146,174600,174599,172832,169937,166342,157094,157092,182274,175800
 }}