Difference between revisions of "T88445: MediaWiki active/active datacenter investigation and work (tracking)"

Latest revision as of 22:12, 19 October 2021

Phabricator Link	T88445
Status	open
Priority	Medium (orange)
Points
Phabricator Task Created	2015/02/03 12:30 PM
Wiki Page Created	2019/06/09 01:22 PM
Phabricator Task Last Modified	2021/10/19 09:06 PM
Wiki Page Last Updated	2021/10/19 11:12 PM
Phabricator Task Closed
Authored By	aaron (Aaron Schulz)
Assigned To
Projects	Epic Performance-Team (Radar) (Watching) Platform Team Initiatives (Multi-DC (TEC1)) Platform Team Workboards (Initiatives) (Later) Sustainability (MediaWiki-MultiDC) (Later) User-mobrovac (Backlog)
Platform Engineering Initiative Column	Multi-DC (TEC1)
Platform Team Workboards Column	Initiatives: Later
Subtasks	T219592: Frequent Echo DB_MASTER write queries on HTTP GET (open) T212129: Move MainStash out of Redis to a simpler multi-dc aware solution (open) T221159: FY18/19 TEC1.6 Q4: Improve or replace the usage of GTID_WAIT with pt-heartbeat in MW (declined) T206016: Create a service for session storage (resolved) T114271: Audit all datastores to see how DR failover is handled (open) T164860: Update Echo's caching strategy for multi-dc compatibility (resolved) T120543: Investigate write mode flag cleanup for Flow API modules (resolved) T92863: Configure 'enqueue' job queue type to be DC-local and in a dedicated runner loop (declined) T89389: Try to avoid doCascadeProtectionUpdates() call on page view (resolved) T88506: Make FileRepo file cache misses use slaves instead of the master (resolved) T137326: Log post-send DB updates and make sure their frequency is acceptable (resolved) T124795: codfw is in read only according to mediawiki (resolved) T120009: Flow: Use WAN cache delete() and slave populations to avoid merge() (resolved) T119642: Create a Master-master topology between datacenters for easier failover (setup circular replication dallas -> eqiad for mysql databases) (resolved) T114273: Cleanup active-DC based MW config code and make it more robust and easy to change (resolved) T111266: Make LoadBalancer slave lag check and read-only mode more robust (for example, using pt-heartbeat) (resolved) T94028: DB master connections requested by Flow on GET/HEAD requests (resolved) T91869: Implement a replication strategy for Swift (resolved) T220170: Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment (resolved) T204245: Run MediaWiki media originals active/active (open) T135027: MASTER_POS_WAIT() alternative that works cross-DC (resolved) T134809: Apache <=> mariadb SSL/TLS for cross-datacenter writes (stalled) T127455: Enable HTTPS for Swift traffic (resolved) T114398: Make and document DC switch-over scripts (duplicate) T113204: Smart caching logic for handling cross-DC network outages (resolved) T113192: Consider using delayed rebound purges for CDN (resolved) T112708: Support for master/slave FileBackend distinction in FileBackendMultiwrite (resolved) T111575: Make $wgSessionCacheType and $wgMainStash caches multi-DC ready (declined) T134811: Consider REST with SSL (HyperSwitch/Cassandra) for session storage (resolved) T108253: Make sure CentralAuth login tokens work with two datacenters (resolved) T99208: Make sure $wgUseLocalMessageCache works with multiple DCs (resolved) T97686: Keep TorExitNodes node list cache populated in both datacenters (resolved) T97620: Convert various core/extension cache users to ReplicatedBagOStuff (resolved) T97562: WANObjectCache relay daemon or mcrouter support (resolved) T97335: Make CachingSiteStore use WANObjectCache (resolved) T95126: Make RedisBagOStuff support tag name hashing (resolved) T94480: Audit and fix extensions that call saveSettings() to lazy-set preferences when loaded (resolved) T94448: Echo doing master queries on simple GET requests (resolved) T93144: Check multi-datacenter compatibility with GettingStarted's use of Redis (resolved) T93141: Convert various core/extension cache users to the WAN cache as needed (resolved) T93006: Figure out a cross-DC strategy for the BloomFilter caches (declined) T92357: Fix database master queries from HTTP GET/HEAD before active-active multi-dc (open) T92007: Investigate a better alternative for making user_newtalk talk updates (resolved) T91870: Figure out a replication strategy for ElasticSearch (resolved) T91837: Consider moving various DB writes on page views to using local jobs (resolved) T91820: Create HTTP verb and sticky cookie DC routing in VCL (open) T91816: Add code to enable setting sticky DC cookies for POST requests (resolved) T91819: Use local jobqueue class for jobs enqueued on pages views (resolved) T91815: Audit cache usage to see what can be DC-local and what needs cross-DC purges (resolved) T91284: Avoid wl_notificationtimestamp updates on page views (resolved) T91279: Augment user_touched with an in-memory field (resolved) T89308: Create a local job class that enqueues jobs to final location (resolved) T89184: Remove cache anti-dependencies (resolved) T88666: RFC: Master/slave datacenter strategy for MediaWiki (resolved) T88635: ObjectCacheSessionHandler should avoid pointless writes in write() (resolved) T88634: Make a ReplicatedBagOStuff class for multi-DC use (resolved) T88493: Devise stashing strategy for multi-DC mediawiki (resolved) T88492: Devise caching (memcached) strategy for multi-DC mediawiki (resolved) T130946: Make ?action=markpatrolled require POST (resolved) T119641: Split-brain strategy for services that use config managed by etcd (open) T270223: FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking) (open)
Parent Tasks

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
MediaWiki-Core-Team	⇒ In Dev/Progress	3 February 2015
MediaWiki-Core-Team	⇐ In Dev/Progress	5 February 2015
MediaWiki-Core-Team	⇒ In Dev/Progress	5 February 2015
TechCom-RFC	⇐ P1: Define	6 March 2015
TechCom-RFC	⇒ Under discussion	6 March 2015
MediaWiki-Core-Team	⇐ In Dev/Progress	6 March 2015
MediaWiki-Core-Team	⇒ In Dev/Progress	6 March 2015
Sustainability	⇐ Tag	23 April 2015
Sustainability	⇒ Tag	23 April 2015
Sustainability	⇐ Tag	23 April 2015
Sustainability	⇒ Tag	23 April 2015
Sustainability	⇐ Tag	23 April 2015
Sustainability	⇒ Tag	23 April 2015
Sustainability	⇐ Tag	23 April 2015
Sustainability	⇒ Tag	23 April 2015
Sustainability	⇐ Tag	27 April 2015
Sustainability	⇒ Doing	27 April 2015
Sustainability	⇐ Doing	27 September 2015
Sustainability	⇒ Doing	27 September 2015
Performance-Team	⇐ Inbox	5 October 2015
Performance-Team	⇒ Doing (old)	5 October 2015
Performance-Team	⇐ Doing (old)	5 October 2015
Performance-Team	⇒ Doing (old)	5 October 2015
Performance-Team	⇐ Doing (old)	5 October 2015
Performance-Team	⇒ Doing (old)	5 October 2015
Performance-Team	⇐ Doing (old)	19 October 2015
Performance-Team	⇒ Doing (old)	19 October 2015
Performance-Team	⇐ Doing (old)	21 December 2015
Performance-Team	⇒ Doing (old)	21 December 2015
Performance-Team	⇐ Doing (old)	1 September 2016
Performance-Team	⇒ Doing (old)	1 September 2016
Sustainability	⇐ Doing	3 May 2017
Sustainability	⇒ MediaWiki-MultiDC	3 May 2017
Sustainability (MediaWiki-MultiDC)	⇐ Later	4 May 2017
Sustainability (MediaWiki-MultiDC)	⇒ Current: Performance Team	4 May 2017
Performance-Team	⇐ Doing (old)	18 January 2018
Performance-Team	⇒ Next: Goal / Oct-Dec '21	18 January 2018
⇓ Performance-Team		5 December 2018
⇑ Performance-Team (Radar)		5 December 2018
Performance-Team (Radar)	⇐ Limbo	20 January 2019
Performance-Team (Radar)	⇒ Watching	20 January 2019
⇑ Services (designing)		7 February 2019
⇑ Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1))		7 February 2019
⇑ User-mobrovac		7 February 2019
⇑ Platform Team Legacy (Epic)		7 February 2019
⇓ Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1))		27 February 2019
⇑ Platform Engineering (Multi-DC (TEC1))		27 February 2019
⇓ Platform Team Legacy (Epic)		5 July 2019
⇓ Services (designing)		5 July 2019
⇓ Platform Engineering (Multi-DC (TEC1))		26 July 2019
Platform Engineering	⇐ Multi-DC (TEC1)	26 July 2019
Platform Engineering	⇒ mop	26 July 2019
⇑ Platform Team Initiatives (Multi-DC (TEC1))		26 July 2019
⇑ Platform Team Workboards (Initiatives)		24 March 2020
Sustainability (MediaWiki-MultiDC)	⇐ Current: Performance Team	22 December 2020
Sustainability (MediaWiki-MultiDC)	⇒ Later	22 December 2020

@@ Line 5: / Line 5: @@
 |points=
 |dateCreated=1422984613
-|dateModified=1576130570
+|dateModified=1634692019
 |author={{Phabricator User
 |type=author
-|username=aaron
-|realName=Aaron Schulz
-}}
-|owner={{Phabricator User
-|type=owner
 |username=aaron
 |realName=Aaron Schulz
@@ Line 20: / Line 15: @@
 |entryDate=1422984613
 }}{{Phabricator Project
-|name=Availability (MediaWiki-MultiDC)
+|name=Platform Team Workboards (Initiatives)
-|column=Doing
+|column=Later
-|entryDate=1493926818
+|entryDate=1422984613
+}}{{Phabricator Project
+|name=Sustainability (MediaWiki-MultiDC)
+|column=Later
+|entryDate=1608676882
 }}{{Phabricator Project
-|name=CPT Initiatives
+|name=Platform Team Initiatives
 |column=Multi-DC (TEC1)
 |entryDate=1422984613
@@ Line 36: / Line 35: @@
 |entryDate=1422984613
 }}
-|subtasks=219592,212129,164860,120543,92863,89389,88506,221159,137326,124795,120009,119642,114273,111266,94028,91869,88044,220170,206016,204245,135027,134809,127455,114398,114271,113204,113192,112708,111575,108253,99208,97686,97620,97562,97335,95126,94480,94448,93144,93141,93006,92357,92007,91870,91837,91820,91819,91816,91815,91284,91279,89308,89184,88666,88635,88634,88493,88492,134811,130946,119641
+|subtasks=164860,120543,92863,89389,88506,270223,221159,137326,124795,120009,119642,114273,111266,94028,91869,220170,219592,212129,206016,204245,135027,134809,127455,114398,114271,113204,113192,112708,111575,108253,99208,97686,97620,97562,97335,95126,94480,94448,93144,93141,93006,92357,92007,91870,91837,91820,91819,91816,91815,91284,91279,89308,89184,88666,88635,88634,88493,88492,134811,130946,119641
 |transitions={{Transition
+|date=1608676882
+|type=Entered Column
+|project=Sustainability (MediaWiki-MultiDC)
+|column=Later
+}}{{Transition
+|date=1608676882
+|type=Exited Column
+|project=Sustainability (MediaWiki-MultiDC)
+|column=Current: Performance Team
+}}{{Transition
+|date=1585088956
+|type=Added Project
+|project=Platform Team Workboards (Initiatives)
+}}{{Transition
 |date=1564166864
 |type=Removed Project
-|project=Multi-DC (TEC1)
+|project=Platform Engineering (Multi-DC (TEC1))
 }}{{Transition
 |date=1564166864
 |type=Added Project
-|project=Multi-DC (TEC1)
+|project=Platform Team Initiatives (Multi-DC (TEC1))
 }}{{Transition
 |date=1564166864
 |type=Entered Column
-|project=Core Platform Team
+|project=Platform Engineering
 |column=mop
 }}{{Transition
 |date=1564166864
 |type=Exited Column
-|project=Core Platform Team
+|project=Platform Engineering
 |column=Multi-DC (TEC1)
 }}{{Transition
 |date=1562355077
 |type=Removed Project
-|project=designing
+|project=Services (designing)
 }}{{Transition
 |date=1562355077
 |type=Removed Project
-|project=Epic
+|project=Platform Team Legacy (Epic)
 }}{{Transition
 |date=1551310141
 |type=Removed Project
-|project=Needs Cleaning - Security, stability, performance, and scalability (TEC1)
+|project=Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1))
 }}{{Transition
 |date=1551310141
 |type=Added Project
-|project=Multi-DC (TEC1)
+|project=Platform Engineering (Multi-DC (TEC1))
 }}{{Transition
 |date=1549581558
 |type=Added Project
-|project=Epic
+|project=Platform Team Legacy (Epic)
 }}{{Transition
 |date=1549581558
 |type=Added Project
-|project=Needs Cleaning - Security, stability, performance, and scalability (TEC1)
+|project=Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1))
 }}{{Transition
 |date=1549581558
 |type=Added Project
-|project=designing
+|project=Services (designing)
 }}{{Transition
 |date=1549581558
@@ Line 104: / Line 117: @@
 |date=1544029677
 |type=Added Project
-|project=Radar
+|project=Performance-Team (Radar)
 }}{{Transition
 |date=1516291007
 |type=Entered Column
 |project=Performance-Team
-|column=Next In This Quarter / Jan-Mar 2020
+|column=Next: Goal / Oct-Dec '21
 }}{{Transition
 |date=1516291007
 |type=Exited Column
 |project=Performance-Team
-|column=Doing
+|column=Doing (old)
 }}{{Transition
 |date=1493926818
 |type=Entered Column
-|project=Availability (MediaWiki-MultiDC)
+|project=Sustainability (MediaWiki-MultiDC)
-|column=Doing
+|column=Current: Performance Team
 }}{{Transition
 |date=1493926818
 |type=Exited Column
-|project=Availability (MediaWiki-MultiDC)
+|project=Sustainability (MediaWiki-MultiDC)
-|column=Backlog
+|column=Later
 }}{{Transition
 |date=1493841737
 |type=Entered Column
-|project=Availability
+|project=Sustainability
 |column=MediaWiki-MultiDC
 }}{{Transition
 |date=1493841737
 |type=Exited Column
-|project=Availability
+|project=Sustainability
 |column=Doing
 }}{{Transition
@@ Line 139: / Line 152: @@
 |type=Entered Column
 |project=Performance-Team
-|column=Doing
+|column=Doing (old)
 }}{{Transition
 |date=1472758374
 |type=Exited Column
 |project=Performance-Team
-|column=Doing
+|column=Doing (old)
 }}{{Transition
 |date=1450727679
 |type=Entered Column
 |project=Performance-Team
-|column=Doing
+|column=Doing (old)
 }}{{Transition
 |date=1450727679
 |type=Exited Column
 |project=Performance-Team
-|column=Doing
+|column=Doing (old)
 }}{{Transition
 |date=1445277895
 |type=Entered Column
 |project=Performance-Team
-|column=Doing
+|column=Doing (old)
 }}{{Transition
 |date=1445277895
 |type=Exited Column
 |project=Performance-Team
-|column=Doing
+|column=Doing (old)
 }}{{Transition
 |date=1444092113
 |type=Entered Column
 |project=Performance-Team
-|column=Doing
+|column=Doing (old)
 }}{{Transition
 |date=1444092113
 |type=Exited Column
 |project=Performance-Team
-|column=Doing
+|column=Doing (old)
 }}{{Transition
 |date=1444068481
 |type=Entered Column
 |project=Performance-Team
-|column=Doing
+|column=Doing (old)
 }}{{Transition
 |date=1444068481
 |type=Exited Column
 |project=Performance-Team
-|column=Doing
+|column=Doing (old)
 }}{{Transition
 |date=1444068253
 |type=Entered Column
 |project=Performance-Team
-|column=Doing
+|column=Doing (old)
 }}{{Transition
 |date=1444068253
@@ Line 198: / Line 211: @@
 |date=1443387069
 |type=Entered Column
-|project=Availability
+|project=Sustainability
 |column=Doing
 }}{{Transition
 |date=1443387069
 |type=Exited Column
-|project=Availability
+|project=Sustainability
 |column=Doing
 }}{{Transition
 |date=1430153849
 |type=Entered Column
-|project=Availability
+|project=Sustainability
 |column=Doing
 }}{{Transition
 |date=1430153849
 |type=Exited Column
-|project=Availability
+|project=Sustainability
-|column=Backlog
+|column=Tag
 }}{{Transition
 |date=1429815199
 |type=Entered Column
-|project=Availability
+|project=Sustainability
-|column=Backlog
+|column=Tag
 }}{{Transition
 |date=1429815199
 |type=Exited Column
-|project=Availability
+|project=Sustainability
-|column=Backlog
+|column=Tag
 }}{{Transition
 |date=1429815196
 |type=Entered Column
-|project=Availability
+|project=Sustainability
-|column=Backlog
+|column=Tag
 }}{{Transition
 |date=1429815196
 |type=Exited Column
-|project=Availability
+|project=Sustainability
-|column=Backlog
+|column=Tag
 }}{{Transition
 |date=1429815194
 |type=Entered Column
-|project=Availability
+|project=Sustainability
-|column=Backlog
+|column=Tag
 }}{{Transition
 |date=1429815194
 |type=Exited Column
-|project=Availability
+|project=Sustainability
-|column=Backlog
+|column=Tag
 }}{{Transition
 |date=1429815192
 |type=Entered Column
-|project=Availability
+|project=Sustainability
-|column=Backlog
+|column=Tag
 }}{{Transition
 |date=1429815192
 |type=Exited Column
-|project=Availability
+|project=Sustainability
-|column=Backlog
+|column=Tag
 }}{{Transition
 |date=1425689785
@@ Line 274: / Line 287: @@
 |type=Exited Column
 |project=TechCom-RFC
-|column=Inbox
+|column=P1: Define
 }}{{Transition
 |date=1423169245