Difference between revisions of "T88445: MediaWiki active/active datacenter investigation and work (tracking)"

Revision as of 08:11, 2 March 2020

Phabricator Link	T88445
Status	open
Priority	Medium (orange)
Points
Phabricator Task Created	2015/02/03 12:30 PM
Wiki Page Created	2019/06/09 01:22 PM
Phabricator Task Last Modified	2019/12/12 01:02 AM
Wiki Page Last Updated	2021/10/19 11:12 PM
Phabricator Task Closed
Authored By	aaron (Aaron Schulz)
Assigned To
Projects	Epic Performance-Team (Radar) (Watching) Platform Team Initiatives (Multi-DC (TEC1)) Platform Team Workboards (Initiatives) (Later) Sustainability (MediaWiki-MultiDC) (Later) User-mobrovac (Backlog)
Platform Engineering Initiative Column	Multi-DC (TEC1)
Platform Team Workboards Column	Initiatives: Later
Subtasks	T219592: Frequent Echo DB_MASTER write queries on HTTP GET (open) T212129: Move MainStash out of Redis to a simpler multi-dc aware solution (open) T221159: FY18/19 TEC1.6 Q4: Improve or replace the usage of GTID_WAIT with pt-heartbeat in MW (declined) T206016: Create a service for session storage (resolved) T114271: Audit all datastores to see how DR failover is handled (open) T164860: Update Echo's caching strategy for multi-dc compatibility (resolved) T120543: Investigate write mode flag cleanup for Flow API modules (resolved) T92863: Configure 'enqueue' job queue type to be DC-local and in a dedicated runner loop (declined) T89389: Try to avoid doCascadeProtectionUpdates() call on page view (resolved) T88506: Make FileRepo file cache misses use slaves instead of the master (resolved) T137326: Log post-send DB updates and make sure their frequency is acceptable (resolved) T124795: codfw is in read only according to mediawiki (resolved) T120009: Flow: Use WAN cache delete() and slave populations to avoid merge() (resolved) T119642: Create a Master-master topology between datacenters for easier failover (setup circular replication dallas -> eqiad for mysql databases) (resolved) T114273: Cleanup active-DC based MW config code and make it more robust and easy to change (resolved) T111266: Make LoadBalancer slave lag check and read-only mode more robust (for example, using pt-heartbeat) (resolved) T94028: DB master connections requested by Flow on GET/HEAD requests (resolved) T91869: Implement a replication strategy for Swift (resolved) T220170: Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment (resolved) T204245: Run MediaWiki media originals active/active (open) T135027: MASTER_POS_WAIT() alternative that works cross-DC (resolved) T134809: Apache <=> mariadb SSL/TLS for cross-datacenter writes (stalled) T127455: Enable HTTPS for Swift traffic (resolved) T114398: Make and document DC switch-over scripts (duplicate) T113204: Smart caching logic for handling cross-DC network outages (resolved) T113192: Consider using delayed rebound purges for CDN (resolved) T112708: Support for master/slave FileBackend distinction in FileBackendMultiwrite (resolved) T111575: Make $wgSessionCacheType and $wgMainStash caches multi-DC ready (declined) T134811: Consider REST with SSL (HyperSwitch/Cassandra) for session storage (resolved) T108253: Make sure CentralAuth login tokens work with two datacenters (resolved) T99208: Make sure $wgUseLocalMessageCache works with multiple DCs (resolved) T97686: Keep TorExitNodes node list cache populated in both datacenters (resolved) T97620: Convert various core/extension cache users to ReplicatedBagOStuff (resolved) T97562: WANObjectCache relay daemon or mcrouter support (resolved) T97335: Make CachingSiteStore use WANObjectCache (resolved) T95126: Make RedisBagOStuff support tag name hashing (resolved) T94480: Audit and fix extensions that call saveSettings() to lazy-set preferences when loaded (resolved) T94448: Echo doing master queries on simple GET requests (resolved) T93144: Check multi-datacenter compatibility with GettingStarted's use of Redis (resolved) T93141: Convert various core/extension cache users to the WAN cache as needed (resolved) T93006: Figure out a cross-DC strategy for the BloomFilter caches (declined) T92357: Fix database master queries from HTTP GET/HEAD before active-active multi-dc (open) T92007: Investigate a better alternative for making user_newtalk talk updates (resolved) T91870: Figure out a replication strategy for ElasticSearch (resolved) T91837: Consider moving various DB writes on page views to using local jobs (resolved) T91820: Create HTTP verb and sticky cookie DC routing in VCL (open) T91816: Add code to enable setting sticky DC cookies for POST requests (resolved) T91819: Use local jobqueue class for jobs enqueued on pages views (resolved) T91815: Audit cache usage to see what can be DC-local and what needs cross-DC purges (resolved) T91284: Avoid wl_notificationtimestamp updates on page views (resolved) T91279: Augment user_touched with an in-memory field (resolved) T89308: Create a local job class that enqueues jobs to final location (resolved) T89184: Remove cache anti-dependencies (resolved) T88666: RFC: Master/slave datacenter strategy for MediaWiki (resolved) T88635: ObjectCacheSessionHandler should avoid pointless writes in write() (resolved) T88634: Make a ReplicatedBagOStuff class for multi-DC use (resolved) T88493: Devise stashing strategy for multi-DC mediawiki (resolved) T88492: Devise caching (memcached) strategy for multi-DC mediawiki (resolved) T130946: Make ?action=markpatrolled require POST (resolved) T119641: Split-brain strategy for services that use config managed by etcd (open) T270223: FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking) (open)
Parent Tasks

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
MediaWiki-Core-Team	⇒ In Dev/Progress	3 February 2015
MediaWiki-Core-Team	⇐ In Dev/Progress	5 February 2015
MediaWiki-Core-Team	⇒ In Dev/Progress	5 February 2015
TechCom-RFC	⇐ P1: Define	6 March 2015
TechCom-RFC	⇒ Under discussion	6 March 2015
MediaWiki-Core-Team	⇐ In Dev/Progress	6 March 2015
MediaWiki-Core-Team	⇒ In Dev/Progress	6 March 2015
Sustainability	⇐ Tag	23 April 2015
Sustainability	⇒ Tag	23 April 2015
Sustainability	⇐ Tag	23 April 2015
Sustainability	⇒ Tag	23 April 2015
Sustainability	⇐ Tag	23 April 2015
Sustainability	⇒ Tag	23 April 2015
Sustainability	⇐ Tag	23 April 2015
Sustainability	⇒ Tag	23 April 2015
Sustainability	⇐ Tag	27 April 2015
Sustainability	⇒ Doing	27 April 2015
Sustainability	⇐ Doing	27 September 2015
Sustainability	⇒ Doing	27 September 2015
Performance-Team	⇐ Inbox	5 October 2015
Performance-Team	⇒ Doing (old)	5 October 2015
Performance-Team	⇐ Doing (old)	5 October 2015
Performance-Team	⇒ Doing (old)	5 October 2015
Performance-Team	⇐ Doing (old)	5 October 2015
Performance-Team	⇒ Doing (old)	5 October 2015
Performance-Team	⇐ Doing (old)	19 October 2015
Performance-Team	⇒ Doing (old)	19 October 2015
Performance-Team	⇐ Doing (old)	21 December 2015
Performance-Team	⇒ Doing (old)	21 December 2015
Performance-Team	⇐ Doing (old)	1 September 2016
Performance-Team	⇒ Doing (old)	1 September 2016
Sustainability	⇐ Doing	3 May 2017
Sustainability	⇒ MediaWiki-MultiDC	3 May 2017
Sustainability (MediaWiki-MultiDC)	⇐ Later	4 May 2017
Sustainability (MediaWiki-MultiDC)	⇒ Current: Performance Team	4 May 2017
Performance-Team	⇐ Doing (old)	18 January 2018
Performance-Team	⇒ Next: Goal / Oct-Dec '21	18 January 2018
⇓ Performance-Team		5 December 2018
⇑ Performance-Team (Radar)		5 December 2018
Performance-Team (Radar)	⇐ Limbo	20 January 2019
Performance-Team (Radar)	⇒ Watching	20 January 2019
⇑ Platform Team Legacy (Epic)		7 February 2019
⇑ Services (designing)		7 February 2019
⇑ Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1))		7 February 2019
⇑ User-mobrovac		7 February 2019
⇓ Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1))		27 February 2019
⇑ Platform Engineering (Multi-DC (TEC1))		27 February 2019
⇓ Services (designing)		5 July 2019
⇓ Platform Team Legacy (Epic)		5 July 2019
⇓ Platform Engineering (Multi-DC (TEC1))		26 July 2019
Platform Engineering	⇐ Multi-DC (TEC1)	26 July 2019
Platform Engineering	⇒ mop	26 July 2019
⇑ Platform Team Initiatives (Multi-DC (TEC1))		26 July 2019
⇑ Platform Team Workboards (Initiatives)		24 March 2020
Sustainability (MediaWiki-MultiDC)	⇐ Current: Performance Team	22 December 2020
Sustainability (MediaWiki-MultiDC)	⇒ Later	22 December 2020

@@ Line 38: / Line 38: @@
 |subtasks=219592,212129,164860,120543,92863,89389,88506,221159,137326,124795,120009,119642,114273,111266,94028,91869,88044,220170,206016,204245,135027,134809,127455,114398,114271,113204,113192,112708,111575,108253,99208,97686,97620,97562,97335,95126,94480,94448,93144,93141,93006,92357,92007,91870,91837,91820,91819,91816,91815,91284,91279,89308,89184,88666,88635,88634,88493,88492,134811,130946,119641
 |transitions={{Transition
+|date=1564166864
+|type=Removed Project
+|project=Multi-DC (TEC1)
+}}{{Transition
+|date=1564166864
+|type=Added Project
+|project=Multi-DC (TEC1)
+}}{{Transition
+|date=1564166864
+|type=Entered Column
 |project=Core Platform Team
 |column=mop
-|transitionDate=1564166864
-|transitionType=Entered Column
 }}{{Transition
+|date=1564166864
+|type=Exited Column
 |project=Core Platform Team
 |column=Multi-DC (TEC1)
-|transitionDate=1564166864
-|transitionType=Exited Column
 }}{{Transition
+|date=1562355077
+|type=Removed Project
+|project=designing
+}}{{Transition
+|date=1562355077
+|type=Removed Project
+|project=Epic
+}}{{Transition
+|date=1551310141
+|type=Removed Project
+|project=Needs Cleaning - Security, stability, performance, and scalability (TEC1)
+}}{{Transition
+|date=1551310141
+|type=Added Project
+|project=Multi-DC (TEC1)
+}}{{Transition
+|date=1549581558
+|type=Added Project
+|project=Epic
+}}{{Transition
+|date=1549581558
+|type=Added Project
+|project=Needs Cleaning - Security, stability, performance, and scalability (TEC1)
+}}{{Transition
+|date=1549581558
+|type=Added Project
+|project=designing
+}}{{Transition
+|date=1549581558
+|type=Added Project
+|project=User-mobrovac
+}}{{Transition
+|date=1548037024
+|type=Entered Column
 |project=Performance-Team (Radar)
 |column=Watching
-|transitionDate=1548037024
-|transitionType=Entered Column
 }}{{Transition
+|date=1548037024
+|type=Exited Column
 |project=Performance-Team (Radar)
 |column=Limbo
-|transitionDate=1548037024
-|transitionType=Exited Column
 }}{{Transition
+|date=1544029677
+|type=Removed Project
+|project=Performance-Team
+}}{{Transition
+|date=1544029677
+|type=Added Project
+|project=Radar
+}}{{Transition
+|date=1516291007
+|type=Entered Column
 |project=Performance-Team
 |column=Next In This Quarter / Jan-Mar 2020
-|transitionDate=1516291007
-|transitionType=Entered Column
 }}{{Transition
+|date=1516291007
+|type=Exited Column
 |project=Performance-Team
 |column=Doing
-|transitionDate=1516291007
-|transitionType=Exited Column
 }}{{Transition
+|date=1493926818
+|type=Entered Column
 |project=Availability (MediaWiki-MultiDC)
 |column=Doing
-|transitionDate=1493926818
-|transitionType=Entered Column
 }}{{Transition
+|date=1493926818
+|type=Exited Column
 |project=Availability (MediaWiki-MultiDC)
 |column=Backlog
-|transitionDate=1493926818
-|transitionType=Exited Column
 }}{{Transition
+|date=1493841737
+|type=Entered Column
 |project=Availability
 |column=MediaWiki-MultiDC
-|transitionDate=1493841737
-|transitionType=Entered Column
 }}{{Transition
+|date=1493841737
+|type=Exited Column
 |project=Availability
 |column=Doing
-|transitionDate=1493841737
-|transitionType=Exited Column
 }}{{Transition
+|date=1472758374
+|type=Entered Column
 |project=Performance-Team
 |column=Doing
-|transitionDate=1472758374
-|transitionType=Entered Column
 }}{{Transition
+|date=1472758374
+|type=Exited Column
 |project=Performance-Team
 |column=Doing
-|transitionDate=1472758374
-|transitionType=Exited Column
 }}{{Transition
+|date=1450727679
+|type=Entered Column
 |project=Performance-Team
 |column=Doing
-|transitionDate=1450727679
-|transitionType=Entered Column
 }}{{Transition
+|date=1450727679
+|type=Exited Column
 |project=Performance-Team
 |column=Doing
-|transitionDate=1450727679
-|transitionType=Exited Column
 }}{{Transition
+|date=1445277895
+|type=Entered Column
 |project=Performance-Team
 |column=Doing
-|transitionDate=1445277895
-|transitionType=Entered Column
 }}{{Transition
+|date=1445277895
+|type=Exited Column
 |project=Performance-Team
 |column=Doing
-|transitionDate=1445277895
-|transitionType=Exited Column
 }}{{Transition
+|date=1444092113
+|type=Entered Column
 |project=Performance-Team
 |column=Doing
-|transitionDate=1444092113
-|transitionType=Entered Column
 }}{{Transition
+|date=1444092113
+|type=Exited Column
 |project=Performance-Team
 |column=Doing
-|transitionDate=1444092113
-|transitionType=Exited Column
 }}{{Transition
+|date=1444068481
+|type=Entered Column
 |project=Performance-Team
 |column=Doing
-|transitionDate=1444068481
-|transitionType=Entered Column
 }}{{Transition
+|date=1444068481
+|type=Exited Column
 |project=Performance-Team
 |column=Doing
-|transitionDate=1444068481
-|transitionType=Exited Column
 }}{{Transition
+|date=1444068253
+|type=Entered Column
 |project=Performance-Team
 |column=Doing
-|transitionDate=1444068253
-|transitionType=Entered Column
 }}{{Transition
+|date=1444068253
+|type=Exited Column
 |project=Performance-Team
 |column=Inbox
-|transitionDate=1444068253
-|transitionType=Exited Column
 }}{{Transition
+|date=1443387069
+|type=Entered Column
 |project=Availability
 |column=Doing
-|transitionDate=1443387069
-|transitionType=Entered Column
 }}{{Transition
+|date=1443387069
+|type=Exited Column
 |project=Availability
 |column=Doing
-|transitionDate=1443387069
-|transitionType=Exited Column
 }}{{Transition
+|date=1430153849
+|type=Entered Column
 |project=Availability
 |column=Doing
-|transitionDate=1430153849
-|transitionType=Entered Column
 }}{{Transition
+|date=1430153849
+|type=Exited Column
 |project=Availability
 |column=Backlog
-|transitionDate=1430153849
-|transitionType=Exited Column
 }}{{Transition
+|date=1429815199
+|type=Entered Column
 |project=Availability
 |column=Backlog
-|transitionDate=1429815199
-|transitionType=Entered Column
 }}{{Transition
+|date=1429815199
+|type=Exited Column
 |project=Availability
 |column=Backlog
-|transitionDate=1429815199
-|transitionType=Exited Column
 }}{{Transition
+|date=1429815196
+|type=Entered Column
 |project=Availability
 |column=Backlog
-|transitionDate=1429815196
-|transitionType=Entered Column
 }}{{Transition
+|date=1429815196
+|type=Exited Column
 |project=Availability
 |column=Backlog
-|transitionDate=1429815196
-|transitionType=Exited Column
 }}{{Transition
+|date=1429815194
+|type=Entered Column
 |project=Availability
 |column=Backlog
-|transitionDate=1429815194
-|transitionType=Entered Column
 }}{{Transition
+|date=1429815194
+|type=Exited Column
 |project=Availability
 |column=Backlog
-|transitionDate=1429815194
-|transitionType=Exited Column
 }}{{Transition
+|date=1429815192
+|type=Entered Column
 |project=Availability
 |column=Backlog
-|transitionDate=1429815192
-|transitionType=Entered Column
 }}{{Transition
+|date=1429815192
+|type=Exited Column
 |project=Availability
 |column=Backlog
-|transitionDate=1429815192
-|transitionType=Exited Column
 }}{{Transition
+|date=1425689785
+|type=Entered Column
 |project=MediaWiki-Core-Team
 |column=In Dev/Progress
-|transitionDate=1425689785
-|transitionType=Entered Column
 }}{{Transition
+|date=1425689785
+|type=Exited Column
 |project=MediaWiki-Core-Team
 |column=In Dev/Progress
-|transitionDate=1425689785
-|transitionType=Exited Column
 }}{{Transition
+|date=1425679554
+|type=Entered Column
 |project=TechCom-RFC
 |column=Under discussion
-|transitionDate=1425679554
-|transitionType=Entered Column
 }}{{Transition
+|date=1425679554
+|type=Exited Column
 |project=TechCom-RFC
 |column=Inbox
-|transitionDate=1425679554
-|transitionType=Exited Column
 }}{{Transition
+|date=1423169245
+|type=Entered Column
 |project=MediaWiki-Core-Team
 |column=In Dev/Progress
-|transitionDate=1423169245
-|transitionType=Entered Column
 }}{{Transition
+|date=1423169245
+|type=Exited Column
 |project=MediaWiki-Core-Team
 |column=In Dev/Progress
-|transitionDate=1423169245
-|transitionType=Exited Column
 }}{{Transition
+|date=1422984613
+|type=Entered Column
 |project=MediaWiki-Core-Team
 |column=In Dev/Progress
-|transitionDate=1422984613
-|transitionType=Entered Column
 }}
 }}