T88445: MediaWiki active/active datacenter investigation and work (tracking)

Phabricator Link	T88445
Status	open
Priority	Medium (orange)
Points
Phabricator Task Created	2015/02/03 12:30 PM
Wiki Page Created	2019/06/09 01:22 PM
Phabricator Task Last Modified	2021/10/19 09:06 PM
Wiki Page Last Updated	2021/10/19 11:12 PM
Phabricator Task Closed
Authored By	aaron (Aaron Schulz)
Assigned To
Projects	Epic Performance-Team (Radar) (Watching) Platform Team Initiatives (Multi-DC (TEC1)) Platform Team Workboards (Initiatives) (Later) Sustainability (MediaWiki-MultiDC) (Later) User-mobrovac (Backlog)
Platform Engineering Initiative Column	Multi-DC (TEC1)
Platform Team Workboards Column	Initiatives: Later
Subtasks	T219592: Frequent Echo DB_MASTER write queries on HTTP GET (open) T212129: Move MainStash out of Redis to a simpler multi-dc aware solution (open) T221159: FY18/19 TEC1.6 Q4: Improve or replace the usage of GTID_WAIT with pt-heartbeat in MW (declined) T206016: Create a service for session storage (resolved) T114271: Audit *all* datastores to see how DR failover is handled (open) T164860: Update Echo's caching strategy for multi-dc compatibility (resolved) T120543: Investigate write mode flag cleanup for Flow API modules (resolved) T92863: Configure 'enqueue' job queue type to be DC-local and in a dedicated runner loop (declined) T89389: Try to avoid doCascadeProtectionUpdates() call on page view (resolved) T88506: Make FileRepo file cache misses use slaves instead of the master (resolved) T137326: Log post-send DB updates and make sure their frequency is acceptable (resolved) T124795: codfw is in read only according to mediawiki (resolved) T120009: Flow: Use WAN cache delete() and slave populations to avoid merge() (resolved) T119642: Create a Master-master topology between datacenters for easier failover (setup circular replication dallas -> eqiad for mysql databases) (resolved) T114273: Cleanup active-DC based MW config code and make it more robust and easy to change (resolved) T111266: Make LoadBalancer slave lag check and read-only mode more robust (for example, using pt-heartbeat) (resolved) T94028: DB master connections requested by Flow on GET/HEAD requests (resolved) T91869: Implement a replication strategy for Swift (resolved) T220170: Address Database hardware infrastructure blockers on datacenter switchover & multi-dc deployment (resolved) T204245: Run MediaWiki media originals active/active (open) T135027: MASTER_POS_WAIT() alternative that works cross-DC (resolved) T134809: Apache <=> mariadb SSL/TLS for cross-datacenter writes (stalled) T127455: Enable HTTPS for Swift traffic (resolved) T114398: Make and document DC switch-over scripts (duplicate) T113204: Smart caching logic for handling cross-DC network outages (resolved) T113192: Consider using delayed rebound purges for CDN (resolved) T112708: Support for master/slave FileBackend distinction in FileBackendMultiwrite (resolved) T111575: Make $wgSessionCacheType and $wgMainStash caches multi-DC ready (declined) T134811: Consider REST with SSL (HyperSwitch/Cassandra) for session storage (resolved) T108253: Make sure CentralAuth login tokens work with two datacenters (resolved) T99208: Make sure $wgUseLocalMessageCache works with multiple DCs (resolved) T97686: Keep TorExitNodes node list cache populated in *both* datacenters (resolved) T97620: Convert various core/extension cache users to ReplicatedBagOStuff (resolved) T97562: WANObjectCache relay daemon or mcrouter support (resolved) T97335: Make CachingSiteStore use WANObjectCache (resolved) T95126: Make RedisBagOStuff support tag name hashing (resolved) T94480: Audit and fix extensions that call saveSettings() to lazy-set preferences when loaded (resolved) T94448: Echo doing master queries on simple GET requests (resolved) T93144: Check multi-datacenter compatibility with GettingStarted's use of Redis (resolved) T93141: Convert various core/extension cache users to the WAN cache as needed (resolved) T93006: Figure out a cross-DC strategy for the BloomFilter caches (declined) T92357: Fix database master queries from HTTP GET/HEAD before active-active multi-dc (open) T92007: Investigate a better alternative for making user_newtalk talk updates (resolved) T91870: Figure out a replication strategy for ElasticSearch (resolved) T91837: Consider moving various DB writes on page views to using local jobs (resolved) T91820: Create HTTP verb and sticky cookie DC routing in VCL (open) T91816: Add code to enable setting sticky DC cookies for POST requests (resolved) T91819: Use local jobqueue class for jobs enqueued on pages views (resolved) T91815: Audit cache usage to see what can be DC-local and what needs cross-DC purges (resolved) T91284: Avoid wl_notificationtimestamp updates on page views (resolved) T91279: Augment user_touched with an in-memory field (resolved) T89308: Create a local job class that enqueues jobs to final location (resolved) T89184: Remove cache anti-dependencies (resolved) T88666: RFC: Master/slave datacenter strategy for MediaWiki (resolved) T88635: ObjectCacheSessionHandler should avoid pointless writes in write() (resolved) T88634: Make a ReplicatedBagOStuff class for multi-DC use (resolved) T88493: Devise stashing strategy for multi-DC mediawiki (resolved) T88492: Devise caching (memcached) strategy for multi-DC mediawiki (resolved) T130946: Make ?action=markpatrolled require POST (resolved) T119641: Split-brain strategy for services that use config managed by etcd (open) T270223: FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking) (open)
Parent Tasks

Column Transitions: (Added project: ⇑, Removed project: ⇓,Entered column: ⇒, Exited column: ⇐)

Project	Column	Date
MediaWiki-Core-Team	⇒ In Dev/Progress	3 February 2015
MediaWiki-Core-Team	⇐ In Dev/Progress	5 February 2015
MediaWiki-Core-Team	⇒ In Dev/Progress	5 February 2015
TechCom-RFC	⇐ P1: Define	6 March 2015
TechCom-RFC	⇒ Under discussion	6 March 2015
MediaWiki-Core-Team	⇐ In Dev/Progress	6 March 2015
MediaWiki-Core-Team	⇒ In Dev/Progress	6 March 2015
Sustainability	⇐ Tag	23 April 2015
Sustainability	⇒ Tag	23 April 2015
Sustainability	⇐ Tag	23 April 2015
Sustainability	⇒ Tag	23 April 2015
Sustainability	⇐ Tag	23 April 2015
Sustainability	⇒ Tag	23 April 2015
Sustainability	⇐ Tag	23 April 2015
Sustainability	⇒ Tag	23 April 2015
Sustainability	⇐ Tag	27 April 2015
Sustainability	⇒ Doing	27 April 2015
Sustainability	⇐ Doing	27 September 2015
Sustainability	⇒ Doing	27 September 2015
Performance-Team	⇐ Inbox	5 October 2015
Performance-Team	⇒ Doing (old)	5 October 2015
Performance-Team	⇐ Doing (old)	5 October 2015
Performance-Team	⇒ Doing (old)	5 October 2015
Performance-Team	⇐ Doing (old)	5 October 2015
Performance-Team	⇒ Doing (old)	5 October 2015
Performance-Team	⇐ Doing (old)	19 October 2015
Performance-Team	⇒ Doing (old)	19 October 2015
Performance-Team	⇐ Doing (old)	21 December 2015
Performance-Team	⇒ Doing (old)	21 December 2015
Performance-Team	⇐ Doing (old)	1 September 2016
Performance-Team	⇒ Doing (old)	1 September 2016
Sustainability	⇐ Doing	3 May 2017
Sustainability	⇒ MediaWiki-MultiDC	3 May 2017
Sustainability (MediaWiki-MultiDC)	⇐ Later	4 May 2017
Sustainability (MediaWiki-MultiDC)	⇒ Current: Performance Team	4 May 2017
Performance-Team	⇐ Doing (old)	18 January 2018
Performance-Team	⇒ Next: Goal / Oct-Dec '21	18 January 2018
⇓ Performance-Team		5 December 2018
⇑ Performance-Team (Radar)		5 December 2018
Performance-Team (Radar)	⇐ Limbo	20 January 2019
Performance-Team (Radar)	⇒ Watching	20 January 2019
⇑ Platform Team Legacy (Epic)		7 February 2019
⇑ Services (designing)		7 February 2019
⇑ Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1))		7 February 2019
⇑ User-mobrovac		7 February 2019
⇓ Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1))		27 February 2019
⇑ Platform Engineering (Multi-DC (TEC1))		27 February 2019
⇓ Services (designing)		5 July 2019
⇓ Platform Team Legacy (Epic)		5 July 2019
⇓ Platform Engineering (Multi-DC (TEC1))		26 July 2019
Platform Engineering	⇐ Multi-DC (TEC1)	26 July 2019
Platform Engineering	⇒ mop	26 July 2019
⇑ Platform Team Initiatives (Multi-DC (TEC1))		26 July 2019
⇑ Platform Team Workboards (Initiatives)		24 March 2020
Sustainability (MediaWiki-MultiDC)	⇐ Current: Performance Team	22 December 2020
Sustainability (MediaWiki-MultiDC)	⇒ Later	22 December 2020